Spatial floating image display device

Abstract

To suitably display an image for a user of a spatial floating image display device, and to provide necessary information thereto, and to contribute to "3. Good Health and Well-Being", "9. Industry, Innovation and Infrastructure", and "11. Sustainable Cities and Communities" of SDGs.SOLUTION: A spatial floating image display device includes an image display device 10, a lenticular lens 1103 arranged on an image light emission side of the image display device 10, and a retroreflective member 330 for reflecting the image light from the image display device 10, and forming a spatial floating image in the air by the reflected light, wherein the image display device 10 displays an image including at least two objects, fixes the position of an arbitrary object among at least the two objects, and displays a plurality of images obtained by mutually shifting the positions of the objects other than the arbitrary object in a prescribed direction, between different multi-viewpoint images, as multi-viewpoint images 1902.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The present invention relates to a space floating image display device. [Background technology]

[0002] As a space-floating image display device, an image display device and a display method that directly displays an image as a spatial image toward the outside are already known. In addition, a detection system that reduces false detections of operations on the operation surface of the displayed spatial image is also described, for example, in JP 2019-128722 A (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-128722 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the disclosure of Patent Document 1 does not fully consider specific techniques for displaying a floating-in-space image in a three-dimensional shape.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a space floating image display device that can generate multi-viewpoint images in a simpler manner and display a space floating image that can be viewed stereoscopically. [Means for solving the problem]

[0006] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present invention includes a plurality of means for solving the above problem, but an example is as follows. The space-floating image display device includes an image display device that displays an image, a lenticular lens arranged on the image light output side of the image display device, and a retroreflective member that reflects the image light from the image display device and forms a space-floating image in the air with the reflected light, and the image display device displays an image including at least two objects, fixes the position of an arbitrary object among the at least two objects, and displays multiple images obtained by shifting the positions of objects other than the arbitrary object in a predetermined direction between different multi-view images as multi-view images. Effect of the Invention

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other objects, configurations, and effects will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of an internal configuration of a space floating image display device according to an embodiment. [Diagram 3] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment; [Figure 4] 11A and 11B are diagrams showing another example of the main part configuration and the retroreflection part configuration of the space floating image display device according to the embodiment; [Diagram 5] FIG. 2 is a perspective view showing an example of the arrangement of members for blocking extraordinary rays generated by retroreflection in one embodiment. [Figure 6] 4 is a cross-sectional view showing an example of the arrangement of a member for blocking extraordinary rays generated by retroreflection in one embodiment. FIG. [Figure 7] 1 is an explanatory diagram of a first sensing technique used in a space floating image display device according to an embodiment; [Figure 8] 11 is an explanatory diagram of a second sensing technique used in a space floating image display device according to an embodiment. FIG. [Figure 9] 1 is an explanatory diagram of the operation and device of a sensing system used in a space floating image display device according to an embodiment. FIG. [Figure 10] FIG. 2 is a diagram showing the characteristics of the spectral irradiance of sunlight. [Figure 11] FIG. 1 is a diagram showing the reflection characteristics of polarized light incident on a medium with a refractive index of 1.5 versus the angle of incidence of the light. [Figure 12] 1 is a diagram showing a configuration of a main part of a space floating image display device according to an embodiment; [Figure 13] FIG. 2 is a diagram showing a main configuration of another space floating image display device according to an embodiment. [Figure 14] FIG. 1 is a diagram showing the principle of displaying a multi-viewpoint image. [Figure 15] FIG. 2 is a diagram showing an example of a camera arrangement for generating a multi-view video. [Figure 16] FIG. 2 is a diagram showing an example of an image displayed by a multi-viewpoint image display device. [Figure 17] FIG. 1 is a diagram showing an example of how a multi-viewpoint video appears as a floating image in space. [Figure 18] FIG. 11 is a diagram showing another example of how a multi-viewpoint video appears as a floating image in space. [Figure 19] FIG. 2 is a diagram showing an example of a multi-view image according to an embodiment; [Figure 20] FIG. 2 is a diagram showing an example of how a multi-view image appears in one embodiment. [Figure 21] FIG. 1 is a diagram showing an example of how a multi-viewpoint video appears as a floating image in space according to an embodiment. [Figure 22] FIG. 11 is a diagram showing another example of a multi-view image according to an embodiment. [Figure 23] A figure showing another example of how a multi-view image appears in one embodiment. [Figure 24] FIG. 11 is a diagram showing another example of how a multi-viewpoint video appears as a space floating image in accordance with an embodiment. [Diagram 25]FIG. 13 is a supplementary explanatory diagram regarding generation of a multi-view image according to an embodiment. [Figure 26] 1 is a diagram illustrating an example of the appearance of a vending machine according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention.

[0010] For the sake of explanation, when describing processing by a program, the program, functions, processing units, etc. may be described as the main components, but the main hardware components of these are the processor, or a controller, device, computer, system, etc. that is configured with the processor. The computer executes processing according to the program read into the memory by the processor, appropriately using resources such as memory and communication interfaces. This realizes predetermined functions, processing units, etc. The processor is configured with semiconductor devices such as a CPU / MPU or GPU, for example. Processing is not limited to software program processing, and can also be implemented by dedicated circuits. Dedicated circuits such as FPGA, ASIC, CPLD, etc. can be used.

[0011] The program may be pre-installed as data in the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or a disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information are composed of structures such as, for example, tables and lists, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are mutually replaceable.

[0012] When displaying a floating-in-space image as a three-dimensional shape, in order to display an image with a three-dimensional effect based on the motion parallax of multi-viewpoint images, it is necessary to use multiple images of a display object such as a push button taken from multiple different directions, or to generate images observed from different directions by rendering. In this case, there is an issue that it takes a lot of time and effort to generate a multi-viewpoint image. Therefore, when displaying an HMI with a relatively simple shape such as a push button as a three-dimensional floating-in-space image, a technology for creating a multi-viewpoint image in a simpler way is desired.

[0013] Therefore, according to the following embodiment, a space-floating display device that can generate a multi-viewpoint image in a simpler manner and display an object that can be viewed stereoscopically can be provided. The space-floating image display device (hereinafter, sometimes simply referred to as the device) of one embodiment is configured to improve visibility by eliminating ghost images that significantly reduce the visibility of the space-floating image and improving the brightness of the space-floating image. In addition, when a space-floating image having a three-dimensional shape is used as, for example, a signage (electronic signboard), it is expected to have the effect of increasing people's interest in the products and services displayed by the signage. When a man-machine interface (HMI; Human Machine Interface, or sometimes simply called a user interface or UI; User Interface) such as a push button (digit button) is displayed as the space-floating image, a non-contact HMI that looks stereoscopic, that is, has a sense of depth, can be realized, so that indirect contact between an unspecified number of people can be prevented compared to the case of using a physical push button, and the risk of infection, etc. can be reduced.

[0014] In addition, when a three-dimensional push button is displayed as a floating-in-space image, the push button appears three-dimensional with a sense of depth, i.e., it appears to pop out or, conversely, recedes, compared to a push button displayed as a flat floating-in-space image (floating-in-space image based on a two-dimensional image / picture), which has the effect of making it easier to use as an HMI for people who are seeing a floating-in-space image for the first time or who are unfamiliar with how to use it.

[0015] For example, the device of one embodiment is applied to numeric buttons for selecting products in a vending machine, buttons on a telephone, etc., and provides a user interface with a screen of floating images. In addition, the system of one embodiment displays the numeric buttons for selecting products or buttons on a telephone on the screen of the floating images when a user approaches the housing of the device.

[0016] In one embodiment, when a user approaches a floating image in space or performs some operation on the floating image in space, the floating image display device automatically displays a welcome message or a figure explaining how to use the image, and then transitions or changes to an operation menu screen that has a plurality of push buttons (number buttons) and a decision button, etc., and allows the user to press these buttons. In addition, when the device determines that the user does not understand how to operate the floating image in space (when the user does not perform any operation for a long time, etc.), the device may provide the user with detailed guidance on how to operate the image.

[0017] In addition, the space floating image display device of one embodiment has a function to identify and specify the user, for example, by face recognition using a camera. The system refers to user attribute information such as age and system usage history for the user specified by the function. The system controls to change the size of the characters of the operation method guide displayed as the space floating image according to the user's attributes.

[0018] In the following description of the embodiments, an image that floats in space or an image that is displayed in the air may be expressed by the term "space-floating image". Instead of this term, it may be expressed as "aerial image", "spatial image", "floating image", "space-floating optical image of displayed image", "floating optical image of displayed image", etc. The term "space-floating image" that is mainly used in the description of the embodiments is used as a representative example of these terms.

[0019] <Space-floating image display device> The present disclosure relates to a display device capable of displaying an image generated by image light from a large-area image light source as a floating image inside or outside a store space by transmitting the image through a transparent member that divides the space, such as glass of a show window. The present disclosure also relates to a large-scale digital signage system configured using a plurality of such display devices.

[0020] According to the following embodiment, for example, a high-resolution image can be displayed in a floating state on the glass surface of a show window or a light-transmitting plate. At this time, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, it is possible to efficiently reflect only the normal reflected light to the retroreflective member (retroreflective member) or retroreflector. Therefore, according to the embodiment, the light utilization efficiency is high, and it is possible to suppress ghost images that occur in addition to the main floating image in space, which was a problem in the conventional retroreflection method, and a clear floating image in space can be obtained.

[0021] In addition, a device including the light source of the present disclosure can provide a novel and highly usable space floating image display device that can significantly reduce power consumption. In addition, according to the technology of the present disclosure, a space floating image display device for a vehicle can be provided that can display a so-called unidirectional space floating image that can be viewed from outside the vehicle through a shield glass including the front glass, rear glass, and side glass of the vehicle.

[0022] On the other hand, the conventional floating image display device combines an organic EL panel or a liquid crystal display panel (sometimes referred to as a liquid crystal panel) as a high-resolution color display image source with a retroreflective member. In the conventional floating image display device, the image light is diffused at a wide angle. Therefore, when the retroreflective member 2 in the first embodiment composed of a polyhedron shown in FIG. 3(B) is used, in addition to the reflected light normally reflected by the retroreflective member 2 (the normal floating image caused by it), a ghost image is generated by the image light obliquely incident on the retroreflective member 2 (retroreflective portion 2a) shown in FIG. 3(C). This reduces the image quality of the floating image. In addition, in the conventional floating image display device, in addition to the normal floating image, multiple ghost images are generated according to the number of reflecting surfaces. Therefore, the same floating image in the space, which is a ghost image, is also viewed by people other than the viewer, which is a major problem from the viewpoint of security.

[0023] <First Configuration Example of the Space Floating Image Display Device> Fig. 1(A) shows an example of the usage form of the space-floating image display device of the embodiment, and is an explanatory diagram of the overall configuration of the space-floating image display device. In Fig. 1(A), for example, in a store, a space is partitioned by a show window (also called window glass) 105, which is a translucent material (also described as a transparent material) such as glass. According to this space-floating image display device, it is possible to display a space-floating image in one direction to the outside of the store space through such a transparent material.

[0024] Specifically, according to this system, light with a narrow-angle directional characteristic and specific polarization is emitted as an image light beam from the image display device 10. The emitted image light beam is once incident on the retroreflective member 2, retroreflected, and passes through the window glass 105 to form a real image, a floating image (aerial image) 3, outside the store space. In FIG. 1(A), the inside of the store inside the transparent member (window glass in this case) 105 is shown as the depth direction, and the outside of the window glass 105 (e.g., the sidewalk) is shown as the foreground. On the other hand, a member that reflects specific polarization can be provided on the window glass 105, and the image light beam can be reflected by such a member to form an aerial image at a desired position in the store.

[0025] 1(B) shows the internal configuration of the video display device 10. The video display device 10 includes a video display unit 1102 that displays an original aerial image, a video control unit 1160 that converts an input video to match the resolution of the panel, and a video / audio signal receiving unit 1130 that receives and inputs a video / audio signal.

[0026] Among these, the video / audio signal receiving unit 1130 plays a role of handling wired input signals through an input interface such as HDMI (High-Definition Multimedia Interface) (registered trademark) and wireless input signals such as Wi-Fi (Wireless Fidelity) (registered trademark). The video / audio signal receiving unit 1130 can also function independently as a video receiving / displaying device. Furthermore, the video / audio signal receiving unit 1130 can also display / output video / audio information from a tablet terminal, a smartphone, or the like. Furthermore, the video / audio signal receiving unit 1130 can also be connected to a processor (arithmetic processing device) such as a stick PC as necessary, in which case the video / audio signal receiving unit as a whole can have the capabilities of calculation processing, video analysis processing, and the like.

[0027] [Functional blocks of the space floating image display device] 2 shows a functional block diagram of the space floating image display device 1. The image display unit 1102 generates an image by modulating light passing through the image display unit 1102 based on a video signal. The image display unit 1102 may be called a display panel, a liquid crystal panel, or a liquid crystal display panel. The image display unit 1102 may be, for example, a transmissive display panel, or in some cases, may be configured using a reflective display panel that modulates light reflected on the panel based on a video signal, a DMD panel (DMD: Digital Micromirror Device, registered trademark), or the like.

[0028] The space floating image display device 1 has a lenticular lens 1103 as shown in FIG. 2. A lenticular is a printed matter in which a pattern changes or a three-dimensional effect is obtained depending on the viewing angle by using a sheet-like lenticular lens. The lenticular lens is a collection of lenses having a semi-cylindrical surface (in other words, a semi-elliptical cylindrical shape; see FIG. 14 described later), and an image display unit 1102 that displays different images corresponding to the number of viewpoints of a multi-viewpoint image (or image) is arranged under one lens (semi-cylindrical lens). In this embodiment, the lenticular lens 1103 is arranged on the image light output side of the image display unit 1102. Specifically, the lenticular lens 1103 is arranged at a predetermined distance from the image light output side of the image display unit 1102. In addition, the space floating image display device 1 displays a multi-view image (or multi-view image) by passing the image light emitted from the image display unit 1102 through a lenticular lens 1103, and the user can observe the multi-view image (or multi-view image).

[0029] 2, the user can see different images (or videos) from each position by moving in the direction in which the semi-cylindrical lenses forming the lenticular lens 1103 are arranged (for example, left and right). Therefore, the different images (or videos) are taken as images (or videos) of a single subject captured in different shooting directions. This allows the user to view the images (or videos) displayed on the liquid crystal panel constituting the video display unit 1102 as a multi-viewpoint stereoscopic image with motion parallax through the lenticular lens.

[0030] The retroreflector 1101 retroreflects the light modulated by the image display unit 1102. Of the light reflected from the retroreflector 1101, the light outputted to the outside of the space-floating image display device 1 forms the space-floating image 3. The light source 1105 generates light for the image display unit 1102. For example, a solid-state light source such as an LED light source or a laser light source is used as the light source 1105. The power source 1106 converts an AC current inputted from the outside into a DC current and supplies power to the light source 1105. Furthermore, the power source 1106 supplies the DC current required by each of the other units.

[0031] The light guide 1104 guides light generated by the light source 1105 and irradiates the light to the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can be called a backlight for the image display unit 1102. Various methods are possible for combining the light guide 1104 and the light source 1105. As shown in FIG. 2, a portion consisting of the three components, the image display unit 1102, the light guide 1104, and the light source 1105, is particularly called the image display device 10.

[0032] The aerial operation detection sensor 1351 is a sensor that senses an area that overlaps at least a part of the display area of ​​the floating in space image 3 or an area that overlaps the entire display area in order to detect an operation (also referred to as an aerial operation) of the floating in space image 3 by the user's fingers. The aerial operation detection sensor 1351 may have a specific sensor configuration such as a distance sensor using invisible light such as infrared rays, invisible light laser, ultrasonic waves, or a configuration that can detect coordinates on a two-dimensional plane by combining a plurality of these. The aerial operation detection sensor 1351 may also be configured as a LiDAR (Light Detection and Ranging) of a ToF (Time of Flight) method described later.

[0033] The aerial operation detection unit 1350 acquires a sensing signal acquired by the aerial operation detection sensor 1351, and based on this, calculates whether or not the user's fingers have touched the floating in space image 3, and the position of the touch on the floating in space image 3. The aerial operation detection unit 1350 may be configured with a circuit such as an FPGA.

[0034] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 (these may be referred to as a sensing system) may be configured to be built into the space-floating image display device 1, or may be provided separately from the space-floating image display device 1. When provided separately, they may be configured to transmit information or signals to the space-floating image display device 1 via a wired or wireless communication connection path or a video signal transmission path. Both the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. In this case, it is possible to build a system in which the space-floating image display device 1 without the aerial operation detection function is the main body, and only the aerial operation detection function can be added as an option. In addition, only the aerial operation detection sensor 1351 may be provided separately, and the aerial operation detection unit 1350 may be built into the space-floating image display device 1. In cases where it is desired to more freely arrange the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1, there is an advantage to a configuration in which only the aerial operation detection sensor 1351 is provided separately.

[0035] The imaging unit 1180 is a so-called camera having an image sensor, and captures an image of the space near the floating-in-space image 3 and / or the user's face, arm, finger, etc. The imaging unit 1180 may use multiple cameras or a camera with a depth sensor depending on the application. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1. When using multiple cameras or a camera with a depth sensor, the imaging unit 1180 may assist the mid-air operation detection unit 1350 in detecting a touch operation on the floating-in-space image 3 by the user, in other words, a mid-air operation that touches the surface of the floating-in-space image 3.

[0036] For example, when the aerial operation detection sensor 1351 is configured as an object intrusion sensor for the plane to which the floating in space image 3 belongs, there are cases where the aerial operation detection sensor 1351 alone cannot detect how close an object (e.g., a user's finger) that has not intruded into the plane is to the plane. In such cases, by using depth calculation information based on the imaging results of multiple cameras in the imaging unit 1180 and depth information based on a depth sensor, it becomes possible to calculate the distance between an object (e.g., a user's finger) that has not intruded into the plane of the floating in space image 3 and the plane. This calculation information can be used for various display controls in the floating in space image 3.

[0037] Alternatively, this system may be configured so that the aerial operation detection section 1350 detects a touch operation (aerial operation) of the floating in space image 3 by the user based on the imaging result of the imaging section 1180, without using the aerial operation detection sensor 1351.

[0038] The imaging unit 1180 may be configured to capture an image of the face of the user operating the space-floating image 3, and the control unit 1110 may perform user identification / specification processing or user authentication processing based on the captured image. Alternatively, the imaging unit 1180 may be configured to capture an image including the surroundings of the user operating the space-floating image 3, in order to determine whether other people are standing around or behind the user operating the space-floating image 3 and peeking at the user's operation of the space-floating image 3.

[0039] The operation input unit 1107 is an operation button or a remote control light receiving unit, and inputs a signal related to a user's operation that is different from the aerial operation on the space floating image 3. The operation input unit 1107 may be used by an administrator of the space floating image display device 1 to operate this system, separately from the above-mentioned user who touches the space floating image 3.

[0040] The video signal input unit 1131 has a function of connecting an external video output device and inputting video data. The audio signal input unit 1133 has a function of connecting an external audio output device and inputting audio data. Meanwhile, the audio signal output unit 1140 has a function of outputting an audio signal based on audio data input to the audio signal input unit 1133. The audio signal output unit 1140 may output an audio signal based on audio data such as numbers and character strings, and other data of operation sounds and error warning sounds, which are recorded in advance in the storage unit 1170. The video signal input unit 1131 and the audio signal input unit 1133 are collectively referred to as the video / audio signal input unit 1130. The video signal input unit 1131 and the audio signal input unit 1133 may be configured separately, or may be combined into one unit.

[0041] The audio signal output unit 1140 is connected to a speaker or an ultra-directional speaker 30. The audio signal output unit 1140 may be connected to a speaker that outputs audio in a normal audible band, but may be connected to an ultra-directional speaker so that the audio cannot be heard by anyone other than the user, particularly when confidentiality is high and security considerations are required. An ultra-directional speaker is a speaker that has the characteristic that audio in an audible band can be heard only by the ears of people present in a specific limited spatial region, and that audio in the audible band cannot be heard by the ears of people present outside the specific spatial region.

[0042] The superdirectional speaker 30 is constructed by arranging a plurality of ultrasonic output elements capable of generating ultrasonic signals of, for example, about 40 kHz on a plane. In this case, the more ultrasonic output elements used, the louder the volume of the sound obtained by the superdirectional speaker. The principle of the superdirectional speaker will be briefly explained. As is well known, ultrasonic waves have a high degree of directivity compared to sounds in the audible range (e.g., human speech). Therefore, by using a 40 kHz ultrasonic signal as a carrier (carrier wave) and modulating the carrier with an audio signal in the audible range (e.g., AM modulation), it becomes possible to make the sound audible only in a specific limited spatial region.

[0043] For example, by using multiple cameras as imaging unit 1180, the positions of the user's face and ears can be identified, and depending on the identification results, the output from superdirectional speaker 30 can be made to allow sound to be heard only in the vicinity of the user's ears. Specifically, by controlling the phase (in other words, delay time) of the ultrasonic signal input to the ultrasonic output elements constituting superdirectional speaker 30, sound can be heard only in a specific, limited spatial region. Also, by arranging multiple ultrasonic output elements not on a flat surface but on, for example, a concave surface, sound can be heard only in a specific, limited spatial region.

[0044] The non-volatile memory 1108 stores various data used by the space floating image display device 1. The data stored in the non-volatile memory 1108 may include data for various operations displayed as the space floating image 3, user interface image information such as icons and buttons, data of objects for user operation, layout information, etc. The memory 1109 stores image data to be displayed as the space floating image 3 and data for controlling the device.

[0045] The control unit 1110 corresponds to a controller of the space floating image display device 1, in other words, a control device, and controls the operation of each unit connected to it. The control unit 1110 includes a device such as a processor. The control unit 1110 executes processing according to a program read from the non-volatile memory 1108 or the storage unit 1170 to the memory 1109 or the built-in memory. This realizes various functions. The control unit 1110 may perform calculation processing based on information acquired from each unit connected to it in cooperation with a program stored in the memory 1109. The control unit 1110 may be implemented in a housing constituting the space floating image display device 1 using a microcomputer or the like, or may be connected and implemented outside the housing.

[0046] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. The communication unit 1132 transmits and receives video, images, audio, and various types of data through this communication.

[0047] The storage unit 1170 records video, images, audio, various data, etc. For example, video, images, audio, various data, etc. may be recorded in advance in the storage unit 1170 at the time of product shipment. Video, images, audio, various data, etc. acquired from an external device or an external server via the communication unit 1132 may be recorded in the storage unit 1170. The video, images, various data, etc. recorded in the storage unit 1170 can be output as the space floating video 3 via the video display unit 1102, the video display device 10, and the retroreflective unit 1101.

[0048] Data such as icons, buttons, objects operated by the user, and data constituting the human figure, which are displayed as a user interface (including an operation menu and a human figure, which will be described later) in the space floating image 3, may also be included in the video and image data recorded in the storage unit 1170. In addition, layout information of the operation menu and the human figure, such as icons, buttons, objects, and the like, which are displayed as a user interface in the space floating image 3, and information such as various metadata related to the operation menu and the human figure, may also be included in the various data recorded in the storage unit 1170. In addition, audio data for the human figure in the space floating image 3 to output audio may also be recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 may be output as an audio signal from the speaker or the superdirectional speaker 30 via the audio signal output unit 1140.

[0049] The control unit 1110, or the video control unit 1160 or audio signal output unit 1140 may appropriately create video data and audio data for displaying and outputting the operation menu and the portrait, based on various data for constructing the operation menu and the portrait, which are stored in the storage unit 1170, the non-volatile memory 1108, etc.

[0050] The video control unit 1160 performs various controls on the video signal input to the video display unit 1102. The video control unit 1160 may perform video switching control, such as determining which video, between the video stored in the memory 1109 and the video input by the video signal input unit 1131, is to be input to the video display unit 1102. Alternatively, the video control unit 1160 may perform control to superimpose the video stored in the memory 1109 and the video input by the video signal input unit 1131 to generate a composite video to be input to the video display unit 1102. The video control unit 1160 may also perform control of image processing on the video data input by the video signal input unit 1131 and the video stored in the memory 1109. Examples of image processing include scaling processing for enlarging, reducing, and transforming an image, brightness adjustment processing for changing the luminance, contrast adjustment processing for changing the contrast curve of an image, and Retinex processing for decomposing an image into light components and changing the weighting of each component.

[0051] Furthermore, the video control unit 1160 may perform special effect video processing for assisting the user's aerial operation on the video input to the video display unit 1102. The special effect video processing may be controlled based on the result of detection of the user's operation by the aerial operation detection unit 1350 and the result of imaging the user by the imaging unit 1180.

[0052] As described above, the space-floating image display device 1 can be equipped with various functions. However, the space-floating image display device 1 does not necessarily have to have all of the above-mentioned configurations. The space-floating image display device 1 may have any configuration as long as it has at least the function of generating the space-floating image 3.

[0053] [The first method for creating floating images in space] FIG. 3 shows the main configuration of the space floating image display device of the embodiment, and also shows an example (referred to as a first method) regarding the formation of the space floating image 3 and the configuration of the retroreflective member 2.

[0054] 3(A), this floating-in-space image display device includes an image display device 10 that diverges image light of a specific polarization at a narrow angle in an oblique direction relative to a transparent member 100, which is a light-transmitting plate such as glass. The image display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic.

[0055] The image light of a specific polarization emitted from the image display device 10 is reflected by a polarization separation member 101 having a film that selectively reflects the image light of a specific polarization, which is provided on a transparent member 100, and the reflected light is incident on the retroreflective member 2. In FIG. 3, the polarization separation member 101 formed in a sheet shape is adhered to the transparent member 100.

[0056] A retroreflective member 2 is provided in the other diagonal direction with respect to the transparent member 100. A λ / 4 plate 21 (in other words, a quarter-wave plate) is provided on the image light incident surface of the retroreflective member 2. The image light is made to pass through the λ / 4 plate 21 twice in total, once when it enters the retroreflective member 2 and once when it leaves, and is thereby polarized and converted from a specific polarization (one polarization) to the other polarization.

[0057] Here, the polarization separation member 101, which selectively reflects image light of a specific polarization, has the property of transmitting the polarized light of the other polarization after the polarization conversion. Therefore, the image light of the other polarization after the polarization conversion passes through the polarization separation member 101. The image light that passes through the polarization separation member 101 forms a space floating image 3, which is a real image, outside the transparent member 100, as shown in the figure.

[0058] The light that forms the floating image 3 is a collection of light rays that converge from the retroreflective member 2 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directionality, unlike the diffuse image light formed on a screen by a general projector or the like.

[0059] 3, the levitating image 3 is perceived as a bright image when viewed by a user from the direction of arrow A, but when viewed by another person from, for example, the direction of arrow B, the levitating image 3 is completely invisible. These characteristics of the levitating image 3 are extremely suitable for use in systems that display images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0060] Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, a part of the image light with the misaligned polarization axis is reflected by the above-mentioned polarization separation member 101 and returns to the image display device 10. This part of the image light is reflected again by the image display surface of the liquid crystal display panel 11 constituting the image display device 10, generating a ghost image. This may be a factor in causing a deterioration in the image quality of the spatially floating image 3.

[0061] Therefore, in this embodiment, an absorptive polarizing plate 12 is provided on the image display surface of the image display device 10. The absorptive polarizing plate 12 transmits the image light emitted from the image display device 10 and absorbs the reflected light returning from the polarization separation member 101, thereby suppressing re-reflection. Therefore, according to this embodiment using the absorptive polarizing plate 12, it is possible to prevent or suppress degradation of image quality due to ghost images of the floating-in-space image 3.

[0062] The above-mentioned polarization separation member 101 may be formed, for example, of a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0063] Fig. 3(B) shows an example of the configuration of the retroreflective member 2 used in the first method. Fig. 3(B) shows the surface shape of a retroreflective member manufactured by Nippon Carbide Industries Co., Ltd., used in this study as a representative retroreflective member 2. This retroreflective member 2 has regularly arranged hexagonal prism retroreflective parts (retroreflective elements) 2a on the surface. A light ray incident on the inside of the hexagonal prism is reflected by the wall and bottom surfaces of the hexagonal prism, and is emitted as retroreflected light in a direction corresponding to the incident light, and a space floating image 3, which is a real image, is displayed based on the image displayed on the image display device 10.

[0064] The resolution of the floating image 3 depends largely on the outer shape D and pitch P of the retroreflective portion 2a of the retroreflective member 2 shown in FIG. 3(B) in addition to the resolution of the liquid crystal display panel 11. For example, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel 11 is used, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion 2a is 240 μm and the pitch P is 300 μm, one pixel of the floating image 3 is equivalent to 300 μm. Therefore, the effective resolution of the floating image 3 is reduced to about 1 / 3. Therefore, in order to make the resolution of the floating image 3 equivalent to the resolution of the image display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion 2a close to one pixel of the liquid crystal display panel 11. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective member 2a and the pixels of the liquid crystal display panel 11, it is advisable to design the pitch ratio of each to be a different from an integral multiple of one pixel.

[0065] The retroreflective portion 2 a may be shaped so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel 11 .

[0066] On the other hand, in order to manufacture the retroreflective member 2 at a low cost, it is preferable to use a roll press method. Specifically, this method is a method in which the retroreflective portions 2a are aligned and shaped on a film. In this method, the inverse shape of the shape to be shaped is formed on the roll surface, an ultraviolet curable resin is applied onto a base material for fixing, and the required shape is formed by passing the resin between the rolls, and the resin is cured by irradiating ultraviolet rays to obtain the retroreflective member 2 of the desired shape.

[0067] [The second method for creating floating images in space] Next, Fig. 4 shows another example (referred to as the second method) of the formation of the space-floating image 3 in the space-floating image display device of this embodiment and the configuration of the retroreflective member. Fig. 4(A) shows an overview of the formation of the space-floating image 3 using the retroreflective member 330 in the second method. With respect to the retroreflective member 330, light from an object P (corresponding point P) in one space (in this example, the space below in the Z direction) is incident on the retroreflective member 330 and is retroreflected to form a space-floating image 331 (corresponding point Q) in the other space (in this example, the space above in the Z direction).

[0068] FIG. 4(B) shows the surface shape of a typical retroreflective member 330 manufactured by Asukanet Co., Ltd., which was used in this study. The retroreflective member 330 has four-sided structures (in other words, tetrahedrons) 330A regularly arranged on the surface (XY plane in the figure). A plurality of structures 330A are arranged between side walls 330B. The four-sided structure 330A is, for example, a micromirror having a rectangular prism shape extending in the Z direction. For example, light (also described as object light) from an object P enters the four-sided structure 330A. The light beam that enters the four-sided structure 330A is reflected by two of the walls of the four-sided structure 330A (for example, the reflecting surface RS1 and the reflecting surface RS2). The reflected light beam (both the light beam emitted upward from the reflecting surface RS1 and the light beam emitted upward from the reflecting surface RS2) is shown as reflected light R0. The reflected light R0 is emitted as retroreflected light in a direction corresponding to the incident light, and forms and displays a space floating image 331, which is a real image based on an object P, as shown in FIG. 4(A).

[0069] The resolution of this floating image 331 also depends heavily on the outer shape (diameter) DS and pitch PT of the retroreflective part (four-sided structure 330A) of the retroreflective part 330, as with the first type retroreflective member 2 in FIG. 3 described above. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the outer shape (diameter) DS of the retroreflective part is 120 μm and the pitch PT is 150 μm, one pixel of the floating image 331 in space is equivalent to 150 μm. For this reason, the effective resolution of the floating image 331 in space is reduced to about half.

[0070] Therefore, in order to make the resolution of the spatial floating image 331 equivalent to that of the image display device 10, it is desirable to make the diameter DS and pitch PT of the retroreflective portion (structure 330A) close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective member 330 and the pixels of the liquid crystal display panel, it is preferable to design the pitch ratio of each to be a different integer multiple of one pixel, as described above. In addition, it is preferable to arrange the shape so that none of the sides of the retroreflective portion (structure 330A) overlaps with any of the sides of one pixel of the liquid crystal display panel.

[0071] The light forming the floating image 331 is a collection of light rays that converge from the retroreflective member 330 to the optical image of the floating image 331, and these light rays continue to travel straight even after passing through the optical image of the floating image 331. Therefore, the floating image 331 is an image with high directionality, unlike the diffuse image light formed on a screen by a general projector or the like.

[0072] In the configuration of Fig. 4, when a user views the image from the direction of the arrow A, the floating image 331 in space is seen as a bright image, but when another person views the image from the direction of the arrow B, the floating image 331 in space cannot be seen at all. The characteristics of the floating image 331 in space, like the floating image using the first type of retroreflective member 2 described above, are very suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people facing the user.

[0073] In the second type retroreflective member 330, as shown in FIG. 4B, the light from the object P is incident on one side (lower side in the Z direction) of the retroreflective member 330, reflected by two reflecting surfaces (RS1, RS2) provided on the four walls constituting the retroreflective member 330, and forms a space floating image 331 as reflected light R0 on the other side (upper side in the Z direction) at the position of point Q. At this time, the two reflecting surfaces (RS1, RS2) generate abnormal light R1, R2 as light reflected in a different direction from the reflected light R0. The abnormal light R1, R2 generated by the two reflecting surfaces (RS1, RS2) generates ghost images 332, 333 as shown in FIG. 4A. For this reason, the ghost images 332, 333 may be a factor that causes a deterioration in the image quality of the space floating image 331.

[0074] As described above, the retroreflective member 2 in the first method generates ghost images according to the number of reflective surfaces. In contrast, the retroreflective member 330 in the second method generates ghost images only in two specific directions depending on the angle of incidence of the object light. Therefore, the retroreflective member 330 in the second method is less affected by ghost images and can display a high-quality spatial image. Therefore, the following description of the space floating image display device will be limited to the case where the retroreflective member 330 in the second method is applied.

[0075] [Technical measures to reduce ghost images] In order to realize a spatial image display device capable of forming a high-quality spatial floating image with reduced ghost images, it is advisable to provide an image light control sheet on the exit surface of the liquid crystal display panel in order to control the divergence angle of the image light from the liquid crystal display panel as the image display element and bend it in a desired direction. Furthermore, it is advisable to provide an image light control sheet on the light exit surface or light entrance surface of the retroreflective member 330, or on both surfaces thereof, to absorb the extraordinary light R1, R2 (FIG. 4(B)) that causes ghost images.

[0076] FIG. 5 shows a specific method and configuration example of applying the image light control sheet to a space floating image display device. In FIG. 5, an image light control sheet 334 is provided on the emission surface of a liquid crystal display panel 335, which is an image display element. In FIG. 5, the emission surface of the liquid crystal display panel 335 is shown as a plane (XY plane) formed by the X-axis and Y-axis shown in the figure. The image light control sheet 334 has a transmissive portion and a light absorbing portion on the main surface (XY plane). In this case, moire may occur due to interference caused by the pitch between the pixels of the liquid crystal display panel 335 and the transmissive portions and light absorbing portions of the image light control sheet 334. The following two methods are effective for reducing this moire.

[0077] (1) As a first method, the vertical stripes (diagonal lines shown in the figure) caused by the transmissive and light-absorbing parts of the image light control sheet 334 are arranged at a predetermined angle (tilt) θ0 with respect to the pixel arrangement (X-axis and Y-axis) of the liquid crystal display panel 335.

[0078] (2) In the second method, when the pixel size of the liquid crystal display panel 335 is A and the pitch of the vertical stripes of the image light control sheet 334 is B, the ratio (B / A) between them is selected to be a value other than an integer multiple. Since one pixel of the liquid crystal display panel 335 is made up of three sub-pixels of the colors RGB arranged in parallel and is generally square, it is not possible to suppress the occurrence of the above-mentioned moire over the entire screen. For this reason, the inclination θ0 shown in the first method of (1) may be optimized within the range of 5 degrees to 25 degrees so that the position where the moire occurs can be intentionally shifted to a position where the spatial floating image is not displayed.

[0079] Although the above-mentioned moire reduction has been described using a liquid crystal display panel and an image light control sheet 334 as a subject, the same method and configuration can be applied to the moire that occurs between the retroreflective member 330 and the image light control sheet 334 when the image light control sheet 334 is provided on the retroreflective member 330. Since both the retroreflective member 330 and the image light control sheet 334 are linear structures, it is sufficient to optimally tilt the image light control sheet 334 with attention paid to the X-axis and Y-axis of the retroreflective member 330. This makes it possible to reduce large-scale moire that is visible to the naked eye and has a long wavelength and low frequency.

[0080] 6(A) shows a vertical cross-sectional view of an image display device 10 having a configuration in which an image light control sheet 334 is arranged on an image light exit surface 3351 of a liquid crystal display panel 335. The image light control sheet 334 has light transmitting portions 336 and light absorbing portions 337 arranged alternately on a main surface thereof, and is adhesively fixed to the image light exit surface 3351 of the liquid crystal display panel 335 by an adhesive layer 338.

[0081] As described above, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used as the image display device 10, even if one pixel (one triplet) (indicated by A in the figure) is about 80 μm, the ghost images 332 and 333 appearing on both sides of the spatial floating image 331 in FIG. 4(A) can be reduced by the following configuration. For example, the pitch B of the image light control sheet 334 is set to 340 μm, which is composed of the distance d2 of the light transmitting portion 336 being 300 μm and the distance d1 of the light absorbing portion 337 being 40 μm. In this case, the image light control sheet 334 can reduce ghost images by controlling sufficient transmission characteristics and the diffusion characteristics of the image light from the image display device 10 that causes abnormal light. In this case, if the thickness of the image light control sheet 334 is set to 2 / 3 or more of the pitch B, the ghost reduction effect is significantly improved.

[0082] 6(B) shows a vertical cross-sectional view of a configuration in which an image light control sheet 334 is disposed on the image light exit surface of the retroreflective member 330 (FIG. 4). The image light control sheet 334 is configured with light transmitting portions 336 and light absorbing portions 337 alternately arranged, and is inclined with a predetermined inclination angle θ1 with respect to the retroreflective member 330 in accordance with the exit direction of the retroreflected light 3341. As a result, the image light control sheet 334 absorbs the abnormal light R1, R2 (FIG. 4(B)) generated by the retroreflection described above, while allowing the normally reflected light to pass through without loss as the retroreflected light 3341.

[0083] The retroreflective member 330 has spaces 3301 arranged therein, which correspond to the retroreflective portion formed by the four-sided structure 330A (FIG. 4) described above. The spaces 3301 corresponding to the retroreflective portions are partitioned by the surfaces of the side walls 330B. The spaces 3301 have, for example, reflective surfaces R1 and R2. For example, light a1 incident on the retroreflective member 330 from below is reflected by, for example, the reflective surface RS1 of the spaces 3301, and the reflected light a2 is further reflected by, for example, the reflective surface RS2, and is emitted to the upper side of the retroreflective member 330. The emitted light is incident on the image light control sheet 334 and is emitted as retroreflected light 3341.

[0084] When a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used, even if one pixel (one triplet) is about 80 μm, the configuration shown in FIG. 4(A) can reduce ghost images 332 and 333 that occur on both sides of a floating image 331 in space. As shown in FIG. 6(B), for example, the pitch B of the image light control sheet 334 is 420 μm, with the distance d2 of the light transmitting portion 336 of the retroreflective member 330 being 400 μm and the distance d1 of the light absorbing portion 337 being 20 μm. In this case, the image light control sheet 334 provides sufficient transmission characteristics and controls the diffusion characteristics of the image light from the image display device 10 that causes abnormal light to occur in the retroreflective member 330, thereby reducing ghost images.

[0085] The image light control sheet 334 described above also prevents external light from entering the space floating image display device, which leads to improved reliability of the components. For example, a viewing angle control film (VCF) by Shin-Etsu Polymer Co., Ltd. is suitable as the image light control sheet 334. The VCF has a sandwich structure in which transparent silicon and black silicon are alternately arranged, and synthetic resin is arranged on the light entrance and exit surfaces. Therefore, when this VCF is applied as the image light control sheet 334 of this embodiment, the above-mentioned effects can be expected.

[0086] [Technology for sensing operations on floating images in space] A user (sometimes referred to as a user, operator, observer, etc.) is bidirectionally connected to the space-floating image display device 1 via the space-floating image 3 (FIG. 2, etc.) by the space-floating image display device 1. In other words, the user uses the application of the space-floating image display device 1, for example, a function as a telephone displaying a numeric keypad or the like as a space-floating image, by watching and operating the space-floating image 3. For this purpose, a sensing technology is required for the user to pseudo-operate the space-floating image 3 and sense the operation. An example of this sensing technology will be described below with a specific example. The "sensing technology" here includes the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 described using FIG. 2, and is particularly a technology for detecting the user's operation in three-dimensional space (in other words, an aerial operation). The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be described as a sensing system.

[0087] FIG. 7(A) shows a principle diagram for explaining the first sensing technology. As shown in FIG. 7(A), the sensing areas a0, a1, a2, and a3 for the space floating image FI are each divided into a plurality of areas (in other words, regions). In this embodiment, the sensing areas a0, a1, a2, and a3 are vertically and horizontally divided into 12 areas (3×4=12). In FIG. 7(A), the surface of the space floating image FI is shown as an xy surface, and the front-back direction with respect to the surface is shown as the z direction. For example, the illustrated area A301 is one area in the upper left of the sensing surface a3.

[0088] In the first sensing technology, a first distance measuring device 340 is provided that incorporates a TOF (Time of Flight) system corresponding to each area of ​​the floating in space image FI. The first distance measuring device 340 is provided in place of the aerial operation detection sensor 1351 in FIG. 2. A light emitting section of the first distance measuring device 340 emits light from a near-infrared light emitting LED (Light Emitting Diode) that serves as a light source in synchronization with a system signal. An optical element for controlling the divergence angle is provided on the light emitting side of the LED, and a pair of highly sensitive avalanche diodes (ABD) with a picosecond time resolution as light receiving elements are aligned in four columns and three rows to correspond to the 12 areas. The LED light source emits light in synchronization with a signal from the system, and during the time it takes for the light to reflect off the object to be measured (here, the tip of the user's finger UH) and return to the light receiving unit, a phase shift occurs, i.e., a time lag between the timing of light emission and the timing of light reception, more specifically, Δt0 to Δt11 in Figure 9, which will be described later.

[0089] The arithmetic unit of the sensing system shown in FIG. 9(B) receives a signal from the system and a signal generated by an avalanche diode, which is a light receiving unit of the first distance measuring device 340, and calculates the phase shift from these signals to calculate the distance to the object. The distance is calculated for each TOF system (TOF1 to TOF12) associated with each area. In FIG. 7(A), sensing surfaces a3, a2, and a1 (also written as the first sensing surface a3, the second sensing surface a2, and the third sensing surface a1) are shown in order of proximity to the object as the measurement layer of the distance measuring device 340 on the side closer to the object (finger UH) in the z direction with respect to the surface of the floating image FI in space, and sensing surface a0 is shown as the measurement layer on the side farther from the floating image FI in space. Distance L1 indicates the distance to the sensing surface a0, distance L2 indicates the distance to the sensing surface a1, distance L3 indicates the distance to the sensing surface a2, and distance L4 indicates the distance to the sensing surface a3.

[0090] Next, the sensing system can recognize the direction of movement of the object (finger UH) by recognizing which of the 12 areas it passed through on each measurement level (sensing surfaces a3 to a1) and calculating the movement time on each measurement level using the method described above.

[0091] FIG. 9(A) shows the timing of light emission from the LED light source and the timing of light reception by the light receiving element for each of the 12 measurement areas. SU1 to SU12 show the timing of light emission and the timing of light reception for each sensing unit associated with each area and TOF. As shown in FIG. 9(A), it can be seen that in sensing unit SU1, the time difference between the timing of light emission and the timing of light reception is Δt0, in sensing unit SU2, the time difference between the timing of light emission and the timing of light reception is Δt1, and in sensing unit SU12, the time difference between the timing of light emission and the timing of light reception is Δt11. Here, the sensing system standardizes the individual data by delaying the timing of light emission from the LED for each of the 12 measurement areas.

[0092] In reality, suppose that the user willfully extends the fingers UH toward the floating image FI in order to be connected to the system bidirectionally. In this case, the sensing system obtains a first sensing signal S1 sensed, for example, in area A301 on the sensing surface a3 that is the furthest from the floating image FI, a second sensing signal S2 sensed, for example, in a specific area on the sensing surface a2, and a third sensing signal S3 sensed, for example, in a specific area on the third sensing surface a1. The sensing system calculates and processes the contact position with the floating image FI from the movement direction of the fingers UH and the time difference when they crossed each sensing surface using these sensing signals (S1 to S3).

[0093] In order to obtain even more accurate position information, a sensing surface a0 is set at a position farther back from the floating image FI. Based on the sensing at the sensing surface a0, the sensing system detects the passage of the finger UH through the floating image FI as an end signal, and obtains the contact point with the floating image FI as a three-dimensional coordinate from the position coordinate of the detection and the two sensing signals described above.

[0094] Also, FIG. 7(B) shows the action of selecting a part of the floating image FI by the user's finger UH (particularly the fingertip) and the action of the user's finger UH moving away from a part of the floating image FI. As shown in FIG. 7(B), in the first sensing technology, when the user moves his / her finger UH back after touching the desired position coordinate of the floating image FI, the following occurs. That is, the sensing system sequentially transmits the first sensing signal S1 sensed by the first sensing surface a1, the second sensing signal S2 sensed by the second sensing surface a2, and the third sensing signal S3 sensed by the third sensing surface a3 to the arithmetic unit of the sensing system and performs calculation processing. As a result, the system recognizes that the user's finger UH has moved away from a specific coordinate of the floating image FI.

[0095] Next, we will explain a more accurate sensing technology for artificially manipulating floating images in space.

[0096] FIG. 8(A) shows a principle diagram for explaining the second sensing technology. The second sensing technology differs from the first sensing technology shown in FIG. 7(A) in that in addition to the first distance measuring device 340, a second distance measuring device 341 is arranged in parallel to achieve more accurate sensing. In the second sensing technology, the second distance measuring device 341 (particularly a CMOS sensor) is used as a second sensing system in combination with the first sensing system using the first distance measuring device 340. As shown in FIG. 8(A), the second distance measuring device 341 senses the same range (sensing planes a1, a2, a3, a0) as the first distance measuring device 340.

[0097] As described above, the first distance measuring device 340 has a built-in TOF system corresponding to each of a plurality of areas, for example, divided into 12 areas, in the space floating image FI (the sensing system in FIG. 9(B) is the first sensing system). On the other hand, the second distance measuring device 341 uses a two-dimensional image sensor, for example, a 1 / 4-inch CMOS sensor for sensing cameras. The aspect ratio of this CMOS sensor is generally 3:4. For this reason, in this embodiment, in accordance with the aspect ratio of the CMOS sensor, the sensing area of ​​the TOF system of the first distance measuring device 340 is also divided vertically into 3 and horizontally into 4, as described above, for a total of 12 areas.

[0098] In addition, the resolution of the CMOS sensor is sufficient even with about 1 million pixels, but unlike a normal camera system, there is no need to provide an RGB color separation filter. Therefore, the CMOS sensor can not only be made smaller and more sensitive with the same number of pixels, but also has high sensitivity to near-infrared light. Therefore, in the second sensing technology, the object to be measured (the tip of the finger UH) is illuminated by the light source light of the TOF system of the first distance measuring device 340 at a timing determined for each area, thereby significantly improving the detection accuracy. Although a detailed explanation is omitted, FIG. 9(B) shows the above-mentioned system as a functional block diagram.

[0099] Fig. 8(B) shows sensing planes a1, a2, and a3 by the first distance measuring device 340, and sensing planes b1, b2, and b3 by the second distance measuring device 341 that correspond to the sensing planes. Fig. 8(B) also shows the action of selecting a part of the floating-in-space image FI by the fingers UH and the action of moving away from the part on those sensing planes.

[0100] As shown in FIG. 8B, in the space-floating image display device using the second sensing technology, when the user willfully extends his / her finger UH toward the space-floating image FI, the following occurs. In this case, in addition to the 3D information by the first distance measuring device 340 described above, 3D information by the second distance measuring device 341 is obtained. The planar resolution of the sensing surface b3 of the second distance measuring device 341 corresponding to the sensing surface a3 of the first distance measuring device 340 that is the furthest from the air-floating image FI can be made highly accurate according to the resolution of the CMOS sensor used. Similarly, the sensing surface b2 corresponds to the sensing surface a2, and the sensing surface b1 corresponds to the sensing surface a1. This makes it possible to realize a sensing system with significantly improved resolution in the planar direction.

[0101] At this time, the contact point with the floating image FI is calculated based on the time difference between the moving direction of the target object (the tip of the user's finger UH) and the time when the target object crosses the sensing planes of the first distance measuring device 340 and the second distance measuring device 341. In order to obtain more accurate position information, a sensing plane a0 is set away from the floating image FI. The sensing system detects the passage of the finger UH through the floating image FI as an end signal, and can calculate the contact point with the floating image FI as a more precise three-dimensional coordinate from the position coordinate on the sensing plane a0 and the two detection signals described above.

[0102] Furthermore, when the frame rate of the CMOS sensor is increased from 1 / 20 seconds to 1 / 30 seconds or 1 / 120 seconds, the resolution is significantly improved because the amount of planar information captured per unit time increases in addition to the detection accuracy in the planar direction. At this time, the detection information by the second sensing technology is systemized with the position information by the first sensing technology by a synchronization signal supplied from the system.

[0103] Furthermore, as shown in Fig. 8(B), when the user touches the desired position coordinates of the floating image FI and then returns the finger UH, the first sensing signal S1 sensed by the first sensing surface a1, the second sensing signal S2 sensed by the second sensing surface a2, and the third sensing signal S3 sensed by the third sensing surface a3 are sequentially transmitted to the arithmetic unit of the sensing system, as in the first sensing technology described above. Then, the system recognizes that the user's finger UH has left the specific coordinates of the floating image FI by the calculation process in the arithmetic unit.

[0104] The LED light source used in the TOF sensor of the first ranging device 340 of the sensing system described above should prevent a decrease in accuracy of the ranging device due to external light such as sunlight, and should use near-infrared light with high light energy in the range beyond the visible light range (380 nm to 780 nm) that cannot be seen by the naked eye.

[0105] A characteristic diagram of the spectral irradiance of sunlight is shown in Fig. 10. As the wavelength of the light source light of the LED of the TOF sensor, it is preferable to use light of wavelength λ1 of 920 nm, which has a low energy of the spectral irradiance of sunlight shown in Fig. 10.

[0106] <Second Configuration Example of the Space Floating Image Display Device> FIG. 12(A) shows a main part configuration of a space-floating image display device 1 according to an embodiment. FIG. 12(B) is an enlarged view of a lenticular lens 1103 arranged on the image light output side of the image display device 10 shown in FIG. 12(A), that is, on the image light output side of the liquid crystal display panel 11. The space-floating image display device 1 shown in FIG. 12(A) is a system suitable for a user who is a monitor (viewer) to observe the space-floating image 3 from diagonally above. In the coordinate system (X, Y, Z) in FIG. 12(A), the housing 350 of the space-floating image display device 1 is arranged on a horizontal plane (XY plane), and the space-floating image 3 is formed at a slight angle in the front-back direction (Y direction) with respect to the vertical direction (Z direction). When the surface of the space-floating image 3 is viewed favorably from the user's viewpoint E, the viewpoint E is arranged slightly diagonally above the surface of the space-floating image 3 in the Y direction in line with the optical axis J2 as shown in the figure. The user can best view the floating in space image 3 by gazing slightly diagonally downward from viewpoint E in the Y direction.

[0107] On the other hand, Fig. 12(B) is a view seen from the opposite direction to the direction of light emitted from the liquid crystal display panel 11. As shown in Fig. 12(B), the lenticular lens 1103 is disposed approximately parallel or parallel to the light emission surface of the liquid crystal display panel 11, and is disposed on the side of the image light emitted from the liquid crystal display panel 11 surface. Also, the multiple semi-cylinders (semi-cylindrical lenses) of the lenticular lens 1103 are disposed side by side, extending vertically when viewed from the XZ plane. The above coordinate system (X, Y, Z) is common to Figs. 12(A) and (B).

[0108] The image display device 10 and other components are arranged in a predetermined positional relationship inside the housing 350. The upper surface (XY surface) of the housing 350 has an opening, and a retroreflective member 330 is arranged at a predetermined angle α1. The optical axis J1 of the image display device 10 faces obliquely upward at a predetermined angle β1 with respect to the Y direction.

[0109] The image display device 10 is configured to include a liquid crystal display panel 11 as an image display element, and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic. The liquid crystal display panel 11 can be applied to a small one with a screen size of about 5 inches to a large one exceeding 80 inches, and is configured to include a panel selected from them. Image light from the liquid crystal display panel 11 is emitted toward a retroreflective member 330 (also described as a retroreflective portion or a retroreflective plate) on an optical axis J1. Light from the light source device 13 with a narrow divergence angle is made incident on the liquid crystal display panel 11. This generates an image light flux φ1 with a narrow divergence angle. The image light flux φ1 with a narrow divergence angle is made incident on the retroreflective member 330 from the lower side in the Z direction along the optical axis J1. Due to retroreflection at this retroreflective member 330, an image light flux φ2 with a narrow divergence angle is generated on the upper side in the Z direction with respect to the retroreflective member 330, in the direction of the optical axis J2, according to the principle described in FIG. 4 above. The image light flux φ2 produces a space floating image 3 (space floating image 331 in FIG. 4) at a predetermined position outside the housing 350. The optical axis J2 faces obliquely upward at a predetermined angle β2 with respect to the Y direction.

[0110] The floating image 3 is formed at a symmetrical position on the image display device 10 with the retroreflective member 330 as a symmetrical plane. The surface of the image display device 10 and the surface of the floating image 3 are arranged in roughly symmetrical or symmetrical positions with respect to the surface of the retroreflective member 330 arranged at an angle. On the surface of the floating image 3, r2 indicates the center position corresponding to the optical axis J2, r1 indicates the lower end position corresponding to the lower end light ray of the image light beam φ2, and r3 indicates the upper end position corresponding to the upper end light ray of the image light beam φ2.

[0111] In this configuration, in order to obtain a high-quality spatial floating image 3 by eliminating the ghost images 332 and 333 generated by the retroreflective member 330 as described in Fig. 4, an image light control sheet 334 (see Fig. 5 and Fig. 6(A) for details) is provided on the exit side of the liquid crystal display panel 11. This controls the diffusion characteristics in unnecessary directions.

[0112] Furthermore, as shown in FIG. 11, the image light from the liquid crystal display panel 11 can theoretically have a high reflectance on a reflective member such as a retroreflective member 330, so it is preferable to use S-polarized waves (electromagnetic waves whose electric field components are perpendicular to the incident surface, S stands for Senkrecht). However, when a user uses polarized sunglasses, the floating image 3 is reflected or absorbed by the polarized sunglasses, so to deal with this, it is preferable to use P-polarized waves (electromagnetic waves whose electric field components are parallel to the incident surface, P stands for parallel). For this purpose, a depolarizing element 339 is provided as an element that optically converts a part of the image light of a specific polarization into the other polarization and converts it into pseudo-natural light. For example, the depolarizing element 339 is disposed on the exit side of the image light control sheet 334. This allows the user to view a good floating image 3 even when using polarized sunglasses.

[0113] Commercially available products of the depolarizing element 339 include Cosmoshine SRF (manufactured by Toyobo Co., Ltd.) and depolarizing adhesive (manufactured by Nagase & Co., Ltd.). In the case of Cosmoshine SRF (manufactured by Toyobo Co., Ltd.), by attaching the adhesive to the image display device, it is possible to reduce reflection at the interface and improve brightness. In addition, in the case of the depolarizing adhesive (manufactured by Nagase & Co., Ltd.), it is used by attaching a colorless transparent plate and an image display device via the depolarizing adhesive.

[0114] In this embodiment, an image light control sheet 338B (similar to the image light control sheet 338, see FIG. 6B for details) is also provided on the image exit surface of the retroreflective member 330. This eliminates ghost images 332, 333 (FIG. 4) that are generated on both sides of the normal image of the spatial floating image 3 due to unnecessary light.

[0115] In this embodiment, the retroreflective member 330 is inclined at a predetermined angle α1 with respect to the horizontal axis (Y direction), and the floating image 3 is generated obliquely with respect to the horizontal axis (particularly, at an angle closer to the vertical plane than to the horizontal plane). Not limited to this, by changing the arrangement of the components, the position and inclination of the floating image 3 can be designed.

[0116] In this embodiment, a first distance measuring device 340 (FIG. 7) is attached to a predetermined position of the housing 350. That is, the same sensing technology as in FIG. 7 is implemented in this system. This allows the user to access and interact with the floating in space image 3. The first sensing system including the first distance measuring device 340 detects the state of operation (air operation) by the user's fingers or the like on the floating in space image 3. Furthermore, a second sensing system including a second distance measuring device 341 may be added, as in FIG. 8 and FIG. 9(B).

[0117] The mounting position and viewing angle α3 of the first distance measuring device 340 may be appropriately selected so as to sufficiently cover the size of the floating image 3. In this example, the first distance measuring device 340 is mounted at the illustrated position on the rear side of the housing 350 in the Y direction (the rear side relative to the user and the floating image 3), on the extension of the inclined surface of the retroreflective member 330, and at a position slightly away so as not to block the image luminous flux of the image light. In this example, the viewing angle α3 (range from the upper end A to the lower end B) of the first distance measuring device 340 is set to a sufficiently wide viewing angle so as to cover the entire floating image 3 and the area including the face of the user viewing it from the viewpoint E of the reference position (directly facing position). The viewing angle α3 includes the viewing angle α2 that captures the entire floating image 3. The viewing angle α2 corresponds to the sensing planes a0, a1, a2, and a3 in FIG. 7, for example.

[0118] The TOF sensor of the first distance measuring device 340 uses a distance measuring system in which the sensing surface of the space floating image 3 is divided into multiple areas as shown in FIG. 7 (or FIG. 8). This improves the resolution of each sensing area. Furthermore, when using the second sensing technology using a CMOS sensor as shown in FIG. 8 and FIG. 9(B), the detection accuracy can be further improved.

[0119] In this embodiment, a light source that emits visible light having a narrow-angle directional characteristic is used as the light source device 13, and the first distance measuring device 340 (and the second distance measuring device 341) is disposed at an outer position with respect to the narrow-angle image light beam on the housing 350 side. The second distance measuring device 341 may also be disposed in a similar manner. This can eliminate adverse effects on the sensing accuracy of the image light that forms the space floating image 3.

[0120] In the above configuration shown in FIG. 12(A), a lenticular lens 1103 is disposed on the image light emission side (position indicated by diagonal lines) of the liquid crystal display panel 11. More specifically, the lenticular lens 1103 is disposed on the image light emission side of the liquid crystal display panel 11 so as to be oriented as shown in FIG. 12(B). On the surface of the lenticular lens 1103 (here, the xy surface), semi-cylindrical lenses extending in the y direction (vertical) are arranged as a plurality of semi-cylindrical lenses in the x direction (horizontal). The x direction corresponds to the horizontal direction in the screen of the display panel 11, and the y direction corresponds to the vertical direction in the screen of the display panel 11. With this configuration, as described later, a user can move in the direction (x direction, X direction) in which the semi-cylindrical lenses forming the lenticular lens 1103 are arranged, and the user can view different images (or videos) from each position. That is, by displaying parallax images as images or videos, motion parallax occurs, and the images or videos displayed on the liquid crystal display panel 11 can be recognized as a stereoscopic image. Multi-viewpoint images and motion parallax will be described later.

[0121] 12(A), the space-floating image 3 is a real image formed at a symmetrical position on the image display device 10 or the liquid crystal display panel 11 with the retroreflective member 330 as a symmetrical plane, so that the user can visually recognize the space-floating image 3 as a stereoscopic image with motion parallax. That is, according to the above configuration in which the lenticular lens 1103 is arranged, the space-floating image 3 can be displayed not simply as a two-dimensional image displayed on the liquid crystal display panel 11, but as a stereoscopic image with motion parallax.

[0122] <Third Configuration Example of the Space Floating Image Display Device> FIG. 13(A) shows another embodiment of the space-floating image display device. FIG. 13(B) is an enlarged view of the lenticular lens 1103 arranged on the image light output side of the image display device 10 shown in FIG. 13(A), that is, on the light output side of the liquid crystal display panel 11. In the coordinate system (X, Y, Z) in FIG. 13(A), the housing 350 of the space-floating image display device 1 is arranged on a horizontal plane (XY plane), and the space-floating image 3 is formed at a slight incline in the front-back direction (Y direction) with respect to the vertical direction (Z direction). When the surface of the space-floating image 3 is viewed favorably from the user's viewpoint E by facing it directly, the viewpoint E is arranged slightly diagonally upward in the Y direction in line with the optical axis J2 with respect to the surface of the space-floating image 3, as shown in the figure. The user can favorably view the space-floating image 3 with a line of sight slightly diagonally downward from the viewpoint E in the Y direction.

[0123] 13B, the lenticular lens 1103 is disposed approximately parallel to or parallel to the light exit surface of the liquid crystal display panel 11, and on the side of the image light exiting from the liquid crystal display panel 11. The multiple semi-cylinders (semi-cylindrical lenses) of the lenticular lens 1103 are arranged side by side, extending vertically when viewed from the XZ plane. The above coordinate system (X, Y, Z) is common to both Figs. 13A and 13B.

[0124] The image display device 10, mirror 360, and the like are arranged in a predetermined positional relationship inside the housing 350. In the opening of the housing 350, which in this example has a surface (XZ surface) that stands approximately vertically, a retroreflective member 330 is arranged at a predetermined angle γ1 (slightly inclined downward) with respect to the Z direction. The mirror 360 is a flat mirror.

[0125] In this embodiment, the image light from the image display device 10 is reflected by the mirror 360 and then enters the retroreflective member 330. The housing 350 has a portion that protrudes upward in the Z direction, and the image display device 10 is disposed within that portion. The optical axis J1 of the image display device 10 faces downward in the Z direction and toward the rear in the Y direction, and is obliquely downward at a predetermined angle δ1 with respect to the Z direction.

[0126] The image display device 10 is configured to include a liquid crystal display panel 11 as an image display element, and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic. The liquid crystal display panel 11 can be applied to a small one with a screen size of about 5 inches to a large one exceeding 80 inches, and is configured to include a panel selected from these. The image light from the liquid crystal display panel 11 is folded back on the optical axis J1 by mirror 360, which is an optical path folding mirror, and is emitted toward the retroreflective member 330 on the folded optical axis J1B.

[0127] Light from the light source device 13 with a narrow divergence angle is incident on the liquid crystal display panel 11. This generates an image light beam φ1 with a narrow divergence angle. The image light beam φ1 with a narrow divergence angle is reflected by the mirror 360 and becomes an image light beam φ1B. The image light beam φ1B with a narrow divergence angle is incident on the retroreflective member 330 from the right side in the Y direction shown in the figure along the optical axis J1B. Due to retroreflection by this retroreflective member 330, an image light beam φ2 with a narrow divergence angle is generated in the direction of the optical axis J2 on the left side in the Y direction with respect to the retroreflective member 330 according to the principle described in FIG. 4 above. The image light beam φ2 provides a floating image 3 (a floating image 331 in FIG. 4) at a predetermined position outside the opening of the housing 350. The optical axis J2 is oriented obliquely upward at a predetermined angle δ2 with respect to the Y direction (angle (90 degrees-δ2) with respect to the Z direction).

[0128] The floating image 3 is formed at a position roughly symmetrical to the mirror 360, with the retroreflective member 330 as a plane of symmetry. In this embodiment, the mirror 360 is used to turn back the optical path, so the image display device 10 is disposed above the floating image 3 in the Z direction. As a result, a system can be realized in which the image light beam enters the retroreflective member 330 from obliquely above and exits obliquely above, forming the floating image 3 tilted obliquely as shown in the figure.

[0129] Moreover, in order to form the floating image 3 obliquely upward (on the optical axis J2 shown in the figure) on the housing 350, the retroreflective member 330 can be arranged to be tilted at a predetermined angle γ1 with respect to the vertical axis (Z direction) of the bottom surface of the housing 350 as shown in the figure. Also, as a result of such a configuration in which the emission axis of the retroreflective member 330 is tilted slightly obliquely downward, it is possible to prevent deterioration in the image quality of the floating image 3 that may occur when external light enters the retroreflective member 330 and enters the inside of the housing 350.

[0130] In order to eliminate ghost images (FIG. 4) that may occur in the space floating image 3 and obtain a higher quality space floating image 3, in the same manner as in the second configuration example (FIGS. 12(A) and (B)), in this embodiment, the diffusion characteristics in unnecessary directions may be controlled by providing an image light control sheet 334 (FIGS. 5 and 6(A)) on the exit side of the liquid crystal display panel 11. In addition, by providing an image light control sheet 334 (FIG. 6(B)) on the image exit surface of the retroreflective member 330, ghost images that occur on both sides of the normal image of the space floating image 3 due to unnecessary light may be eliminated.

[0131] By disposing the above-described structures inside the housing 350, it is possible to prevent external light from being incident on the retroreflective member 330, and thus to prevent the occurrence of ghost images.

[0132] In this embodiment, the image light from the liquid crystal display panel 11 may use S-polarized waves as in FIG. 12(A), or, in the case of compatibility with polarized sunglasses, a depolarization element 339 may be provided using P-polarized waves.

[0133] In this embodiment, the retroreflective member 330 is inclined at a predetermined angle γ1 with respect to the vertical axis (Z direction), and the floating image 3 is generated obliquely with respect to the horizontal axis (particularly, at an angle closer to the vertical plane than to the horizontal plane). Not limited to this, by changing the arrangement of the components, the position and inclination of the arrangement of the floating image 3 can be designed and adjusted.

[0134] In this embodiment, a first distance measuring device 340 (FIG. 7) is attached to a predetermined position of the housing 350. That is, the same sensing technology as in FIG. 7 is implemented in this system. This allows the user to access and interact with the floating-in-space image 3. The first sensing system including the first distance measuring device 340 detects the state of operation (air operation) by the user's fingers or the like on the floating-in-space image 3.

[0135] The mounting position and viewing angle γ3 of the first distance measuring device 340 may be appropriately selected so as to sufficiently cover the size of the floating image 3. In this example, the first distance measuring device 340 is mounted at the bottom of the housing 350, in the vicinity of the retroreflective member 330 in the Y direction, and at a position slightly away so as not to block the image light flux of the image light, as shown in the figure. In this example, the viewing angle γ3 of the first distance measuring device 340 is set to a sufficiently wide viewing angle so as to cover the entire floating image 3 and the area including the face of the user viewing it from the viewpoint E of the reference position. The viewing angle γ3 includes a viewing angle that captures the entire floating image 3.

[0136] In addition to the first sensing system including the first distance measuring device 340, a configuration may be provided in which a second sensing system including a second distance measuring device 341 (particularly a CMOS sensor) is added, as in Figures 8 and 9(B).

[0137] In this embodiment, a light source that emits visible light having a narrow-angle directional characteristic is used as the light source device 13, and the first distance measuring device 340 (and the second distance measuring device 341) is disposed at an outer position with respect to the narrow-angle image light beam on the housing 350 side. This can eliminate adverse effects on the sensing accuracy of the image light that forms the space floating image 3.

[0138] Furthermore, in this embodiment, a capacitive touch panel 361 may be fixed between the space floating image 3 and the retroreflective member 330 by a support member 362 as shown in the figure. The support member 362 is, for example, in the shape of a frame and supports the touch panel 361 on the inside. The support member 362 is, for example, fixed to the bottom surface of the housing 350. This touch panel 361 is made of a member that transmits image light for forming the space floating image 3 and light from the first distance measuring device 340.

[0139] This touch panel 361 is a capacitive type and detects the state of proximity of the user's finger to the surface of the touch panel. Alternatively, this touch panel 361 detects the state of contact of the user's finger to the surface of the touch panel. By using the third sensing technology including this touch panel 361 together with the first sensing technology, etc., the detection accuracy can be further improved. Similarly, the size and mounting position of this capacitive type touch panel 361 should be selected so that it can sufficiently cover the floating image 3 in space.

[0140] As the capacitive touch panel 361 capable of capturing highly accurate position information, for example, a projected capacitive touch panel can be adopted. This type of touch panel is manufactured by patterning, for example, ITO, which is a transparent electrode (Y-axis electrode) having a fine line-to-line distance, and a copper thin film, which is a transparent electrode (X-axis electrode) having a fine line-to-line distance, on both sides of a transparent glass substrate by photolitho-etching. Therefore, when an object (for example, a finger tip) approaches this transparent glass substrate, the change in capacitance is captured by each of the X-axis electrode and the Y-axis electrode, and the relative coordinates of the object are obtained. In addition, this method can obtain a higher resolution as the line-to-line distance of the transparent electrodes becomes shorter, so that multi-point detection is possible. Therefore, this method also allows simultaneous input by multiple fingers.

[0141] In the above configuration shown in FIG. 13(A), as shown in FIG. 13(B), a lenticular lens 1103 is arranged on the image light output side (position indicated by diagonal lines) of the liquid crystal display panel 11, similarly to the space-floating image display device 1 shown in FIG. 12(A). On the surface of the lenticular lens 1103 (here, the xy surface), semi-cylindrical lenses extending in the y direction (vertical) are arranged as a plurality of semi-cylindrical lenses in the x direction (horizontal). With this configuration, the user can recognize the space-floating image 3 as a stereoscopic image accompanied by motion parallax. That is, according to the above configuration in which the lenticular lens 1103 is arranged, it is possible to display a stereoscopic image as the space-floating image 3, rather than a two-dimensional image simply displayed on the liquid crystal display panel 11.

[0142] Here, the fact that the user can recognize the floating image 3 as a three-dimensional image brings about a new effect not found in conventional systems in which the floating image in space is a two-dimensional plane, especially when the displayed three-dimensional image is a person (particularly a face). For example, as described later, a person (particularly a face) displayed as a floating image in space brings about a new effect that it always faces the user no matter where the user is located around the floating image in space. This gives the user the feeling that the person displayed as a floating image in space is speaking only to the user, which is particularly suitable in situations where the displayed person explains something to the user or provides some kind of assistance (support) to the user.

[0143] [Technology for generating and displaying multi-viewpoint images] As already mentioned, it is well known that motion parallax can be obtained by a multi-view image or a multi-view video using a lenticular lens. A lenticular lens is a collection of lenses (semi-cylindrical lenses) having a semi-cylindrical shape on their surface, which are arranged to extend in a predetermined direction, and a liquid crystal display panel that displays different images corresponding to the number of viewpoints of the multi-view image or the multi-view video is arranged below one semi-cylindrical lens. In this embodiment (the second configuration example in FIG. 12 or the third configuration example in FIG. 13), the predetermined direction (the direction in which the axis of the semi-cylindrical lens extends) is the vertical direction (y direction) as described above.

[0144] FIG. 14(A) is a diagram showing the principle of generating a multi-view image using a lenticular lens 1103 in the present embodiment (second and third configuration examples). FIG. 14(B) is a schematic diagram showing the lenticular lens 1103 as viewed from diagonally above in order to more clearly show the configuration of the lenticular lens 1103. Here, a case of nine viewpoints as a multi-view image will be described. In FIG. 14(A), the pixels 1401 of the liquid crystal display panel 11 form a nine-view multi-view image with nine pixels 1401 numbered 1 to 9 as one group. In FIG. 14(A), the numbers 1 to 9 of the pixels 1401 that appear as images to the observer's eyes are indicated, and may be written as pixels 1 to 9. In FIG. 14(B), the lenticular lens 1103 has a plurality of lenses (semi-cylindrical lenses) 1103a that are repeatedly arranged in the X direction.

[0145] On the other hand, it is known that the distance between human eyes, i.e., the distance between the pupils, is almost constant, and for example, the average interpupillary distance PD for Japanese people is about 64 mm. By making the pitch of the lenticular lens 1103, i.e., the interval between the semi-cylindrical lenses 1103a, approximately the same as half the distance between human eyes, i.e., approximately 32 mm, for example, light from different pixels 1401 reaches the right eye and left eye of the observer (user), as shown in FIG. 14(A). More specifically, light from an image displayed at pixel 6 reaches the observer's right eye, and light from an image displayed at pixel 4 reaches the observer's left eye.

[0146] For this reason, the observer sees images displayed on different pixels 1401 with his / her right and left eyes, and if images of the same object or person captured from different viewpoints are displayed on each pixel 1401, parallax occurs between the observer's eyes. As a result, the observer can recognize the captured image as three-dimensional. As described above, in a configuration in which the lenticular lens 1103 is disposed on the light exit side of the liquid crystal display panel 11, light from different pixels 1401 reaches the observer's right and left eyes, so the observer can recognize a three-dimensional image.

[0147] Here, when the observer (particularly the face) moves left and right (X direction), the light from the pixel 1401 reaches the right eye and the left eye of the observer different from the light from before the movement. More specifically, as shown in FIG. 14(A), when the observer moves one pixel to the right, the light from the image displayed in pixel 7 reaches the right eye of the observer, and the light from the image displayed in pixel 5 reaches the left eye of the observer. That is, as the observer moves (moves), the light from the pixel 1401 different from the light from before the movement reaches the eye. As a result, the observer can obtain an effect equivalent to the same object or person seen from a different angle, that is, motion parallax, as he or she moves left and right. Therefore, in a configuration in which the lenticular lens 1103 is disposed on the light exit side of the liquid crystal display panel 11, the movement of the observer's eyes can obtain an effect equivalent to the image with a three-dimensional effect seen from a different angle.

[0148] FIG. 15 is a schematic diagram showing an example of an apparatus for capturing an image for generating the above-mentioned motion parallax, that is, a multi-viewpoint image. FIG. 15 shows a state in which a person (particularly the face of the person) as a subject 1500 is captured from nine different viewpoints. More specifically, as shown in FIG. 15, nine cameras 1501, cameras No. 1 to No. 9, are arranged at positions on a semicircle that are positions at a predetermined distance from the subject 1500 and shifted by a predetermined angle from each other, and capture the image. In this embodiment, cameras No. 1 to No. 9 are arranged at positions at equal distances from the subject 1500 and shifted by 22.5 degrees from each other, in other words, nine positions obtained by dividing 180 degrees into eight. The distance and angle from the subject 1500 may be changed according to the number of viewpoints.

[0149] At this time, if the subject 1500 is stationary, it is also possible to capture a multi-viewpoint image by moving one camera 1501 to the positions of cameras No. 1 to No. 9 in order to capture the subject. If the subject 1500 is moving, for example the face of a person who is talking and moving their mouth while changing their facial expression, it is also possible to use nine cameras 1501, fix the cameras 1501 in their respective positions, and capture the subject as a video (in other words, a moving image or video).

[0150] The images 1502 (or videos) of the nine cameras 1501 captured as described above are assigned to the nine pixels 1401 of the video display unit, here the liquid crystal display panel 11, and displayed. As shown in FIG. 15, a multi-view image (or multi-view video) with motion parallax can be obtained by displaying a video of one subject 1500 as images 1502 (or videos) captured from different angles. In the example shown in FIG. 15, the face of a person, who is the subject 1500, is captured from different angles by nine cameras No. 1 to No. 9, and the images 1502 captured by the nine cameras 1501 are assigned to the pixels 1401 (pixels 1 to 9) of the liquid crystal display panel 11 and displayed.

[0151] Furthermore, a lenticular lens 1103 is disposed on the light output side of the liquid crystal display panel 11 to obtain a multi-view image (or video) with motion parallax. The method of obtaining a multi-view image (or video) is not limited to the method using one or more cameras 1501 as described above, but may be a method of rendering a multi-view image (or video) by computer graphics (CG). By generating CG by rendering, a large-scale shooting device using multiple cameras is not required, and a multi-view image (or video) can be obtained more simply, without restrictions on the number of viewpoints due to the number of cameras, and in a short time, which is particularly suitable for generating multi-view images and videos.

[0152] FIG. 16 is a diagram showing a display example by a multi-viewpoint image display device. Here, the multi-viewpoint image display device refers to a display device having a configuration in which a lenticular lens 1103 is arranged on the image light output side of an image display device 10 composed of a liquid crystal display panel 11, a light source device 13, etc. Specifically, the multi-viewpoint image display device has a light source device 13, a liquid crystal display panel 11 which is an image display unit, and a lenticular lens 1103. The image display device 10 displays an image including at least two objects, and displays a plurality of images obtained by fixing or shifting the position of an arbitrary object of the at least two objects in the left-right direction (a predetermined direction) as a multi-viewpoint image. In other words, the image display device 10 displays an image including at least two objects, and displays a plurality of images obtained by fixing the position of an arbitrary object (a first object) of the at least two objects and shifting the position of an object (a second object) other than the arbitrary object in the left-right direction between different multi-viewpoint images as a multi-viewpoint image.

[0153] The left-right direction (predetermined direction) here means the left-right direction with respect to the user's viewpoint (X direction in FIG. 14), and corresponds to the direction in which a plurality of semi-cylindrical lenses 1103a in lenticular lens 1103 are repeatedly arranged. Here, the image output surface of liquid crystal display panel 11, which is the image display unit, and the incident surface of lenticular lens 1103 are parallel. Also, the image output surface of liquid crystal display panel 11, which is the image display unit, and the incident surface of lenticular lens 1103 are arranged with a predetermined distance between them. In this embodiment, the predetermined distance between the light incident surface of lenticular lens 1103 and the light exit surface of liquid crystal display panel 11 is adjusted and arranged based on the focal length inherent to lenticular lens 1103. At this time, when the focal length of the lenticular lens 1103 is a relatively large value, the above-mentioned predetermined distance is increased, and conversely, when the focal length of the lenticular lens 1103 is a relatively small value, the above-mentioned predetermined distance is decreased by adjusting the distance between the light entrance surface of the lenticular lens 1103 and the light exit surface of the liquid crystal display panel 11, i.e., the above-mentioned predetermined distance. This makes it possible to display a suitable multi-viewpoint image.

[0154] FIG. 16 shows a multi-viewpoint image display device that displays multi-viewpoint images having nine different viewpoints. FIG. 16(A) shows a case where images 1502 taken by cameras 1501 (No. 1 to No. 9) are arranged in the order of capture for the above-mentioned pixels 1401 (pixels 1 to 9) of the liquid crystal display panel 11. On the other hand, FIG. 16(B) shows a case where images 1502 taken by cameras 1501 (No. 9 to No. 1) are arranged in the reverse order to that of FIG. 16(A) for pixels 1 to 9 of the liquid crystal display panel 11. The difference in effect between FIG. 16(A) and (B) is as follows. First, in FIG. 16(A), when a user (user) moves from the left side to the right side of the multi-viewpoint image display device, the user can observe an image of the subject (person's face) seen from the left side from the left side, and the user can observe an image of the subject (person's face) seen from the right side from the right side. That is, the user can observe the same subject as when the user observes the subject from the left side or the right side, with the actual subject at the center.

[0155] On the other hand, in Fig. 16(B), the situation is reversed from Fig. 16(A), and when a user (observer) moves from the left to the right with respect to the multi-viewpoint video display device, from the left side, an image of the subject (person's face) viewed from the right side of Fig. 16(A) can be observed, and from the right side, an image of the subject (person's face) viewed from the left side of Fig. 16(A) can be observed. As a result, in Fig. 16(B), when a user looks at a subject (person's face), no matter what position (relative angle) the user is in with respect to the subject, the user feels as if the person, the subject, is always looking toward the user.

[0156] 16(B) above, that is, the feature that the subject always seems to be looking toward the user regardless of the user's position, creates an effect that makes the user feel that the subject is always facing him / her (the user) and talking to him / her. This effect is particularly suitable in a scene where the subject is explaining or guiding something to only the user.

[0157] Here, in a multi-viewpoint image display device using a lenticular lens, so-called reverse viewing often becomes a problem. As shown in FIG. 14(A), light from an image displayed on pixel 6, for example, reaches the right eye of the observer, and light from an image displayed on pixel 4 reaches the left eye of the observer, so that the observer can recognize a stereoscopic image. Reverse viewing refers to a phenomenon in which, due to the positional relationship between the observer's eyes and the lenticular lens, light from pixel 4 reaches the right eye of the observer, even though it should be light from pixel 6, and light from pixel 6 reaches the left eye of the observer, even though it should be light from pixel 4. When such reverse viewing occurs, the observer cannot recognize the stereoscopic image that should be observed.

[0158] The occurrence of the above-mentioned reverse viewing can be effectively prevented by using the image display device 10 including the liquid crystal display panel 11 as the image display element and the light source device 13 having a narrow-angle diffusion characteristic. More specifically, the diffusion angle of the lenticular lens for displaying a multi-viewpoint image is generally 40 degrees to 60 degrees (±20 to 30 degrees from the center), whereas the occurrence of reverse viewing can be preferably prevented by using the light source device 13 having a narrow-angle diffusion characteristic with a diffusion angle of 30 degrees (±15 degrees from the center) as the light source of the image display device 10, or by using the image light control sheet 334 shown in FIG. 6.

[0159] Next, Fig. 17 and Fig. 18 are schematic diagrams showing a state in which a space-floating image 3 is generated by passing the image light emitted from the multi-viewpoint image display device equipped with the lenticular lens 1103 shown in Fig. 16(A) and Fig. 16(B) through a retroreflector (retroreflecting member, retroreflective member) 330. Fig. 17 and Fig. 18 are the same in that the space-floating image 3 is generated by the multi-viewpoint image display device and the retroreflector 330.

[0160] The difference between the two embodiments of FIG. 17 and FIG. 18 is that the order of the multi-viewpoint images on the multi-viewpoint image display device is different. That is, FIG. 17 corresponds to FIG. 16(A), and images 1502 by cameras No. 1 to No. 9 are assigned from left to right of the multi-viewpoint image display device as seen by the user, that is, to pixels 1 to 9 of the liquid crystal display panel 11. As a result, in the space floating image 3 generated through the retroreflector 330, the order of the multi-viewpoint images is reversed as seen by the user, and a multi-viewpoint image 1503 corresponding to the images 1502 by cameras No. 1 to No. 9 is displayed from right to left. On the other hand, FIG. 18 corresponds to FIG. 16(B), and images 1502 by cameras No. 1 to No. 9 are assigned from right to left of the multi-viewpoint image display device, that is, to pixels 1 to 9 of the liquid crystal display panel 11. As a result, in the spatial floating image 3 generated via the retroreflector 330, the order of the multi-viewpoint images is reversed from that of FIG. 17 when viewed from the user, with multi-viewpoint images 1503 corresponding to images 1502 taken by cameras No. 1 to No. 9 being displayed from left to right.

[0161] As described above, in the space floating image 3 (multiple viewpoint image 1503) generated via the retroreflector 330, the order of the multi-view images displayed on the image display device 10 and the order of the multi-view images by the space floating image 3 are recognized by the user in a reversed order by arranging the lenticular lens 1103 between the image display device 10 and the retroreflector 330. That is, when providing the user with the space floating image 3 having motion parallax, the order of the images 1502 by the cameras No. 1 to No. 9 may be appropriately determined and arranged with respect to the pixels 1401 on the liquid crystal display panel 11 according to the purpose of what kind of multi-view image should be provided to the user.

[0162] <Example 1 for displaying multi-viewpoint images as floating images in space> Next, a first embodiment of the present invention for displaying a multi-viewpoint video as a space floating image will be described. Fig. 19 is a diagram showing a first embodiment of the present invention for generating a multi-viewpoint image using a lenticular lens 1103.

[0163] Here, FIG. 19 is compared with FIG. 15. First, in FIG. 15, nine cameras 1501 are arranged at positions at a predetermined distance from a subject 1500 (a person's face) and shifted by a predetermined angle (specifically, 22.5 degrees) from each other to capture the subject 1500. In this way, an image 1502 captured by cameras No. 1 to No. 9, which is a multi-viewpoint image, is generated. Then, the images 1502 captured by cameras No. 1 to No. 9 are assigned to pixels 1 to 9, which are nine pixels 1401 of the liquid crystal display panel 11, to form one pixel group. One pixel group is a plurality of pixels 1401 that fit within one lens 1103a among the lenticular lens group (a plurality of semi-cylindrical lenses 1103a) of the lenticular lens 1103, as shown in FIG. 14.

[0164] 19 shows a different method for generating a multi-viewpoint image by not photographing one object from different angles. As an example, the position of the number 0 among five numbers 0, 1, 2, 3, and 4 in the object 1900 is fixed, and the positions of the four numbers 1, 2, 3, and 4 are moved little by little from left to right in the X direction to generate a multi-viewpoint image 1902, which is nine images 1902 shown as image No. 1 to image No. 9. Images No. 1 to No. 9 generated in this way are assigned to pixels 1 to 9, which are pixels 1401 of one pixel group of the liquid crystal display panel 11.

[0165] FIG. 25 is a supplementary explanatory diagram regarding the generation of the multi-viewpoint image 1902, which is the nine images 1902 shown as image No. 1 to image No. 9. Here, only three camera images 1902, image No. 1 of camera No. 1, image No. 2 of camera No. 2, and image No. 9 of camera No. 9, are shown with the same position in the X direction. Of the multiple objects in image No. 1, object 2501 with number 1 will be described as an object (second object) to be moved left and right in the X direction. Assume that object 2501 with number 1 is moved from the leftmost position to the rightmost position in the X direction. In image No. 1, the leftmost position is X1, and in image No. 9, the rightmost position is X9. The moving distance from the leftmost position X1 to the rightmost position X9 is D. The moving distance D is divided by 8 according to the number (9) of images 1902 of the multiple viewpoints (9 viewpoints), and a unit moving distance (D / 8) is obtained. For the object 2501 with the number 1, the position X2 in image No. 2 is a position moved by one unit movement distance (D / 8) from the position X1 in image No. 1. The positions in the other images can be obtained in a similar manner.

[0166] In forming a multi-view image 1902 of a subject 1900, the examples in Figures 19 and the like show a case in which multiple images 1902 are assigned to only one pixel group corresponding to one cylindrical lens 1103a (Figure 14), but this is not limited to the above, and the multi-view image 1902 of the subject 1900 can be constructed using multiple adjacent pixel groups.

[0167] Next, FIG. 20 shows that, as described above, the position of the number 0 (first object) is fixed with respect to the pixels 1 to 9 of the liquid crystal display panel 11, and the numbers 1 to 4 (second object) are simply moved little by little from left to right to generate a multi-viewpoint image 1902, and further, a lenticular lens 1103 is disposed on the image light output side of the liquid crystal display panel 11, so that motion parallax can be obtained. That is, as the viewpoint of the user shown in the figure moves from left to right in the X direction, the position of the number 0 as an image does not change, but the numbers 1 to 4 as images appear to move from left to right. As a result, the user sees the number 0 as being relatively far away (farther back), and the numbers 1 to 4 as being relatively close (farther forward).

[0168] The phenomenon in which the number 0 appears to be far away (farther back) from the observer's (user's) perspective, and the numbers 1 to 4 appear to be close (farther forward), in other words the phenomenon of feeling a sense of perspective and three-dimensionality, can be explained as an analogy with the following physical phenomenon: when a passenger on a train looks out the window of a moving train, distant objects such as mountains and clouds do not change position, but nearby objects such as buildings and fields change position significantly depending on the speed of the train.

[0169] Considering the analogy of the view seen from the window of a moving train as explained above, distant mountains and clouds correspond to the number 0 in FIG. 20. That is, the position of the number 0 does not change even if the user moves from left to right or vice versa. On the other hand, nearby buildings and fields correspond to the numbers 1 to 4 in FIG. 20. That is, the position of the numbers changes significantly when the user moves from left to right or vice versa. This has the effect of making the user feel as if the number 0 is far away, i.e., relatively far back, and the numbers 1 to 4 are close, i.e., relatively closer to the user.

[0170] As described above, when attempting to obtain motion parallax using multi-view images, it is not necessarily necessary to use images 1502 captured by cameras 1501 arranged at different positions on a circle as multi-view images as shown in Fig. 15. Multi-view images can also be generated by simply shifting the relative positions of multiple subjects (numbers 0 to 4 of subject 1900 in Fig. 19) in the left-right direction as shown in Fig. 19. Among subjects 1900 in Fig. 19, number 0 is an example of a first object whose position is fixed, and numbers 1 to 4 are examples of second objects whose positions are shifted left-right.

[0171] Here, the method shown in Fig. 19, that is, the method of generating a multi-viewpoint image by simply shifting the relative positions of the numbers 0 to 4 as objects (second objects) in the same image in the left-right direction, can generate a multi-viewpoint image much more easily than the method of generating a multi-viewpoint image using multiple cameras 1501 arranged on a circumference shown in Fig. 15. Moreover, it has the characteristic that it is possible to obtain motion parallax for multiple objects (here, numbers 0 to 4) displayed in one image.

[0172] Fig. 20 shows an example of display by a multi-viewpoint image display device equipped with an image display device 10 and a lenticular lens 1103, using a multi-viewpoint image 1902 generated by the method shown in Fig. 19. As already described, by combining the multi-viewpoint image 1902 obtained by simply shifting the relative positions of the numbers 0 to 4 in the left-right direction with the lenticular lens 1103, the user can obtain motion parallax and can see the displayed objects three-dimensionally. In the case of Fig. 20, by the user moving left and right in the X direction, the number 0 appears to be present in the back, and conversely, the numbers 1 to 4 appear to be present in the foreground.

[0173] <Example 2 for displaying multi-viewpoint images as floating images in space> 21 shows a case in which, contrary to the case of Fig. 19, the position of the number 0 of the five numbers 0, 1, 2, 3, and 4 in the subject 1900 is fixed, and the positions of the four numbers 1, 2, 3, and 4 are gradually moved from right to left to generate a multi-viewpoint image 1902 consisting of nine images 1902 shown as image No. 1 to image No. 9. Images No. 1 to No. 9 generated in this way are assigned to pixels 1 to 9 of the liquid crystal display panel 11 as one pixel group.

[0174] Fig. 22 shows that, as described above, a multi-viewpoint image 1902 is generated by fixing the position of the number 0 with respect to pixels 1 to 9 and gradually moving the numbers 1 to 4 from right to left, and further, a lenticular lens 1103 is disposed on the image light output side of the liquid crystal display panel 11, thereby enabling motion parallax to be obtained. That is, as the viewpoint of the user shown in the figure moves from left to right in the X direction, the position of the number 0 does not change, but the numbers 1 to 4 appear to move from right to left. In this case, contrary to Fig. 20, the number 0 appears to be in a closer position (toward the viewer), and the numbers 1 to 4 appear to be in a farther position (backward, at the back) to the user.

[0175] The phenomenon in which the number 0 appears to be closer (closer) and the numbers 1 to 4 appear to be further away (farther back) as described above can be explained as an analogy to the following physical phenomenon. That is, in FIG. 22, when the user is on the left side, the numbers 1 to 4 appear to be to the right of the number 0 when viewed from the user, and when the user is in front, the numbers 0 and 1 to 4 appear to exist on the same plane, and conversely, when the user is on the right side, the numbers 1 to 4 appear to be to the left of the number 0 when viewed from the user. In other words, the position of the number 0 appears to always remain at the reference position (center), whereas the numbers 1 to 4 appear to always exist farther away (farther back, further back) than the number 0.

[0176] That is, in the cases of Figures 20 and 22, similarly to the cases of Figures 19 and 21, the position of the number 0 is fixed as the first object, and the numbers 1 to 4 are arranged as the second object in pixels 1 to 9, shifting them slightly to the left or right or right or left, so that the user sees the numbers 0 and 1 to 4 shifted in the depth direction. More specifically, in the cases of Figures 19 and 20, the number 0 appears to be located farther away than the numbers 1 to 4, and conversely, in the cases of Figures 21 and 22, the number 0 appears to be located closer than the numbers 1 to 4.

[0177] The above is one of the key points of the first embodiment of the present invention. That is, when motion parallax is obtained using a multi-view image, in the conventional technology, the multi-view image needs to be generated by shooting the subject from a position where a camera is arranged on the circumference of the subject at different angles, whereas in the first embodiment of the present invention, by simply linearly shifting the relative positional relationship of a plurality of objects constituting the multi-view image in a predetermined direction (left and right direction), one object among the plurality of objects appears to exist at a relatively farther position (backward) than the other objects, and conversely, the other objects among the plurality of objects appear to exist at a relatively closer position (frontward) than the one object.

[0178] The second point of the first embodiment of the present invention appears when the multi-viewpoint image 1902 generated as described above is further displayed as a space floating image 3.

[0179] Fig. 23 is a diagram showing a state in which a multi-viewpoint image 1902 displayed by a multi-viewpoint image display device equipped with the image display device 10 and the lenticular lens 1103 shown in Fig. 20 is displayed as a space-floating image 3 via a retroreflector 330. As already described with reference to Figs. 17 and 18, in the space-floating image 3 generated via the retroreflector 330, the left-right order of the multi-viewpoint image (which may be a multi-viewpoint video) displayed by the image display device 10 and the left-right order of the multi-viewpoint image by the space-floating video 3 are recognized by the user as being exactly the opposite order.

[0180] That is, when the multi-viewpoint image 1902 is generated in such an order that image No. 1 corresponds to pixel 1 on the far left, image No. 2 corresponds to pixel 2 on the right of pixel 1, and the last image No. 9 corresponds to pixel 9 on the far right as shown in FIG. 19, the multi-viewpoint image display device displays the multi-viewpoint image 1902 thus generated as a space-floating image 3 via the image display device 10, the lenticular lens 1103, and the retroreflector 330 as shown in FIG. 23. In the image display device 10 and the lenticular lens 1103, images No. 1 to No. 9 are arranged in order from left to right in the X direction. Then, from the user's perspective, in the space-floating image 3, images No. 1 to No. 9, which are the multi-viewpoint image 1903, are arranged in the reverse order to that in the image display device 10, and image No. 1 is recognized as being on the far right, image No. 2 is recognized as being on the far left of image No. 1, and the last image No. 9 is recognized as being on the far left.

[0181] As described above, in the embodiment shown in Fig. 23, the multi-viewpoint image 1903 that the user recognizes as the space-floating image 3 has the same arrangement as the order shown in Fig. 22. That is, from the user who recognizes (observes) the space-floating image 3, the position of the number 0 appears to always remain at the reference position (center), and the numbers 1 to 4 appear to exist in positions relatively farther away (backward, deeper) than the number 0.

[0182] 24 is a diagram showing a state in which the multi-viewpoint image 1902 displayed by the multi-viewpoint image display device equipped with the image display device 10 and the lenticular lens 1103 shown in FIG. 22 is displayed as a space-floating image 3 via a retroreflector 330. In this case, as in FIG. 23, in the space-floating image 3 generated via the retroreflector 330, the left-right order of the multi-viewpoint image 1902 (which may be a multi-viewpoint image) displayed by the image display device 10 and the left-right order of the multi-viewpoint image 1902 by the space-floating image 3 are recognized by the user as being in the reverse order.

[0183] The process is the same as in Fig. 23, so a detailed explanation will be omitted, but in the embodiment shown in Fig. 24, the multi-viewpoint image 1903 that the user recognizes as the space-floating image 3 has the same arrangement as the order shown in Fig. 20. That is, from the user who recognizes (observes) the space-floating image 3, the position of the number 0 appears to always remain at the reference position (center), and the numbers 1 to 4 appear to exist in relatively closer positions (front side) than the number 0.

[0184] As explained above with reference to Fig. 23 and Fig. 24, the left-right order of the multi-viewpoint image 1902 displayed by the image display device 10 and the left-right order of the multi-viewpoint image 1903 by the space floating image 3 are recognized by the user as being exactly the opposite order. More specifically, in the case of Fig. 23, the position of the number 0 always appears to stay at the reference position (center), whereas the numbers 1 to 4 always appear to exist in a position relatively farther away (backward, on the far side) than the number 0. On the other hand, in the case of Fig. 24, the position of the number 0 always appears to stay at the reference position (center), whereas the numbers 1 to 4 always appear to exist in a position relatively closer (toward the viewer) than the number 0.

[0185] In a space floating image display device, which is a multi-viewpoint image display device, when it is desired to display a multi-viewpoint image 1903 as shown in Fig. 24, a first state in which the objects of numbers 1 to 4 are displayed in the foreground, it is sufficient to display a multi-viewpoint image 1902 as shown in Fig. 22, and arrange the numbers 1 to 4 in reverse order (from right to left) as moving second objects. Conversely, when it is desired to display a multi-viewpoint image 1903 as shown in Fig. 23, a second state in which the objects of numbers 1 to 4 are displayed in the background, it is sufficient to display a multi-viewpoint image 1902 as shown in Fig. 20, and arrange the numbers 1 to 4 in order (from left to right) as moving second objects.

[0186] That is, according to the first embodiment of the present invention, when, for example, a plurality of numbers are displayed as objects to be displayed as the space-floating image 3, a multi-viewpoint image can be generated by not changing the position of any number among the plurality of numbers and relatively shifting the positions of the other numbers in the left and right direction. That is, the space-floating image display device that forms the space-floating image 3 in the air includes an image display device 10 that displays images of at least two objects, a lenticular lens 1103 arranged on the image light output side of the image display device, and an optical member (retroreflective member 330) for forming the image light from the image display device 10 as the space-floating image 3 in the air, and the image display device 10 displays, for example, a push button corresponding to the number as the object as a multi-viewpoint image obtained by simply moving it in the left and right direction as seen from the user.

[0187] As a result, when a user observes the generated floating image 3, it creates an effect in which the positions of the objects (in this case, the push buttons corresponding to the numbers) in the depth direction, that is, the relative positions of the objects in the foreground and background, appear to be different. In other words, it is possible to create a sense of depth or three-dimensionality in the floating image 3. The objects displayed as this image are not limited to numbers, but may be any characters or figures.

[0188] As a result, according to this embodiment, when the floating image 3 is displayed as an HMI or GUI with numbers as push buttons (in other words, number buttons), when any number button is touched (operated in mid-air) by the user, the display control causes only the touched number button to recede to the back, while the other number buttons remain relatively in front, resulting in a new effect. As a specific display control, the floating image display device may control the touched number button (for example, the object of the number 1) to change from the first state in which it is displayed in front as described above to the second state in which it is displayed in the back.

[0189] In the examples shown in Figs. 23 and 24, an example is shown in which the number 0 (a first object whose position is fixed) is displayed in front of the other numbers 1 to 4 (a second object whose position is moved) (Fig. 23), and conversely, an example is shown in which the other numbers 1 to 4 are displayed in front of the number 0 (Fig. 24). The present invention is not limited to this, and for example, it is possible to display any one or more numbers among the numbers 0 to 9 in front or behind the other numbers. In addition, in the above example, the case where only two positions, the front side and the back side, are provided as the positions of the object in the depth direction of the multi-viewpoint image has been described, but the present invention is not limited to this, and it is possible to similarly realize the positions of the object in the depth direction of the multi-viewpoint image as multi-stage positions by designing the distance by which the image is shifted in a predetermined direction.

[0190] The principle of the first embodiment of the present invention has been explained above using the numbers 0 to 4 as objects, but as already described, the objects displayed as the floating image 3 in space are not limited to numbers, and may be any characters or figures. Therefore, the scope of application of the first embodiment of the present invention is wide. For example, it can be applied to a telephone equipped with push buttons for the numbers 0 to 9, a decision button, a call on / off button, etc., and it can also be applied to an elevator equipped with number buttons indicating the elevator floors and a door open / close button.

[0191] According to the first embodiment of the present invention, by simply moving each of a plurality of objects constituting a multi-view image linearly to the left and right, that is, by generating a plurality of images (e.g., image 1902 in FIG. 23) in which the positional relationship of the objects is changed by the movement, the objects can be displayed with different positions in the depth direction (whether they are at the back or the front). Therefore, in the first embodiment, a multi-view image can be generated by a simple method compared to the conventional method of generating a multi-view image. Moreover, since the user can visually confirm in three dimensions whether the button (object) that he or she operated is responding reliably, the user can reliably grasp whether or not there is an operation error, and further, since the button can be operated as a floating image in space, the user can operate a device in which an unspecified number of users perform touch operations with peace of mind.

[0192] <Embodiments of the present invention relating to vending machines> Next, an example in which the space-floating image display device is applied to a vending machine will be described as an embodiment of the present invention with reference to Fig. 26. Fig. 26 is a diagram showing a case in which the multi-viewpoint image display device (space-floating image display device) of the present invention is applied to a vending machine for drinks, for example.

[0193] 26, the vending machine main body 2600 is equipped with a space-floating image display unit 2620. This space-floating image display unit 2620 also has the internal configuration shown in FIG. 12 or 13, although not shown, and the space-floating image 3 by the space-floating image display unit 2620 is generated based on a multi-view image by the image display device 10 and the lenticular lens 1103. In addition, this vending machine main body 2600 is equipped with a drink display unit 2680 that displays drinks sold by the vending machine main body 2600, a bill insertion slot 2681 for inserting bills, a coin insertion slot 2682 for inserting coins, a change removal slot 2683 for removing change, and a drink removal slot 2684 for removing drinks purchased by the user.

[0194] The vending machine body 2600 is equipped with a human sensor or camera 2630. The human sensor or camera 2630 is a device for detecting the approach of a user to the vending machine main body 2600. When a user approaches the vending machine main body 2600, the space-floating image display device detects the user's approach based on the detection result by the human sensor or camera 2630, and activates the space-floating image display unit 2620. Next, as shown in (a) of FIG. 26, a person image 2621 appears on the space-floating image display unit 2620 as a space-floating image (multi-view image), and a voice such as "Welcome. Thank you for your patronage. The screen will change to numeric buttons. Please select the product number you would like" is issued to the user. After that, the person image 2621 disappears from the space-floating image display unit 2620, and then, as shown in (b), numeric buttons 2622 and a decision button 2623 are displayed. At this time, although not shown, in addition to the numeric buttons 2622 and the decision button 2623, a cancel button and a back button may be displayed.

[0195] Here, the image of the person image 2621 displayed on the space-floating image display unit 2620 shown in Fig. 26 may be a space-floating image 3 (1503) based on a multi-viewpoint image 1502 with motion parallax as shown in Fig. 17. This allows the user to visually recognize the image of the person image 2621 as a stereoscopic image. Furthermore, when the user moves around the vending machine main body 2600, the user will see the person image 2621 always looking at the user and speaking to him / her as the user moves, for example, in the left and right directions. This brings about an effect that the user can get the feeling that the image of the person image 2621 is speaking to him / her personally.

[0196] The user operates the number buttons 2622 and decision button 2623 displayed on the floating-in-space image display unit 2620 to select a drink, and then inserts a specified amount of money into the bill insertion unit 2681 or coin insertion unit 2682, and the drink is dispensed in a form that can be removed from the drink removal port 2684.

[0197] Here, the numeric buttons 2622 and the decision button 2623 use the multi-view images shown in Figs. 23 and 24. With this, for example, of the numeric buttons 0 to 9, the user can select the drink represented by the number 12 by performing a touch operation (air operation) on numeric buttons 1 and 2 in this order, and then performing a touch operation (air operation) on the decision button 2623. Also, here, numeric buttons 1 and 2 and the decision button 2623 are displayed as if they are recessed further back than the other buttons, i.e., numeric buttons 0 and 3 to 9, as in Fig. 25. This allows the user to clearly recognize that he or she has selected the drink represented by the number 12.

[0198] When a drink is taken out from the drink outlet 2684, the number buttons 0 to 9 and the enter button 2623 disappear from the floating-in-space image display unit 2620, and as shown in (c), the figure 2621 appears again, emitting a voice such as, for example, "Thank you very much. We look forward to seeing you again." Even in this case, the voice may be emitted from a normal speaker, or may be emitted from the superdirectional speaker described above so that only the user can hear it.

[0199] Through the above series of operations, the user can purchase the desired drink. In the example of FIG. 26, the vending machine main body 2600 is provided with only the space-floating image display unit among the above-mentioned components, but both the image display device and the space-floating image display unit may be provided, and the space-floating image display unit may be provided in two or more places instead of one place. In addition, when the space-floating image display unit is provided in two places, for example, a human image may be displayed as a multi-view image with motion parallax on one of the space-floating image display units, and a number button and a decision button may be displayed on the other space-floating image display unit.

[0200] Furthermore, a plurality of different human figures or animated characters of different ages or genders may be displayed as the human figure 2621. Data for displaying the plurality of different human figures or animated characters of different ages or genders is stored in the non-volatile memory 1108 of FIG. 2, and one of the plurality of human figures or animated characters may be appropriately selected and displayed on the floating-in-space image display unit. In this case, it may be determined which human figure or character to display according to the attributes of the user (e.g., age, etc.).

[0201] As described above, in this embodiment, since the floating-in-space image display unit 2620 based on the multi-view image (or image) with motion parallax is provided, the user can select and purchase a product without contact. In addition, when the user approaches the vending machine, the floating-in-space image display unit 2620 can automatically display the floating-in-space image, and further, the floating-in-space image display unit 2620 can display the human image 2621 recognized as a stereoscopic image based on the display of the multi-view image (or image) with motion parallax. As a result, the user can get the feeling that an actual person is present there, and that the person is always speaking to the user no matter where the user moves. Furthermore, when any of the number buttons 2622 (number buttons 0 to 9) and the decision button 2623 are touched, the number button 2622 and the decision button 2623 selected (touched) by the user are displayed as if they are recessed further back than the other buttons, as described above, so that the user can get the effect of being able to clearly recognize the number of the drink he or she has selected.

[0202] Furthermore, as described above, in this embodiment, multi-viewpoint images are displayed as floating-in-space images, and in particular, push buttons as HMI are displayed three-dimensionally, so that it appears to the user as if he or she is pressing an actual button, resulting in the effect of being able to provide floating-in-space images that are ideal as an HMI.

[0203] Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-mentioned embodiments and can be modified in various ways without departing from the gist of the invention. In each embodiment, components can be added, deleted, or replaced, except for essential components. Unless otherwise specified, each component can be singular or plural. Forms in which each embodiment is combined are also possible.

[0204] The technology according to the embodiment displays high-resolution, high-brightness floating images in a floating state in space, allowing users to operate the system without worrying about contact infection. If the technology according to the embodiment is used in a system used by an unspecified number of users, it is possible to provide a contactless user interface that reduces the risk of contact infection and can be used without worry. The present invention provides such technology and contributes to the "3 Health and Well-being for All" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0205] In addition, the technology according to the embodiment reduces the divergence angle of the emitted image light and aligns it to a specific polarization, so that only the normal reflected light is efficiently reflected by the retroreflective material, resulting in a high light utilization efficiency and a bright and clear floating image in space. The technology according to the embodiment can provide a non-contact user interface with excellent usability that can significantly reduce power consumption. The present invention, which provides such technology, contributes to the Sustainable Development Goals (SDGs) proposed by the United Nations, "9 Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote technological innovation" and "11 Make cities and towns sustainable."

[0206] Furthermore, the technology according to the embodiment enables the formation of a floating image by imaging light with high directionality (linearity). The technology according to the present embodiment makes it possible to provide a non-contact user interface with low risk of people other than the user looking at the floating image by displaying highly directional imaging light, even when displaying images that require high security such as images displayed on so-called kiosk terminals, or highly confidential images that should be concealed from people directly facing the user. By providing the above-mentioned technology, the present invention contributes to "Sustainable cities and towns" of the Sustainable Development Goals (SDGs) proposed by the United Nations. [Explanation of symbols]

[0207] 1...space floating image display device, 2...retroreflective member, 3...space floating image, 10...image display device, 11...liquid crystal display panel, 13...light source device, 330...retroreflector (retroreflective member, retroreflective member), 1103...lenticular lens, 1401...pixel, 1902...image (multiple viewpoint image), 1903...image (multiple viewpoint image).

Claims

1. A space floating image display device, a video display device that displays a video; a lenticular lens disposed on the image light exit side of the image display device; a retroreflective member that reflects image light from the image display device and forms a floating image in the air using the reflected light; the image display device displays an image including at least two objects, fixes the position of an arbitrary object among the at least two objects, and shifts the positions of objects other than the arbitrary object in a predetermined direction between different multi-viewpoint images, thereby displaying a plurality of images obtained by doing so as multi-viewpoint images; A floating image display device.

2. 2. The space floating image display device according to claim 1, The predetermined direction is a left-right direction with respect to a viewpoint of a user viewing the floating image in space. A floating image display device.

3. 2. The space floating image display device according to claim 1, the lenticular lens is disposed between the image display device and the retroreflective member. A floating image display device.

4. 2. The space floating image display device according to claim 1, the video display device has a display panel, the lenticular lens is disposed at a predetermined distance from the exit surface of the display panel. A floating image display device.

5. 5. The space floating image display device according to claim 4, The distance between the lenticular lens and the image display device is adjusted by the focal length of the lenticular lens. A floating image display device.

6. 2. The space floating image display device according to claim 1, the video display device has a display panel, the exit surface of the display panel and the entrance surface of the lenticular lens are parallel to each other; A floating image display device.

7. 2. The space floating image display device according to claim 1, The object is a number, a letter, or a graphic. A floating image display device.

8. 2. The space floating image display device according to claim 1, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, The object is any one of a number, a character, and a graphic. the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; A floating image display device.

9. 2. The space floating image display device according to claim 1, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; the object displayed on the image display device has a positional relationship in the predetermined direction that is opposite to that of the object displayed as the space floating image; A floating image display device.

10. 2. The space floating image display device according to claim 1, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; When it is detected that the user has approached the housing, the object is displayed as the floating image in space. A floating image display device.

11. 11. The space floating image display device according to claim 10, Equipped with a human presence sensor, Detecting the approach of the user based on the output result of the human presence sensor. A floating image display device.

12. 11. The space floating image display device according to claim 10, An imaging device is provided, detecting the approach of the user based on an image of the user captured by the imaging device; A floating image display device.

13. A space floating image display device, a video display device that displays a video; a lenticular lens disposed on the image light exit side of the image display device; a retroreflective member that reflects image light from the image display device and forms a floating image in the air using the reflected light; the image display device displays an image including at least two objects, and displays a plurality of images obtained by shifting relative positions of a first object and a second object among the at least two objects in a predetermined direction between different multi-viewpoint images as multi-viewpoint images; A floating image display device.

14. In the space floating image display device according to claim 13, The predetermined direction is a left-right direction with respect to a viewpoint of a user viewing the floating image in space. A floating image display device.

15. In the spatial floating image display device according to claim 13, the lenticular lens is disposed between the image display device and the retroreflective member. A floating image display device.

16. In the spatial floating image display device according to claim 13, the video display device has a display panel, the lenticular lens is disposed at a predetermined distance from the exit surface of the display panel. A floating image display device.

17. In the spatial floating image display device according to claim 16, The distance between the lenticular lens and the image display device is adjusted by the focal length of the lenticular lens. A floating image display device.

18. In the spatial floating image display device according to claim 13, the video display device has a display panel, the exit surface of the display panel and the entrance surface of the lenticular lens are parallel to each other; A floating image display device.

19. In the spatial floating image display device according to claim 13, The object is a number, a letter, or a graphic. A floating image display device.

20. In the spatial floating image display device according to claim 13, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, The object is any one of a number, a character, and a graphic. the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; A floating image display device.

21. The spatial floating image display device according to claim 13, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; the object displayed on the image display device has a positional relationship in the predetermined direction that is opposite to that of the object displayed as the space floating image; A floating image display device.

22. The spatial floating image display device according to claim 13, the lenticular lens is disposed between the image display device and the retroreflective member, a housing that houses the image display device and the retroreflective member; a control device that executes a predetermined process based on a predetermined operation, the object has motion parallax in accordance with the movement of a user viewing the floating-in-space image; When it is detected that the user has approached the housing, the object is displayed as the floating image in space. A floating image display device.

23. In the spatial floating image display device according to claim 22, Equipped with a human presence sensor, Detecting the approach of the user based on the output result of the human presence sensor. A floating image display device.

24. In the spatial floating image display device according to claim 22, An imaging device is provided, detecting the approach of the user based on an image of the user captured by the imaging device; A floating image display device.

Images