Space suspension image information display system

Abstract

The present invention appropriately displays an image to a user of a self-service terminal or the like, and provides necessary information. According to the invention, it is possible to contribute to the sustainable development goals' 3 good health and welfare ',' 9 industry, innovation and infrastructure ', and' 11 sustainable cities and communities'. A spatial floating image information display system includes an image display device that displays an image of at least one object, a lenticular lens disposed on an image light emission side of the image display device, and a retro-reflection member that forms a spatial floating image in the air by retro-reflecting image light from the image display device, as a target, the video display device displays a multi-viewpoint image captured from a plurality of viewpoints or rendered.

Description

Technical Field

[0001] The invention relates to a space suspended image information display system. Background Art

[0002] As a spatial floating image information display system, an image display device and a display method are known that directly display an image as a spatial image facing the outside. In addition, for operations performed on an operation surface of a displayed spatial image, for example, Japanese Patent Publication No. 2019-128722 (Patent Document 1) describes a detection system that can reduce erroneous detection of the operation.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-128722 Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] Patent Document 1 describes that the purpose is to reduce erroneous detection of operations on an image formed in the air, and that the purpose is achieved by accepting a user's action as an operation when the user approaches the image from a predetermined direction.

[0008] In addition, paragraph 0011 of Patent Document 1 states that “the aerial image may be a still image or a dynamic image” and that the aerial image “may be a three-dimensional shape such as a curved surface, a sphere, or a cube.” However, Patent Document 1 does not describe a specific technology or method for forming the aerial image into a three-dimensional shape.

[0009] By forming a spatial floating image (also called an aerial image or an aerial image. These names are described later) into a three-dimensional shape, the user can have a stronger sense of presence when observing the spatial floating image compared to a spatial floating image generated based on a flat image, and can generate a spatial floating image that makes the user feel as if it is actually floating in space.

[0010] Therefore, when a spatial floating image with a three-dimensional shape is used as a sign (electronic sign), for example, it is expected to further increase people's attention to the goods and services displayed on the sign. In addition, if a virtual assistant (concierge, a person who provides users with operation guidance and various services to users) is formed into a three-dimensional shape as a spatial floating image, it can produce a feeling that the user is talking to a real person, compared with a display based on a two-dimensional spatial floating image, so the user is more likely to feel close to the virtual assistant, and it is expected to achieve the effect of being able to perform smoother operations according to the guidance of the virtual assistant.

[0011] However, as described above, the prior art example such as Patent Document 1 does not disclose a technology for displaying a spatial floating image as a three-dimensional shape, and there is a need to realize a technology for displaying a human figure, especially a virtual assistant, as a three-dimensional spatial floating image.

[0012] Therefore, an object of the present invention is to provide a spatial floating image display system capable of displaying three-dimensional objects.

[0013] Problem Solving Techniques

[0014] In order to solve the above-mentioned problem, for example, the structure described in the technical solution claimed for protection is adopted. The present invention includes a variety of means for solving the above-mentioned problem, one example of which is as follows. A spatial suspended image information display system for forming a spatial suspended image in the air, comprising: an image display device, which displays the image of at least one object; a cylindrical lens, which is arranged on the image light emitting side of the image display device; and a retroreflective component, which is used to make the image light from the image display device retroreflect to form a spatial suspended image in the air, wherein, as an object, the image display device displays a multi-viewpoint image obtained by shooting from multiple viewpoints or by rendering.

[0015] Effects of the Invention

[0016] According to a representative embodiment of the present disclosure, the spatial floating image information display system can display multi-viewpoint images as spatial floating images, and can display characters such as virtual assistants in three dimensions, so that the user can be provided with a suitable spatial floating image with a sense of presence, as if talking to a real person. Other problems, features, effects, etc. other than the above are shown in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing an example of how the spatial floating image information display system according to one embodiment is used.

[0018] Figure 2 This is a diagram showing an example of the internal structure of a spatial floating image information display system according to one embodiment.

[0019] Figure 3 This is a diagram showing an example of the main structure of a spatial floating image information display system and a retroreflective unit structure according to an embodiment.

[0020] Figure 4 This is a diagram showing another example of the main structure of the spatial floating image information display system and the retroreflective unit structure according to one embodiment.

[0021] Figure 5This is a perspective view showing an example of arrangement of a member for shielding an anomalous light generated by retro-reflection according to an embodiment.

[0022] Figure 6 It is a cross-sectional view showing an example of arrangement of a member for shielding an extraordinary light generated by retro-reflection according to one embodiment.

[0023] Figure 7 This is an explanatory diagram of a first sensing technology used in a spatial floating image information display system according to an embodiment.

[0024] Figure 8 This is an explanatory diagram of a second sensing technology used in a spatial floating image information display system according to an embodiment.

[0025] Fig. 9 This is an explanatory diagram of the operation and device of a sensing system used in a spatial floating image information display system according to one embodiment.

[0026] Fig.10 It is a graph showing the characteristics of the spectral irradiance of sunlight.

[0027] Fig.11 This is a diagram showing the reflection characteristics of polarized light incident on a medium with a refractive index of 1.5, depending on the incident angle of the light.

[0028] Fig.12 This diagram is an explanatory diagram of a technique for inputting / drawing text, etc. on a spatial floating image according to one embodiment.

[0029] Fig.13 It is a diagram showing the main part structure of a spatial floating image information display system according to one embodiment of the present invention.

[0030] Fig.14 It is a diagram showing the main part structure of another spatial floating image information display system according to one embodiment of the present invention.

[0031] Fig.15 A diagram showing the principle of displaying multi-viewpoint images.

[0032] Fig.16 This is a diagram showing an example of a camera arrangement for generating multi-viewpoint videos.

[0033] Fig.17 This is a diagram showing an example of an image displayed by a multi-viewpoint image display device.

[0034] Fig.18 This is a diagram showing an example of the visual effect of a multi-viewpoint video as a spatial floating image.

[0035] Fig.19This is a diagram showing another example of the visual effect of a multi-viewpoint video as a spatial floating image.

[0036] Fig. 20 This is a diagram showing an example of a self-service terminal.

[0037] Fig.21 This is a diagram showing an example of the appearance of a kiosk terminal according to an embodiment (Embodiment 1).

[0038] Fig. 22 This is a diagram showing a cross-sectional configuration example of a kiosk terminal according to the first embodiment.

[0039] Fig.23 This is a diagram showing an example of the appearance of a kiosk terminal according to an embodiment (Embodiment 2).

[0040] Fig.24 This is a diagram showing a cross-sectional configuration example of a kiosk terminal according to the second embodiment.

[0041] Fig.25 This is a diagram showing a self-service terminal and a user according to an embodiment (Implementation 3).

[0042] Fig.26 This is a diagram showing a display example of a kiosk terminal according to an embodiment (Implementation 3).

[0043] Fig. 27 This is a diagram showing a first operation flow of the kiosk terminal according to the third embodiment.

[0044] Fig.28 This is a diagram showing a second operation flow of the kiosk terminal according to the third embodiment.

[0045] Fig.29 This is a diagram showing a cooperative operation between a kiosk terminal and a portable terminal according to an embodiment (Implementation 4).

[0046] Fig.30 This is a diagram showing an operation / display example of a signature on a self-service terminal according to an embodiment (Implementation 5).

[0047] Fig.31 This is a diagram showing a display example on a kiosk terminal according to a modification example such as the third embodiment.

[0048] Fig.32 It is a cross-sectional view showing a specific structural example of the light source device.

[0049] Fig.33 It is a structural diagram showing a specific structural example of the light source device.

[0050] Fig.34 It is a diagram showing a specific configuration example of a light source device of another embodiment.

[0051] Fig.35 This is a diagram showing an example of the appearance of a vending machine according to an embodiment (another embodiment). DETAILED DESCRIPTION

[0052] The following is a detailed description of the embodiments of the present invention (also referred to as examples) with reference to the accompanying drawings. In principle, the same reference numerals are used for the same parts in the drawings, and repeated descriptions are omitted. In the drawings, in order to make the invention easier to understand, the expressions of the constituent elements sometimes do not represent the actual position, size, shape, range, etc.

[0053] In the description, when describing the processing performed by a program, sometimes the program, function or processing unit is used as the main body for description, but their hardware main body is a processor or a controller, device, computer, system, etc. composed of the processor. The computer uses the processor to appropriately use resources such as memory and communication interface to execute processing in accordance with the program read into the memory. Thereby, the specified functions and processing units are realized. The processor is composed of semiconductor devices such as CPU (Central Processing Unit) and GPU. The processor is composed of devices and circuits that can perform specified operations. The processing is not limited to software program processing, and can also be implemented by a dedicated circuit. The dedicated circuit can apply FPGA (field-programmable gate array), ASIC (application specific integrated circuit), CPLD (Complex Programmable Logic Device), etc.

[0054] The program can be pre-installed as data on the target computer, or it can be distributed as data from the program distribution source to the target computer. The program sending source can be a program distribution server on a communication network, or it can be a non-temporary computer-readable storage medium (such as a memory card). The program can be composed of multiple modules. The computer system can be composed of multiple devices. The computer system can be composed of a client server system, a cloud computing system, etc. Various data and information are structured as tables, lists, etc., but are not limited to this. Expressions such as identification information, identifiers, IDs (identification), names, numbers, etc. can be interchangeable.

[0055] A spatial floating image information display system (hereinafter sometimes referred to as the system) of one embodiment first has a structure that improves the recognizability by eliminating ghost images that significantly reduce the recognizability of the spatial floating image and increasing the brightness of the spatial floating image. A system of one embodiment is applied to a self-service terminal (kiosk terminal), etc., and uses a screen formed by the spatial floating image to provide a user interface such as an operation menu. On this basis, when a user approaches the shell of the system (self-service terminal), the system of one embodiment first displays a virtual assistant (sometimes referred to as a virtual assistant image, etc.) as a prescribed character image on the screen of the spatial floating image. The system uses the virtual assistant of the spatial floating image to greet, guide, explain, etc. to the user.

[0056] In a spatial floating image information display system of one embodiment, when a user approaches a spatial floating image or operates the spatial floating image, the virtual assistant image is automatically transferred and changed to an operation menu screen having a prescribed operation menu, such as a plurality of numeric buttons and option buttons, and the button can be pressed. The system appropriately uses the virtual assistant of the spatial floating image to guide and explain the user to operate the menu. In particular, when the system determines that the user is a first-time user or an elderly user, the virtual assistant guides the user in detail on the operation menu. In addition, when the system determines that the user does not quite understand the operation method of the spatial floating image, the virtual assistant guides the user in detail on the operation menu.

[0057] In addition, a spatial floating image information display system of one embodiment has a function of identifying / determining a user by, for example, performing facial verification using a camera. The system refers to user attribute information such as age and system usage history for the user determined by this function. The system controls according to the user's attributes and changes the manner and content of guidance provided by the virtual assistant of the spatial floating image.

[0058] In addition, the spatial floating image information display system of one embodiment also provides a function that allows the user to input any text or graphics on the surface of the spatial floating image using a finger. The system detects the contact state of the finger on the surface of the spatial floating image and draws the input line on the surface of the spatial floating image. The system obtains the input line as, for example, the user's signature.

[0059] In the following description of the embodiment, the term "spatial floating image" is used to express an image suspended in space or an image displayed in the air. This term may be replaced by "aerial image", "space image", "air suspended image", "spatial suspended optical image of displayed image", "air suspended optical image of displayed image", etc. The term "spatial floating image" mainly used in the description of the embodiment is used as a representative example of these terms.

[0060] <Space-suspended image information display system>

[0061] The present disclosure relates to an information display system, for example, capable of allowing an image formed by image light from a large-area image light source to be transmitted through a transparent component that separates a space, such as glass of a display window, and displayed as a spatially suspended image inside or outside a store space. In addition, the present disclosure relates to a large-scale digital signage system composed of a plurality of such information display systems.

[0062] According to the following embodiments, for example, high-resolution image information can be displayed in a spatially suspended state on a glass surface of a display window or a light-transmitting plate. At this time, by reducing the divergence angle of the emitted image light to an acute angle and then unifying it into a specific polarization, it is possible to efficiently reflect only normal reflected light to the retroreflective component (retroreflective component) or retroreflective plate. Therefore, according to the embodiments, the efficiency of light utilization is improved, ghost images generated outside the main spatially suspended image, which are problematic in the existing retroreflective method, can be suppressed, and a clear spatially suspended image can be obtained.

[0063] In addition, by including the light source of the present disclosure, a novel spatial floating image information display system with excellent usability and which can significantly reduce power consumption can be provided. In addition, according to the technology of the present disclosure, for example, a spatial floating image information display system for a vehicle can be provided, which can realize a so-called one-way spatial floating image display that can be viewed from outside the vehicle through the windshield including the front window glass, rear window glass and side window glass of the vehicle.

[0064] On the other hand, the existing spatial floating image information display system combines an organic EL panel, a liquid crystal display panel (sometimes referred to as a liquid crystal panel, etc.) as a high-resolution color display image source with a retroreflective component. In the existing spatial floating image display device, the image light is diffused at a wide angle. Therefore, when using Figure 3 In the case of the first embodiment of the retro-reflective component 2 composed of a polyhedron as shown in (B), in addition to the normal reflected light in the retro-reflective component 2 (which forms a regular spatial floating image), there is also Figure 3 Ghost images are generated by the image light obliquely incident on the retroreflective component 2 (retroreflective portion 2a) as shown in (C). As a result, the image quality of the spatial suspended image is reduced. In addition, the spatial suspended image display device of the prior art will generate ghost images corresponding to the number of reflective surfaces in addition to the normal spatial suspended image. Therefore, people other than the viewer will also see the same ghost image as the spatial suspended image, which also poses a major problem in terms of safety.

[0065] <First Configuration Example of Spatial Floating Image Information Display System>

[0066] Figure 1 (A) shows an example of the use of the spatial floating image information display system of the embodiment and is an explanatory diagram showing the overall structure of the spatial floating image information display system. Figure 1 In (A), for example, in a store, a space is divided by a light-transmitting member such as glass (also referred to as a transparent member), namely a display window (also referred to as window glass) 105. According to the spatial floating information display system, a spatial floating image can be displayed unidirectionally to the outside of the store space through the transparent member.

[0067] Specifically, according to the present system, light with narrow-angle directivity and specific polarization is emitted as an image beam from the image display device 10. The emitted image beam first enters the retro-reflective component 2, and after retro-reflection, passes through the window glass 105 to form a real image of a space floating image (aerial image) 3 outside the store space. Figure 1 In (A), the inside of the store inside the transparent member (window glass in this case) 105 is represented as the deep direction, and the outside of the window glass 105 (for example, the sidewalk) is represented as the front side. On the other hand, a member that reflects a specific polarization may be provided on the window glass 105, and the image beam may be reflected by the member to form an aerial image at a desired position in the store.

[0068] Figure 1 (B) shows the internal structure of the image display device 10. The image display device 10 includes an image display unit 1102 for displaying an original image of an aerial image, an image control unit 1160 for converting an input image according to the resolution of a panel, and an image / audio signal receiving unit 1130 for receiving / inputting an image / audio signal.

[0069] Among them, the role of the image / sound signal receiving unit 1130 is, for example, to support wired input signals through input interfaces such as HDMI (High-Definition Multimedia Interface) (registered trademark), and to support wireless input signals such as Wi-Fi (Wireless Fidelity) (registered trademark). In addition, the image / sound signal receiving unit 1130 can also function independently as an image receiving / display device. Furthermore, the image / sound signal receiving unit 1130 can also display / output image / sound information from tablet terminals, smart phones, etc. Furthermore, the image / sound signal receiving unit 1130 can also be connected to a processor (computation processing device) such as a stick-shaped PC as needed. In this case, as the entire image signal / sound signal receiving unit, it can also be enabled to have the capabilities of calculation processing and image analysis processing.

[0070] [Functional modules of the spatial suspended image information display system]

[0071] Figure 2 The functional block diagram of the spatial floating image information display system 1 is shown. The image display unit 1102 modulates the light passing through the image display unit 1102 based on the image signal, thereby generating an image. The image display unit 1102 is sometimes also referred to as a display panel, a liquid crystal panel, or a liquid crystal display panel. The image display unit 1102 can be composed of, for example, a transmissive display panel, or a reflective display panel that modulates the light reflected on the panel based on the image signal, a DMD panel (DMD: Digital Micromirror Device, registered trademark), etc., depending on the situation.

[0072] Space suspended image information display system 1 Figure 2 The device shown has a lenticular lens 1103. Lenticular refers to a printed material (lenticular printing) that is composed of sheet-like lenticular lenses and has a pattern that changes depending on the viewing angle or can obtain a three-dimensional effect. The lenticular lens is a collection of lenses with a semi-cylindrical "Kamaboko-shaped" shape on the surface, and an image display unit 1102 is configured to display different images corresponding to the number of viewpoints of a multi-viewpoint image (or video) at the bottom of one semi-cylindrical lens. In the invention of the present application, the lenticular lens 1103 is configured on the image light emitting side of the image display unit 1102. Specifically, the lenticular lens 1103 is configured to be separated from the image light emitting side of the image display unit 1102 by a predetermined distance. In addition, the image display unit 1102 causes the emitted image light to pass through the lenticular lens to display a multi-viewpoint image (or a multi-viewpoint video), and the user can observe the multi-viewpoint image (or a multi-viewpoint video).

[0073] according to Figure 2 The user can view different images (or videos) from different positions by moving the semi-cylindrical lenses forming the cylindrical lens 1103 in the arrangement direction (e.g., left and right direction). Thus, images (or videos) taken from different shooting directions of one subject are used as the above-mentioned different images (or videos). In this way, the user can view the images (or videos) displayed on the liquid crystal panel constituting the image display unit 1102 in the form of multi-viewpoint stereoscopic images with motion parallax through the cylindrical lenses.

[0074] The retroreflection unit 1101 retroreflects the light modulated by the image display unit 1102. The light output to the outside of the spatial floating image information display system 1 from the reflected light from the retroreflection unit 1101 forms a spatial floating image 3. The light source 1105 generates light for the image display unit 1102. For example, a solid light source such as an LED light source or a laser light source is used for the light source 1105. The power supply 1106 converts the AC power input from the outside into a DC power supply and supplies power to the light source 1105. Furthermore, the power supply 1106 supplies necessary DC current to each of the other parts.

[0075] The light guide 1104 guides the light generated by the light source 1105 and illuminates the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight source of the image display unit 1102. Various forms can be considered for the combination of the light guide 1104 and the light source 1105. Specific structural examples are described later. In addition, Figure 2 As shown in the figure, the portion consisting of three components, namely, the image display unit 1102 , the light guide 1104 , and the light source 1105 , is specifically referred to as an image display device 10 .

[0076] The air operation detection sensor 1351 is a sensor for detecting the operation (also referred to as air operation) performed by the user's finger on the space floating image 3, and sensing in a range that overlaps at least a part of the display range of the space floating image 3, or in a range that overlaps the entire display range. As a specific sensor structure, the air operation detection sensor 1351 can be a distance sensor composed of non-visible light such as infrared, non-visible laser, ultrasonic wave, etc., or a combination of multiple thereof can detect the coordinates of a two-dimensional plane. In addition, the air operation detection sensor 1351 can also be configured as a ToF (Time of Flight) method LiDAR (Light Detection and Ranging) described later.

[0077] The aerial operation detection unit 1350 obtains the sensing signal obtained by the aerial operation detection sensor 1351, determines whether the user's finger touches the spatial floating image 3 based on the sensing signal, and calculates the contact position on the spatial floating image 3. The aerial operation detection unit 1350 may also be formed by a circuit such as an FPGA.

[0078] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 (sometimes referred to as a sensing system) can be built into the spatial floating image information display system 1, but can also be set outside the spatial floating image information display system 1. In the case of separate settings, it is configured to be able to transmit information and signals to the spatial floating image information display system 1 via a wired or wireless communication connection path and an image signal transmission path. The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 can also be set separately. In this case, a system can be constructed in which the aerial operation detection function can be optionally added to the spatial floating image information display system 1 that does not have an aerial operation detection function as the main body. In addition, it is also possible to separate only the aerial operation detection sensor 1351 and to build the aerial operation detection unit 1350 into the spatial floating image information display system 1. In the case where it is desired to configure the aerial operation detection sensor 1351 more freely than the setting position of the spatial floating image information display system 1, the structure in which only the aerial operation detection sensor 1351 is separated has an advantage.

[0079] The camera unit 1180 is a so-called camera with an image sensor, which captures the space near the spatial suspended image 3 and / or the user's face, arms, fingers, etc. The camera unit 1180 can use a plurality of cameras or a camera with a depth sensor according to the purpose. The camera unit 1180 can also be provided separately from the spatial suspended image information display system 1. In the case where the camera unit 1180 uses a plurality of cameras or a camera with a depth sensor, the detection of the user's touch operation on the spatial suspended image 3, in other words, the operation of contacting the surface of the spatial suspended image 3, can provide assistance to the aerial operation detection unit 1350.

[0080] For example, when the aerial operation detection sensor 1351 targets the plane to which the spatial floating image 3 belongs and is configured so that an object in the plane enters the sensor, there is a case where the aerial operation detection sensor 1351 alone cannot detect how close an object that has not entered the plane (e.g., a user's finger) is to the plane. In such a case, by using depth calculation information obtained based on the camera results of multiple cameras in the camera unit 1180 and depth information obtained based on the depth sensor, the distance between an object that has not entered the plane of the spatial floating image 3 (e.g., a user's finger) and the plane can be calculated. This calculation information can be used for various display controls of the spatial floating image 3.

[0081] Alternatively, the system may not use the mid-air operation detection sensor 1351 but may be configured such that the mid-air operation detection unit 1350 detects the user's touch operation on the spatial floating image 3 based on the image pickup result of the image pickup unit 1180 .

[0082] The camera unit 1180 may be configured to capture the face of the user operating the spatial floating image 3, and based on the captured image, the control unit 1110 may perform user identification / determination processing or user authentication processing. Alternatively, in order to determine whether there are other people standing around or behind the user operating the spatial floating image 3 and peeping at the user's operation of the spatial floating image 3, the camera unit 1180 may be configured to capture the surrounding area of ​​the user operating the spatial floating image 3.

[0083] The operation input unit 1107 is an operation button or a remote control signal receiving unit, and inputs a signal regarding an operation performed by the user that is different from the mid-air operation of the spatial floating image 3. The operation input unit 1107 can also be used by an administrator of the spatial floating image information display system 1, who is different from the above-mentioned user who performs touch operation on the spatial floating image 3, to operate the system.

[0084] The image signal input unit 1131 has a function of connecting to an external image output device and inputting image data. The sound signal input unit 1133 has a function of connecting to an external sound output device and inputting sound data. On the other hand, the sound signal output unit 1140 has a function of outputting a sound signal obtained based on the sound data input to the sound signal input unit 1133. In addition, the sound signal output unit 1140 can also output a sound signal obtained based on sound data such as numbers and character strings pre-recorded in the storage unit 1170, other operation sounds, and error warning sounds. In addition, the image signal input unit 1131 and the sound signal input unit 1133 are collectively referred to as the image / sound signal input unit 1130. The image signal input unit 1131 and the sound signal input unit 1133 can be separate structures or combined into one.

[0085] The sound signal output unit 1140 is connected to a speaker or a super-directional speaker 30. The sound signal output unit 1140 can be connected to a speaker that outputs sound in a normal auditory frequency band, but as in the embodiments such as the self-service terminal described later, in the case where high confidentiality is particularly required and security is considered, it can also be connected to a super-directional speaker so that people other than the user cannot hear it. The super-directional speaker refers to a speaker having such a characteristic that only human ears in a specific limited spatial area can hear the sound in the auditory frequency band, and human ears outside the specific spatial area cannot hear the sound in the auditory frequency band.

[0086] The super-directional loudspeaker 30 is constructed by arranging a plurality of ultrasonic output elements capable of generating ultrasonic signals of about 40kHz on a plane, for example. At this time, the more ultrasonic output elements are used, the greater the volume of the sound obtained by the super-directional loudspeaker. The principle of the super-directional loudspeaker is briefly explained. It is well known that ultrasonic waves have a better straight-line performance than sounds in the auditory frequency band (such as human speech). Thus, by using a 40kHz ultrasonic signal as a carrier and modulating the carrier according to a sound signal in the auditory frequency band (such as AM modulation), the sound can be heard only in a specific limited spatial area.

[0087] For example, by using a plurality of cameras as the camera unit 1180, the positions of the user's face and ears are determined, and the sound is outputted from the super-directional speaker 30 accordingly according to the determination result, so that the sound can be heard only in the vicinity of the user's ears. Specifically, by controlling the phase (in other words, the delay time) of the ultrasonic signal input to the ultrasonic output element constituting the super-directional speaker 30, the sound can be heard only in a specific limited space area. In addition, by adopting a structure in which the plurality of ultrasonic output elements are arranged not on a plane but on a concave surface, for example, the sound can also be heard only in a specific limited space area.

[0088] The nonvolatile memory 1108 stores various data used in the spatial floating image information display system 1. The data stored in the nonvolatile memory 1108 may include data for various operations to be displayed as the spatial floating image 3, user interface image information such as icons and buttons, data and layout information of objects for user operations, etc. The memory 1109 stores image data to be displayed as the spatial floating image 3 and device control data.

[0089] The control unit 1110 is equivalent to the controller (in other words, the control device) of the spatial floating image information display system 1, and controls the operation of each connected unit. The control unit 1110 has devices such as a processor. The control unit 1110 executes processing in accordance with the program read from the non-volatile memory 1108 or the storage unit 1170 to the memory 1109 or the built-in memory. In this way, various functions are realized. The control unit 1110 can also cooperate with the program stored in the memory 1109 to perform calculations based on the information obtained from the connected units. The control unit 1110 can be implemented using a microcomputer or the like inside the shell constituting the spatial floating image information display system 1, or it can be connected / implemented outside the shell.

[0090] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. The communication unit 1132 transmits and receives images, pictures, sounds, and various data through the communication.

[0091] The storage unit 1170 records images, pictures, sounds, various data, etc. For example, images, pictures, sounds, various data, etc. may be recorded in the storage unit 1170 in advance when the product is shipped. Images, pictures, sounds, various data, etc. obtained from an external device, an external server, etc. via the communication unit 1132 may also be recorded in the storage unit 1170. The images, pictures, various data, etc. recorded in the storage unit 1170 are output as a space floating image 3 via the image display unit 1102, the image display device 10, and the retro-reflection unit 1101.

[0092] The data of the images and pictures recorded in the storage unit 1170 may also include data of icons, buttons, objects for user operation, etc. displayed as a user interface (including the operation menu and virtual assistant image described later) in the spatial floating image 3, and data constituting the virtual assistant image. In addition, information such as the layout information of the operation menu and the virtual assistant of the icons, buttons, objects, etc. displayed as a user interface in the spatial floating image 3, and various metadata about the operation menu and the virtual assistant may also be included in the various data recorded in the storage unit 1170. In addition, sound data for causing the virtual assistant of the spatial floating image 3 to output sound may also be recorded in the storage unit 1170. The sound data recorded in the storage unit 1170 may be output as a sound signal from the speaker or the super-directional speaker 30 via the sound signal output unit 1140.

[0093] The control unit 1110 or the image control unit 1160 and the sound signal output unit 1140 can appropriately generate image data and sound data for displaying / outputting the operation menu and the virtual assistant based on the various data for constituting the operation menu and the virtual assistant stored in the storage unit 1170 or the non-volatile memory 1108.

[0094] The image control unit 1160 performs various controls on the image signal input to the image display unit 1102. For example, the image control unit 1160 can perform image switching control to determine which image is used as the image input to the image display unit 1102, among the image stored in the memory 1109 and the image input through the image signal input unit 1131. Alternatively, the image control unit 1160 can also perform control to generate a composite image input to the image display unit 1102 by superimposing the image stored in the memory 1109 and the image input through the image signal input unit 1131. In addition, the image control unit 1160 can also perform image processing control on the image data input through the image signal input unit 1131, the image stored in the memory 1109, and the like. As the image processing, there are, for example, scaling processing for enlarging, reducing, deforming, etc., brightness adjustment processing for changing the brightness, contrast adjustment processing for changing the contrast curve of the image, and Retinex processing for decomposing the image into light components and changing the weight of each component.

[0095] In addition, the image control unit 1160 may also perform special effect image processing for assisting the user's mid-air operation on the image input to the image display unit 1102. The special effect image processing may be controlled based on the detection result of the user's operation by the mid-air operation detection unit 1350 and the result of the user's video recording by the video camera unit 1180.

[0096] As described above, the spatial floating image information display system 1 can be equipped with various functions. However, the spatial floating image information display system 1 does not necessarily need to have all of the above structures. The spatial floating image information display system 1 can have any structure as long as it has at least the function of generating the spatial floating image 3.

[0097] [The first method of forming a suspended image in space]

[0098] Figure 3 The main part structure of the spatial floating image information display system of the embodiment is shown, and an example (as a first embodiment) of the formation of the spatial floating image 3 and the structure of the retroreflective member 2 is shown.

[0099] like Figure 3 As shown in (A), the spatial floating image information display system includes an image display device 10, which diffuses image light of specific polarization at a narrow angle in an inclined direction relative to a transparent plate such as glass, i.e., a transparent member 100. The image display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light of specific polarization having a narrow-angle diffusion characteristic.

[0100] The image light of specific polarization emitted from the image display device 10 is reflected by the polarization separation member 101 provided on the transparent member 100 and having a film for selectively reflecting the image light of specific polarization, and the reflected light is incident on the retroreflective member 2 . Figure 3 In the embodiment, the polarization separation member 101 formed in a sheet shape is attached to the transparent member 100 .

[0101] A retro-reflective member 2 is provided in the other oblique direction with respect to the transparent member 100. A λ / 4 plate 21 (in other words, a quarter wave plate) is provided on the image light incident surface of the retro-reflective member 2. The image light is polarized twice by passing through the λ / 4 plate 21 when incident and when emitted with respect to the retro-reflective member 2, thereby converting the image light from a specific polarization (one polarization) into another polarization.

[0102] Here, the polarization separation component 101 that selectively reflects the image light of a specific polarization has the property of transmitting the polarized light of another polarization after polarization conversion. Therefore, the image light of another polarization after polarization conversion passes through the polarization separation component 101. The image light after passing through the polarization separation component 101 forms a real image of the space floating image 3 outside the transparent component 100 as shown in the figure.

[0103] In addition, the light forming the suspended image 3 is a collection of light rays that converge from the retro-reflective member 2 to the optical image of the suspended image 3, and these light rays continue to travel straight after passing through the optical image of the suspended image 3. Therefore, the suspended image 3 is different from the diffused image light formed on the screen by a normal projector, etc., and is an image with high directivity.

[0104] therefore, Figure 3 In the structure, when the user views from the direction of arrow A, the air-floating image 3 appears as a bright image, but when other persons view from the direction of arrow B, for example, the air-floating image 3 cannot be viewed as an image at all. Such characteristics of the air-floating image 3 are very suitable for use in systems that display images requiring high security and display highly confidential images that are to be kept secret from persons facing the user.

[0105] In addition, the polarization axis of the image light after reflection may not be uniform depending on the performance of the retro-reflective component 2. In this case, a portion of the image light with a non-uniform polarization axis is reflected on the polarization separation component 101 and then returns to the image display device 10. This portion of the image light is reflected again on the image display surface of the liquid crystal display panel 11 constituting the image display device 10, generating a ghost image. Therefore, it may become a cause of the image quality of the spatial floating image 3 being degraded.

[0106] Therefore, in this embodiment, an absorption type polarizing plate 12 is provided on the image display surface of the image display device 10. The absorption type polarizing plate 12 allows the image light emitted from the image display device 10 to pass through the absorption type polarizing plate 12, and absorbs the reflected light returning from the polarization separation component 101 by the absorption type polarizing plate 12, thereby suppressing the re-reflection. Therefore, according to this embodiment using the absorption type polarizing plate 12, it is possible to prevent or suppress the image quality degradation caused by the ghost image of the spatial floating image 3.

[0107] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects a specific polarization.

[0108] Figure 3 (B) shows a structural example of the retroreflective component 2 used in the first embodiment. Figure 3 In (B), the surface shape of the retroreflective component manufactured by Japan Carbide Industries Co., Ltd. used in this study is shown as a representative retroreflective component 2. The retroreflective component 2 has a retroreflective portion (retroreflective element) 2a of regularly arranged hexagonal prisms on the surface. The light incident on the inside of the hexagonal prism is reflected on the wall and bottom of the hexagonal prism to become retroreflected light, which is emitted in a direction corresponding to the incident light, and a spatial floating image 3 of a real image is displayed based on the image displayed on the image display device 10.

[0109] The resolution of the spatial floating image 3 depends not only on the resolution of the liquid crystal display panel 11, but also largely on Figure 3 The shape D and pitch P of the retroreflective portion 2a of the retroreflective component 2 shown in (B). For example, in the case of a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel 11, even if 1 pixel (1 triplet) is about 80μm, if, for example, the diameter D of the retroreflective portion 2a is 240μm and the pitch P is 300μm, 1 pixel of the spatial suspended image 3 is equivalent to 300μm. Therefore, the effective resolution of the spatial suspended image 3 is reduced to about 1 / 3. Therefore, in order to make the resolution of the spatial suspended image 3 the same as the resolution of the image display device 10, it is preferred to make the diameter D and pitch P of the retroreflective portion 2a close to 1 pixel of the liquid crystal display panel 11. On the other hand, in order to suppress the moiré caused by the pixels of the retroreflective component 2a and the liquid crystal display panel 11, the respective pitch ratios can be designed to deviate from an integer multiple of 1 pixel.

[0110] In addition, the shape may be arranged so that any side of the retroreflective portion 2 a does not overlap with any side of one pixel of the liquid crystal display panel 11 .

[0111] On the other hand, in order to manufacture the retroreflective component 2 at low cost, a roller pressing method can be used for forming. Specifically, the method is a method for arranging the retroreflective portion 2a and shaping it on a film. The method forms a shape opposite to the shape to be shaped on the roller surface, applies an ultraviolet curing resin on a base material for fixing and passes it between rollers, thereby giving the necessary shape, and then irradiates ultraviolet rays to cure it, thereby obtaining a retroreflective component 2 of the desired shape.

[0112] [Regarding the second method of forming a suspended image in space]

[0113] then, Figure 4 Another example (as a second embodiment) of the formation of the spatial floating image 3 and the configuration of the retro-reflective member in the spatial floating image information display system of the present embodiment will be described. Figure 4 (A) shows an overview of forming a spatial floating image 3 using the second method of the retroreflective component 330. For the retroreflective component 330, light from an object P (corresponding to point P) in a space on one side (the space on the lower side in the Z direction in this example) is incident on the retroreflective component 330, and after retroreflection, a spatial floating image 331 (corresponding to point Q) is formed in the space on the other side (the space on the upper side in the Z direction in this example).

[0114] As a representative retroreflective member 330, a retroreflective member manufactured by Asukanet Co., Ltd. was used in this study. Figure 4 (B) represents a surface shape used to illustrate its working principle. The retro-reflective component 330 has a four-sided structure (in other words, a tetrahedron) 330A that is regularly arranged on the surface (the XY plane shown in the figure). A plurality of structures 330A are arranged between the side walls 330B. The four-sided structure 330A is, for example, a micro-reflector having a quadrangular prism shape extending in the Z direction. For example, light from an object P (denoted as object light) is incident on the interior of the four-sided structure 330A. The light incident on the interior of the four-sided structure 330A is reflected on two surfaces (for example, the reflection surface RS1 and the reflection surface RS2) among the wall surfaces of the four-sided structure 330A. The reflected light (both the light emitted from the reflection surface RS1 to the upper side and the light emitted from the reflection surface RS2 to the upper side) is represented as reflected light R0. The reflected light R0 is emitted as retro-reflected light in a direction corresponding to the incident light, as shown in FIG. Figure 4 As shown in (A), a spatial floating image 331 based on the real image of object P is formed / displayed.

[0115] The resolution of the space suspended image 331 is also the same as the above Figure 3The first embodiment of the retroreflective member 2 is similarly dependent on the outer shape (diameter) DS and the pitch PT of the retroreflective portion (four-sided structure 330A) of the retroreflective member 330. For example, in the case of a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the outer shape (diameter) DS of the retroreflective portion is 120 μm and the pitch PT is 150 μm, one pixel of the spatial suspended image 331 is equivalent to 150 μm. Therefore, the effective resolution of the spatial suspended image 331 is reduced to about 1 / 2.

[0116] Therefore, in order to make the resolution of the spatial suspended image 331 the same as the resolution of the image display device 10, it is preferred to make the diameter DS and the pitch PT of the retro-reflective portion (structure 330A) close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the moiré caused by the pixels of the retro-reflective component 330 and the liquid crystal display panel, the respective pitch ratios can be designed to deviate from an integer multiple of one pixel as described above. In addition, the shape can be configured so that any side of the retro-reflective portion (structure 330A) does not overlap with any side of one pixel of the liquid crystal display panel.

[0117] In addition, the light forming the suspended image 331 in space is a collection of light rays that converge from the retro-reflective member 330 to the optical image of the suspended image 331 in space, and these light rays continue to travel straight after passing through the optical image of the suspended image 331 in space. Therefore, the suspended image 331 in space is different from the diffused image light formed on the screen by a normal projector, etc., and is an image with high directivity.

[0118] Figure 4 In the structure, when the user views from the direction of arrow A, the space floating image 331 appears as a bright image, but when other people view from the direction of arrow B, for example, the space floating image 331 cannot be viewed as an image at all. The characteristics of such a space floating image 331 are the same as those of the air floating image using the retro-reflective member 2 of the first embodiment, and are very suitable for use in systems that display images requiring high security and images with high confidentiality that are to be kept secret from people facing the user.

[0119] In addition, in the retro-reflective component 330 of the second embodiment, the light from the object P is as follows: Figure 4As shown in (B), the incident light is incident from one side (lower side in the Z direction) relative to the retro-reflective component 330, and is reflected on the two reflection surfaces (RS1, RS2) provided on the four walls constituting the retro-reflective component 330, and is imaged as a spatial floating image 331 at the position of point Q on the other side (upper side in the Z direction) as reflected light R0. At this time, as light with a different reflection direction from the reflected light R0, abnormal light R1, R2 is generated on the two reflection surfaces (RS1, RS2). Due to the abnormal light R1, R2 generated on the two reflection surfaces (RS1, RS2), the following is generated: Figure 4 Therefore, the ghost images 332 and 333 may cause the image quality of the spatial floating image 331 to deteriorate.

[0120] As described above, the first mode of the retro-reflective component 2 generates ghost images corresponding to the number of reflective surfaces. In contrast, the second mode of the retro-reflective component 330 generates ghost images only in two specific directions according to the incident angle of the object light. Therefore, the ghost image of the second mode of the retro-reflective component 330 is less affected, and a high-quality spatial image can be displayed. Therefore, as the spatial floating image display device and the spatial floating image information display system shown below, the description is limited to the case of applying the second mode of the retro-reflective component 330.

[0121] [Technical means to reduce ghost images]

[0122] In order to realize a spatial image display device that can form a high-quality spatial floating image with reduced ghost images, etc., an image light control sheet can be provided on the emission surface of the image display element, i.e., the liquid crystal display panel, to control the divergence angle of the image light from the liquid crystal display panel so that it is deflected in a desired direction. Furthermore, an image light control sheet can be provided on the light emission surface or the light incident surface or both surfaces of the retroreflective component 330 to absorb the abnormal light R1, R2 ( Figure 4 (B)).

[0123] Figure 5 A specific method and a structural example of applying the above-mentioned image light control sheet to a spatial floating image display device are shown. Figure 5 In the embodiment, the image light control sheet 334 is arranged on the emission surface of the image display element, namely the liquid crystal display panel 335. Figure 5 In the figure, the emission surface of the liquid crystal display panel 335 is represented as a plane (XY plane) formed by the X-axis and the Y-axis shown in the figure. The image light control sheet 334 has a transmission portion and a light absorption portion on the main surface (XY plane). In this case, moiré may be generated due to interference caused by the spacing between the pixels of the liquid crystal display panel 335 and the transmission portion and the light absorption portion of the image light control sheet 334. In order to reduce the moiré, the following two methods are effective.

[0124] (1) In a first method, the vertical stripes (diagonal lines in the figure) generated by the transmissive portion and the light absorbing portion of the image light control sheet 334 are arranged to be inclined at a predetermined angle (inclination angle) θ0 relative to the arrangement of pixels (X axis and Y axis) of the liquid crystal display panel 335 .

[0125] (2) The second method is to select a ratio (B / A) of A to a non-integer multiple when the pixel size of the liquid crystal display panel 335 is A and the pitch of the vertical stripes of the image light control sheet 334 is B. One pixel of the liquid crystal display panel 335 is formed by the arrangement of sub-pixels of three colors, RGB, which are usually square, so the above-mentioned moiré cannot be suppressed in the entire screen. Therefore, the inclination angle θ0 shown in the first method of (1) can be optimized within the range of 5 degrees to 25 degrees, so that the moiré generation position can be deliberately offset and configured in a place where the spatial floating image is not displayed.

[0126] In order to reduce the above-mentioned moiré, the liquid crystal display panel and the image light control sheet 334 are used as the subject for description. However, when the image light control sheet 334 is provided on the retro-reflective member 330, the same method / structure can also be applied to the moiré generated between the retro-reflective member 330 and the image light control sheet 334. Since both the retro-reflective member 330 and the image light control sheet 334 are linear structures, it is sufficient to tilt the image light control sheet 334 optimally with attention paid to the X-axis and Y-axis of the retro-reflective member 330. In this way, it is possible to reduce the coarse moiré with a long wavelength and a low frequency that can be seen visually.

[0127] Figure 6 (A) is a vertical cross-sectional view of the image display device 10, which has a structure in which an image light control sheet 334 is arranged on the image light exit surface 3351 of the liquid crystal display panel 335. The image light control sheet 334 is composed of light transmission parts 336 and light absorption parts 337 alternately arranged on the main surface, and is bonded and fixed to the image light exit surface 3351 of the liquid crystal display panel 335 by an adhesive layer 338.

[0128] In addition, as described above, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used as the image display device 10, even if one pixel (one triplet) (indicated by A in the figure) is about 80 μm, the following structure can be used to reduce the generation of the two sides of the space floating image 331. Figure 4(A) ghost images 332, 333. For example, as the pitch B of the image light control sheet 334, the pitch B formed by the distance d2 of 300 μm of the transmission part 336 and the distance d1 of 40 μm of the light absorption part 337 is set to 340 μm. In this case, the image light control sheet 334 is used to achieve sufficient transmission characteristics, and the diffusion characteristics of the image light from the image display device 10, which is the cause of the generation of abnormal light, are controlled, so that ghost images can be reduced. In this case, if the thickness of the image control sheet 334 is set to 2 / 3 or more of the pitch B, the ghost reduction effect is greatly improved.

[0129] Figure 6 (B) indicates that the retro-reflective component 330 ( Figure 4 ) is a vertical cross-sectional view of a structure in which an image light control sheet 334 is arranged on the image light exit surface. The image light control sheet 334 is composed of light transmission parts 336 and light absorption parts 337 arranged alternately, and is arranged obliquely relative to the retro-reflective component 330 so as to have a predetermined inclination angle θ1 corresponding to the exit direction of the retro-reflected light 3341. As a result, the image light control sheet 334 can absorb the abnormal light R1, R2 ( Figure 4 (B)), on the other hand, allows the normal reflected light to be transmitted without loss as retro-reflected light 3341.

[0130] The retro-reflective member 330 includes a structure 330A ( Figure 4 ) is formed. The space 3301 corresponding to the retro-reflective portion is divided by the surface of the side wall 330B. The space 3301 includes, for example, a reflection surface R1 and a reflection surface R2. For example, light a1 incident from the lower side to the retro-reflective component 330 is reflected on the reflection surface R1 of the space 3301, and the reflected light a2 is further reflected on the reflection surface R2, and is emitted toward the upper side of the retro-reflective component 330. The emitted light is incident on the image light control sheet 334 and is emitted as retro-reflected light 3341.

[0131] In the case of a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, the following structure can be used to reduce the Figure 4 As shown in (A), ghost images 332 and 333 are generated on both sides of the spatial suspended images 331 and 330. Figure 6As shown in (B), for example, as the pitch B of the image light control sheet 334, the pitch B formed by the distance d2 of 400 μm of the transmission part 336 of the retroreflective member 330 and the distance d1 of 20 μm of the light absorption part 337 is set to 420 μm. In this case, the image light control sheet 334 can realize sufficient transmission characteristics, and the diffusion characteristics of the image light from the image display device 10, which is the cause of the generation of abnormal light on the retroreflective member 330, can be controlled, so that ghost images can be reduced.

[0132] On the other hand, the image light control sheet 334 prevents external light from the outside from entering the space suspended image display device, so it also helps to improve the reliability of the components. As the image light control sheet 334, for example, the viewing angle control film (VCF) of Shin-Etsu Polymer Co., Ltd. is suitable. The structure of the VCF is to alternately configure transparent silicone and black silicone, and configure synthetic resin on the light input and output surfaces to form a sandwich structure. Therefore, when the VCF is applied to the image light control sheet 334 of this embodiment, it is expected to achieve the above-mentioned effect.

[0133] [Sensing technology for operating suspended images in space]

[0134] A user (sometimes also referred to as a user, operator, etc.) displays a spatial suspended image 3 ( Figure 2 In other words, the user uses the system's applications (such as the ticketing function of the self-service terminal) by viewing and operating the spatial floating image 3. Therefore, the user simulates the operation of the spatial floating image 3, and a sensing technology for sensing (detecting) the operation is required. The following is an example of the sensing technology. The so-called "sensing technology" here refers to the use of Figure 2 The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 described above are technologies for detecting user operations in three-dimensional space (in other words, aerial operations). The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are sometimes referred to as a sensing system.

[0135] Figure 7 (A) is a schematic diagram for explaining the first sensing technology. Figure 7 As shown in (A), the sensing areas a0, a1, a2, and a3 corresponding to the spatial floating image FI are divided into multiple areas (in other words, ranges). In this embodiment, the sensing areas a0, a1, a2, and a3 are divided into 3×4=12 areas in length and width. Figure 7 In (A), the plane of the spatial floating image FI is represented as the xy plane, and the front-back direction of the plane is represented by the z direction. For example, the illustrated area A301 is an upper left area in the sensing surface a3.

[0136] The first sensing technology is provided with a first distance measuring device 340, which has a built-in TOF (Time of Flight) system corresponding to each area of ​​the spatial floating image FI. Figure 2 The aerial operation detection sensor 1351 is provided. From the light emitting part of the first distance measuring device 340, a near-infrared light emitting LED (Light Emitting Diode) serving as a light source emits light in synchronization with the signal of the system. An optical element for controlling the divergence angle is provided on the light emitting side of the LED, and a pair of highly sensitive avalanche diodes (ABD) with a picosecond time resolution are used as light receiving elements, which are arranged in 4 columns vertically and 3 rows horizontally corresponding to the 12 regions. The LED serving as a light source emits light in synchronization with the signal from the system, and the light is reflected on the object to be measured (here, the front end of the user's finger UH) and returns to the light receiving part, generating a phase deviation (phase shift) corresponding to the time, that is, a time deviation between the light emitting moment and the light receiving moment, more specifically, as described later. Fig. 9 Δt0~Δt11 in.

[0137] Fig. 9 The computing unit of the sensing system shown in (B) receives the signal from the system and the signal generated by the avalanche diode, which is the light receiving part of the first distance measuring device 340, and calculates the phase shift based on these signals, thereby calculating the distance to the object. The distance is calculated for each TOF system (TOF1 to TOF12) corresponding to each area. Figure 7 In (A), the side close to the object (finger UH) in the z direction relative to the plane of the spatial suspended image FI is shown as the measurement layer of the distance measuring device 340, and the sensing surfaces a3, a2, and a1 (also referred to as the first sensing surface a3, the second sensing surface a2, and the third sensing surface a1) are shown in order from near to far from the object, and the sensing surface a0 is shown as the measurement layer on the side far from the spatial suspended image FI. The distance L1 represents the distance to the sensing surface a0, the distance L2 represents the distance to the sensing surface a1, the distance L3 represents the distance to the sensing surface a2, and the distance L4 represents the distance to the sensing surface a1.

[0138] Next, the sensing system identifies which area of ​​the 12 areas the object (finger UH) has passed through on each measurement layer (sensing surfaces a3 to a1) in the direction of movement, and can perform system identification by calculating the movement time of each measurement layer using the above method.

[0139] exist Fig. 9In (A), the light emission timing of the LED light source and the light receiving timing of the light receiving element of each of the 12 measurement areas are shown. SU1 to SU12 represent the light emission timing and light receiving timing of the sensing units corresponding to each area and TOF. Fig. 9 As shown in (A), in the sensing unit SU1, the time difference between the light emission time and the light reception time is Δt0, in the sensing unit SU2, the time difference between the light emission time and the light reception time is Δt1, and in the sensing unit SU12, the time difference between the light emission time and the light reception time is Δt11. Here, the sensing system delays the light emission time of the LED according to the 12 measurement areas, thereby unifying the data.

[0140] In fact, in order to connect with the system bidirectionally, the user consciously extends the finger UH toward the suspended image FI in space. In this case, the sensing system obtains, on the sensing surface a3 farthest from the suspended image FI in the air, a first sensing signal S1 sensed in the area A301, a second sensing signal S2 sensed in a specific area of ​​the sensing surface a2, and a third sensing signal S3 sensed in a specific area of ​​the third sensing surface a1. The sensing system uses the sensing signals (S1 to S3) to calculate and process the position of the contact point with the suspended image FI according to the moving direction of the finger UH and the time difference of crossing each sensing surface.

[0141] In order to obtain more accurate position information, a sensing surface a0 is set at a position away from the spatial suspended image FI. Based on the sensing on the sensing surface a0, the sensing system detects that the finger UH has passed through the spatial suspended image FI as an end signal, and according to the sensed position coordinates and the above two sensing signals, the contact point with the spatial suspended image FI is obtained in three-dimensional coordinates.

[0142] in addition, Figure 7 (B) indicates an action of selecting a part of the spatial floating image FI by the user's finger UH (especially the fingertip), and an action of the user's finger UH leaving the part of the spatial floating image FI. Figure 7 As shown in (B), in the first sensing technology, when the user retracts the finger UH after touching the required position coordinates of the spatial suspended image FI, the following is described. That is, the sensing system sequentially transmits the first sensing signal S1 sensed on the first sensing surface a1 to the computing unit of the sensing system based on the third sensing signal S3 sensed on the third sensing surface a3 and the second sensing signal S2 sensed on the second sensing surface a2, and performs calculation processing. As a result, the system recognizes that the user's finger UH has left the specific coordinates of the spatial suspended image FI.

[0143] Next, a higher-precision sensing technology for simulating the operation of a spatial floating image is described below.

[0144] Figure 8 (A) is a schematic diagram for explaining the second sensing technology. Figure 7 The difference between the first sensing technology shown in (A) is that in addition to the first distance measuring device 340, a second distance measuring device 341 is also configured to achieve higher precision sensing. The first distance measuring device 340 and the second distance measuring device 341 can also be set together. In the second sensing technology, the second distance measuring device 341 (especially a CMOS sensor) is used as a second sensing system in combination with the first sensing system. Figure 8 As shown in (A) of FIG. 1 , the second distance measuring device 341 performs sensing in the same range (sensing surfaces a1, a2, a3, a0) as the first distance measuring device 340.

[0145] As described above, the first distance measuring device 340 has a built-in TOF system corresponding to each of the plurality of regions divided into, for example, 12 regions in the spatial floating image FI. Figure 8 On the other hand, the second distance measuring device 341 uses a two-dimensional image sensor, such as a 1 / 4-inch CMOS sensor for a sensor camera. The aspect ratio of the CMOS sensor is usually 3:4. Therefore, in this embodiment, corresponding to the aspect ratio of the CMOS sensor, the sensing area of ​​the TOF system of the first distance measuring device 340 is also divided into 3 vertical columns and 4 horizontal rows as described above, forming a total of 12 areas.

[0146] In addition, regarding the resolution of the CMOS sensor, a sufficient resolution can be obtained with about 1 million pixels, but unlike the usual camera system, there is no need to set up an RGB color separation filter. Therefore, the CMOS sensor can not only achieve miniaturization and high sensitivity with the same number of pixels, but also has a high sensitivity to near-infrared light. Therefore, in the second sensing technology, the object to be measured (the tip of the finger UH) is illuminated by the light source of the TOF system of the first ranging device 340 at a time determined according to each area, so the detection accuracy is greatly improved. Detailed explanation is omitted. Fig. 9 The system described above is shown in (B) as a functional block diagram.

[0147] Figure 8 (B) represents the sensing surfaces a1, a2, a3 of the first distance measuring device 340 and the sensing surfaces b1, b2, b3 of the second distance measuring device 341 provided corresponding thereto. Figure 8 (B) shows the action of the finger UH selecting a part of the spatial floating image FI on these sensing surfaces, and the action of leaving the part.

[0148] like Figure 8As shown in (B), in a spatial suspended image information display system using the second sensing technology, when the user consciously extends a finger UH toward the spatial suspended image FI, the following is described. In this case, in addition to the three-dimensional information obtained by the above-mentioned first ranging device 340, three-dimensional information based on the second ranging device 341 can also be obtained. The planar resolution of the sensing surface b3 of the second ranging device 341 corresponding to the sensing surface a3 of the first ranging device 340 farthest from the aerial suspended image FI can be highly accurate according to the resolution of the CMOS sensor used. Similarly, the sensing surface b2 corresponds to the sensing surface a2, and the sensing surface b1 corresponds to the sensing surface a1. Thereby, a sensing system with greatly improved resolution in the planar direction can be realized.

[0149] At this time, regarding the moving direction of the object (the front end of the user's finger UH), the contact point position with the spatial suspended image FI is calculated based on the time difference between the first distance measuring device 340 and the second distance measuring device 341. In order to obtain more accurate position information, a sensing surface a0 is set away from the spatial suspended image FI to the deep side. The sensing system detects that the finger UH has passed through the spatial suspended image FI as an end signal, and calculates the contact point with the spatial suspended image FI as a more precise three-dimensional coordinate based on the position coordinates on the sensing surface a0 and the above two sensing signals.

[0150] In addition, when the frame rate of the CMOS sensor is increased from 1 / 20 second to 1 / 30 second or 1 / 120 second, in addition to the detection accuracy of the plane direction, the plane information introduced per unit time is also increased, so the resolution is greatly improved. At this time, the detection information of the second sensing technology is associated with the position information obtained by the first sensing technology based on the synchronization signal supplied from the system.

[0151] Furthermore, if Figure 8 As shown in (B), when the user retracts the finger UH after touching the desired position coordinate of the spatial suspended image FI, similarly to the first sensing technology described above, the third sensing signal S3 sensed on the third sensing surface a3, the second sensing signal S2 sensed on the second sensing surface a2, and the first sensing signal S1 sensed on the first sensing surface a1 are sequentially transmitted to the computing unit of the sensing system. Then, through the calculation processing in the computing unit, the system recognizes that the user's finger UH has left the specific coordinates of the spatial suspended image FI.

[0152] In order to prevent the ranging device from having its accuracy reduced due to external light such as sunlight, the LED light source used in the TOF sensor of the first ranging device 340 of the sensing system described above can use near-infrared light with higher energy in the range that is invisible to the naked eye and exceeds the visible light range (380nm~780nm).

[0153] Fig.10 A characteristic diagram showing the spectral irradiance of sunlight. As the wavelength of the light source light of the LED of the TOF sensor, it is possible to use Fig.10 The spectral irradiance of sunlight is shown as the light with a wavelength λ1 of 920nm with smaller energy.

[0154] [Technology for inputting text and graphics into floating images in space]

[0155] Next, as a spatial floating image information display system of an embodiment, a technology that allows a user to input arbitrary images such as text and graphics into a spatial floating image will be described. By using this technology, a user can input, for example, a signature into a spatial floating image.

[0156] By using the above Figure 2 Sensing technology using the aerial operation detection sensor 1351, and Figure 7 and Figure 8 The sensing technology of the TOF system can be used to obtain the fingertip position and movement direction of the user's finger as the object, the contact point position with the spatial floating image, the position where the finger leaves the spatial floating image (recorded as the departure position), etc. Therefore, the spatial floating image information display system can draw a line segment connecting these two points as an image on the surface of the spatial floating image based on the detection information of the contact point position and the departure position.

[0157] In this embodiment, by using the above-mentioned sensing technology, the movement of the user's finger, in other words, the movement of the position of the contact point, is detected on the surface of the spatial floating image, and a line of text or the like can be input / displayed.

[0158] For example, the spatial floating image information display system detects the contact point position between the spatial floating image and the user's fingertip, and then the fingertip moves to another position within the surface, and detects that the fingertip leaves the spatial floating image and the departure point at the position after the movement. As a result, graphics such as line segments and curves, or text such as letters and numbers can be drawn on the surface of the spatial floating image. As a result, graphics and text can be input on the surface of the spatial floating image just as graphics or text are drawn on the surface of a liquid crystal display with a touch panel.

[0159] In addition, although not shown in the figure, the spatial floating image information display system can use sensing technology to sample the movement of the user's fingertip relative to the spatial floating image at a predetermined time interval (e.g., 100 milliseconds), and draw a line on the spatial floating image in a manner of connecting each sampling point. Thus, in the case of a text or graphic that can be written in one stroke, such as the numbers 1, 2, 3, the letters L and M, etc., it can be drawn as a spatial floating image.

[0160] Furthermore, in the case of characters and graphics that cannot be written in one stroke, such as 4 and H, it is possible to deal with it as described below. That is, when the user inputs the character H, for example, the system first detects / draws a longitudinal line (first line) generated by the continuous contact of the finger with the surface of the spatially suspended image. Next, the system detects that the user temporarily leaves the surface and then touches another position with the finger again, and detects / draws a transverse line (second line). Finally, the system detects that the user leaves the surface and then touches another position again with the finger, and detects / draws a longitudinal line (third line). In this way, the character H can be drawn on the surface of the spatially suspended image.

[0161] Fig.12 An explanatory diagram showing a technique for inputting / drawing an image of text, graphics, etc. on the above-mentioned spatial floating image using a user's finger. Fig.12 A schematic diagram showing a situation where a spatial suspended image FI displayed by the spatial suspended image information display system 1 is viewed from the user's viewpoint. In this example, the spatial suspended image FI is arranged obliquely relative to the horizontal plane of the system housing 1201, and the spatial suspended image FI is viewed from the user's viewpoint slightly obliquely downward. In this example, the spatial suspended image FI includes a display of a quadrilateral frame 1202 (this is not a frame-shaped device). First, the frame 1202 is displayed in a specified color (for example, green), and only a specified background color is displayed in the frame 1202.

[0162] Fig.12 In the example, the situation in which the letter L is drawn as a simple text on the surface of the spatial suspended image FI by the fingertip UF of the user's finger is shown. At this time, the user notifies the spatial suspended image information display system 1 of the intention to draw text through some method. For example, in one embodiment, as a physical button, a text input button 1203 is set at any place of the spatial suspended image information display system 1, in this case, it is the lower right position on the shell 1201. When the user presses the text input button 1203, the system regards it as the user's intention to input text, etc. into the spatial suspended image FI, and transfers to the mode of inputting / drawing the spatial suspended image FI (also recorded as the text input mode). In addition, it is also possible to notify the spatial suspended image information display system 1 of the intention to draw text by sound instead of using a physical button.

[0163] In another embodiment, a text input button may be displayed as a user interface in a part of the space floating image FI provided by the space floating image information display system 1. When the user touches the text input button, the system is similarly transferred to a mode for inputting / drawing the space floating image FI.

[0164] In the text input mode, the spatial suspended image information display system 1 changes the entire screen of the spatial suspended image FI to, for example, a white display. In addition, in the text input mode, the system changes the frame 1202 of the spatial suspended image FI to a specified color, such as red. The system three-dimensionally senses the movement and contact point position of the user's finger UF relative to the surface of the spatial suspended image FI as described above. Then, based on the sensing, the system draws the contact point position corresponding to the movement of the finger UF on the surface of the spatial suspended image FI using a specified color different from the white background, such as black or blue. As a result, the trajectory of the movement of the finger UF is drawn as a line on the white background surface of the spatial suspended image FI. In this way, the user himself can draw the required text and graphics while watching the line drawn by himself in the spatial suspended image FI.

[0165] The system controls the display so that the display state of the spatial suspended image FI is different when the finger UF does not touch the surface of the spatial suspended image FI (in other words, the fingertip is located in front of the surface) and when the finger UF touches the surface of the spatial suspended image FI (in other words, the fingertip is inserted deep into the surface). As a result, the user can more easily identify whether his or her finger UF touches the surface of the spatial suspended image FI.

[0166] Fig.12 The following describes the steps when a user draws the letter L on the surface of the spatial suspended image FI. First, as step S121, the user inserts the finger UF in the deep direction (in other words, maintains contact) at any position on the spatial suspended image FI. Then, as step S122, the user moves the finger UF downward while maintaining contact on the spatial suspended image FI. The downward arrow indicates the direction of movement of the finger UF (below the surface). As the movement is detected, the system draws a vertical line (first line) 1211 in the letter L as part of the spatial suspended image FI.

[0167] Next, step S122 indicates the state where the finger UF reaches the end position below the first longitudinal line 1211. At this point, the first longitudinal line 1211 has been drawn. The contact of the finger UF is maintained in this state. Next, in step S123, from the moment when the first line 1211 has been drawn, the user moves the finger UF to the right. As the movement is detected, the system draws the horizontal line (second line 1212) in the character L. The position of the finger UF reaches the right end position of the second horizontal line 1212.

[0168] Next, as step S124, the user moves the finger UF located at the right end of the second line 1212 away from the surface of the spatial floating image FI toward the front side (the side close to the user). The system detects the departure of the finger UF and regards the drawing of the second line 1212 as completed. Through the above series of operations, the user can draw the character L on the spatial floating image FI in one stroke.

[0169] When the user presses the text input button 1203 to enter the text input mode, not only the entire screen (background) of the spatial floating image FI changes to white, but also the frame 1202 of the spatial floating image FI changes from green to red, for example. This is preferred because the user can easily recognize that they are in the text input mode. In addition, when the user wants to end the text input mode, the user can press the text input button 1203 again. In this case, the system ends the text input mode and returns to the original mode. In the above example, the entire screen of the spatial floating image FI changes to white, but it is not limited to this, and it can also be a color other than white, such as black or other colors.

[0170] In addition, after the transition to the above-mentioned text input mode, the user may touch (contact the finger UF) the spatial floating image FI, so that part of the frame 1202 of the spatial floating image FI changes to red, for example. By changing part of the frame 1202 of the spatial floating image FI to red, the user can recognize that his finger is in contact with the surface of the spatial floating image FI and is in a state where text can be drawn.

[0171] Through the above series of actions, the user can input / draw the required text, graphics, etc. as the image of the spatial floating image FI by moving the fingertips. In addition, the frame 1202 of the spatial floating image FI can also be a color other than red, such as blue or other colors. Furthermore, in the above example, the entire screen of the spatial floating image FI is made into an area where images of graphics, text, etc. can be input / drawn. Without being limited to this, the area where images can be input / drawn can also be a specified part of the screen of the spatial floating image FI.

[0172] The user can not only use the spatial floating image FI as a user interface such as an operation menu, but can also input / draw any text or graphics on the spatial floating image FI with his or her own finger (or a pen made of conductive material, etc.) in the above-mentioned text input mode. This can further expand the application range of the spatial floating image and the spatial floating image display device than before, which is preferred.

[0173] <Second Configuration Example of Spatial Floating Image Information Display System>

[0174] Fig.13(A) shows the main structure of a spatial floating image information display system 1 according to an embodiment. Fig.13 (B) is an enlarged view of the cylindrical lens 1103, which is arranged at Fig.13 The image light emitting side of the image display device 10 shown in (A) is the image light emitting side of the display panel 11. Fig.13 The spatial floating image information display system 1 shown in (A) is a system suitable for a viewer, i.e., a user, to observe the spatial floating image 3 from obliquely above. Fig.13 In the coordinate system (X, Y, Z) of (A), the shell 350 of the spatial suspended image information display system 1 is arranged on a horizontal plane (XY plane), and the spatial suspended image 3 is formed slightly tilted in the front-to-back direction (Y direction) relative to the vertical direction (Z direction). When the user's viewpoint E is properly viewed from the surface of the spatial suspended image 3 facing the surface, the viewpoint E is configured to be slightly tilted upward in the Y direction relative to the surface of the spatial suspended image 3 and the optical axis J2 as shown in the figure. The user can properly view the spatial suspended image 3 from the viewpoint E with a line of sight that is slightly tilted downward in the Y direction. On the other hand, Fig.13 (B) is a view viewed in the direction opposite to the direction of light emitted from the liquid crystal panel. Fig.13 As shown in (B), the cylindrical lens 1103 is substantially parallel or parallel to the light emitting surface of the liquid crystal panel and is arranged on the side of the image light emitted from the liquid crystal panel surface. In addition, the multiple semi-cylinders of the cylindrical lens in the shape of "Japanese fish cake" are arranged vertically in the XZ plane. In addition, the above coordinate system is Fig.13 (A) and (B) are common.

[0175] The image display device 10 and the like are arranged in a predetermined positional relationship in the housing 350. The top surface (XY plane) of the housing 350 forms an opening, and the retroreflective member 330 is arranged at a predetermined angle α1. The optical axis J1 of the image display device 10 is directed obliquely upward at a predetermined angle β1 relative to the Y direction.

[0176] The image display device 10 is constructed to include a liquid crystal display panel 11 as an image display element, and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic. The liquid crystal display panel 11 can be applied to a small panel with a screen size of about 5 inches to a large panel with a screen size of more than 80 inches, and is composed of a panel selected therefrom. The image light from the liquid crystal display panel 11 is emitted toward the retroreflective component 330 (also referred to as a retroreflective portion or a retroreflective plate) on the optical axis J1. Light from the light source device 13 with a narrow divergence angle described later is made incident on the liquid crystal display panel 11. Thus, an image light beam φ1 with a narrow divergence angle is generated. The image light beam φ1 with a narrow divergence angle is made incident on the retroreflective component 330 from the lower side in the Z direction along the optical axis J1. By retroreflection in the retroreflective component 330, according to the above Figure 4 According to the principle described in the above, an image beam φ2 with a narrow divergence angle is generated in the direction of the optical axis J2 relative to the retro-reflective component 330 in the Z direction. The spatial floating image 3 ( Figure 4 The optical axis J2 is oriented obliquely upward at a predetermined angle β2 relative to the Y direction.

[0177] The spatial floating image 3 is formed at a symmetrical position of the image display device 10 with the retro-reflective component 330 as a symmetric plane. The surface of the image display device 10 and the surface of the spatial floating image 3 are arranged at a substantially symmetrical position or a symmetrical position relative to the surface of the retro-reflective component 330 arranged obliquely. On the surface of the spatial floating image 3, r2 represents the center position corresponding to the optical axis J2, r1 represents the lower end position corresponding to the light ray at the lower end of the image beam φ2, and r3 represents the upper end position corresponding to the light ray at the upper end of the image beam φ2.

[0178] In this structure, in order to eliminate Figure 4 The high-quality spatial floating image 3 is obtained by the ghost images 332 and 333 generated by the retro-reflective component 330, and an image light control sheet 334 (specifically, the above-mentioned Figure 5 or Figure 6 (A)). Thus, the diffusion characteristics in the unnecessary direction are controlled.

[0179] Furthermore, the image light from the liquid crystal display panel 11 is as follows Fig.11 As shown, since the reflectivity in the reflective components such as the retro-reflective component 330 can be improved in principle, S polarization (electromagnetic waves with electric field components perpendicular to the incident surface, S is the abbreviation of senkrecht) can be used. However, when the user uses polarized sunglasses, the suspended image 3 in the air will be reflected or absorbed by the polarized sunglasses, so P polarization (electromagnetic waves with electric field components parallel to the incident surface, P is the abbreviation of parallel) can be used when measures are taken for this. For this reason, as an element that optically converts a part of the image light of a specific polarization into another polarization and thus simulates the conversion into natural light, a depolarizing element 339 shown in the figure is provided. For example, the depolarizing element 339 is arranged on the exit side of the image light control sheet 334. As a result, the user can also watch a good suspended image 3 in the space when using polarized sunglasses.

[0180] As commercially available products of the depolarizing element 339, COSMOSHINE SRF (manufactured by Toyobo Co., Ltd.) and depolarizing adhesive (manufactured by Nagase Sangyo Co., Ltd.) can be cited. When COSMOSHINE SRF (manufactured by Toyobo Co., Ltd.) is used, by laminating the adhesive on the image display device, the reflection of the interface can be reduced and the brightness can be improved. In addition, when the depolarizing adhesive (manufactured by Nagase Sangyo Co., Ltd.) is used, a colorless transparent plate and an image display device are laminated via the depolarizing adhesive.

[0181] In addition, in this embodiment, an image light control sheet 338B (similar to the image light control sheet 338, specifically, the image light control sheet 338B) is also provided on the image emitting surface of the retro-reflective component 330. Figure 6 Thus, the ghost images 332 and 333 generated on both sides of the normal image of the spatial floating image 3 due to the extra light are eliminated. Figure 4 ).

[0182] The structure adopted in this embodiment is to tilt the retro-reflective component 330 at a predetermined angle α1 relative to the horizontal axis (Y direction), and to generate the space floating image 3 tilted relative to the horizontal axis (especially, tilted at an angle closer to the vertical plane than to the horizontal plane). Not limited to this, by changing the arrangement of the components, the position and inclination angle of the arrangement of the space floating image 3 can be designed.

[0183] In addition, in this embodiment, a first distance measuring device 340 ( Figure 7 ). That is, the system achieves Figure 7 The same sensing technology is used. Thus, a system is formed in which a user can access and interact with the space floating image 3. The first sensing system including the first distance measuring device 340 can detect the state of the user's finger or the like operating the space floating image 3. Furthermore, it can also be configured as follows: Figure 8 and Fig. 9 (B) Similarly, a second sensing system including a second distance measuring device 341 is added.

[0184] The installation position and the viewing angle α3 of the first distance measuring device 340 can be appropriately selected in such a manner as to be able to fully cover the size of the spatial suspended image 3. In this example, the first distance measuring device 340 is installed in the housing 350 at the deep side in the Y direction (the deep side relative to the position of the user and the spatial suspended image 3), located on the extension line of the inclined surface of the retroreflective component 330, and slightly away from the position so as not to block the image beam of the image light, that is, the position shown in the figure. In this example, the viewing angle α3 (the range from the upper end A to the lower end B) of the first distance measuring device 340 is set to a sufficiently wide viewing angle in such a manner as to be able to cover the entire spatial suspended image 3 and the facial area of ​​the user viewing the suspended image from the viewpoint E of the reference position (the opposite position). The viewing angle α3 includes a viewing angle α2 that can capture the entire spatial suspended image 3. The viewing angle α2 is, for example, Figure 7 The sensing surfaces a0, a1, a2, and a3 correspond to each other.

[0185] The TOF sensor of the first distance measuring device 340 is as follows Figure 7 (or Figure 8 ), a ranging system is used to divide the sensing surface of the space floating image 3 into multiple areas. Thus, the resolution of each sensing area can be improved. Figure 8 and Fig. 9 In the case of the second sensing technology based on the CMOS sensor shown in (B), the detection accuracy can be further improved.

[0186] In addition, in this embodiment, a light source that emits visible light with narrow-angle directivity is used as the light source device 13, and the first distance measuring device 340 (and the second distance measuring device 341) are arranged on the side of the housing 350 at a position located outside the narrow-angle image beam. In addition, the second distance measuring device 341 may also be arranged. In this way, it is possible to eliminate the adverse effect of the image light used to form the spatial floating image 3 on the sensing accuracy.

[0187] exist Fig.13 In the above structure shown in (A), a cylindrical lens 1103 is arranged on the image light emitting side (the position shown by the oblique lines) of the liquid crystal display panel 11. More specifically, on the image light emitting side of the liquid crystal display panel 11, a cylindrical lens 1103 is arranged on the image light emitting side of the liquid crystal display panel 11. Fig.13 The cylindrical lens 1103 is arranged in the direction shown in (B). By adopting this structure, Figure 2 As described above, by moving in the arrangement direction of the semi-cylindrical lenses ("Japanese fish cake"-shaped lenses) forming the cylindrical lenses, the user can see different images (or videos) from different positions. That is, by displaying parallax images as images or videos, motion parallax can be generated, and the images or videos displayed on the liquid crystal display panel 11 can be recognized as three-dimensional images. Multi-viewpoint images and motion errors will be described later.

[0188] exist Fig.13 In (A), the spatial suspended image 3 is a real image formed at a symmetrical position of the image display device 10 or the liquid crystal display panel 11 with the retro-reflective component 330 as a symmetrical plane, so the user can see a three-dimensional image with motion parallax as the spatial suspended image 3. That is, by configuring the above structure with a cylindrical lens, the spatial suspended image 3 can not only display a simple two-dimensional image displayed on the liquid crystal display panel 11, but also display a three-dimensional image with motion parallax.

[0189] <Third Configuration Example of Spatial Floating Image Information Display System>

[0190] Fig.14 (A) represents another embodiment of the spatial floating image information display system. In addition, Fig.14 (B) is an enlarged view of the cylindrical lens 1103, which is arranged at Fig.14 The image light emitting side of the image display device 10 shown in (A) is the light emitting side of the display panel 11. Fig.14 In the coordinate system (X, Y, Z) of (A), the shell 350 of the spatial suspended image information display system 1 is arranged on a horizontal plane (XY plane), and the spatial suspended image 3 is formed slightly tilted in the front-to-back direction (Y direction) relative to the vertical direction (Z direction). When the user's viewpoint E is properly viewed from the surface of the spatial suspended image 3 facing the surface, the viewpoint E is arranged to be slightly tilted upward in the Y direction relative to the surface of the spatial suspended image 3 and the optical axis J2 as shown in the figure. The user can properly view the spatial suspended image 3 from the viewpoint E with a line of sight that is slightly tilted downward in the Y direction. On the other hand, Fig.14 In (B), the cylindrical lens 1103 is substantially parallel or parallel to the light emitting surface of the liquid crystal panel and is arranged on the side of the image light emitted from the liquid crystal panel. In addition, the multiple semi-cylinders of the cylindrical lens in the shape of "Japanese fish cake" are arranged vertically in the XZ plane. In addition, the above coordinate system is Fig.14 (A) and (B) are common.

[0191] The image display device 10, the reflector 360, etc. are arranged in a predetermined positional relationship in the housing 350. The retro-reflective component 330 is arranged at a predetermined angle γ1 (slightly inclined downward) relative to the Z direction in the opening of the housing 350, which in this example is an opening having a surface (XZ surface) that stands substantially in the vertical direction. The reflector 360 is a plane reflector.

[0192] In this embodiment, the image light from the image display device 10 is reflected on the reflector 360 and then incident on the retro-reflective component 330. The housing 350 has a portion protruding convexly toward the upper side in the Z direction, and the image display device 10 is arranged in this portion. The optical axis J1 of the image display device 10 is on the lower side in the Z direction and the deep side in the Y direction, and is directed obliquely downward at a predetermined angle δ1 relative to the Z direction.

[0193] The image display device 10 is configured to include a liquid crystal display panel 11 as an image display element and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic. The liquid crystal display panel 11 can be configured by a panel selected from a small panel having a screen size of about 5 inches to a large panel having a screen size of more than 80 inches. The image light from the liquid crystal display panel 11 is deflected on the optical axis J1 by a reflector 360 as an optical path deflection reflector, and is emitted toward the retro-reflection unit 330 on the optical axis J1B after deflection.

[0194] Light from the narrow divergence angle light source device 13 described later is incident on the liquid crystal display panel 11. Thus, an image beam φ1 with a narrow divergence angle is generated. The image beam φ1 with a narrow divergence angle is reflected on the reflector 360 to become an image beam φ1B. The image beam φ1B with a narrow divergence angle is incident on the retro-reflective component 330 from the right side in the Y direction as shown in the figure along the optical axis J1B. Through retro-reflection in the retro-reflective component 330, the image beam φ1B is reflected as described above. Figure 4 According to the principle described in the above, an image beam φ2 with a narrow divergence angle is generated in the direction of the optical axis J2 on the left side of the retroreflective component 330 in the Y direction. Through this image beam φ2, a spatial floating image 3 ( Figure 4 The optical axis J2 is oriented obliquely upward at a predetermined angle δ2 relative to the Y direction ((90 degrees - δ2) relative to the Z direction).

[0195] The spatial suspended image 3 is formed at a position that is substantially symmetrical with respect to the reflector 360 with the retro-reflective component 330 as a symmetry plane. In this embodiment, since the reflector 360 is used to fold the optical path, the image display device 10 is arranged above the spatial suspended image 3 in the Z direction. As a result, the image light is incident from obliquely above and emitted obliquely above relative to the retro-reflective component 330, and a system that forms an inclined spatial suspended image 3 as shown in the figure can be realized.

[0196] In addition, in order to image the spatial floating image 3 obliquely upward (on the optical axis J2 shown in the figure) relative to the housing 350, this can be achieved by arranging the retro-reflective component 330 at a predetermined angle γ1 relative to the vertical axis (Z direction) of the bottom surface of the housing 350 as shown in the figure. In addition, such a structure in which the emission axis of the retro-reflective component 330 is slightly tilted downward can prevent the image quality of the spatial floating image 3 from being reduced due to external light being incident on the retro-reflective component 330 and entering the housing 350.

[0197] In order to eliminate the possible ghost images in the space floating image 3 ( Figure 4 ) to obtain a high-quality spatial floating image 3, which is different from the second embodiment ( Fig.13 Similarly, in this embodiment, an image light control sheet 334 ( Figure 5 , Figure 6 In addition, an image light control sheet 334B ( Figure 6 (B)) is used to eliminate ghost images on both sides of the normal image of the spatial floating image 3 caused by excess light.

[0198] By disposing the above-described structure inside the housing 350 , it is possible to prevent external light from entering the retroreflective member 330 , thereby preventing the generation of ghost images.

[0199] In this embodiment, Fig.13 (A) Similarly, the image light from the liquid crystal display panel 11 can use S polarization. In the case of supporting polarized sunglasses, P polarization can also be used and a depolarizing element 339 can be provided.

[0200] The structure adopted in this embodiment is to tilt the retro-reflective component 330 at a predetermined angle γ1 relative to the vertical axis (Z direction), and to generate the space floating image 3 tilted relative to the horizontal axis (especially, tilted at an angle closer to the vertical plane than to the horizontal plane). Without being limited to this, the position and tilt angle of the configuration of the space floating image 3 can be designed and adjusted by changing the configuration of the components.

[0201] In addition, in this embodiment, a first distance measuring device 340 ( Figure 7 ). That is, the system achieves Figure 7 The same sensing technology is used. Thus, a system is formed in which a user can access and interact with the space floating image 3. The first sensing system including the first distance measuring device 340 can detect the state of the user's finger or the like operating the space floating image 3. Furthermore, it can also be configured as follows: Figure 8 and Fig. 9 (B) Similarly, a second sensing system including a second distance measuring device 341 is added.

[0202] The installation position and the viewing angle γ3 of the first distance measuring device 340 can be appropriately selected in such a manner that the size of the space floating image 3 can be fully covered. In this example, the first distance measuring device 340 is installed on the bottom surface of the housing 350, in the Y direction, near the front side of the retro-reflective component 330 and slightly away from the image beam of the image light, that is, the position shown in the figure. In this example, the viewing angle γ3 of the first distance measuring device 340 is set to a sufficiently wide viewing angle in such a manner that it can cover the entire space floating image 3 and the face area of ​​the user who views the floating image from the viewpoint E of the reference position. The viewing angle γ3 includes a viewing angle that can capture the entire space floating image 3.

[0203] It is possible to use not only the first distance measuring device 340 but also further use Figure 8 and Fig. 9 (B) shows the second sensing technology based on CMOS sensor.

[0204] In addition, in this embodiment, a light source that emits visible light with narrow-angle directivity is used as the light source device 13, and the first distance measuring device 340 (and the second distance measuring device 341) are arranged on one side of the housing 350 at a position located outside the narrow-angle image beam. In this way, the image light used to form the spatial floating image 3 can be eliminated from having a negative impact on the sensing accuracy.

[0205] Furthermore, in this embodiment, between the space floating image 3 and the retro-reflective component 330, as shown in the figure, a touch panel 361 of an electrostatic capacitance method can also be fixed and arranged by a supporting component 362. The supporting component 362 is, for example, frame-shaped and supports the touch panel 361 on the inner side. The supporting component 362 is, for example, fixed to the bottom surface of the housing 350. The touch panel 361 is composed of a component that can transmit the image light used to form the space floating image 3 and the light from the first distance measuring device 340.

[0206] The touch panel 361 detects the state of the user's finger approaching the surface of the touch panel in an electrostatic capacitance manner. Alternatively, the touch panel 361 detects the state of the user's finger contacting the surface of the touch panel. By using a third sensing technology including the touch panel 361 together with the first sensing technology, the detection accuracy can be further improved. The size and installation position of the electrostatic capacitance touch panel 361 can also be selected in a manner that can fully cover the spatial floating image 3.

[0207] As a touch panel 361 of an electrostatic capacitance method capable of obtaining high-precision position information, for example, a projected electrostatic capacitance method can be used. The touch panel of this method is manufactured, for example, by forming an ITO transparent electrode (Y-axis electrode) with a fine line spacing and a copper thin film transparent electrode (X-axis electrode) with a fine line spacing on both sides of a transparent glass substrate by photolithography. Therefore, when an object (such as a fingertip) is close to the transparent glass substrate, the change in electrostatic capacitance is captured by the X-axis electrode and the Y-axis electrode respectively, and the relative coordinates of the object can be obtained. In addition, in this method, the shorter the line spacing of the transparent electrode, the higher the resolution can be obtained, so multi-point detection is possible. Therefore, this method can also be used for simultaneous input with multiple fingers.

[0208] exist Fig.14 In the above structure shown in (A), Fig.13 Similarly, in the spatial floating image information display system 1 shown in (A), a cylindrical lens 1103 is arranged on the image light emitting side (the position shown by the oblique lines) of the liquid crystal display panel 11. More specifically, Fig.14 As shown in (B), a cylindrical lens 1103 is arranged on the image light emitting side of the liquid crystal display panel 11. By adopting this structure, the user can recognize the spatial suspended image 3 as a stereoscopic image with motion parallax. That is, by configuring the above structure with the cylindrical lens 1103, as the spatial suspended image 3, not only a two-dimensional image displayed on the liquid crystal display panel 11 can be simply displayed, but also a stereoscopic image can be displayed.

[0209] Here, the user can recognize the spatial suspended image 3 as a stereoscopic image, especially when the displayed stereoscopic image is a person (especially a face), which brings a new effect that is not available in the existing system where the spatial suspended image is a two-dimensional plane. For example, as described later, such a new effect is brought about: as long as the user is located in the vicinity of the spatial suspended image, no matter where it exists, the person (face) displayed as the spatial suspended image is always facing the direction of the user. As a result, the user has the feeling that the person displayed as the spatial suspended image is only talking to himself, which is particularly preferred in the scene where the displayed person, for example, explains something to the user, or provides some kind of assistance like a virtual assistant.

[0210] [Technology for displaying multi-viewpoint images as floating images in space]

[0211] As described above, motion parallax can be obtained by using a multi-viewpoint image (or multi-viewpoint video) using a cylindrical lens. The cylindrical lens is a collection of lenses having a semi-cylindrical surface and a "Japanese fish cake" shape arranged in a predetermined direction. A liquid crystal panel that displays different images corresponding to the number of viewpoints of the multi-viewpoint image (or video) is arranged below one semi-cylindrical lens. The predetermined direction in this embodiment is the vertical direction.

[0212] Fig.15 (A) is a diagram showing the principle of generating a multi-viewpoint image using a cylindrical lens 1103 in this embodiment. Fig.15 (B) is a diagram showing the lenticular lens 1103 as viewed from above obliquely in order to more easily understand the structure of the lenticular lens 1103. In addition, here, a case of nine viewpoints as a multi-viewpoint image is described. Fig.15 In (A), the pixels of the liquid crystal panel 11 are grouped into nine pixels represented by numbers 1 to 9, thereby forming a multi-viewpoint image of nine viewpoints.

[0213] On the other hand, it is known that the distance between the pupils of human eyes is approximately constant (for example, about 64 mm). By making the spacing of the cylindrical lenses, that is, the spacing of one semi-cylindrical shape, approximately equal to half of the spacing of the human eyes, that is, about 32 mm, Fig.15 As shown in (A), light from different pixels reaches the right eye and left eye of the observer (user). More specifically, light from the image displayed at pixel 6 reaches the right eye of the observer, and light from the image displayed at pixel 4 reaches the left eye of the observer.

[0214] Therefore, the observer sees images displayed at different pixels with his right eye and left eye, and if images of the same object or person shot from different viewpoints are displayed at each pixel, parallax occurs between the observer's eyes. As a result, the observer can recognize the shot image as three-dimensional. That is, Fig.15 (A) shows a situation where light from an image displayed at pixel 6 reaches the right eye of the observer, and light from an image displayed at pixel 4 reaches the left eye of the observer. As described above, by configuring a cylindrical lens on the light emitting side of the liquid crystal panel, light from different pixels reaches the right eye and left eye of the observer, and the observer can recognize a three-dimensional image.

[0215] Here, when the observer moves left and right, light from pixels different from before the movement reaches the observer's right and left eyes. More specifically, Fig.15 As shown in (A), when the observer moves to the right, the light from the image displayed at pixel 7 reaches the observer's right eye, and the light from the image displayed at pixel 5 reaches the observer's left eye. That is, as the observer moves (moves), the light reaching the human eye becomes the light of different pixels than before the movement. As a result, as the observer moves left and right, the observer can obtain the same effect as viewing the same object or person from different angles, that is, motion parallax. Therefore, by configuring a cylindrical lens on the light emitting side of the liquid crystal panel, the same effect as viewing a three-dimensional image at a different angle can be obtained through the movement of the observer's eyes.

[0216] Fig.16This is a diagram showing an example of a device for capturing an image capable of generating the above-mentioned motion parallax, that is, a multi-viewpoint image. Fig.16 FIG. 1 shows a state where a person (face part) of a subject is photographed from nine different viewpoints. More specifically, nine cameras No. 1 to No. 9 are arranged as follows: Fig.16 As shown, the cameras are arranged at a predetermined distance from the subject and the angles therebetween are successively moved by predetermined angles to perform photography. In the present embodiment, the cameras are arranged at an equal distance from the subject and the angles therebetween are successively moved by 22.5 degrees, but the distance and angle from the subject may also be changed according to the number of viewpoints. At this time, if the subject is stationary, multi-viewpoint images can be captured by moving one camera sequentially to the positions of cameras No. 1 to No. 9 and performing photography. However, when the subject is moving, such as the face of a person who changes his expression and speaks at the same time, 9 cameras are used and the cameras are fixed at each position to capture dynamic images (dynamic images, images).

[0217] As described above, the images (or videos) captured by the nine cameras are respectively assigned to the nine types of pixels of the image display unit (here, the liquid crystal panel) for display. Fig.16 As shown, by displaying an image of a subject as images (or images) taken from different angles, a multi-viewpoint image (or multi-viewpoint image) with motion parallax can be obtained. Fig.16 The illustrated embodiment shows an example of obtaining a multi-viewpoint image (or video) by photographing a person (the face part) as a subject with nine cameras No. 1 to No. 9 at different angles. However, the method of obtaining the multi-viewpoint image (or video) is not limited to the method of using a plurality of cameras as described above, but may also be a method of rendering the multi-viewpoint image (or video) with computer graphics (CG).

[0218] Here, rendering refers to the generation or editing of images or dynamic images (videos) into CG through computer processing. By using rendering to generate CG, it is not necessary to use a large-scale shooting device based on multiple cameras, and it is possible to obtain multi-viewpoint images (or videos) more simply, without the limitation of the number of viewpoints determined by the number of cameras, and in a short time, which is particularly suitable for generating multi-viewpoint images and multi-viewpoint videos.

[0219] Fig.17: is a diagram showing a display example of a multi-view image display device. Here, the multi-view image display device refers to a display device having a structure in which a cylindrical lens 1103 is arranged on the image light emitting side of the image display device 10 (composed of a liquid crystal panel 11, a light source 13, etc.). Specifically, the multi-view image display device has a light source, an image display unit (liquid crystal panel), and a cylindrical lens. Here, the image emitting surface of the image display unit (liquid crystal panel) 11 is parallel to the incident surface of the cylindrical lens 1103. In addition, the image emitting surface of the image display unit (liquid crystal panel) 11 is arranged at a predetermined distance from the incident surface of the cylindrical lens 1103. In the present embodiment, based on the inherent focal length of the cylindrical lens, the predetermined distance between the light incident surface of the cylindrical lens 1103 and the light emitting surface of the liquid crystal panel 11 is adjusted for arrangement. At this time, if the focal length of the cylindrical lens 1103 is a relatively large value, the above-mentioned specified distance is increased. Conversely, if the focal length of the cylindrical lens 1103 is a relatively small value, the above-mentioned specified distance is reduced. By adjusting the distance between the light incident surface of the cylindrical lens 1103 and the light exit surface of the liquid crystal panel 11, that is, the above-mentioned specified distance, an appropriate multi-viewpoint image can be obtained.

[0220] Fig.17 A multi-viewpoint image display device capable of displaying multi-viewpoint images having 9 different viewpoints is shown. Fig.17 (A) shows a case where images of cameras No. 1 to No. 9 are arranged for pixels 1 to 9 of the liquid crystal panel. On the other hand, Fig.17 (B) shows a case where the images of cameras No. 9 to No. 1 are arranged for pixels 1 to 9 of the liquid crystal panel in the reverse order of (A). Fig.17 The difference between the effects of (A) and (B) is as follows. First, Fig.17 In (A), when the user (user) moves from the left to the right relative to the multi-viewpoint image display device, an image of the subject (person's face) viewed from the left can be observed from the left, and an image of the subject (person's face) viewed from the right can be observed from the right. That is, what is observed is the same subject as when the actual subject is placed at the center and the user observes from the left and right sides of the subject.

[0221] In contrast, Fig.17 In (B), Fig.17 (A) On the contrary, when the user (observer) moves from the left to the right relative to the multi-viewpoint image display device, the user (observer) can observe the image from the left to the right. Fig.17 The image of the subject (the face of the person) is viewed from the right side of (A), and the Fig.17 The image of the subject (the face of the person) is viewed from the left side of (A). Fig.17In (B), when the user looks at the subject (the face of a person), no matter where the user is relative to the subject (relative angle), it can be felt that the person as the subject always looks toward the user.

[0222] above Fig.17 The visual effect (B), that is, the subject always looks toward the user regardless of the user's position, will give the user the feeling that the subject is always speaking to him / her (the user). This effect is particularly preferred in a scene where the subject is only explaining or guiding the user.

[0223] then, Fig.18 , Fig.19 is a schematic representation of the Fig.17 The diagram shows a state in which the image light emitted from the multi-viewpoint image display device (composed of the image display device 10 and the lenticular lens 1103 ) passes through the retro-reflective plate 330 to generate a spatial floating image 3 . Fig.18 , Fig.19 The similarity of the illustrated embodiments is that they all use a multi-viewpoint image display device and a retro-reflective plate 330 to generate a spatial floating image 3 .

[0224] Above Fig.18 and Fig.19 The difference between these two embodiments is that the order of the multi-viewpoint images on the multi-viewpoint image display device is different. Fig.18 and Fig.17 Corresponding to (A), from the user's point of view, in the multi-view image display device, the images of cameras No. 1 to No. 9 are allocated from left to right, that is, for pixels 1 to 9 of the liquid crystal panel, the order of the multi-view images in the spatial suspended image 3 generated by the retroreflective plate 330 is reversed from the user's point of view, and the images of cameras No. 1 to No. 9 are displayed from right to left.

[0225] on the other hand, Fig.19 and Fig.17 In the multi-view image display device, the images of cameras No. 1 to No. 9 are allocated from right to left, that is, to pixels 1 to 9 of the liquid crystal panel. As a result, from the user's point of view, the order of the multi-view images in the spatial floating image 3 generated by the retro-reflective plate 330 is the same as Fig.18 On the contrary, images from cameras No. 1 to No. 9 are displayed from left to right.

[0226] As described above, in the spatial suspended image 3 generated by the retroreflector 330, by arranging the cylindrical lens 1103 between the image display device 10 and the retroreflector 330, the order of the multi-viewpoint images of the spatial suspended image 3 is recognized by the user in the reverse order of the order of the multi-viewpoint images displayed on the image display device 10. That is, in the case of providing the spatial suspended image with motion parallax to the user, the order of the images of the cameras No. 1 to No. 9 can be appropriately determined and arranged for the pixels on the liquid crystal panel 11 according to the purpose of which multi-viewpoint image should be provided to the user.

[0227] To explain this in more detail, Fig.18 In the embodiment of FIG. 1 , when the user views the spatial floating image 3 with the face of the virtual assistant as the subject, the order of the multi-viewpoint images is to recognize the images of cameras No. 1 to No. 9 from right to left. In this case, regardless of the user's position, the virtual assistant always keeps its sight line toward the user. On the other hand, in Fig.19 In the embodiment of FIG. 1 , when the user views the spatial floating image 3 with the face of the virtual assistant as the subject, the order of the multi-viewpoint images is such that the images of cameras No. 1 to No. 9 are recognized from left to right. In this case, regardless of the position of the user, the virtual assistant always directs his / her sight in a certain direction, that is, the front direction.

[0228] As mentioned above, in Fig.18 In an embodiment, when a user views a virtual assistant as a floating image in space, the virtual assistant always faces the user, which is particularly preferred in scenarios where certain instructions or guidance are conveyed accompanied by voice. The user can easily empathize with the virtual assistant, which can result in the user being able to focus more on the instructions and guidance given by the virtual assistant.

[0229] On the other hand, Fig.19 As shown, when the user watches the virtual assistant as a spatial floating image, the virtual assistant always faces a certain direction, i.e., the front, and speaks. In this case, the user recognizes that the virtual assistant is not only speaking to himself, but feels that it is giving some kind of speech to an uncertain number of people around the spatial floating image 3, for example.

[0230] That is, in the case of displaying a virtual assistant as a spatially suspended image, in the two embodiments described above, Fig.18 As shown in FIG. 1 , the virtual assistant always speaks to the user (referred to as “display mode A”). Fig.19 The display form shown is one in which the virtual assistant always speaks facing forward (referred to as "display form B"), and the above two display forms can be appropriately selected.

[0231] <About implementation method 1 of the self-service terminal>

[0232] As mentioned above, use Figures 15 to 19 , two forms of displaying a virtual assistant as a spatial floating image, namely "display form A" and "display form B" are described. Next, as an embodiment of the present invention, a case where the spatial floating image information display system 1 is applied to a so-called self-service terminal (referred to as embodiment 1) is described below.

[0233] A self-service terminal is an information terminal that allows an indefinite number of people to access necessary information or use various services through a human-machine interface such as a touch panel operation or a user interface. Self-service terminals are installed in public facilities, transportation agencies, amusement parks and other entertainment facilities, and in recent years, so-called convenience stores. Self-service terminals are also used for the sale of various tickets and administrative services (such as the issuance of residence certificates).

[0234] In the following description of the embodiment, the term "self-service terminal" is used to express an information terminal having a specific structure. The term "self-service terminal" may be replaced by "information terminal", "information display device", "information processing terminal", "ticket issuing terminal", "document issuing terminal", "administrative terminal", "service terminal", etc. The term "self-service terminal" mainly used in the description of the embodiment is used as a representative example of these terms.

[0235] exist Fig. 20 , an example of the appearance of a common self-service terminal based on the prior art is shown. The self-service terminal 1400, for example, has a metal housing 1450 with a height of about 120 to 50 cm, and a liquid crystal display screen 1410 and input buttons 1420 are provided on the surface of the housing 1450 (the surface on the side opposite to the user, especially the inclined surface 1451). The liquid crystal display screen 1410 is a part of the liquid crystal display device, and is a screen (i.e., a screen) with a touch panel that displays various information and accepts touch operations by the user. The input button 1420 is a physical button for the user to input a unique authentication password, etc., or a touch button within the screen composed of a touch panel. In addition, a take-out port 1430 is provided on a part of the surface of the housing 1450. The take-out port 1430 is a take-out port for the user to take out items such as tickets and administrative documents issued by operating the self-service terminal 1400.

[0236] exist Fig.21 , an example of the appearance of a self-service terminal of the spatial floating image information display system of Embodiment 1 is shown. Fig.21 2 shows an overview of the kiosk terminal 1500 when viewed from the diagonal right. Fig.21 Self-service terminal 1500 and Fig. 20 The differences of the self-service terminal 1400 are as follows. Fig.21 The self-service terminal 1500 has, on the surface (particularly the inclined surface 1570) of the housing 1550, in addition to the liquid crystal display screen 1510 of the liquid crystal display device, a space floating image display unit 1520 for displaying the above-mentioned space floating image 3 at the bottom. That is, the self-service terminal 1500 has two screens (screens), the liquid crystal display screen 1510 and the space floating image display unit 1520, and is divided into two display parts, the liquid crystal display screen 1510 and the space floating image display unit 1520, on the inclined surface 1570.

[0237] exist Fig.21 In the structure shown, the screen of the space floating image display unit 1520 (hereinafter also referred to as the first screen) is mainly used among the two screens. In the first screen, a virtual assistant and an operation menu are displayed as a user interface formed by the space floating image 3. The first screen of the space floating image display unit 1520 has an area of ​​predetermined dimensions in both vertical and horizontal directions. In this example, the first screen has a dimension that is slightly longer in horizontal direction.

[0238] On the other hand, the liquid crystal display screen 1510 (hereinafter also referred to as the second screen) can display any image, and is used for displaying advertisements, etc., similar to a common kiosk terminal. The liquid crystal display screen 1510 may be, for example, a liquid crystal touch panel screen having a touch sensor.

[0239] In addition, as a modification of the first embodiment, the second screen of the liquid crystal display screen 1510 and the first screen of the spatial floating image display unit 1520 may be used together as a user interface such as an operation menu.

[0240] In addition, as another modification example, a configuration in which the liquid crystal display screen 1510 , ie, the second screen is not provided may be adopted.

[0241] Can be Fig.21 In the first screen of the spatial floating image display unit 1520, both the virtual assistant and the operation menu are displayed as one spatial floating image 3. However, since the size of the first screen is limited, when both are displayed in the first screen, the displayed content is small and may be difficult to see. Therefore, Fig.21 The embodiment can perform display switching control to display one of the virtual assistant and the operation menu larger in the first screen.

[0242] Of course, the positional relationship between the liquid crystal display screen 1510 and the space floating image display unit 1520 is not limited to Fig.21In other words, the space floating image display unit 1520 may be arranged on the upper part of the inclined surface 1570, and the liquid crystal display screen 1510 may be arranged on the lower part. In addition, the inclined surface 1570 may be arranged side by side. However, as described later, Fig. 22 As shown in the internal structure, in a structure in which the self-service terminal 1500 has a spatial floating image display unit 1520 in addition to the liquid crystal display screen 1510, from the perspective of component configuration within the shell 1550 (efficiency in the use of space), it is more appropriate to configure the liquid crystal display screen 1510 at the top and the spatial floating image display unit 1520 at the bottom.

[0243] In addition, in Fig.21 In the case of a structure with two screens composed of two display units, in order to make it easy for the user to know that the screens of the two display units are the liquid crystal display screen 1510 and the space floating image display unit 1520, a notification display such as "This is a liquid crystal screen" or "This is a space floating image" may be displayed on each screen. This makes it easier for the user to use. In addition, instead of displaying it on the screen, a mark such as "liquid crystal screen" or "space floating image" may be physically marked in advance near the frame of each screen.

[0244] Fig.21 In the example shown, a virtual assistant 1521 (also referred to as a person image or a virtual assistant image) is displayed as a spatial floating image 3 on the spatial floating image display unit 1520 . Fig.21 In the illustrated structure, in addition to the image or video information displayed on the liquid crystal display screen 1510, the user can also operate the screen (especially the operation menu displayed on the spatial floating image display unit 1520) according to the operation guidance given by the virtual assistant 1521 displayed on the spatial floating image display unit 1520, which will be described in detail later. The virtual assistant 1521 provides the user with operation guidance (also referred to as operation guide, operation instructions, etc.) through images and sounds.

[0245] Furthermore, in Embodiment 1, the virtual assistant 1521 displayed on the spatial floating image display unit 1520 is displayed as a multi-viewpoint image with motion parallax, so compared with the case where a two-dimensional plane image is simply displayed as a spatial floating image, the user can feel as if a real person (virtual assistant) exists on the kiosk terminal 1500. Furthermore, the virtual assistant carefully explains to the user how to operate the kiosk terminal 1500. Therefore, even if it is the first time for a user to use the kiosk terminal, it can operate the kiosk terminal 1500 more easily without confusion and can receive the requested service.

[0246] Here, in Fig.21 In the space floating image display unit 1520 shown in FIG. 1 , as a method for forming the space floating image 3, the above-mentioned Figure 3 The structure using the retro-reflective component 2 or the like, or the above-mentioned Fig.13 and Fig.14 The structure using the retro-reflective component 330 and the like. In addition, other methods capable of displaying a space floating image in the air can also be applied without being limited to the above-mentioned methods. Fig.14 The structure of the case (internal structure refers to Fig. 22 ).

[0247] Furthermore, regarding Fig.21 In the embodiment of the present invention, the detection method (sensing technology) for detecting the user's operation on the spatial floating image 3 can apply the above Figure 2 The above-mentioned combination of the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 can also be applied. Figure 7 and Figure 8 In addition, other methods capable of detecting aerial operations on a space floating image may also be applied without limitation to the above methods.

[0248] In addition, Fig.21 In the self-service terminal 1500, two cameras 1541 and 1542 are provided at the left and right positions of the housing 1550 (especially the inclined surface 1570). These cameras are Figure 2 The cameras 1541 and 1542 capture the face and iris of a user (e.g., a user operating the spatial floating image 3) located near the front (the front side in the Y direction in the figure) of the self-service terminal 1500. The spatial floating image information display system 1, i.e., the control unit 1110 ( Figure 2 ) is configured to detect the user approaching the self-service terminal 1500, identify / determine the user, or authenticate the user based on the images captured by the cameras 1541 and 1542. In this embodiment, two cameras 1541 and 1542 are configured, but it can also be one or more than two. In addition, a camera can also be configured inside the housing 1550, and it can also be configured to send the video information captured by the camera configured outside to the self-service terminal 1500 by wire or wirelessly.

[0249] In addition, stereoscopic shooting can be performed using the left and right cameras 1541 and 1542, and the user's face can be photographed as a stereoscopic image, and the distance to the user's face and its three-dimensional shape can be calculated. Therefore, compared with the case where the user's face is photographed as a two-dimensional image using one camera, the system can improve the accuracy of user face authentication, which is also suitable from the perspective of preventing illegal use.

[0250] in addition, Fig.21 The self-service terminal 1500 has two speakers at the left and right positions of the housing 1550, in particular, two super-directional speakers 1551, 1552 (corresponding to Figure 2 Speaker / super-directional speaker 30). By having a super-directional speaker, it is possible to emit a sound with very high directivity that can only be heard by the user who wants to operate the self-service terminal 1500. The self-service terminal 1500 may also have a speaker that outputs sound in the normal auditory frequency band, but in the case described below, in order to prevent others other than the target user from hearing it, it is particularly preferred to have super-directional speakers 1551, 1552. That is, in the case where the sound emitted from the self-service terminal 1500 in response to the user's operation, etc., is, for example, the user's name, date of birth, input number information, etc., it is particularly necessary to improve confidentiality and consider security. In such a case, it is preferred to use a super-directional speaker.

[0251] in addition, Fig.21 In the embodiment, the super-directional speakers 1551 and 1552 are arranged to extend outside the housing 1550, but are not limited thereto and may be arranged inside the housing 1550. The super-directional speakers 1551 and 1552 may also be used to output a sound signal indicating that a button or the like of an operation menu of the spatial floating image 3 has been pressed in a manner that can only be heard by the user.

[0252] Fig. 22 (A) indicates Fig.21 FIG. 1 is an explanatory diagram of the internal structure of the self-service terminal 1500. Fig. 22 (A) shows the view from the right side Fig.21 The YZ cross section of the interior of the case 1550 is a perspective view. The upper portion of the case 1550 has a shape having an inclined surface 1570. In the upper portion, the Fig.14 The components of the spatial floating image information display system 1 described in the above are the image display device 10, the lenticular lens 1103, the reflector 360, the retro-reflective component 330, etc. On the other hand, Fig. 22 (B) indicates that Fig. 22 The cylindrical lens 1103 when viewed in the Z-axis direction of the coordinate axis. Fig. 22The structure of (A) and (B) enables the user to see the spatial floating image 3 as a multi-view image with motion parallax, which is similar to the use of Fig.14 Same as described.

[0253] In the lower part of the housing 1550, the Figure 2 The control device of the control unit 1110, the communication device of the communication unit 1132, and the power supply device and other components. In addition, a human body sensor 1560 may also be provided at the lower part (e.g., the front surface) of the housing 1550 as shown in the figure. The human body sensor 1560 detects whether a person approaches the self-service terminal 1500.

[0254] and Fig.14 same, Fig. 22 The structure of (A) changes the direction of the image light emitted from the image display device 10 in the housing 1550 by using the reflector 360 and makes it incident on the retro-reflective component 330. Then, the retro-reflected light generated by the retro-reflective component 330 is emitted obliquely upward on the side opposite to the image display device 10 (the front side in the Y direction). As a result, the spatial floating image 3 is generated in the spatial floating image display unit 1520.

[0255] The inclined surface 1570 of the shell 1550 has a specified angle ε1 relative to the horizontal plane (Y direction). On the inclined surface 1570, an opening portion (indicated by a dotted line) of the spatial floating image display unit 1520 is provided at a position obliquely below relative to the liquid crystal display screen 1510 above. A transparent component or the like may also be provided at the opening portion. The image light from the retroreflective component 330 passes through the opening portion obliquely upward to form a real image of a spatial floating image 3 at a specified position outside the inclined surface 1570. The spatial floating image 3 is formed to be inclined to the front side at a specified angle ε2 relative to the inclined surface 1570. Therefore, from the user's point of view, the spatial floating image 3 appears to be suspended protruding toward the front side relative to the inclined surface 1570. The user can interact with Fig.14 Similarly, the spatial floating image 3 displayed on the spatial floating image display unit 1502 is appropriately viewed from the obliquely upward viewpoint E with a line of sight directed obliquely downward.

[0256] In addition, the self-service terminal 1500 is applied Fig.14 In the case of the structure of the housing 1550, the optical path is folded back by the reflector 360 in the housing 1550, so the optical path from the image display device 10 to the retro-reflective component 330 in the housing 1550 can be made longer. As a result, the effect of increasing the protrusion amount of the spatial floating image 3 protruding obliquely forward and upward relative to the retro-reflective component 330 can be obtained. In addition, with this structure, the thickness dimension of the housing 1550 in the depth direction (Y direction) can be suppressed.

[0257] In addition, Fig. 22 In the structure of (A), the first distance measuring device 340 (including the TOF sensor, etc.) senses the range covering the entire space floating image 3 through the opening of the space floating image display unit 1520. By using the first distance measuring device 340, the user's touch operation on the space floating image 3 is mainly sensed. In addition, Fig. 22 In (A), the lower end B of the sensing range of the first distance measuring device 340 corresponds to the opening and is obliquely upward relative to the horizontal plane (Y direction), but is not limited to this and can also be close to the horizontal direction (Y direction). By changing the position of the opening of the housing 1550 and the configuration position of the first distance measuring device 340, its sensing range can be designed.

[0258] In addition, the sensing system including the first distance measuring device 340 can also be used to detect whether the user is sufficiently close to the self-service terminal 1500 (the inclined surface 1570 of the housing 1550, or the spatial floating image 3, etc.), or assist in the detection together with the camera. Fig. 22 The structure of (A) is set as follows Figure 8 In the case of the first distance measuring device 340 and the second distance measuring device 341 shown, they can be arranged at positions to the left and right of the retroreflective member 330 in the X direction so that the optical axis of distance measurement is close to the horizontal plane (Y direction).

[0259] The self-service terminal 1500 uses a sensing system including a first distance measuring device 340 to sense the operation of the user's finger on the spatial floating image 3 of the spatial floating image display unit 1520. The control unit 1110 of the self-service terminal 1500 controls the display content of the spatial floating image 3 in the spatial floating image display unit 1520 (the virtual assistant 1521, the user interface such as the operation menu described later) according to the detection information including the sensing result.

[0260] As described above, according to the spatial suspended image information display system of embodiment 1, i.e., the self-service terminal, the spatial suspended image 3 can be displayed as a multi-view image with motion parallax. Thus, compared with the case where the spatial suspended image 3 is displayed as a flat image, a better interface and necessary information implemented by the spatial suspended image can be provided to users of the self-service terminal, etc. According to the system of embodiment 1, even users who operate the user interface such as the operation menu displayed as a spatial suspended image for the first time, users who are not accustomed to the operation, or elderly users, etc., can provide the user with operation guidance by a virtual assistant composed of friendly characters displayed as a spatial suspended image and a multi-view image. As a result, it is possible to avoid or reduce the user's erroneous input and erroneous operation of the interface of the spatial suspended image, and to achieve more reliable operation.

[0261] <About implementation method 2 of self-service terminals>

[0262] Fig.23 An example of the external structure of another embodiment (referred to as Embodiment 2) of the kiosk terminal is shown. Fig.23 The self-service terminal 1600 shown is Fig.21 Unlike the self-service terminal shown in FIG. 1 , the self-service terminal does not have a liquid crystal display screen 1510, and a space floating image display unit 1620 is provided on the inclined surface 1670 of the housing 1650, which is substantially the entire surface. The screen formed by the space floating image display unit 1620 is larger than the screen formed by the liquid crystal display screen 1510. Fig.21 The screen size of the spatial floating image display unit 1520 is larger. Fig.23 In the embodiment of the present invention, the space floating image display unit 1620 has one vertically long screen, and displays one space floating image 3. In this example, the virtual assistant and the operation menu are displayed side by side in the space floating image 3 of the one screen, and the virtual assistant provides guidance on the operation of the operation menu.

[0263] In the spatial floating image display unit 1620, the spatial floating image 3 is displayed on the front side (the side closer to the user) relative to the inclined surface 1670. In this example, in the spatial floating image 3, the virtual assistant 1621 appears in the upper part, and the operation menu 1622 appears in the lower part. The operation menu 1622 includes, for example, operation buttons (represented by circles and triangles). The virtual assistant 1621 explains and guides the operation method of the operation menu 1622. Fig.23 In the case of the structure, also with Fig.21 Similarly, the user can easily operate the self-service terminal 1600, especially the operation menu 1622, according to the guidance given by the virtual assistant 1621 displayed as a multi-viewpoint spatial floating image, and can receive the required service.

[0264] Fig.23 The structure also includes cameras 1641, 1642, super-directional speakers 1651, 1652 and a human body sensor 1660 in a housing 1650. Fig.21 The structure is the same.

[0265] Fig.24 (A) means Fig.23 FIG. 1 is a diagram showing an example of the internal structure of the self-service terminal 1600, showing a YZ cross section of the internal perspective when the self-service terminal 1600 is viewed from the right side. The self-service terminal 1600 accommodates the above-mentioned Fig.13 The components of the spatial floating image information display system 1 are shown in FIG. That is, the image display device 10, the cylindrical lens 1103, the retro-reflective component 330, etc. are arranged in the upper part of the housing 1650. On the other hand, Fig.24 (B) indicates that Fig.24 The cylindrical lens 1103 is viewed in the Z-axis direction of the coordinate axis. Fig.24 The structure of (A) and (B) enables the user to see the spatial suspended image 3 in a multi-view image with motion parallax, which is also similar to the use of Fig.13 Same as described.

[0266] Fig.24 In the structure of (A), the image display device 10 is arranged along a substantially horizontal plane (specifically, slightly tilted upward on the deep side in the Y direction) on the upper bottom surface portion in the housing 1650, and the optical axis J1 of the image display device 10 is oriented substantially vertically upward (in the Z direction). In addition, an opening portion (indicated by a dotted line) of the spatially suspended image display unit 1620 is provided on the upper inclined surface 1670 of the housing 1650. In the upper portion of the housing 1650, a retro-reflective component 330 is arranged near the opening portion at a prescribed angle. The inclined surface 1670 has a prescribed angle ε1, and the retro-reflective component 330 is arranged at a prescribed angle ε2 relative to the inclined surface 1670.

[0267] Fig.24 In the structure of (A), Fig.13 In the same manner as in the case of the image display device 10, the image light from the image display device 10 is incident on the retro-reflective component 330, so that the retro-reflected light is emitted in the direction of the opening of the inclined surface 1670, which is the opposite side of the image display device 10. As a result, a real image of the spatial suspended image 3 is generated as a multi-viewpoint image at a predetermined position outside the opening of the spatial suspended image display unit 1620 of the inclined surface 1670. The spatial suspended image 3 is tilted at a predetermined angle ε2 relative to the inclined surface 1670. With respect to the spatial suspended image 3, the user can appropriately view the spatial suspended image 3 from the obliquely upper viewpoint E with an obliquely lower line of sight.

[0268] In addition, Fig.24 In the structure of (A), a first distance measuring device 340 (including a TOF sensor, etc.) is provided at a predetermined position in the upper part of the housing 1650, in this example, a position near the upper end of the retro-reflective component 330. The optical axis of the first distance measuring device 340 is oriented toward the direction of the spatial suspended image 3, and the viewing angle is set to a size that can cover the entire spatial suspended image 3. The operation of the user's finger UH on the spatial suspended image 3 of the spatial suspended image display unit 1620 is sensed by the sensing system including the first distance measuring device 340 installed in the self-service terminal 1600.

[0269] exist Fig.23 , Fig.24In the second embodiment shown, the image of the virtual assistant 1621 as the spatial floating image 3 and the operation menu 1622 as the spatial floating image 3 are both displayed as multi-viewpoint images with motion parallax in one spatial floating image display unit 1620. In this case, only the virtual assistant 1621 may be displayed as a multi-viewpoint stereoscopic image, and the operation menu 1622 may be displayed as a flat image. In this case, even in a state where the lenticular lens 1103 is arranged, the spatial floating image 3 can be displayed as a multi-viewpoint stereoscopic image or as a flat image simply by switching the image displayed on the image display device 10 to be a multi-viewpoint stereoscopic image or a flat image.

[0270] The display method of the space floating image in the self-service terminal is not limited to this. As a variation, it is also possible to display the space floating image in one screen of the space floating image display unit 1620 instead of Fig.23 Instead of performing the display of the virtual assistant and the display of the operation menu simultaneously as shown, the display of the virtual assistant and the display of the operation menu are switched appropriately.

[0271] <Use scenarios of self-service terminals>

[0272] Two embodiments of the self-service terminal have been described above. Next, actual usage scenarios of the self-service terminal by actual users will be described. Fig.25 This is a diagram showing a scene in which one user uses a kiosk terminal as the spatial floating image information display system according to the first embodiment. Fig.25 In the figure, on the self-service terminal 1500, a virtual assistant (a female face) is displayed as a multi-viewpoint image suspended in space.

[0273] Here, regarding the virtual assistant as a multi-viewpoint image suspended in space, if the user located near the self-service terminal 1500 is within the range where the virtual assistant can be seen, the virtual assistant can be seen always looking in the direction of the user (the user) regardless of the user's position. More specifically, if Fig.25 As shown, when the user stands on the oblique left side of the self-service terminal, the user can see the virtual assistant facing the direction where the user stands, that is, the oblique left side. On the other hand, when the user stands on the oblique right side of the self-service terminal, the user can see the virtual assistant facing the direction where the user stands, that is, the oblique right side.

[0274] As described above, in order to display the virtual assistant as a floating image in space, it is possible to use Fig.18As a result, the images of the virtual assistant from cameras No. 1 to No. 9 are sequentially displayed as a multi-viewpoint spatial floating image with 9 viewpoints and motion parallax, starting from the right side when the user is facing the kiosk terminal 1500, that is, the spatial floating image.

[0275] In addition, when the user moves from the right to the left facing the self-service terminal 1500, the user can recognize that Fig.15 The motion parallax unique to the multi-viewpoint image shown can identify the virtual assistant as a spatially suspended image as a stereoscopic image (or stereoscopic video). As a result, compared to simply displaying the spatially suspended image as a two-dimensional image, the user can see the virtual assistant through a stereoscopic multi-viewpoint image with motion parallax that looks like a real person. Furthermore, the user can obtain a new effect that has not been seen before: when moving near the self-service terminal 1500, the user can get the feeling that the virtual assistant always speaks in the direction of the user (the user) with his / her eyes.

[0276] <About implementation method 3 of self-service terminals>

[0277] Fig.26 An example of display of a spatial floating image 3 in a kiosk terminal as one embodiment (Implementation 3) is shown. Fig.26 The self-service terminals are Fig.21 Take the structure of the self-service terminal 1500 as an example. Fig.26 The self-service terminal is a self-service terminal that uses the space floating image 3 formed by the space floating image information display system 1 as a user interface. In addition, Fig. 27 , Fig.28 Indicates user operation Fig.26 The action flow in the case of a self-service terminal.

[0278] Fig. 27 express Fig.26 The first action flow when the self-service terminal is started. The first action flow represents basic actions. In step S100, when the process starts, the self-service terminal is in a standby state (in other words, a dormant state). In step S101, the self-service terminal detects whether the user is close to the self-service terminal. A user who wants to use the self-service terminal will approach the self-service terminal, or the user will stand at a specified position near the self-service terminal. When the user approaches the self-service terminal, based on the shooting of cameras 1541 and 1542, it is detected that the user is close to the self-service terminal (especially the front of the inclined surface 1570). Alternatively, instead of the camera, a Fig.21The human body sensor 1560 using infrared rays, etc., detects that the user is close to the self-service terminal 1500. When it is detected that the user is close to the self-service terminal ("Yes"), in step S102, the self-service terminal is transferred to the startup state. Thereafter, in the startup state, the user can use the self-service terminal.

[0279] In step S103, the self-service terminal displays a virtual assistant 1521 or an operation menu 1522 on the spatial floating image display unit 1520, provides guidance to the user on the operation method, etc., and accepts the operation (in other words, input) on the operation menu 1522, and performs a prescribed process (application or service process) corresponding to the detected operation. Fig.26 As shown in (A), a virtual assistant 1521 is displayed on the space floating image display unit 1520, and the user is greeted and guided through the image and sound of the virtual assistant 1521. Then, when the greeting is finished, as shown in FIG. Fig.26 As shown in (B), the display of the spatial floating image display unit 1520 automatically shifts from the virtual assistant 1521 to the operation menu 1522. The user operates the operation menu 1522 with a finger. The self-service terminal detects the operation using sensing technology and performs processing corresponding to the detected operation.

[0280] In addition, the image of the virtual assistant 1521 is a spatial floating image based on a multi-viewpoint image with motion parallax as described above. The multi-viewpoint image may be a dynamic image obtained by shooting a real person from multiple viewpoints, i.e., multiple cameras corresponding to the number of viewpoints, or a CG character image generated by computer rendering, or an animation representing a person's image. In addition, it is not limited to humans, and may be replaced by animals or fictional characters.

[0281] In step S104, it is detected whether the user has left the self-service terminal. If the user has left the self-service terminal, it is detected that the user has left the self-service terminal (especially the front inclined surface 1570) based on the shooting of the cameras 1541 and 1542. Alternatively, it is also possible to use a camera instead of a camera. Fig.21 The human body sensor 1560 and the like shown detect that the user has left the self-service terminal. When it is detected that the user has left the self-service terminal ("Yes"), in step S105, the self-service terminal shifts to a standby state.

[0282] Fig.28 express Fig.26 The second action flow when the self-service terminal is started in the implementation mode 3 of the present invention. The second action flow represents a more detailed embodiment than the first action flow. The second action flow is an action after the self-service terminal becomes the started state, and represents an action flow when the user authentication is performed.

[0283] At step S200, at the beginning of the process, the self-service terminal is in a standby state. The user approaches the front of the self-service terminal, or stands at a predetermined position near the self-service terminal. At step S201, the self-service terminal detects whether the user is close to the self-service terminal (especially the front of the inclined surface 1570) by performing stereoscopic shooting with two cameras. If the user is detected ("yes"), at step S202, the self-service terminal displays the space floating image display unit 1520 as shown in FIG. Fig.26 As shown in (A), the virtual assistant 1521 is first displayed, and the user is greeted by the image and sound of the virtual assistant 1521, such as "Welcome, thank you for using me.", and a greeting message may also be displayed.

[0284] Next, in step S203, the self-service terminal authenticates the detected user. At this time, the self-service terminal also displays a message indicating that the authentication is in progress, such as "authenticating" as a spatial floating image 3. As an authentication, the self-service terminal can perform facial authentication of a person approaching (i.e., the detected user) based on the images (especially facial images) captured by the two cameras 1551 and 1552, thereby identifying / determining the user. Alternatively, authentication based on the iris image of the user's eyes can also be performed. The authentication here refers to using two cameras to capture the face or iris of the user within a specified range in front of the front of the self-service terminal, and based on the stereoscopic image captured, determining whether the user is a legitimate user of the self-service terminal who has been pre-registered in the system.

[0285] Here, authentication based on the user image can also be performed by shooting with one camera, but by using the stereoscopic images of the two cameras 1551 and 1552, the accuracy is higher than that of authentication using images from one camera, and user authentication that is difficult to be used illegally can be performed. For example, it is possible to detect illegal authentication by printing an image of the face shot only from the front on paper and using the printed face image.

[0286] Not limited to the above-mentioned face recognition, the following method can also be applied as another method of user authentication. User authentication can be performed by having the self-service terminal read the information of the user's personal identification card, driver's license, etc. held by the user. In addition, user authentication can also be performed by transmitting user information stored in the portable terminal held by the user through short-range wireless communication between the portable terminal and the self-service terminal.

[0287] In addition, the following method can also be applied. Code information such as a QR code (registered trademark) that can identify the user's personal information (such as name, address, date of birth, and history of use of the self-service terminal) is displayed on a portable terminal such as a smartphone held by the user, and the code information is presented to the spatial floating image display unit 1520 of the self-service terminal (for example, presented to the cameras 1551 and 1552). Then, the user information presented above is read by the cameras 1551 and 1552 of the self-service terminal to perform user authentication.

[0288] In step S204, the self-service terminal confirms whether the above user authentication is successful, and proceeds to step S206 if successful, and proceeds to step S205 if failed. In step S205, the self-service terminal recommends other authentication methods, and returns to step S203 to try to authenticate with other authentication methods. After the above user authentication process, the self-service terminal can present a prescribed operation menu screen (an operation menu corresponding to an uncertain majority of users) on the spatial floating image display unit 1520, or can also present the menu screen that is optimal for each user determined by authentication (an operation menu for individuals).

[0289] In a specific example, after successful authentication, Fig.26 As shown in (A) and (B), the display of the spatial floating image display unit 1520 is changed / transferred from the virtual assistant 1521 to the operation menu 1522. In the operation menu 1522 of (B), a message (text image) such as "Please touch to select an item" is displayed as a content example, and buttons such as "1 Issuance of residence certificate", "2 Ticket purchase", "3 Registration information", and "4 Other menus" are displayed as options and items. "1 Issuance of residence certificate" is an item about administrative procedures such as issuance of residence certificates. "2 Ticket purchase" is an item about purchasing and receiving tickets for concerts, movies, or trams. "3 Registration information" is an item about confirming and changing registration information about the user. "4 Other menus" are items for transferring to other menus. The user presses the button of the "2 Ticket purchase" item according to his purpose. The self-service terminal transfers to the display of the next operation menu accordingly according to the button pressed. A more detailed example is shown below.

[0290] [Operational guidance by virtual assistant]

[0291] exist Fig.28 In step S206, the space floating image information display system 1 of the self-service terminal communicates with the communication unit 1132 ( Figure 2) accesses the self-service server on the communication network and notifies the user of the authentication result information. In addition, by obtaining / referring to the user's attribute information from the user database in the self-service server, it is determined whether the user has used the self-service terminal in the past. Alternatively, the self-service terminal can also obtain / refer to the user's attribute information based on the code information from the portable terminal of the user. The user's attribute information includes, for example, the user's gender, age, and past use history of the self-service terminal.

[0292] In addition, in step S206, the self-service terminal may also obtain user information including user attribute information from the user's portable terminal through wireless communication. In addition, the self-service terminal may not only obtain user attribute information, but also obtain other specified data about the user and the operation menu, such as ticket booking information, which has been previously input / registered by the user, from an external device such as a self-service server. In addition, in step S206, the self-service terminal may also determine the user's age and other attributes based on the images of the cameras 1541 and 1542. The user attribute information obtained in step S206 is used in the determination of the next step S207.

[0293] In the next step S207, the self-service terminal determines whether the user is a "first-time user" or "elderly user" using the self-service terminal based on the user's attribute information obtained in step S206. If it is determined that the user meets such specified conditions ("Yes"), the process proceeds to step S208, and if it is determined that the user does not meet the conditions ("No"), the process proceeds to step S209.

[0294] The system branches the response accordingly according to the judgment result of step S207. There are roughly two responses here. One is the first processing flow (the flow starting from step S209) assuming that the user is an accustomed user, and the other is the second processing flow (the flow starting from step S208) assuming that the user is an unaccustomed user (a first-time user or an elderly user). In the first processing flow, the self-service terminal omits the operation guidance provided by the virtual assistant, or replaces it with simplified operation guidance compared to the second processing flow. In the second processing flow, the self-service terminal provides detailed operation guidance through the virtual assistant.

[0295] In step S208, the self-service terminal performs "operation guidance with voice given by a virtual assistant" in the space floating image display unit 1520. Here, "virtual assistant" refers to a "guide", for example Fig.26As shown in (A), a virtual assistant 1521 (schematically shown in the figure) is displayed as a space floating image 3 based on a multi-viewpoint image (or image) with motion parallax, for example, as an image of the upper body of a person. For users who use the space floating image 3 as a user interface to operate the self-service terminal for the first time, or elderly users who are not accustomed to the operation, the virtual assistant 1521 provides guidance on the operation method, etc. to the user with voice and action. The virtual assistant 1521 explains and guides the user on the part that the user does not understand about the operation of the operation menu 1522 such as (B).

[0296] More specifically, in step S208, first Fig.26 As shown in (A), the image of the virtual assistant 1521 appears on the screen of the space floating image display unit 1520. The virtual assistant 1521 outputs a text image and voice such as "I will guide you on the operation method. Please follow my instructions to operate the menu." The voice is, for example, Fig.26 The super-directional loudspeakers 1551 and 1552 of (A) are provided in a manner that can be heard only by the user. The virtual assistant 1521 provides operation guidance by successively informing the user of how to use the kiosk terminal and the operation menu.

[0297] Next, in either step S208 or step S209, Fig.26 As shown in (A) and (B) of FIG. 200 , the image of the virtual assistant 1521 as the spatial floating image 3 is switched to the image of the operation menu 1522 as the spatial floating image 3. Here, in step S208, after the screen of the spatial floating image 3 is switched to the operation menu 1522, the guidance by the virtual assistant 1521 is also continued. The self-service terminal appropriately switches the display of the virtual assistant 1521 and the display of the operation menu 1522 in the spatial floating image display unit 1520. The user operates the operation menu 1522 on the screen of the spatial floating image display unit 1520 according to the operation guidance of the virtual assistant 1521.

[0298] Next, in step S210, the user performs a series of prescribed operations on the operation menu 1522 of the self-service terminal, such as the authentication password input operation for ticket issuance. The self-service terminal detects the operation and processes it to issue the ticket. The user takes the ticket from the take-out exit 1530. When the operation requested by the user and the corresponding action of the self-service terminal, such as the ticket issuance action, are completed ("Yes"), the process moves to step S212.

[0299] In step S212, the self-service terminal displays the operation menu 1622 again (e.g., the initial menu screen, or the next menu screen if there is a next menu, or the last menu screen, etc.) on the screen of the spatial floating image display unit 1520. Alternatively, the self-service terminal ends the display in the spatial floating image display unit 1520. In addition, when the display ends, the self-service terminal may also display the virtual assistant 1521 on the screen at the end, and output an ending message (such as "Thank you for using") to the user.

[0300] On the other hand, in step S209, the self-service terminal assumes that the user is not operating for the first time but is accustomed to the operation to a certain extent, and displays the information on this basis. The self-service terminal displays the screen of the space floating image display unit 1520 as follows: Fig.26 The operation menu 1522 is switched as shown in (B). After that, the user performs the required operation such as selecting an item according to the operation menu 1622 of the screen. In step S211, similar to step S210, when a series of prescribed operations and corresponding actions (such as ticket issuance) are completed ("Yes"), the process moves to step S212.

[0301] Here, in the case of proceeding to step S209, step S212 is also set in consideration of the possibility that the user's operation is not smooth. In the case that the user's prescribed series of operations in step S211 are not completed ("No"), proceed to step S212. In step S212, the self-service terminal determines whether the user's operation in the operation menu 1522 is smooth. In a specific example, the self-service terminal determines whether the operation performed by the user in the operation menu 1522 has stopped (in other words, there is no input) for a longer time than a prescribed time (for example, 30 seconds). There is a possibility that the user stops the operation when he does not understand the operation of the operation menu 1522 as the spatial floating image 3. The self-service terminal uses a camera and sensing technology to detect the long-term operation stop. In the case of judging that the long-term operation stops, etc. ("Yes"), proceed to step S206, and in the case that it is not the case ("No"), proceed to step S209.

[0302] In another example, in step S212, the self-service terminal may also determine whether the user has made an erroneous operation on the surface (operation menu 1522) of the space floating image 3. For example, it may be determined that the finger is trying to touch a position away from an item button.

[0303] When the process proceeds from step S212 to step S208, the self-service terminal displays a virtual assistant 1521 on the space floating image display unit 1520 for the user, and provides operation guidance through the virtual assistant 1521. At this time, as the operation guidance provided by the virtual assistant 1521, the self-service terminal may replay the operation guidance based on the predetermined content, but it is more preferable to provide the operation guidance on the part of the operation menu 1522 that the user stopped operating in step S212. For example, in Fig.26 When the user stops operating in the operation menu after the "2. Buy Tickets" button in (B) is pressed, the self-service terminal guides the user through the virtual assistant 1521 on how to operate next in the operation menu.

[0304] In addition, the above-mentioned operation example can also be applied to Fig.23 In the embodiment, a configuration example is provided in which both the virtual assistant 1621 and the operation menu 1622 are displayed side by side in a screen of a spatial floating image 3. In this case, the virtual assistant 1621 can point to the operation menu 1622 with a fingertip in the screen to provide operation guidance.

[0305] According to Embodiment 3, the self-service terminal displays the virtual assistant 1521 as a spatial floating image 3 when the user approaches the self-service terminal, and starts the operation guidance by the virtual assistant 1521. The image of the virtual assistant 1521 is a spatial floating image based on a multi-viewpoint image (or image) with motion parallax, so the user can see the image of the virtual assistant 1521 in a stereoscopic image. Therefore, the user can feel the image of the virtual assistant 1521 as if there is a real person, and even users who operate the self-service terminal for the first time and elderly users who are not accustomed to the operation can reliably perform the prescribed operation according to the operation guidance of the virtual assistant 1521 and the sound of the super-directional speaker (operation guidance that cannot be heard by others).

[0306] According to Embodiment 3, users are identified through authentication, and based on the user's age and system usage history, the operation method is carefully explained to users who are operating for the first time, users who are not used to the operation, and elderly users through a virtual assistant. As a result, such users can also perform reliable key input operations on the operation menu as a floating image in space. For accustomed users, efficient services can also be provided by omitting or simplifying the operation guidance.

[0307] Modifications of the third embodiment and the like may be as follows. Fig.31The following is a display example of this variation. The self-service terminal displays the image of the operation menu 1522 as large as possible within one screen of the spatial floating image 3 formed by the spatial floating image display unit 1520. The self-service terminal superimposes and displays the image of the virtual assistant 1521 on the operation menu 1522 in a relatively small size. Then, the self-service terminal guides the operation of the operation menu 1522 through the virtual assistant 1521. The virtual assistant 1521 preferably moves in coordination with actions, gestures, mouth and eye movements, and is configured as a spatial floating image based on a multi-viewpoint image with motion parallax. The self-service terminal may also appropriately change whether the virtual assistant 1521 is displayed, the display position, etc. within the screen of the operation menu 1522.

[0308] <About implementation method 4 of self-service terminals>

[0309] exist Fig.29 In the embodiment, as another embodiment (referred to as Implementation 4), an embodiment in which a self-service terminal and a portable terminal held by a user perform cooperative actions is shown. Fig.29 The kiosk terminal 1700 is a kiosk terminal that uses the space floating image 3 formed by the space floating image information display system 1 as a user interface. When the user 1710 operates the kiosk terminal 1700, the kiosk terminal 1700 cooperates with a portable terminal 1740 such as a smartphone held by the user 1710. Fig.29 The self-service terminal 1700 Fig.23 The structure of the self-service terminal 1600 is based on.

[0310] As an example, the case where a user 1710 uses a self-service terminal 1700 to purchase a ticket (e.g., a ticket for a concert) is described. Fig. 27 , Fig.28 Similarly, when the user 1710 approaches the self-service terminal 1700, the self-service terminal 1700 authenticates the user 1710 and displays the virtual assistant 1721 and the operation menu 1722 on the space floating image display unit 1720. The user operates the operation menu 1722 according to the instructions of the virtual assistant 1721. Fig.26 The user 1720 selects "2 Purchase Ticket" in the operation menu 1522 shown in (B). The self-service terminal displays the operation menu 1722 transferred from "2 Purchase Ticket" in the screen of the space floating image display unit 1720. The user 1720 performs detailed operations for purchasing tickets through the operation menu 1722.

[0311] The ticket is issued in the form of, for example, printed on paper, and the user 1710 receives the paper ticket. Alternatively, as a method of obtaining the ticket, it is also possible to receive it in the form of a so-called electronic ticket (or "e-Ticket") instead of a ticket printed on paper. In this case, the user 1710 uses the portable terminal 1740 held by the user 1710 to receive the same information as the ticket printed on paper (i.e., the electronic ticket) instead of the ticket printed on paper. In this example, when operating through the operation menu 1722, short-range wireless communication is performed between the portable terminal 1740 and the self-service terminal 1700, for example, using a communication interface such as Bluetooth (registered trademark). The above-mentioned electronic ticket is sent directly from the self-service terminal 1700 to the user's portable terminal 1740. At this time, the self-service terminal guides the operation of the communication through the virtual assistant 1721.

[0312] As another example, Fig.29 As shown, the self-service terminal 1700 displays code information 1760 (in other words, a code image) such as a barcode or a QR code (registered trademark) including the information content of the above-mentioned electronic ticket on the screen of the space floating image display unit 1720 as a space floating image 3 (for example, a part of the operation menu 1722). The user 1710 uses his own portable terminal 1740 to read the code information. For example, the user 1710 operates according to the guidance of the virtual assistant 1721 and uses the reader function of the camera of the portable terminal 1740 to read the code information 1760 displayed in the operation menu 1722. The virtual assistant 1721 guides the reading operation. The portable terminal 1740 obtains the electronic ticket based on the read code information 1760 and stores it in the memory inside the portable terminal 1740.

[0313] The items and information that the user 1710 can receive from the self-service terminal 1700 are not limited to paper tickets and electronic tickets. For example, they can also be coupons that can be used for shopping, information on points, and electronic books. Tickets and coupons printed on paper may be lost, but if they are electronic tickets and coupons, they can be kept inside the portable terminal 1740, so the possibility of loss can be reduced. In addition, not only can the possibility of loss be reduced, but by keeping the electronic tickets and coupons inside the portable terminal 1740, compared with tickets and coupons printed on paper, when the tickets are actually used, the following usage methods can be achieved: presenting the portable terminal, or having the ticket issuer read the electronic tickets held in the portable terminal, etc. As a result, the ease of use for users is improved.

[0314] The self-service terminal can send information associated with the operation menu (such as access history information, operation history information, etc.) to the user's portable terminal through wireless communication according to the user's operation on the operation menu of the spatial floating image 3.

[0315] <About implementation method 5 of self-service terminal>

[0316] exist Fig.30 As an embodiment (referred to as embodiment 5), a function in which a user can input / draw any text or graphic on the above-mentioned space floating image 3 is shown ( Fig.12 ) is a detailed embodiment of the present invention. For example, by Fig. 27 , 28 The flowchart shown in the figure illustrates a series of steps. When a user receives a paper ticket or an electronic ticket from a self-service terminal, the user "signs" (signs his / her name) to indicate that the user acknowledges that he / she has received the ticket.

[0317] Fig.30 The self-service terminal 2000 is based on Fig.21 , Fig.23 This is an example based on the structure of a self-service terminal, and only the spatial floating image display unit 2020 is illustrated. Fig.30 (A) shows a state where the virtual assistant 2021 is displayed as a spatial floating image 3 of a multi-viewpoint image on the spatial floating image display unit 2020. (B) shows a state where a user interface for inputting a signature is displayed as a spatial floating image 3. (C) shows a state where a signature is input and drawn by the user in the user interface. (D) shows a state where the virtual assistant 2021 corresponding to the completion of signature input is displayed as a spatial floating image 3.

[0318] After the self-service terminal issues the ticket and the user receives the ticket, the self-service terminal Fig.30 As shown in (A), a virtual assistant 2021 as a space floating image 3 is displayed on the space floating image display unit 2020. The virtual assistant 2021 outputs an image and sound of a message such as "Finally, please sign to confirm that you have received the ticket" to the user. Thus, the user is reminded to sign (in other words, sign for) the space floating image 3.

[0319] Next, the self-service terminal clears the display of the image of the virtual assistant 2021, such as Fig.30 As shown in (B) of FIG. 1 , a signature screen 2022 (in other words, a user interface for inputting a signature) is displayed on the space floating image display unit 2020 as the space floating image 3 .

[0320] exist Fig.30In the example (B), as the text drawn by the space floating image 3, the text "Please sign for the ticket" is displayed on the signature screen 2022. Below the text, a frame line (signature frame line area) 2023 indicating the area for signature is displayed. The frame line 2021 is, for example, based on a predetermined background color (e.g., black) and the frame lines on the four sides are displayed in a predetermined color (e.g., green). When the fingertip is not inserted into the frame line 2021, the frame line 2021 is displayed in the predetermined color (e.g., green).

[0321] Then, if Fig.30 As shown in (C), the user uses the tip of his own finger UH or a predetermined conductive pen to operate within the frame line 2023 to draw his own name, in this example, "Tom" is drawn in handwriting as a signature. At this time, the self-service terminal and the above ( Fig.12 ) Similarly, the mid-air operation on the surface of the spatial floating image 3 is detected using the sensing technology, and a line segment is drawn in real time within the surface of the spatial floating image 3 (within the frame line 2023) according to the detected contact point position of the fingertip. The line segment 2024 is a line drawn in one stroke corresponding to the input / drawn signature ("Tom"). The line segment 2024 is displayed in a predetermined color.

[0322] When the user finishes drawing his or her name as a signature, the user moves the fingertip away from the signature frame area 2023 of the space floating image 3. When the self-service terminal determines / detects that a predetermined time (e.g., 5 seconds) has passed since the fingertip left, the user is deemed to have completed inputting the signature. Fig.30 As shown in (D), the screen 2025 indicating the completion of signature input is transferred. In the screen 2025, for example, a text such as "Thank you for signing" and an output voice are displayed together with the image of the virtual assistant 2021. The above series of operations in which the user "signs" to acknowledge that the user has received the ticket is completed.

[0323] in addition, Fig.30 The technique shown in (C), that is, the technique in which the user uses his fingertips or the like to fill in his name as a signature within the frame 2023 of the space floating image 3, can also be applied in the same manner as the above Fig.12 In addition, Fig.12 In the example described in the description of , when the user inserts his fingertip into the deep side relative to any position on the surface of the space floating image 3, the frame of the entire space floating image 3 changes to red. Fig.30In the example of (C), not the entire frame of the spatial floating image 3 but only a part of the region (signature frame region 2023) within the spatial floating image 3 is used as the region for inputting and drawing the signature. Furthermore, when the self-service terminal detects that the fingertip is inserted into (in other words, touches) the region, the frame region 2023 can be changed to, for example, red.

[0324] In addition, when the user inserts his fingertip into any position on the surface of the space floating image 3, that is, when in contact, the frame line 2023 is not limited to red, and can also be displayed as other specified colors such as blue. In addition, the frame line 2023 can also be flashed in the state of contact. Furthermore, in the state of contact, the background area in the frame line 2023 can also be changed to white, and the color of the line segment written in the background area can be changed to other specified colors such as black. In addition, the background area in the frame line 2023 can also be changed to black, and the color of the line segment written in the background area can be changed to white. That is, the self-service terminal controls the display in the signature frame line area 2023 so that the line segments of the text and graphics drawn according to the action of the user's fingertip in the signature frame line area 2023 are easy for the user to recognize. The self-service terminal controls to change the display mode in the signature frame line area 2023 accordingly according to the state such as whether the fingertip is in contact with the signature frame line area 2023.

[0325] For existing space floating image display devices, the main operation is for the user to select from the buttons of the options in the operation menu formed by the space floating image. In contrast, implementation mode 5 provides a function (air signature function) in which the user can move his finger on the space floating image 3 to input any text as a signature. In addition, in this function, the display of the frame line 2023 and the like changes accordingly depending on whether the fingertip is in contact with the surface of the space floating image 3, so the user can easily understand whether he is in a state where the fingertip is in contact with the surface of the space floating image 3, and it is easy to input the signature. The above example shows an example of using the input of the space floating image 3 for signature, but is not limited to this and can also be used for other applications.

[0326] <First Configuration Example of Image Display Device>

[0327] Next, use Figure 32 to Figure 34 The following figures illustrate a detailed configuration example of the spatial floating image information display system 1. Fig.32 An example of a more specific configuration of the image display device 10 constituting the spatial floating image information display system 1 is shown. Fig.32The light source device 13 is composed of, for example, a housing made of plastic or the like, in which an LED, a collimator, a polarization conversion element, a synthetic diffusion block, a light guide, and the like are housed. A liquid crystal display panel 11 is mounted on the upper surface of the light source device 13. In addition, an LED circuit board 102 is mounted on one side of the housing of the light source device 13, on which an LED element 102A as a semiconductor light source and a control circuit for the LED element are mounted. Furthermore, a heat sink (not shown) is mounted on the outer side of the LED circuit board 102 as a component for cooling the heat generated in the LED element 102A and the control circuit.

[0328] In addition, a liquid crystal display panel frame mounted on the upper surface of the housing is provided with a liquid crystal display panel 11 mounted on the frame, and a flexible printed circuit (FPC) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11 as a liquid crystal display element and the LED element 102A as a solid light source modulate the intensity of the transmitted light based on a control signal from a control circuit constituting the electronic device to generate a display image.

[0329] <First Configuration Example of Light Source Device>

[0330] Next, regarding the configuration example of the optical system such as the light source device housed in the housing in the first configuration example of the image display device, Fig.32 Refer to Fig.33 The details are explained below. Fig.32 In the embodiment, LED 102A constituting the light source is installed at a predetermined position relative to the collimator 15. Fig.32 1 shows a cross section, so only one LED element and collimator can be seen, but multiple LED elements are arranged on the LED circuit board 102, and multiple collimators are arranged correspondingly. The collimators 15 are each formed of a light-transmitting resin such as acrylic resin. The collimator 15 has an outer peripheral surface 156 of a conical convex shape obtained by rotating a parabolic cross section, and has a concave portion 153 with a convex portion (i.e., a convex lens surface) 157 formed in the center of the top (the side in contact with the LED circuit board 102).

[0331] In addition, a convex lens surface 154 protruding outward (or a concave lens surface recessed inward) is provided at the center of the flat surface portion (opposite to the top) of the collimator 15. In addition, the parabola 156 forming the conical outer peripheral surface of the collimator 15 is set within an angle range that allows the light emitted from the LED element 102A in the peripheral direction to be totally reflected inside it, or forms a reflecting surface.

[0332] In addition, the LED element 102A ( Fig.33In (A), the LED elements 14a and 14b are respectively arranged at predetermined positions on the surface of the LED circuit board 102. The LED circuit board 102 is arranged and fixed relative to the collimator 15 so that the LED elements 102A on the surface are respectively located in the center of the recessed portion 153.

[0333] According to this structure, the light emitted from the LED 102A, especially the light emitted from the central part to the upper side (to the right in the figure), is focused by the two convex lens surfaces 157 and 154 forming the outer shape of the collimator 15 to become approximately parallel light. In addition, the light emitted from other parts to the peripheral direction is reflected by the parabola forming the conical outer surface of the collimator 15, and is also focused to become approximately parallel light. In other words, by using the collimator 15 having a convex lens in the central part and a parabola formed in the peripheral part, almost all of the light generated by the LED element 102A can be made into parallel light and exported. As a result, the utilization efficiency of the generated light can be improved.

[0334] In addition, a polarization conversion element 21 (in other words, a polarization conversion component) is provided on the light exit side of the collimator 15. The polarization conversion element 21 is an element that converts the polarization characteristics of the incident light. Fig.33 As shown in (A), a plurality of light-transmitting components are arranged in parallel and in an array on a plane orthogonal to the optical axis of the parallel light from the collimator 15. Furthermore, polarization beam splitters (PBS films) 211 and reflection films 212 are alternately arranged on the interface between adjacent light-transmitting components arranged in the array. In addition, a λ / 2 phase plate 213 (in other words, a half-wave plate) is provided on the exit surface from which the light incident on the polarization conversion element 21 and transmitted through the PBS film 211 is emitted.

[0335] On the exit surface of the polarization conversion element 21, Fig.33 As shown in (A), a rectangular synthetic diffusion block 16 is also provided. That is, the light emitted from the LED 102A becomes parallel light under the action of the collimator 15, and after the polarization characteristics are transformed by the polarization conversion element 21, it enters the synthetic diffusion block 16, is diffused by the texture 161 on the emission side, and then reaches the light guide 17.

[0336] The light guide 17 is Fig.33 As shown in (B), the light guide 17 is a rod-shaped member having a substantially triangular cross section, for example, formed of a light-transmitting resin such as acrylic resin. Fig.32As shown, it includes a light guide light incident portion 171, a light guide light reflecting portion 172 and a light guide light emitting portion 173, wherein the light guide light incident portion 171 includes an incident surface opposite to the exit surface of the synthetic diffusion block 16 via the first diffusion plate 18a, the light guide light reflecting portion 172 includes a reflecting surface formed as an inclined surface, and the light guide light emitting portion 173 includes an exit surface opposite to the liquid crystal display panel 11 via the second diffusion plate 18b.

[0337] In the light guide light reflecting portion 172 of the light guide 17, as shown in (B) of FIG. 33, a plurality of reflecting surfaces 172a and connecting surfaces 172b are formed alternately in a zigzag shape. The reflecting surface 172a (the line segment rising to the right in the figure) forms an angle of αn (n is a natural number, 1 to 130 in this example) with respect to the horizontal plane. As an example, αn is set to be less than 43 degrees (more than 0 degrees).

[0338] The light guide incident portion 171 is formed into a curved convex shape inclined toward the light source side. As a result, parallel light from the exit surface of the synthetic diffusion block 16 is diffusely incident via the first diffusion plate 18a. As shown in the figure, the incident light is slightly bent and deflected upward by the light guide incident portion 171 and reaches the light guide light reflection portion 172 and is reflected there. The reflected light is emitted from the exit surface of the light guide exit portion 173 at the top of the figure and reaches the liquid crystal display panel 11 arranged relative to the exit surface.

[0339] According to the above image display device 10, the light utilization efficiency and uniform illumination characteristics can be further improved, and the modular S-polarized light source device 13 can be manufactured in a small size and at a low cost in the mainland. In addition, in the above description, the polarization conversion element 21 is installed after the collimator 15, but it is not limited to this. The polarization conversion element 21 can be set in the light path reaching the liquid crystal display panel 11.

[0340] In addition, a plurality of reflection surfaces 172a and connection surfaces 172b are alternately formed in a zigzag shape on the light guide light reflection portion 172, and the illumination light beam is totally reflected on each reflection surface 172a and goes upward. Furthermore, a narrow-angle diffusion plate is provided on the light guide light exit portion 173 to make it a substantially parallel diffused light beam, and it is incident on the light direction change panel 54 for adjusting the directional characteristics, and is incident on the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction change panel 54 is provided between the exit surface of the light guide 17 and the liquid crystal display panel 11, but even if the light direction change panel 54 is provided on the exit surface of the liquid crystal display panel 11, the same effect can be obtained.

[0341] <Second Configuration Example of Image Display Device>

[0342] Next, use Fig.34 Another example of the specific structure of the image display device 10 will be described. Fig.34 The light source device of the image display device 10 is the same as the above example, and a plurality of LED elements constituting a light source are arranged on the LED circuit board 102 (however, only one is shown in the figure because it is a cross-sectional view). These LED elements are installed at a predetermined position relative to the collimator 15. The divergent light beam from the LED element (light in which P polarized light and S polarized light are mixed) is converted into a substantially parallel light beam by the collimator 15, and is reflected toward the liquid crystal display panel 11 by the reflective surface of the reflective light guide 304. The reflected light is incident on the reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304.

[0343] The reflective polarizer 49 transmits a specific polarization (e.g., P polarized light) and makes it incident on the liquid crystal display panel 11. The reflective polarizer 49 reflects another polarization (e.g., S polarized light) and makes it go to the reflective light guide 304 again. The reflective polarizer 49 is tilted so as not to be perpendicular to the principal ray of light from the reflection surface of the reflective light guide 304, and the principal ray of light reflected on the reflective polarizer 49 is incident on the transmission surface of the reflective light guide 304.

[0344] The light incident on the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 as a phase difference plate, and is reflected by the reflection plate 271. The light reflected by the reflection plate 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that passes through the transmission surface of the reflective light guide 304 enters the reflective polarizer 49 again.

[0345] At this time, the light incident on the reflective polarizer 49 again passes through the λ / 4 plate 270 twice, and is thus polarized into a polarization (e.g., P polarized light) that can pass through the reflective polarizer 49. Therefore, the light after polarization conversion passes through the reflective polarizer 49 and enters the liquid crystal display panel 11. In addition, regarding the polarization design of polarization conversion, a structure in which the polarization of S polarized light and P polarized light is switched as compared with the above description may be adopted.

[0346] As a result, light from the LED element, which is a light source, is unified into specific polarization (for example, P-polarized light) and enters the liquid crystal display panel 11, and is brightness modulated according to the image signal, so that an image is displayed on the panel surface.

[0347] Fig.34 The collimators 15 in the figure are each formed of, for example, a light-transmitting resin such as acrylic resin or glass. Fig.34 The collimator 15 in the Fig.32Similarly, the collimator 15 in FIG. 1 has a conical convex outer peripheral surface obtained by rotating a parabola section, and may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center at the top. In addition, in the center of the planar portion of the collimator 15, there is a convex lens surface protruding outward (or a concave lens surface recessed inward). The parabola forming the conical outer peripheral surface of the collimator 15 is set within an angle range that allows the light emitted from the LED to the peripheral direction to be totally reflected inside it, or a reflecting surface is formed.

[0348] Fig.34 The LED elements are arranged at predetermined positions on the surface of the LED circuit board 102. The LEDs on the surface of the LED circuit board 102 are arranged and fixed relative to the collimator 15 so as to be located at the center of the top of the conical convex shape (the concave portion if the top has a concave portion).

[0349] According to this structure, the light emitted from the LED element, especially the light emitted from the central part, is focused by the convex lens surface forming the outer shape of the collimator 15 to become parallel light. In addition, the light emitted from other parts in the peripheral direction is reflected by the parabola forming the outer peripheral surface of the conical shape of the collimator 15, and is also focused to become parallel light. In other words, by using the collimator 15 having a convex lens in the central part and a parabola formed in the peripheral part, almost all of the light generated by the LED element can be made parallel light and led out, which can improve the utilization efficiency of the generated light.

[0350] The light source device 13 and other structures described above can be applied to the above Fig.13 and Fig.14 The light source device 13 of the image display device 10 of the spatial floating image information display system 1 shown in FIG.

[0351] Furthermore, Fig.32 The light converted into substantially parallel light by the collimator 15 shown is reflected on the reflective light guide 304. Of the reflected light, light of a specific polarization is transmitted through the reflective polarizer 49 by the action of the reflective polarizer 49, and light of another polarization reflected by the action of the reflective polarizer 49 is transmitted through the light guide 304 again. The light is reflected by the reflective plate 271 at a position opposite to the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized by passing through the λ / 4 plate 270 as a phase difference plate twice.

[0352] The light reflected on the reflector 271 passes through the light guide 304 again and is incident on the reflective polarizer 49 provided on the opposite surface. The incident light has been polarized, so it passes through the reflective polarizer 49 and is incident on the liquid crystal display panel 11 in a uniform polarization direction. As a result, the light of the light source can be fully utilized, so the geometric optical utilization efficiency of the light is doubled. In addition, because the polarization degree (in other words, the extinction ratio) of the reflective polarizer is multiplied by the extinction ratio of the entire system, the contrast of the entire display device is greatly improved by using the light source device 13 of this embodiment.

[0353] In addition, the reflection diffusion angle of light on each reflection surface can be adjusted by adjusting the surface roughness of the reflection surface of the reflective light guide 304 and the surface roughness of the reflection plate 271. The surface roughness of the reflection surface of the reflective light guide 304 and the surface roughness of the reflection plate 271 can be adjusted for each design to make the uniformity of light incident on the liquid crystal display panel 11 better.

[0354] <About the implementation of vending machines>

[0355] Next, as another embodiment of the present invention, using Fig.35 An example of applying the spatial floating image information display system to a vending machine is described. Fig.35 The diagram shows a case where the present invention is applied to, for example, an automatic beverage vending machine.

[0356] Fig.35 In the embodiment, the vending machine body 2900 includes a space floating image display unit 2920. Although not shown in the figure, the space floating image display unit 2920 also includes Fig.13 , Fig.14 The internal structure shown in the figure obtains a multi-viewpoint image through the image display device 10 and the cylindrical lens 1103, and generates a spatial floating image based on the multi-viewpoint image. In addition, the vending machine 2900 has a beverage display unit 2980 for displaying the beverages sold by the vending machine 2900, a banknote input unit 2981 for inputting banknotes, a coin input unit 2982 for inputting coins, a change outlet 2983 for taking out change, and a beverage outlet 2984 for taking out the beverage purchased by the user.

[0357] Similar to the self-service terminal, the vending machine body 2900 is equipped with a camera or a human body sensor. When a user approaches, the spatial floating image display unit 2920 is activated. Fig.35As shown, in the space floating image display unit 2920, a virtual assistant 2921 appears and speaks to the user, for example, "Welcome. Thank you for using. The screen will change to number buttons. Please select the product number you like." After that, the space floating image display unit 2920 clears the image of the virtual assistant 2921 and then displays the number buttons 2922 and the decision button 2923. At this time, although not shown in the figure, a cancel button and a return button may also be displayed in addition to the number buttons 2922 and the decision button 2923.

[0358] As mentioned above, Fig.35 The image of the virtual assistant 2921 displayed in the spatial floating image display unit 2920 is a spatial floating image based on a multi-viewpoint image with motion parallax. As a result, the user can see the image of the virtual assistant 2921 in a stereoscopic image. Furthermore, when the user moves around the vending machine body 2900, from the user's point of view, the virtual assistant 2921 always speaks in the direction of the user with the line of sight as the user moves. Therefore, a new effect can be brought about: the user can get the feeling that the image of the virtual assistant 2921 is as if there is a real person and is talking to the user alone.

[0359] The user selects a drink by operating the number button 2922 and the decision button 2923 displayed in the spatial floating image display unit 2920, and inserts a specified amount of money into the banknote input unit 2981 or the coin input unit 2982, whereby the drink is provided in a form that can be taken out from the drink outlet 2984. After that, the number button and the decision button disappear in the spatial floating image display unit 2920, and the virtual assistant 2921 appears again, for example, saying "Thank you. I look forward to your use again." In this case, as with the self-service terminal, the sound can be emitted from a normal speaker, or it can be emitted by a super-directional speaker in a manner that only the user can hear.

[0360] Through the above series of operations, the user can buy the desired beverage. Fig.35 The example of shows an example of only a spatial floating image display unit, but the vending machine can also have both a liquid crystal display device and a spatial floating image display unit, similar to the example of the self-service terminal, and the spatial floating image display unit can be set in two or more places instead of one place. In addition, when the spatial floating image display unit is set in two places, it can be configured so that the virtual assistant is displayed as a multi-viewpoint image with motion parallax in any one of the spatial floating image display units, and the number buttons and the decision button are displayed in the other spatial floating image display unit.

[0361] In addition, as a virtual assistant 2921, multiple portraits of people of different ages and genders and animated characters can also be displayed. The data for displaying the above multiple portraits of people of different ages and genders and animated characters is stored in Figure 2 In the non-volatile memory 1108, one of the multiple human images or animation characters can be appropriately selected and displayed in the space floating image display unit as a virtual assistant 2921. In this case, which human image or character to be displayed can be determined according to the user's attributes (age, etc.).

[0362] As described above, this embodiment is similar to the case of the self-service terminal, and is provided with a spatial floating image display unit based on a multi-viewpoint image (or image) with motion parallax, so that the user can select and purchase goods contactlessly. In addition, it is possible to detect that the user is approaching the vending machine, automatically display the spatial floating image, and then display the image or role of the virtual assistant 2921, which is obtained based on the multi-viewpoint image (or image) with motion parallax and is recognized as a three-dimensional image, in the spatial floating image display unit. As a result, compared with a conventional vending machine, the user can obtain the following effect: for the image of the virtual assistant 2921, it is possible to obtain a feeling as if a real person exists and is always talking to oneself no matter where the user moves. Therefore, because of the novelty of the display of the virtual assistant 2921 as described above, it can attract more users and is expected to increase the sales of goods of the vending machine.

[0363] The present invention has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments, and various changes can be made without departing from the main purpose. In addition to the necessary components, each embodiment can add, delete, replace, etc. the components. In the absence of special restrictions, each component can be single or multiple. It is also possible to adopt a method of combining the various embodiments.

[0364] In the technology of the embodiment, by making high-resolution and high-brightness image information suspended in space as a space-suspended image, for example, the user can operate without feeling uneasy about the contact transmission of infectious diseases. If the technology of this embodiment is used in a system used by an uncertain number of users, the risk of contact transmission of infectious diseases can be reduced, and a contactless user interface that can be used without feeling uneasy is provided. The present invention that provides such technology contributes to the "3 Good Health and Well-Being" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0365] In addition, in the technology of the implementation mode, by reducing the divergence angle of the emitted image light and then unifying it into a specific polarization, only the normal reflected light is efficiently reflected to the retro-reflective component, so the light utilization efficiency is high, and a bright and clear spatial floating image can be obtained. According to the technology of the implementation mode, a non-contact user interface with excellent usability that can significantly reduce power consumption can be provided. The present invention that provides such technology contributes to the "9 industries, innovations and infrastructure" and "11 sustainable cities and communities" of the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations.

[0366] Furthermore, the technology of the embodiment can form a spatial floating image formed by image light with high directivity (straight-line propagation). The technology of this embodiment can provide a non-contact user interface with low risk of spatial floating images being viewed by people other than the user by displaying highly directional image light when displaying images that require high security such as so-called self-service terminals or displaying highly confidential images that are to be kept secret from people facing the user. The present invention contributes to the "11 Sustainable Cities and Communities" of the Sustainable Development Goals (SDGs) advocated by the United Nations by providing the above-mentioned technology.

[0367] Description of Reference Numerals

[0368] 1……space-suspended image information display system, 2……retro-reflective component, 3……space-suspended image, 10……image display device, 11……liquid crystal display panel, 13……light source device, 330……retro-reflective plate, 1103……cylindrical lens, 1400, 1500, 1600, 1700……self-service terminal, 1510……liquid crystal display screen, 1520, 1620, 1720……space-suspended image display unit, 1521, 1621, 1721, 2921……virtual assistant, 1522, 1622, 1722……operation menu, 1530……exit, 1541, 1542……camera, 1550……housing, 1551, 1552……super-directional loudspeakers, 1560……human body sensor, 1570……inclined surface, 2922……digital buttons, 2923……determination button.

Claims

1. A spatial suspended image information display system, which forms a spatial suspended image in the air, characterized in that: include: an image display device that displays an image of at least one object; A cylindrical lens disposed on the image light emitting side of the image display device; and a retro-reflective component for causing image light from the image display device to retro-reflect and form the spatial floating image in the air; Herein, as the object, the image display device displays multi-viewpoint images captured from multiple viewpoints or obtained by rendering.

2. The spatial suspended image information display system according to claim 1, characterized in that: The cylindrical lens is arranged between the image display device and the retroreflective component.

3. The spatial suspended image information display system according to claim 1, characterized in that: The cylindrical lens is arranged at a predetermined distance from an emission surface of a display panel of the image display device.

4. The spatial suspended image information display system according to claim 3, characterized in that: The distance between the cylindrical lens and the image display device is adjusted according to the focal length of the cylindrical lens.

5. The spatial suspended image information display system according to claim 1, characterized in that: The output surface of the display panel of the image display device is parallel to the incident surface of the cylindrical lens.

6. The spatial suspended image information display system according to claim 1, characterized in that: The object has a shape as a human figure.

7. The spatial suspended image information display system according to claim 1, characterized in that: The cylindrical lens is arranged between the image display device and the retro-reflective component. The spatial suspended image information display system comprises: a housing that accommodates the image display device and the retro-reflective component; and a control device that performs a prescribed process based on a prescribed operation, wherein the object has a shape as a human figure, The human image displayed as the multi-viewpoint image has motion parallax as the user moves.

8. The spatial suspended image information display system according to claim 7, characterized in that: The direction of the face of the person image displayed as the multi-viewpoint image moves so as to face the direction of the user as the user moves.

9. The spatial suspended image information display system according to claim 7, characterized in that: The character image is a virtual assistant that guides the user to perform the operation.

10. The spatial suspended image information display system according to claim 7, characterized in that: When it is detected that the user has approached the housing, a human figure is first displayed as the object as the spatial floating image, and then an operation menu is displayed.

11. The spatial suspended image information display system according to claim 7, characterized in that: When it is detected that the user has approached the housing, both an image of the person being the target and an operation menu are displayed as the spatial floating image.

12. The spatial suspended image information display system according to claim 7, characterized in that: comprising a camera device for photographing the user, Approach of the user is detected based on the image of the user acquired by the imaging device.

13. The spatial suspended image information display system according to claim 7, characterized in that: The control device authenticates the user and, based on a result of the authentication, switches the display of the object from the image of the person to the display of the operation menu.

14. The spatial suspended image information display system according to claim 7, characterized in that: comprising a camera device for photographing the user, The user is authenticated based on the image of the user acquired by the imaging device.

15. The spatial suspended image information display system according to claim 1, characterized in that: As the spatial floating image, an operation menu including at least two or more options is displayed.

16. The spatial suspended image information display system according to claim 9, characterized in that: As the spatial floating image, the virtual assistant and an operation menu including at least two or more options are displayed, The operation of the operation menu is guided by the image and sound of the virtual assistant.

17. The spatial suspended image information display system according to claim 7, characterized in that: A super-directional speaker is included that outputs sound signals in a manner that can only be heard by the user.

18. The spatial suspended image information display system according to claim 7, characterized in that: The control device obtains attribute information of the user, and based on the attribute information, determines at least one of the following conditions: whether the user is a first-time user of the spatial floating image information display system, and whether the user is a user over a specified age. For users who meet the conditions, guidance is provided by the virtual assistant which is the spatial floating image.

19. The spatial suspended image information display system according to claim 1, characterized in that: include: A sensing system, comprising a sensor for detecting a user's operation on the spatial floating image; A housing, which is provided with the image display device and the retro-reflective component; and a control device that performs a predetermined process based on the detected operation, The control device executes a process of issuing a document based on the detected operation, and a portion of the housing has an outlet for taking out the document.

20. The spatial suspended image information display system according to claim 19, characterized in that: The spatial floating image information display system is a self-service terminal, which has the function of issuing tickets or administrative certificates as the document.

21. The spatial suspended image information display system according to claim 7, characterized in that: The device includes a communication unit for performing wireless communication with a portable terminal held by the user.

22. The spatial suspended image information display system according to claim 8, characterized in that: Based on the wireless communication, the information of the user transmitted from the portable terminal is received.

23. The spatial suspended image information display system according to claim 22, characterized in that: The control device executes a process of issuing a document based on the detected operation, Based on the wireless communication, the information of the file is sent to the portable terminal.

24. The spatial suspended image information display system according to claim 7, characterized in that: As the spatial floating image, code information for reading by the user's portable terminal is displayed.

25. The spatial suspended image information display system according to claim 7, characterized in that: comprising a camera device for photographing the user, The code information displayed on the portable terminal of the user is read through the camera device.

26. The spatial suspended image information display system according to claim 1, characterized in that: include: A sensor for detecting a user's operation on the spatial floating image; A housing for housing the image display device and the retro-reflective component; and a control device that performs a predetermined process based on the detected operation, The sensing system is used to detect the state of the surface contact of the user's finger or object with the spatial floating image. A line is drawn at a position where the contact occurs on the surface of the spatial floating image as the spatial floating image.

27. The spatial suspended image information display system according to claim 26, characterized in that: The control device obtains the line drawn as text or graphics on the surface of the spatial floating image and uses it as input information of the user.

28. The spatial suspended image information display system according to claim 26, characterized in that: The control device obtains the line drawn as text or graphics on the surface of the spatial floating image and uses it as the signature of the user.

29. The spatial suspended image information display system according to claim 26, characterized in that: The line drawn on the surface of the spatial floating image is information about the name or attribute of the user.

30. The spatial suspended image information display system according to claim 26, characterized in that: A frame is displayed on the entire or a part of the area of ​​the suspended image in the space, The control device controls the display so as to change the color of the frame according to whether the finger or the object held by the user contacts the partial area on the surface of the spatial floating image.

31. The spatial suspended image information display system according to claim 26, characterized in that: A frame is displayed on the entire or a part of the area of ​​the suspended image in the space, The control device controls the display so as to change the background color of the area inside the frame according to whether the finger or the object held by the user touches the area inside the frame on the surface of the spatial floating image.

32. The spatial suspended image information display system according to claim 26, characterized in that: A frame is displayed on the entire or a part of the area of ​​the suspended image in the space, The control device controls switching between a mode of drawing the line on the spatial floating image and a mode of not drawing the line on the spatial floating image, and controls display to change the color of the frame in the mode of drawing the line and the mode of not drawing the line.

33. The spatial suspended image information display system according to claim 26, characterized in that: A frame is displayed on the entire or a part of the area of ​​the suspended image in the space, The control device controls the display so as to flash the frame according to whether the finger or the object held by the user contacts the surface of the spatial floating image.

34. The spatial suspended image information display system according to claim 26, characterized in that: A physical button is provided at a predetermined position of the housing for shifting to a mode for detecting the operation on the surface of the spatial floating image and drawing the line.

35. The spatial suspended image information display system according to claim 26, characterized in that: A button for shifting to a mode of detecting the operation on the surface of the spatial floating image and drawing the line is displayed as a part of the spatial floating image.

Images