Aerial floating video display device, mobile terminal, and display method

Abstract

To provide a more suitable aerial floating video display device, thereby contributing to the Sustainable Development Goals (SDGs): "3 good health and well-being" "9 industry, innovation, and infrastructure," and "11 sustainable cities and communities."SOLUTION: An aerial floating video display device, based on a communication connection with a user's mobile terminal, displays, for example, a display image displayed on a screen of the mobile terminal as a display image also in an aerial floating video, and based on detection and determination of a user operation on the display image displayed on the screen of the mobile terminal, executes predetermined processing associated with the display image on the screen, and performs control so that the processing is also reflected in the display image of the aerial floating video. In addition, in this system, it is set and controlled which device's operation and detection are valid / invalid.SELECTED DRAWING: Figure 29

Description

Technical Field

[0001] The present invention relates to an airborne floating video display device.

Background Art

[0002] Regarding airborne floating information display technology, for example, it is disclosed in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the disclosure of Patent Document 1, the consideration regarding the configuration for obtaining practical brightness and quality of the airborne floating video, and the configuration for the user to view the airborne floating video more enjoyably, etc., was not sufficient.

[0005] An object of the present invention is to provide a more suitable airborne floating video display device.

Means for Solving the Problems

[0006] To solve the above problems, for example, the configuration described in the claims is adopted. This application includes a plurality of means for solving the above problems. If an example is given, it may be configured as follows. One embodiment is an airborne floating image display device, including an image processing unit that performs image processing, a display unit that displays the image processed by the image processing unit, an optical system that generates an airborne floating image based on the image displayed by the display unit, a user operation detection mechanism that detects an operation by the user with respect to the display range of the airborne floating image, and a communication unit that communicates with the user's mobile terminal. Based on the communication connection with the user's mobile terminal, a first image displayed in the airborne floating image is also displayed as the first image on the screen of the mobile terminal, and / or a second image displayed on the screen of the mobile terminal is also displayed as the second image in the airborne floating image. Based on the detection and determination of the user's operation on the first image displayed in the airborne floating image, a predetermined first process associated with the first image in the airborne floating image is executed, and control is performed to reflect the first process also on the first image on the screen of the mobile terminal, and / or based on the detection and determination of the user's operation on the second image displayed on the screen of the mobile terminal on the mobile terminal side, a predetermined second process associated with the second image on the screen of the mobile terminal is executed, and control is performed to reflect the second process also on the second image in the airborne floating image.

Effect of the Invention

[0007] According to the present invention, a more suitable airborne floating image display device can be realized. Other problems, configurations, and effects will be clarified in the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 2I

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 4F

Figure 4G

Figure 4H

Figure 4I

Figure 4J

Figure 4K

Figure 4L

Figure 4M

Figure 4N

Figure 4O

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18A

Figure 18B

Figure 18C

Figure 18D

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 19E

Figure 19F

Figure 20A

Figure 20B

Figure 20C

Figure 21A

Figure 21B

Figure 22A

Figure 22B

Figure 22C

Figure 23A

Figure 23B

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 25A

Figure 25B

Figure 25C

Figure 26A

Figure 26B

Figure 26C

Figure 27A

Figure 27B

Figure 28A

Figure 28B

Figure 28C

Figure 29

Figure 30

Figure 31

Figure 32A

Figure 32B

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59

Figure 60

Figure 61

Figure 62

Figure 63

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical idea disclosed in this specification. Also, in all the drawings for explaining the present invention, those having the same function are given the same reference numerals, and the repeated description thereof may be omitted.

[0010] The following embodiments relate to a video display device capable of transmitting a video by video light from a video light source through a transparent member that partitions a space such as glass and displaying it as a space floating video outside the transparent member. In the description of the following embodiments, a video floating in space is expressed by the term "space floating video". Instead of this term, it may be expressed as "aerial image", "space image", "aerial floating video", "space floating optical image of the display video", "aerial floating optical image of the display video", etc. The term "space floating video" mainly used in the description of the embodiments is used as a representative example of these terms.

[0011] According to the following embodiments, for example, a video display device suitable for use in bank ATMs, ticket vending machines at stations, digital signage, etc. can be realized. For example, currently, touch panels are usually used in bank ATMs, ticket vending machines at stations, etc. However, by using a transparent glass surface or a light-transmissive plate material, high-resolution video information can be displayed in a state of floating in space on this glass surface or light-transmissive plate material. At this time, by reducing the divergence angle of the emitted video light, that is, making it an acute angle, and further aligning it with a specific polarization, only the regular reflected light is efficiently reflected to the retroreflective plate. Therefore, the light utilization efficiency is high, and it is possible to suppress the ghost image that occurs in addition to the main space-floating image, which has been a problem in the conventional retroreflective method, and a clear space-floating video can be obtained. In addition, by means of the device including the light source of this embodiment, it is possible to provide a novel and highly usable space-floating video display device (space-floating video display system) capable of significantly reducing power consumption. Also, for example, it is possible to provide a space-floating video display device for a vehicle that enables so-called unidirectional space-floating video display that can be visually recognized inside and / or outside the vehicle. <Example 1>

[0012] <An example of the usage form of the space-floating video display device> FIG. 1 is a diagram showing an example of a usage form of a spatial floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the spatial floating image display device according to the present embodiment. The specific configuration of the spatial floating image display device will be described in detail with reference to FIG. 2 and the like. Light with a specific polarization and a sandwiching angle directivity characteristic is emitted as an image light beam from the image display device 1, once enters the retroreflective plate 2 after being reflected, etc. in the optical system inside the spatial floating image display device, is retroreflected, passes through a transparent member 100 (such as glass), and forms a real image, an aerial image (spatial floating image 3), outside the glass surface. In the following embodiments, the retroreflective plate 2 (retroreflective plate) will be used as an example of the retroreflective member. However, the retroreflective plate 2 of the present invention is not limited to a flat plate, and is used as an example of a concept including a sheet-like retroreflective member attached to a flat or non-flat member, or the entire assembly with a sheet-like retroreflective member attached to a flat or non-flat member. Further, since the light beam after reflection by the retroreflective plate 2 has an optical characteristic of forming an image, the retroreflective plate 2 may be expressed as an imaging optical member or an imaging optical plate.

[0013] Also, in a store or the like, a space is partitioned by a show window (also referred to as “window glass”) 105 which is a translucent member such as glass. According to the spatial floating image display device of the present embodiment, it is possible to display a floating image in one direction with respect to the outside and / or the inside of the store (space) through such a transparent member.

[0014] In FIG. 1, the inside (inside the store) of the window glass 105 is shown in the depth direction, and the outside (for example, the sidewalk) is in the front. On the other hand, by providing means for reflecting a specific polarization wave on the window glass 105, it is also possible to reflect and form an aerial image at a desired position inside the store.

[0015] <Configuration example of the optical system of the spatial floating image display device> FIG. 2A is a diagram showing an example of the configuration of an optical system of a spatial floating image display device according to an embodiment of the present invention. Using FIG. 2A, the configuration of the spatial floating image display device will be described more specifically. As shown in FIG. 2A(1), a display device 1 that diverges image light of a specific polarization at a sandwiching angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having a sandwiching angle diffusion characteristic.

[0016] The image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 having a film that selectively reflects the image light of a specific polarization provided on the transparent member 100 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100), and enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light is polarization-converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, when entering and exiting the retroreflector 2. Here, since the polarization separation member 101 that selectively reflects the image light of a specific polarization has the property of transmitting the polarization of the other polarization that has been polarization-converted, the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a spatial floating image 3, which is a real image, outside the transparent member 100. Note that, in FIG. 2A, an example is described in which the principal ray of the image light incident on the retroreflector 2 is incident at 90° with respect to the retroreflector 2. However, the incident angle of the principal ray of the image light with respect to the retroreflector 2 is not limited to 90°, and for example, it can also be used at 90° ± 15°.

[0017] Here, a first example of the polarization design in the optical system of FIG. 2A will be described. For example, the configuration may be such that the S-polarized video light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting S-polarization and transmitting P-polarization. In this case, the S-polarized video light that has reached the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from S-polarization to P-polarization. The video light converted to P-polarization travels toward the polarization separation member 101 again. Here, since the polarization separation member 101 has the property of reflecting S-polarization and transmitting P-polarization, the P-polarized video light passes through the polarization separation member 101 and then through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is the optical image of the display image of the display device 1, is formed at a position that has a mirror relationship with the display image of the display device 1 with respect to the polarization separation member 101. Such a polarization design can preferably form the spatial floating image 3.

[0018] Next, a second example of the polarization design in the optical system of FIG. 2A will be described. For example, the configuration may be such that P-polarized video light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting P-polarization and transmitting S-polarization. In this case, the P-polarized video light that has reached the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from P-polarization to S-polarization. The video light converted to S-polarization travels toward the polarization separation member 101 again. Here, since the polarization separation member 101 has the property of reflecting P-polarization and transmitting S-polarization, the S-polarized video light passes through the polarization separation member 101 and then through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a specular relationship with the display image of the display device 1 with respect to the polarization separation member 101. Such a polarization design can preferably form the spatial floating image 3.

[0019] Note that the light forming the spatial floating image 3 is a collection of light rays converging from the retroreflector 2 to the optical image of the spatial floating image 3, and these light rays continue to travel straight even after passing through the optical image of the spatial floating image 3. Therefore, unlike the diffused video light formed on a screen by a general projector or the like, the spatial floating image 3 is an image with high directivity. Thus, in the configuration of FIG. 2A, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in a system that displays an image that requires high security or an image with high confidentiality that needs to be concealed from a person facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axes of the image light after reflection may become uneven. Also, the reflection angles may become uneven. Such uneven light may not maintain the polarization state and propagation angle assumed in the design. For example, light with a polarization state and propagation angle outside the design assumptions may directly re-enter the image display surface side of the liquid crystal display panel 11 from the position of the retroreflector 2 without passing through the polarization separation member. Light with a polarization state and propagation angle outside the design assumptions may also re-enter the image display surface side of the liquid crystal display panel 11 after being reflected by components within the spatial floating image display device. The light that re-enters the image display surface side of such a liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and potentially degrading the image quality of the spatial floating image. Therefore, in this embodiment, an absorption-type polarizing plate 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorption-type polarizing plate 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorption-type polarizing plate 12, thereby suppressing the above-mentioned re-reflection. Thereby, it is possible to prevent degradation of the image quality due to the ghost image of the spatial floating image. Specifically, if the configuration is such that S-polarized image light is emitted from the display device 1 to the polarization separation member 101, the absorption-type polarizing plate 12 may be a polarizing plate that absorbs P-polarization. Also, if the configuration is such that P-polarized image light is emitted from the display device 1 to the polarization separation member 101, the absorption-type polarizing plate 12 may be a polarizing plate that absorbs S-polarization.

[0021] The above-described polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects a specific polarization wave.

[0022] Next, FIG. 2A(2) shows an example of the surface shape of a representative retroreflector 2 as the retroreflector. Light rays incident inside regularly arranged hexagonal prisms are reflected by the wall surfaces and bottom surfaces of the hexagonal prisms and exit as retroreflected light in a direction corresponding to the incident light, displaying a spatial floating image that is a real image based on the image displayed on the display device 1.

[0023] The resolution of this floating image in space depends greatly on the outer shape D and pitch P of the retroreflective portion of the retroreflector 2 shown in Fig. 2A(2), in addition to the resolution of the liquid crystal display panel 11. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, for example, one pixel of the floating image in space is equivalent to 300 μm. Therefore, the effective resolution of the floating video is reduced to about one-third.

[0024] Therefore, in order to make the resolution of the floating video in space equivalent to the resolution of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion closer to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moiré due to the pixels of the retroreflector and the liquid crystal display panel, it is advisable to design by deviating the respective pitch ratios from an integer multiple of one pixel. Also, the shape should be arranged so that none of the sides of the retroreflective portion overlap any of the sides of one pixel of the liquid crystal display panel.

[0025] Note that the surface shape of the retroreflector according to this embodiment is not limited to the above example. It may have various surface shapes that realize retroreflection. Specifically, a retroreflective element in which triangular pyramid prisms, hexagonal pyramid prisms, other polygonal prisms, or combinations thereof are periodically arranged may be provided on the surface of the retroreflector of this embodiment. Alternatively, a retroreflective element in which these prisms are periodically arranged to form a cube corner may be provided on the surface of the retroreflector of this embodiment. These can also be expressed as a corner reflector array or a multi-faceted reflector array. Alternatively, a capsule lens type retroreflective element in which glass beads are periodically arranged may be provided on the surface of the retroreflector of this embodiment. Since the detailed configuration of these retroreflective elements may use existing technologies, detailed description is omitted. Specifically, the technologies disclosed in JP-A-2001-33609, JP-A-2001-264525, JP-A-2005-181555, JP-A-2008-70898, JP-A-2009-229942, etc. may be used.

[0026] <Another Configuration Example 1 of the Optical System of the Spatial Floating Image Display Device> Another configuration example of the optical system of the spatial floating image display device will be described with reference to FIG. 2B. In FIG. 2B, components labeled with the same reference numerals as in FIG. 2A have the same functions and configurations as those in FIG. 2A. For the sake of simplicity, repeated descriptions of such components will be omitted.

[0027] In the optical system of FIG. 2B, similar to FIG. 2A, image light of a specific polarization is output from the display device 1. The image light of the specific polarization output from the display device 1 is input to the polarization separation member 101B. The polarization separation member 101B is a member that selectively transmits the image light of the specific polarization. Different from the polarization separation member 101 in FIG. 2A, the polarization separation member 101B is not integrated with the transparent member 100 and has an independent plate-like shape. Therefore, the polarization separation member 101B may be referred to as a polarization separation plate. The polarization separation member 101B may be configured, for example, as a reflective polarizing plate formed by attaching a polarization separation sheet to a transparent member. Or it may be formed of a metal multilayer film or the like that selectively transmits a specific polarization of the transparent member and reflects other specific polarizations. In FIG. 2B, the polarization separation member 101B is configured to transmit the image light of the specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflective plate 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflective plate. The image light is polarization-converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, when entering and exiting the retroreflective plate. Here, since the polarization separation member 101B has the property of reflecting the polarization of the other polarization that has been polarization-converted by the λ / 4 plate 21, the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a spatial floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of the polarization design in the optical system of FIG. 2B will be described. For example, the configuration may be such that P-polarized video light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized video light that has reached the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from P-polarized light to S-polarized light. The video light converted to S-polarized light travels toward the polarization separation member 101B again. Here, since the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light, the S-polarized video light is reflected by the polarization separation member 101 and passes through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a mirror relationship with the display image of the display device 1 with respect to the polarization separation member 101B. Such a polarization design can preferably form the spatial floating image 3.

[0030] Next, a second example of the polarization design in the optical system of FIG. 2B will be described. For example, the configuration may be such that the S-polarized video light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized video light that has reached the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from S-polarized light to P-polarized light. The video light converted to P-polarized light travels again toward the polarization separation member 101B. Here, since the polarization separation member 101B has the property of reflecting P-polarized light and transmitting S-polarized light, the P-polarized video light is reflected by the polarization separation member 101 and passes through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that is in a mirror image relationship with the display image of the display device 1 with respect to the polarization separation member 101B. With such a polarization design, the spatial floating image 3 can be preferably formed.

[0031] In Fig. 2B, the image display surface of the display device 1 and the surface of the retroreflective sheet 2 are arranged in parallel. The polarization separation member 101B is arranged at an angle α (e.g., 30°) with respect to the image display surface of the display device 1 and the surface of the retroreflective sheet 2. Then, in the reflection of the polarization separation member 101B, the traveling direction of the image light reflected by the polarization separation member 101B (the direction of the principal ray of the image light) is different from the traveling direction of the image light incident from the retroreflective sheet 2 (the direction of the principal ray of the image light) by an angle β (e.g., 60°). By configuring in this way, in the optical system of Fig. 2B, the image light is output at a predetermined angle shown toward the outside of the transparent member 100, and the spatial floating image 3 which is a real image is formed. In the configuration of Fig. 2B, when the user views from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when another person views from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable when adopted in a system for displaying an image that requires high security or an image with high confidentiality that needs to be concealed from a person facing the user.

[0032] As described above, in the optical system of Fig. 2B, although it is an optical system with a configuration different from that of the optical system of Fig. 2A, it is possible to form a suitable spatial floating image in the same manner as the optical system of Fig. 2A.

[0033] Note that an absorption type polarizing plate may be provided on the surface of the transparent member 100 on the side of the polarization separation member 101B. The absorption type polarizing plate may be an absorption type polarizing plate that transmits the polarization wave of the image light from the polarization separation member 101B and absorbs the polarization wave whose phase is different by 90° from the polarization wave of the image light from the polarization separation member 101B. In this way, while allowing sufficient transmission of the image light for forming the spatial floating image 3, it is possible to reduce the external light incident from the side of the spatial floating image 3 of the transparent member 100 by about 50%. Thereby, it is possible to reduce the stray light in the optical system of Fig. 2B based on the external light incident from the side of the spatial floating image 3 of the transparent member 100.

[0034] <Another Configuration Example of the Optical System of the Spatial Floating Image Display Device 2> Another configuration example of the optical system of the spatial floating image display device will be described with reference to FIG. 2C. In FIG. 2C, components labeled with the same reference numerals as those in FIG. 2B have the same functions and configurations as those in FIG. 2B. For such components, repetitive descriptions will be omitted for the sake of simplicity.

[0035] The difference between the optical system of FIG. 2C and the optical system of FIG. 2B is only the arrangement angle of the polarization separation member 101B with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. Since all other configurations are the same as those of the optical system of FIG. 2B, repetitive descriptions will be omitted. Since the polarization design of the optical system of FIG. 2C is the same as that of the optical system of FIG. 2B, repetitive descriptions will be omitted.

[0036] In the optical system of FIG. 2C, the polarization separation member 101B is arranged at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C, the angle α is 45°. With this configuration, in the reflection of the polarization separation member 101B, the angle β formed by the traveling direction of the image light reflected by the polarization separation member 101B (the direction of the principal ray of the image light) with respect to the traveling direction of the image light incident from the retroreflector 2 (the direction of the principal ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the traveling direction of the image light reflected by the polarization separation member 101B, and the angular relationship of the surfaces constituting the optical system can be simplified. If the surface of the transparent member 100 is arranged perpendicular to the traveling direction of the image light reflected by the polarization separation member 101B, the angular relationship of the surfaces constituting the optical system can be further simplified. In the configuration of FIG. 2C, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in systems that display images requiring high security or highly confidential images that need to be concealed from people facing the user.

[0037] As described above, in the optical system of FIG. 2C, although it is an optical system with a configuration different from the optical systems of FIGS. 2A and 2B, it is possible to form a suitable aerial image in the same manner as the optical systems of FIGS. 2A and 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] Note that an absorption-type polarizing plate may be provided on the surface of the transparent member 100 on the side of the polarization separation member 101B. The absorption-type polarizing plate may be an absorption-type polarizing plate that transmits the polarization wave of the video light from the polarization separation member 101B and absorbs the polarization wave having a phase difference of 90° from the polarization wave of the video light from the polarization separation member 101B. In this way, while allowing sufficient transmission of the video light for forming the aerial image 3, it is possible to reduce the external light incident from the side of the aerial image 3 of the transparent member 100 by about 50%. Thereby, it is possible to reduce the stray light in the optical system of FIG. 2C based on the external light incident from the side of the aerial image 3 of the transparent member 100.

[0039] <Another Configuration Example 3 of the Optical System of the Aerial Image Display Device> Another configuration example of the optical system of the aerial image display device will be described with reference to FIG. 2D. The optical system of FIG. 2D is an optical system using a retroreflector 5, which is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, with reference to FIGS. 2D to 2I, another configuration example 3 of the optical system will be described more specifically. In FIG. 2D, the components denoted by the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. For such components, repetitive description will not be given in order to simplify the explanation.

[0040] FIG. 2D is a diagram showing an example of the main configuration and the retroreflective portion configuration of an aerial image display device according to an embodiment of the present invention. In an oblique direction of a transparent member 100 such as glass, a display device 10 that emits video light is provided. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] The chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and it can also be used, for example, at 45° ± 15°.

[0042] The retroreflector 5 is an optical member having an optical property of retroreflecting at least part of the light rays in some directions. Also, since the reflected light rays have an optical property of forming an image, the retroreflector 5 may be referred to as an imaging optical member or an imaging optical plate.

[0043] The specific configuration of the retroreflector 5 will be described in detail with reference to FIGS. 2E, 2F, etc. By the retroreflector 5, the chief ray 9020 is retroreflectively reflected in the x and y directions while traveling in the z direction. As a result, the reflected ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the chief ray 9020 with reference to the retroreflector 5, passes through the transparent member 100, and forms a three-dimensional floating image 3 as a real image on the imaging surface.

[0044] The light beam forming the three-dimensional floating image 3 is a collection of light rays converging from the retroreflector 5 to the optical image of the three-dimensional floating image 3, and these light rays continue to travel straight even after passing through the optical image of the three-dimensional floating image 3. Therefore, the three-dimensional floating image 3 is an image with high directivity, different from a diffused image formed on a screen by a general projector or the like. Thus, in the configuration of FIG. 2, when viewed by the user from the direction of arrow A, the three-dimensional floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the three-dimensional floating image 3 cannot be viewed as an image at all. This characteristic is suitable for use in a system that displays an image requiring high security or an image with high confidentiality that needs to be concealed from a person facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described with reference to FIGS. 2E and 2F. The retroreflector 5 has a configuration in which a plurality of corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be referred to as a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail with reference to FIGS. 2G, 2H, and 2I. The light rays 9111, 9112, 9113, and 9114 emitted from the light source 9110 are reflected twice by the two mirror surfaces 9041 and 9042 of the corner reflector 9040 to become the reflected light rays 9121, 9122, 9123, and 9124. This double reflection results in a retroreflective reflection that folds back in the same direction as the incident direction (advances in the direction rotated by 180°) with respect to the x and y directions, and a specular reflection in which the incident angle and the reflection angle coincide due to total reflection with respect to the z direction.

[0046] That is, the light rays 9111 to 9114 generate the reflected light rays 9121 to 9124 on a straight line symmetric with respect to the z direction with respect to the corner reflector 9040, and form an aerial image 9120. The light rays 9111 to 9114 emitted from the light source 9110 are four light rays representing the diffused light from the light source 9110. Depending on the diffusion characteristics of the light source 9110, the light rays incident on the retroreflector 5 are not limited to these, but any incident light ray causes a similar reflection and forms an aerial image 9120. For ease of viewing the drawing, the positions of the light source 9110 and the aerial image 9120 in the x direction are shifted in the drawing, but actually, the positions of the light source 9110 and the aerial image 9120 in the x direction are the same position, and they overlap when viewed from the z direction.

[0047] Next, the configuration and effect of the corner reflector 9040 that constitutes the retroreflector 5 will be described with reference to FIGS. 2G, 2H, and 2I. The corner reflector 9040 is a rectangular parallelepiped in which only two specific surfaces are the mirror surfaces 9041 and 9042, and the other four surfaces are formed of a transparent member. The retroreflector 5 has a configuration in which the corner reflectors 9040 are arranged in an array such that their corresponding mirror surfaces face the same direction.

[0048] When viewed from above (+z direction), the light ray 9111 emitted from the light source 9110 is incident on the mirror surface 9041 (or mirror surface 9042) at a specific incident angle, and after total reflection at the reflection point 9130, it is totally reflected again at the reflection point 9132 on the mirror surface 9042 (or mirror surface 9041).

[0049] Assuming that the incident angle of the light ray 9111 with respect to the mirror surface 9041 (or mirror surface 9042) is θ, the incident angle of the first reflected light ray 9131 reflected by the mirror surface 9041 (or mirror surface 9042) with respect to the mirror surface 9042 (or mirror surface 9041) can be expressed as 90° - θ. Therefore, with respect to the light ray 9111, the second reflected light ray 9121 obtains a rotation of 2θ by the first reflection and 2×(90° - θ) by the second reflection, resulting in a total reverse optical path of 180°. On the other hand, when viewed from the side (the intermediate direction between -x and -y), total reflection with respect to the z direction occurs only once. Therefore, assuming that the incident angle with respect to the mirror surface 9041 or mirror surface 9042 is φ, with respect to the light ray 9111, the reflected light ray 9121 obtains a rotation of 2×φ by one reflection.

[0050] From the above, the light rays incident on the corner reflector 9040 produce a recursive reflection with a reverse optical path in the x and y directions, and a regular reflection by total reflection in the z direction. Considering the recursive reflection plate 5, similar reflections are caused in each optical path. Therefore, in the x and y directions, an inverted optical path with convergence forms an image at a point symmetric with respect to the z-axis direction.

[0051] Here, in the optical systems of FIGS. 2A to 2C, the recursive reflection plate 2 has recursive reflection characteristics in three axial directions. As a result, when a diffusive incident light beam is incident on the recursive reflection plate 2, a convergent reflected light beam travels toward the side where the light source of the incident light ray exists with respect to the recursive reflection plate 2. The convergent reflected light beam forms an image in the air to form a spatial floating image 3. The traveling direction of the principal ray of the convergent reflected light beam reflected from the recursive reflection plate 2 is opposite to the traveling direction of the principal ray of the diffusive incident light beam incident on the recursive reflection plate 2.

[0052] In contrast, in the optical system of FIG. 2D, the retroreflective plate 5 has retroreflective characteristics in the biaxial direction and specularly reflects in the other uniaxial direction. Thus, when a diffusive incident light beam is incident on the retroreflective plate 5, the convergent reflected light beam reflected by the corner reflector array travels toward the side opposite to the side where the light source of the incident light ray exists with respect to the retroreflective plate 5. The convergent reflected light beam forms a spatial floating image 3 by imaging in the air.

[0053] The traveling direction of the principal ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflective plate 5 does not become the reverse direction of the traveling direction of the principal ray of the diffusive incident light beam incident on the retroreflective plate 5. The normal direction component of the plate-shaped surface of the retroreflective plate 5 with respect to the traveling direction of the principal ray of the diffusive incident light beam incident on the retroreflective plate 5 and the normal direction component of the plate-shaped surface of the retroreflective plate 5 with respect to the traveling direction of the principal ray after the light beam is reflected by the retroreflective plate 5 and becomes a convergent reflected light beam travel straight without changing before and after reflection by the corner reflector array.

[0054] That is, due to the reflection in the retroreflective plate 5, the diffusive incident light beam is converted into a convergent reflected light beam, but in the normal direction of the plate-shaped surface of the retroreflective plate 5, the light beam travels through the retroreflective plate 5. Here, the diffusive incident light beam incident on the retroreflective plate 5 and the convergent reflected light beam emitted from the retroreflective plate 5 have a geometrically plane-symmetric relationship with respect to the plate-shaped surface of the retroreflective plate 5.

[0055] The resolution of the spatial floating image formed by the light rays from the video output unit 10 depends greatly on the diameter D and pitch P (not shown) of the retroreflective portion of the retroreflective plate 5 shown in FIGS. 2E and 2F in addition to the resolution of the liquid crystal display panel 11. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch P is 300 μm, one pixel of the spatial floating image corresponds to 300 μm. Therefore, the effective resolution of the spatial floating video is reduced to about 1 / 3.

[0056] Therefore, in order to make the resolution of the floating image in space equivalent to that of the display device 10, it is desirable to make the diameter D and the pitch P of the retroreflective portion approach one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moiré due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design by deviating the respective pitch ratios from an integral multiple of one pixel. Also, the shape should be arranged so that none of the sides of the retroreflective portion overlap with any of the sides of one pixel of the liquid crystal display panel.

[0057] Note that the shape of the retroreflective plate (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that realize retroreflectivity. Specifically, various cubic corner bodies, corner reflector arrays may be used, or a slit mirror array, a two-sided corner reflector array, a multi-sided reflector array, or a shape in which combinations of its reflective surfaces are periodically arranged may be used. Alternatively, a capsule lens type retroreflective element in which glass beads are periodically arranged may be provided on the surface of the retroreflective plate of this embodiment. Since the detailed configuration of these retroreflective elements may use existing technologies, detailed description is omitted. Specifically, the technologies disclosed in JP-A-2017-33005, JP-A-2019-133110, JP-A-2017-67933, WO2009 / 131128, etc. may be used.

[0058] Note that in the optical system of Fig. 2D, the video light emitted from the display device 10 may be in any polarization state. There is no problem whether it is S-polarized light or P-polarized light.

[0059] As described above, in the optical system of Fig. 2D, although it is an optical system using a retroreflective plate different from the optical systems of Figs. 2A to 2C, a more suitable floating image in space can be formed in the same manner as the optical systems of Figs. 2A to 2C.

[0060] According to the optical systems of Figs. 2A, 2B, 2C, and 2D described above, a brighter and higher-quality floating image in space can be provided.

[0061] <<Block Diagram of the Internal Configuration of the Spatial Floating Image Display Device>>

[0062] Next, a block diagram of the internal configuration of the spatial floating image display device 1000 will be described. FIG. 3 is a block diagram showing an example of the internal configuration of the spatial floating image display device 1000.

[0063] The spatial floating image display device 1000 includes a retroreflective unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power input interface 1111, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an air operation detection sensor 1351, an air operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. Note that a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112, etc. may also be provided.

[0064] Each component of the spatial floating image display device 1000 is arranged in a housing 1190. Note that the imaging unit 1180 and the air operation detection sensor 1351 shown in FIG. 3 may be provided outside the housing 1190.

[0065] The retroreflective unit 1101 in FIG. 3 corresponds to the retroreflective plate 2 in FIGS. 2A, 2B, and 2C. The retroreflective unit 1101 retroreflects the light modulated by the image display unit 1102. Among the reflected light from the retroreflective unit 1101, the spatial floating image 3 is formed by the light output to the outside of the spatial floating image display device 1000.

[0066] The image display unit 1102 in FIG. 3 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. The light source 1105 in FIG. 3 corresponds to the light source device 13 in FIGS. 2A, 2B, and 2C. And the image display unit 1102, the light guide 1104, and the light source 1105 in FIG. 3 correspond to the display device 1 in FIGS. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating the transmitted light based on the input video signal under the control of the video control unit 1160 described later. The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. As the video display unit 1102, for example, a transmissive liquid crystal panel is used. Also, as the video display unit 1102, for example, a reflective liquid crystal panel or a DMD (Digital Micromirror Device: registered trademark) panel that modulates the reflected light may be used.

[0068] The light source 1105 generates light for the video display unit 1102 and is a solid light source such as an LED light source or a laser light source. The power supply 1106 converts the AC current input from the outside through the external power supply input interface 1111 into a DC current and supplies power to the light source 1105. Also, the power supply 1106 supplies the necessary DC current to each part in the spatial floating video display device 1000. The secondary battery 1112 stores the power supplied from the power supply 1106. Also, the secondary battery 1112 supplies power to the light source 1105 and other components that require power when power is not supplied from the outside through the external power supply input interface 1111. That is, when the spatial floating video display device 1000 is provided with the secondary battery 1112, the user can use the spatial floating video display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides the light generated by the light source 1105 and irradiates the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be referred to as the backlight of the video display unit 1102. The light guide 1104 may mainly be configured using glass. The light guide 1104 may mainly be configured using plastic. The light guide 1104 may be configured using a mirror. Various methods are conceivable for the combination of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The air operation detection sensor 1351 is a sensor that detects the operation of the spatial floating image 3 by the finger of the user 230. The air operation detection sensor 1351 senses, for example, a range that overlaps with the entire display range of the spatial floating image 3. Note that the air operation detection sensor 1351 may sense only a range that overlaps with at least a part of the display range of the spatial floating image 3.

[0071] Specific examples of the air operation detection sensor 1351 include distance sensors using invisible light such as infrared rays, invisible light lasers, ultrasonic waves, etc. Further, the air operation detection sensor 1351 may be configured by combining a plurality of sensors in a plurality of ways so as to be able to detect coordinates in a two-dimensional plane. Further, the air operation detection sensor 1351 may be composed of a ToF (Time of Flight) type LiDAR (Light Detection and Ranging) or an image sensor.

[0072] The air operation detection sensor 1351 only needs to be able to sense, for example, a touch operation by the user's finger on an object displayed as the spatial floating image 3. Such sensing can be performed using existing technologies.

[0073] The air operation detection unit 1350 acquires a sensing signal from the air operation detection sensor 1351, and based on the sensing signal, determines the presence or absence of contact of the object of the spatial floating image 3 by the finger of the user 230, and calculates the position (contact position) where the finger of the user 230 contacts the object, etc. The air operation detection unit 1350 is composed of a circuit such as an FPGA (Field Programmable Gate Array), for example. Further, some functions of the air operation detection unit 1350 may be realized by software by a spatial operation detection program executed by the control unit 1110, for example.

[0074] The air operation detection sensor 1351 and the air operation detection unit 1350 may be incorporated into the spatial floating image display device 1000, or may be provided externally as a separate unit from the spatial floating image display device 1000. When provided as a separate unit from the spatial floating image display device 1000, the air operation detection sensor 1351 and the air operation detection unit 1350 are configured to be able to transmit information and signals to the spatial floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Also, the air operation detection sensor 1351 and the air operation detection unit 1350 may be provided as separate units. Thereby, it is possible to construct a system in which the spatial floating image display device 1000 without the air operation detection function is used as the main body and only the air operation detection function can be added as an option. Also, only the air operation detection sensor 1351 may be a separate unit, and the air operation detection unit 1350 may be incorporated into the spatial floating image display device 1000. When it is desired to arrange the air operation detection sensor 1351 more freely with respect to the installation position of the spatial floating image display device 1000, etc., there is an advantage in a configuration in which only the air operation detection sensor 1351 is a separate unit.

[0076] The imaging unit 1180 is a camera having an image sensor, and images the space near the spatial floating image 3 and / or the face, arm, finger, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180 or by using an imaging unit with a depth sensor, the air operation detection unit 1350 can be assisted during the detection process of the touch operation of the spatial floating image 3 by the user 230. The imaging unit 1180 may be provided as a separate unit from the spatial floating image display device 1000. When the imaging unit 1180 is provided as a separate unit from the spatial floating image display device 1000, it may be configured to be able to transmit an imaging signal to the spatial floating image display device 1000 via a wired or wireless communication connection path or the like.

[0077] For example, when the air operation detection sensor 1351 is configured as an object intrusion sensor that detects the presence or absence of an object entering a plane (intrusion detection plane) including the display surface of the floating image 3 in space, the air operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., the user's finger) that has not entered the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as the depth calculation information of the object based on the captured images of the plurality of imaging units 1180 and the depth information of the object by the depth sensor. Then, these information and various information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating image 3 in space.

[0079] Alternatively, the air operation detection unit 1350 may detect a touch operation of the floating image 3 in space by the user 230 based on the captured image of the imaging unit 1180 without using the air operation detection sensor 1351.

[0080] Further, the imaging unit 1180 may capture the face of the user 230 who operates the floating image 3 in space, and the control unit 1110 may perform identification processing of the user 230. Also, in order to determine whether there is someone standing around or behind the user 230 who operates the floating image 3 in space and whether the person is peeping at the operation of the user 230 on the floating image 3, the imaging unit 1180 may capture a range including the user 230 who operates the floating image 3 in space and the surrounding area of the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal reception unit such as a remote controller, or an infrared light reception unit, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. Separately from the above-mentioned user 230 who performs a touch operation on the floating image 3 in space, the operation input unit 1107 may be used, for example, by an administrator to operate the floating image display device 1000.

[0082] The video signal input unit 1131 connects to an external video output device to input video data. Various digital video input interfaces can be considered for the video signal input unit 1131. For example, it may be configured with a video input interface compliant with the HDMI (Registered Trademark) (High-Definition Multimedia Interface) standard, a video input interface compliant with the DVI (Digital Visual Interface) standard, or a video input interface compliant with the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device to input audio data. The audio signal input unit 1133 may be configured with an audio input interface compliant with the HDMI standard, an optical digital terminal interface, or a coaxial digital terminal interface, etc. In the case of an interface compliant with the HDMI standard, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an integrated interface with integrated terminals and cables. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be composed of speakers.

[0084] Also, the audio output unit 1140 may output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output to an external device as a digital signal, such as the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that picks up sounds around the spatial floating video display device 1000, converts them into signals, and generates audio signals. The microphone may record the voice of a person such as the user's voice, and the generated audio signal may be configured such that the control unit 1110 described later performs speech recognition processing to obtain character information from the audio signal.

[0085] The non-volatile memory 1108 stores various data used in the spatial floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed in the spatial floating image 3, display icons, data of objects to be operated by the user's operations, layout information, and the like. The memory 1109 stores video data to be displayed as the spatial floating image 3, control data of the device, and the like.

[0086] The control unit 1110 controls the operations of each connected unit. Further, the control unit 1110 may perform arithmetic processing based on the information acquired from each unit within the spatial floating image display device 1000 in cooperation with the program stored in the memory 1109.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. When the communication unit 1132 has a wired communication interface, the wired communication interface may be configured by, for example, a LAN interface conforming to the Ethernet standard. When the communication unit 1132 has a wireless communication interface, it may be configured by, for example, a communication interface of the Wi-Fi system, a communication interface of the Bluetooth system, a mobile communication interface such as 4G or 5G. Through the communication via the communication unit 1132, various data such as video data, image data, and audio data are transmitted and received.

[0088] Also, the removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor element memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the spatial floating image 3 via the video display unit 1102 and the retroreflective unit 1101.

[0089] The storage unit 1170 is a storage device for recording various information such as video data, image data, and audio data. The storage unit 1170 may be composed of a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor element memory such as a solid state drive (SSD). For example, various information such as video data, image data, and audio data may be recorded in the storage unit 1170 in advance when the product is shipped. Also, the storage unit 1170 may record various information such as video data, image data, and audio data acquired from an external device or an external server via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the spatial floating image 3 via the video display unit 1102 and the retroreflective unit 1101. The video data, image data, etc. such as display icons and objects for the user to operate, which are displayed as the spatial floating image 3, are also recorded in the storage unit 1170.

[0091] The layout information of display icons, objects, etc. and various metadata information related to the objects, which are displayed as the spatial floating image 3, are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from, for example, the audio output unit 1140.

[0092] The video control unit 1160 performs various controls on the video signal input to the video display unit 1102. The video control unit 1160 may also be referred to as a video processing circuit and may be composed of hardware such as an ASIC, an FPGA, or a video processor. Note that the video control unit 1160 may also be referred to as a video processing unit or an image processing unit. The video control unit 1160 performs controls such as video switching, for example, which video signal among the video signals stored in the memory 1109 and the video signals (video data) input to the video signal input unit 1131 is to be input to the video display unit 1102.

[0093] Further, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal stored in the memory 1109 and the video signal input from the video signal input unit 1131, and perform control to form a composite video as the floating video 3 in space by inputting the superimposed video signal to the video display unit 1102.

[0094] Further, the video control unit 1160 may perform control to perform image processing on the video signal input from the video signal input unit 1131, the video signal stored in the memory 1109, etc. Examples of the image processing include scaling processing such as enlarging, reducing, and deforming an image, brightness adjustment processing for changing the brightness, contrast adjustment processing for changing the contrast curve of the image, and Retinex processing for decomposing an image into light components and changing the weighting for each component.

[0095] Further, the video control unit 1160 may perform special effect video processing or the like to assist the air operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed, for example, based on the detection result of the touch operation of the user 230 by the air operation detection unit 1350 or the captured image of the user 230 by the imaging unit 1180.

[0096] The posture sensor 1113 is a sensor composed of a gravity sensor, an acceleration sensor, or a combination thereof, and can detect the posture in which the spatial floating video display device 1000 is installed. Based on the posture detection result of the posture sensor 1113, the control unit 1110 may control the operations of the connected components. For example, when an unfavorable posture as the user's usage state is detected, the display of the video being displayed on the video display unit 1102 may be stopped, and control may be performed to display an error message to the user. Alternatively, when the posture sensor 1113 detects that the installation posture of the spatial floating video display device 1000 has changed, control may be performed to rotate the display direction of the video being displayed on the video display unit 1102.

[0097] As described so far, the spatial floating video display device 1000 is equipped with various functions. However, the spatial floating video display device 1000 does not necessarily need to have all of these functions, and any configuration may be used as long as it has the function of forming the spatial floating video 3.

[0098] <Configuration Example of Spatial Floating Video Display Device> Next, a configuration example of the spatial floating video display device will be described. The layout of the components of the spatial floating video display device according to this embodiment can have various layouts depending on the usage form. Hereinafter, each layout of FIGS. 4A to 4M will be described. In any of the examples of FIGS. 4A to 4M, the thick line surrounding the spatial floating video display device 1000 shows an example of the housing structure of the spatial floating video display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system of FIG. 2A. In the spatial floating image display device 1000 shown in FIG. 4A, it is installed horizontally so that the surface on which the spatial floating image 3 is formed faces upward. That is, in FIG. 4A, a transparent member 100 is installed on the upper surface of the spatial floating image display device 1000. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected. Here, the x direction is the left-right direction as viewed from the user, the y direction is the front-rear direction (depth direction) as viewed from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, since the definitions of the x direction, y direction, and z direction are the same in each figure of FIG. 4, repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The spatial floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the spatial floating image 3 is formed faces the front (the direction of the user 230) of the spatial floating image display device 1000. That is, in FIG. 4B, a transparent member 100 is installed on the front (the direction of the user 230) of the spatial floating image display device. The spatial floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected. Here, as shown in FIG. 4B, the air operation detection sensor 1351 can use the reflection of the sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, since the nail has a higher reflectivity than the finger pad, the accuracy of touch detection can be improved by configuring it in this way.

[0101] FIG. 4C is a diagram showing an example of the configuration of the airborne floating image display device. The airborne floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. In the airborne floating image display device 1000 shown in FIG. 4C, it is installed horizontally so that the surface on which the airborne floating image 3 is formed faces upward. That is, in FIG. 4C, for the airborne floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The airborne floating image 3 is formed above the surface of the transparent member 100 of the airborne floating image display device 1000. The light of the airborne floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the airborne floating image 3 by the finger of the user 230 can be detected.

[0102] FIG. 4D is a diagram showing an example of the configuration of the airborne floating image display device. The airborne floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The airborne floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the airborne floating image 3 is formed faces the front (the direction of the user 230) of the airborne floating image display device 1000. That is, in FIG. 4D, for the airborne floating image display device 1000, a transparent member 100 is installed on the front (the direction of the user 230) of the device. The airborne floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the airborne floating image display device 1000. The light of the airborne floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the airborne floating image 3 by the finger of the user 230 can be detected. Here, as shown in FIG. 4D, the air operation detection sensor 1351 can use the reflection of the sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, since the nail has a higher reflectivity than the fingertip, the accuracy of touch detection can be improved by configuring it in this way.

[0103] Figure 4E is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 shown in Figure 4E is equipped with an optical system corresponding to the optical system of Figure 2C. In the spatial floating image display device 1000 shown in Figure 4E, it is installed horizontally so that the surface on which the spatial floating image 3 is formed faces upward. That is, in Figure 4E, in the spatial floating image display device 1000, the transparent member 100 is installed on the upper surface of the device. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in the directly upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0104] Figure 4F is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 shown in Figure 4F is equipped with an optical system corresponding to the optical system of Figure 2C. The spatial floating image display device 1000 shown in Figure 4F is installed vertically so that the surface on which the spatial floating image 3 is formed faces the front (the direction of the user 230) of the spatial floating image display device 1000. That is, in Figure 4F, in the spatial floating image display device 1000, the transparent member 100 is installed on the front (the direction of the user 230) of the device. The spatial floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in the direction towards the user's front. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0105] FIG. 4G is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system of FIG. 2C. In the optical systems of the spatial floating image display devices from FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, within the optical systems of the spatial floating image display devices from FIGS. 4A to 4F, the image light traveled in the front-rear direction and the up-down direction as seen by the user. In contrast, in the optical system of the spatial floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, within the optical system of the spatial floating image display device shown in FIG. 4G, the image light travels in the left-right direction and the front-rear direction as seen by the user. In the spatial floating image display device 1000 shown in FIG. 4G, the surface on which the spatial floating image 3 is formed is installed so as to face the front of the device (the direction of the user 230). That is, in FIG. 4G, in the spatial floating image display device 1000, the transparent member 100 is installed on the front of the device (the direction of the user 230). The spatial floating image 3 is formed on the surface of the transparent member 100 of the spatial floating image display device 1000 on the user side. The light of the spatial floating image 3 travels in the direction in front of the user. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0106] FIG. 4H is a diagram showing an example of the configuration of a floating image display device. The floating image display device 1000 in FIG. 4H is different from the floating image display device in FIG. 4G in that it has a window with a transparent plate 100B such as glass or plastic on the back of the device (the side opposite to the position where the user 230 views the floating image 3, that is, the side opposite to the traveling direction of the image light of the floating image 3 toward the user 230). Since the other configurations are the same as those of the floating image display device in FIG. 4G, repeated descriptions are omitted. The floating image display device 1000 in FIG. 4H includes a window having a transparent plate 100B at a position on the opposite side of the traveling direction of the image light of the floating image 3 with respect to the floating image 3. Therefore, when the user 230 views the floating image 3, the scenery behind the floating image display device 1000 can be recognized as the background of the floating image 3. Therefore, the user 230 can recognize that the floating image 3 is floating in the air in front of the scenery behind the floating image display device 1000. Thereby, the floating feeling of the floating image 3 in the air can be more emphasized.

[0107] Note that depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a part of the image light output from the display device 1 may be reflected by the polarization separation member 101B and may travel toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again on the surface of the transparent plate 100B and may be visually recognized by the user as stray light. Therefore, in order to prevent the stray light, the window on the back of the floating image display device 1000 may be configured not to be provided with the transparent plate 100B.

[0108] FIG. 4I is a diagram showing an example of the configuration of a floating image display device. The floating image display device 1000 in FIG. 4I is different from the floating image display device in FIG. 4H in that an opening / closing door 1410 for light shielding is provided in the window of the transparent plate 100B disposed on the back of the device (the side opposite to the position where the user 230 views the floating image 3). Since the other configurations are the same as those of the floating image display device in FIG. 4H, repeated descriptions are omitted.

[0109] The opening / closing door 1410 of the spatial floating image display device 1000 in Fig. 4I has, for example, a light-shielding plate, and is provided with a mechanism for moving (sliding), rotating, or attaching / detaching the light-shielding plate, so that the window (rear-side window) of the transparent plate 100B located on the back side of the spatial floating image display device 1000 can be switched between an open state and a light-shielding state. The movement (sliding) or rotation of the light-shielding plate by the opening / closing door 1410 may be an electric type driven by a motor (not shown). The control unit 1110 in Fig. 3 may control the motor. In the example of Fig. 4I, an example of two light-shielding plates of the opening / closing door 1410 is disclosed. In contrast, the number of light-shielding plates of the opening / closing door 1410 may be one.

[0110] For example, when the scenery visible behind the window of the transparent plate 100B of the spatial floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the spatial floating image 3 may become too bright, and the visibility of the spatial floating image 3 for the user 230 may decrease. In such a case, if the rear-side window is shielded from light by moving (sliding), rotating, or attaching the light-shielding plate of the opening / closing door 1410, the background of the spatial floating image 3 will become dark, so that the visibility of the spatial floating image 3 can be relatively increased. Such a shielding operation by the light-shielding plate of the opening / closing door 1410 may be directly performed by the force of the user 230's hand. The control unit 1110 may control a motor (not shown) according to an operation input via the operation input unit 1107 in Fig. 3 to perform a shielding operation by the light-shielding plate of the opening / closing door 1410.

[0111] In addition, an illuminance sensor may be provided on the back side (opposite side of the user 230) of the spatial floating image display device 1000, such as near the rear-side window, to measure the brightness of the space in front of the rear-side window. In this case, the control unit 1110 in Fig. 3 may control a motor (not shown) according to the detection result of the illuminance sensor to perform an opening / closing operation by the light-shielding plate of the opening / closing door 1410. By controlling the opening / closing operation by the light-shielding plate of the opening / closing door 1410 in this way, it is possible to more preferably maintain the visibility of the spatial floating image 3 without the user 230 manually performing the opening / closing operation of the light-shielding plate of the opening / closing door 1410.

[0112] In addition, the light-shielding plate by the opening and closing door 1410 may be manually detachable. Depending on the usage purpose and installation environment of the spatial floating image display device 1000, the user can select whether to keep the rear window in an open state or a light-shielded state. If it is planned to use the rear window in a light-shielded state for a long time, the detachable light-shielding plate can be fixed in the light-shielded state. Also, if it is planned to use the rear window in an open state for a long time, it can be used with the detachable light-shielding plate removed. For the attachment and detachment of the light-shielding plate, screws, a hook structure, or a fitting structure may be used.

[0113] Note that in the example of the spatial floating image display device 1000 in FIG. 4I as well, depending on the polarization distribution of the video light output from the display device 1 and the performance of the polarization separation member 101B, a part of the video light output from the display device 1 may be reflected by the polarization separation member 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again on the surface of the transparent plate 100B and be visually recognized by the user as stray light. Therefore, in order to prevent such stray light, the window on the back of the spatial floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening and closing door 1410 may be provided in the window without the transparent plate 100B. In order to prevent the stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening and closing door 1410 has a coating or material with a low light reflectance.

[0114] FIG. 4J is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 in FIG. 4J is different in that an electronically controlled variable transmittance device 1620 is arranged in the rear window of the spatial floating image display device in FIG. 4H instead of arranging a transparent plate 100B made of glass or plastic. Since the other configurations are the same as those of the spatial floating image display device in FIG. 4H, repeated descriptions are omitted. An example of the electronically controlled variable transmittance device 1620 is a liquid crystal shutter or the like.

[0115] That is, the liquid crystal shutter can control the transmitted light by voltage-controlling the liquid crystal element sandwiched between two polarizing plates. Therefore, if the liquid crystal shutter is controlled to increase the transmittance, the background of the spatial floating image 3 will be in a state where the scenery through the rear window can be seen through. Also, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be made invisible as the background of the spatial floating image 3. In addition, since the liquid crystal shutter can control the intermediate length, it can also be set to a state such as a transmittance of 50%. For example, the control unit 1110 may control the transmittance of the electronically controlled variable transmittance device 1620 according to the operation input via the operation input unit 1107 in FIG. 3. With such a configuration, when one wants to see the scenery through the rear window as the background of the spatial floating image 3, but the scenery through the rear window as the background is too bright and the visibility of the spatial floating image 3 is reduced, the transmittance of the electronically controlled variable transmittance device 1620 can be adjusted to adjust the visibility of the spatial floating image 3.

[0116] An illuminance sensor may be provided on the rear side (the side opposite to the user 230) of the spatial floating image display device 1000, such as near the rear window, to measure the brightness of the space in front of the rear window. In this case, the control unit 1110 in FIG. 3 may control the transmittance of the electronically controlled variable transmittance device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 in FIG. 3, the transmittance of the electronically controlled variable transmittance device 1620 can be adjusted according to the brightness of the space in front of the rear window, so that the visibility of the spatial floating image 3 can be maintained more suitably.

[0117] Also, in the above example, an example of a liquid crystal shutter was described as the electronically controlled variable transmittance device 1620. In contrast, as another example of the electronically controlled variable transmittance device 1620, electronic paper may be used. Even when using electronic paper, the same effects as described above can be obtained. Moreover, electronic paper has very low power consumption for maintaining the intermediate tone state. Therefore, compared with the case of adopting a liquid crystal shutter, a spatial floating image display device with low power consumption can be realized.

[0118] FIG. 4K is a diagram showing an example of the configuration of the floating image display device. The floating image display device 1000 in FIG. 4K is different from the floating image display device in FIG. 4G in that it has a transmissive self-emitting image display device 1650 instead of the transparent member 100. Since the other configurations are the same as those of the floating image display device in FIG. 4G, repeated descriptions are omitted.

[0119] In the floating image display device 1000 of FIG. 4K, after the image light beam passes through the display surface of the transmissive self-emitting image display device 1650, the floating image 3 is formed outside the floating image display device 1000. That is, when an image is being displayed on the transmissive self-emitting image display device 1650, which is a two-dimensional plane display, the floating image 3 can be displayed as an image that pops out further on the user side of the image of the transmissive self-emitting image display device 1650. At this time, the user 230 can simultaneously view two images with different depths. The transmissive self-emitting image display device 1650 may be configured using existing technologies such as a transmissive organic EL panel disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although not shown in FIG. 3, the transmissive self-emitting image display device 1650 may be configured to be connected to other processing units such as the control unit 1110 as a component of the floating image display device 1000 in FIG. 3.

[0120] Here, if an effect such as moving only the object such as a character to the floating image 3 on the front side after displaying both the background and the object such as a character on the transmissive self-emitting image display device 1650 is performed, an effective surprise effect video experience can be provided to the user 230.

[0121] Also, if the interior of the floating image display device 1000 is made light-shielded, the background of the transmissive self-luminous image display device 1650 will become sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, to the user 230, the transmissive self-luminous image display device 1650 appears to be a normal two-dimensional flat display rather than a transmissive display. (Since the floating image 3 in the embodiment of the present invention is displayed as an optical image of a real image in a space without a screen, if the light source of the display device 1 is turned off, the planned display position of the floating image 3 will be a space with nothing. ). Therefore, when an image is being displayed using the transmissive self-luminous image display device 1650 as if it were a general two-dimensional flat display, by suddenly displaying a character, an object, etc. as a floating image 3 in the air, an effective surprise production video experience can be provided to the user 230.

[0122] Note that the darker the interior of the floating image display device 1000 is made, the more the transmissive self-luminous image display device 1650 appears as a two-dimensional flat display. Therefore, on the inner surface of the transmissive self-luminous image display device 1650 on the side of the floating image display device 1000 (the incident surface of the image light reflected by the polarization separation member 101B into the transmissive self-luminous image display device 1650, that is, the surface of the transmissive self-luminous image display device 1650 opposite to the floating image 3), an absorption-type polarizing plate (not shown) that transmits the polarization of the image light reflected by the polarization separation member 101B and absorbs a polarization having a phase difference of about 90° from the polarization may be provided. In this way, the influence on the image light forming the floating image 3 is not so great, but the light incident from the outside into the interior of the floating image display device 1000 through the transmissive self-luminous image display device 1650 can be significantly reduced, the interior of the floating image display device 1000 can be made darker, which is preferable.

[0123] FIG. 4L is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4L is a modified example of the spatial floating image display device in FIG. 4K. The orientation of the configuration arrangement in the spatial floating image display device 1000 is different from that of the spatial floating image display device in FIG. 4K and is similar to the arrangement of the spatial floating image display device in FIG. 4F. Since the functions, operations, etc. of each configuration are the same as those of the spatial floating image display device in FIG. 4K, repeated explanations are omitted.

[0124] In the spatial floating image display device of FIG. 4L as well, after the light beam of the video light passes through the transmissive self-luminous video display device 1650, the spatial floating image 3 is formed on the user 230 side rather than the transmissive self-luminous video display device 1650.

[0125] In both the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L, from the user 230, the spatial floating image 3 is displayed overlapping in front of the image of the transmissive self-luminous video display device 1650. Here, the position of the spatial floating image 3 and the position of the image of the transmissive self-luminous video display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves the head (the position of the viewpoint), the depth of the two images can be recognized due to the parallax. Therefore, by displaying two images with different depth positions, a three-dimensional video experience can be more suitably provided to the user without the need for stereoscopic glasses or the like and with the naked eye.

[0126] FIG. 4M is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4M is provided with a second display device 1680 on the back side as viewed from the user with respect to the polarization separation member 101B of the spatial floating image display device in FIG. 4G. Since the other configurations are the same as those of the spatial floating image display device in FIG. 4G, repeated explanations are omitted.

[0127] In the configuration example shown in FIG. 4M, the second display device 1680 is provided on the back side of the display position of the spatial floating image 3, and the image display surface is directed toward the spatial floating image 3. With such a configuration, from the viewpoint of the user 230, the images displayed at two positions with different depths, namely, the image of the second display device 1680 and the spatial floating image 3, can be visually recognized in an overlapping manner. That is, it can be said that the second display device 1680 is arranged in a direction in which an image is displayed in the direction of the user 230 who views the spatial floating image 3. Although the second display device 1680 is not shown in FIG. 3, it may be configured to be connected to other processing units such as the control unit 1110 as a component of the spatial floating image display device 1000 in FIG. 3.

[0128] Note that the image light of the second display device 1680 of the spatial floating image display device 1000 in FIG. 4M is visually recognized by the user 230 after passing through the polarization separation member 101B. Therefore, in order for the image light of the second display device 1680 to more preferably pass through the polarization separation member 101B, it is desirable that the image light output from the second display device 1680 be polarized light with a polarization direction in which the polarization separation member 101B more preferably passes through. That is, it is desirable that the polarization direction of the image light output from the display device 1 be the same as the polarization direction of the polarization wave. For example, when the image light output from the display device 1 is S-polarized light, it is desirable that the image light output from the second display device 1680 is also S-polarized light. Further, when the image light output from the display device 1 is P-polarized light, it is desirable that the image light output from the second display device 1680 is also P-polarized light.

[0129] The example of the spatial floating image display device in FIG. 4M also has the same effect as the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L in that a second image is displayed behind the spatial floating image 3. However, different from the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L, in the example of the spatial floating image display device in FIG. 4M, the light beam of the image light for forming the spatial floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, and a liquid crystal display which is a two-dimensional plane display may be sufficient. The second display device 1680 may also be an organic EL display. Therefore, in the example of the spatial floating image display device in FIG. 4M, it is possible to realize the spatial floating image display device 1000 at a lower cost than the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a part of the image light output from the display device 1 may be reflected by the polarization separation member 101B and may go toward the second display device 1680. This light (a part of the image light) may be reflected again on the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, in order to prevent such stray light, an absorption type polarizing plate may be provided on the surface of the second display device 1680. In this case, the absorption type polarizing plate may be an absorption type polarizing plate that transmits the polarization wave of the image light output from the second display device 1680 and absorbs the polarization wave whose phase is 90° different from the polarization wave of the image light output from the second display device 1680. When the second display device 1680 is a liquid crystal display, there is also an absorption type polarizing plate on the image emission side inside the liquid crystal display. However, when there is a cover glass (cover glass on the image display surface side) on the further emission surface of the absorption type polarizing plate on the image emission side inside the liquid crystal display, stray light generated by the reflection of the cover glass by light from outside the liquid crystal display cannot be prevented. Therefore, it is necessary to separately provide the above-mentioned absorption type polarizing plate on the surface of the cover glass.

[0132] When displaying an image on the second display device 1680, which is a two-dimensional plane display, the spatial floating image 3 can be displayed as an image further on the user side in front of the image of the second display device 1680. At this time, the user 230 can simultaneously view two images with different depth positions. By displaying a character on the spatial floating image 3 and displaying a background on the second display device 1680, an effect can be provided such that the user 230 visually recognizes the space where the character exists as if it were three-dimensional.

[0133] In addition, after displaying both an object such as a background and a character on the second display device 1680, an effect such as moving only the object such as a character to the spatial floating image 3 on the front side can be performed, thereby providing the user 230 with an effective surprise effect in the video experience.

[0134] Next, FIG. 4N is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4N is a spatial floating image display device that employs the optical system of FIG. 2D. Similar to the example of the spatial floating image display device that employs the optical systems of FIGS. 2A to 2C, the image light that has passed through the transparent member 100 forms an image in the air as the spatial floating image 3. Also, using the sensing light of the air operation detection sensor 1351 disposed on the back side of the transparent member 100 as viewed from the user, the operation of the spatial floating image 3 by the user's finger 9004 can be detected.

[0135] In the example of the spatial floating image display device that employs the optical systems of FIGS. 2A to 2C and in the example of the spatial floating image display device that employs the optical system of FIG. 2D, the spatial floating image 3 forms an image in front of the transparent member 100, and using the sensing light of the air operation detection sensor 1351 disposed on the back side of the transparent member 100 as viewed from the user, the operation of the spatial floating image 3 by the user's finger can be detected. Therefore, the spatial floating image display device that employs the optical system of FIG. 2D has a different optical system from the spatial floating image display device in which the optical systems of FIGS. 2A to 2C are disposed on the back side of the transparent member 100 as viewed from the user.

[0136] However, the usability of the spatial floating image display device employing the optical system of FIG. 2D as seen by the user is almost the same as that of the spatial floating image display device employing the optical systems of FIGS. 2A to 2C.

[0137] Next, FIG. 4O is a diagram showing an example of the configuration of the spatial floating image display device. FIG. 4O is a diagram showing the configuration of the internal optical system visible in the spatial floating image display device 1000 of FIG. 4N. The spatial floating image display device 1000 shown in FIG. 4O is equipped with an optical system corresponding to the optical system of FIG. 2D. In the spatial floating image display device 1000 shown in FIG. 4O, it is installed horizontally so that the surface on the side where the spatial floating image 3 is formed faces upward.

[0138] That is, in FIG. 4O, in the spatial floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0139] Here, compare the configuration of FIG. 4O with the configuration of FIG. 4A and confirm the differences. In FIG. 4A, the display device 1 and the spatial floating image 3 are in a plane-symmetric relationship with respect to the surface of the polarization separation member 101. In contrast, in FIG. 4O, the display device 1 and the spatial floating image 3 are in a plane-symmetric relationship with respect to the surface of the retroreflective plate 5. Also, in the configuration of FIG. 4A, there are a retroreflective plate 2 and a λ / 4 plate 21, but these do not exist in FIG. 4O. Further, in FIG. 4A, it is more preferable to have an absorption-type polarizing plate 12, but in FIG. 4O, the absorption-type polarizing plate 12 is not particularly necessary.

[0140] That is, in order to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and replace it with the configuration of FIG. 4O, the following steps can be taken. That is, the polarization beam splitter member 101 in the configuration of FIG. 4A can be replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 can be removed from the configuration of FIG. 4A. The absorption type polarizing plate 12 may or may not be present. By performing the replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the spatial floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and it can be replaced with a spatial floating image display device equipped with the optical system of FIG. 2D. At this time, in FIGS. 4A and 4B, the polarization beam splitter member 101 may be replaced with the retroreflector 5, and in FIGS. 4C to 4G, the polarization beam splitter member 101B may be replaced with the retroreflector 5.

[0141] By doing so, a spatial floating image display device in which the optical system in the configuration of the spatial floating image display device of FIGS. 4A to 4G is replaced with the optical system of FIG. 2D can be realized. Even in these spatial floating image display devices replaced with the optical system of FIG. 2D, a spatial floating image display device with almost the same usability as the spatial floating image display device of FIGS. 4A to 4G can be realized.

[0142] <Display device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes a light source device 13 that constitutes its light source together with an image display element 11 (liquid crystal display panel). In FIG. 5, the light source device 13 is shown as a developed perspective view together with the liquid crystal display panel.

[0143] This liquid crystal display panel (image display element 11) receives, as shown by the arrow 30 in FIG. 5, an illumination light beam from the light source device 13, which is a backlight device, and has a diffusion characteristic with a sandwiching angle, that is, a characteristic similar to that of a laser beam with strong directivity (linear propagation) and the polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam according to the input video signal. The modulated video light is reflected by the retroreflector 2, transmitted through the transparent member 100, and forms a spatial floating image that is a real image (see FIG. 1).

[0144] Also, in FIG. 5, the liquid crystal display panel 11 constituting the display device 1 further includes a light direction conversion panel 54 for controlling the directivity characteristics of the emitted light beam from the light source device 13, and, if necessary, a sandwiching angle diffusion plate (not shown). That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and video light of a specific polarization modulates the intensity of light according to a video signal and is emitted (see arrow 30 in FIG. 5). As a result, a desired video is projected as light of a specific polarization with high directivity (linear propagation property) through the light direction conversion panel 54 toward the retroreflective plate 2. After being reflected by the retroreflective plate 2, it passes through toward the eyes of a monitor outside the store (space) to form a floating video 3 in the space. Note that a protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the above-described light direction conversion panel 54.

[0145] <Example 1 of Display Device> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, the liquid crystal display panel 11 and the light direction conversion panel 54 are arranged on the light source device 13 of FIG. 5. This light source device 13 is formed of, for example, plastic on the case shown in FIG. 5, and houses an LED element 201 and a light guide 203 inside. On the end face of the light guide 203, as also shown in FIG. 5 and the like, in order to convert the divergent light emitted from each LED element 201 into a substantially parallel light beam, it has a shape in which the cross-sectional area gradually increases toward the opposite side of the light receiving part, and has an action such that the divergence angle gradually decreases by total reflection a plurality of times when propagating inside, and is provided with a lens shape. On the upper surface of the display device 1, the liquid crystal display panel 11 constituting such a display device 1 is attached. Further, on one side surface (the left end face in this example) of the case of the light source device 13, an LED (Light Emitting Diode) element 201 which is a semiconductor light source and an LED substrate 202 on which its control circuit is mounted are attached. A heat sink, which is a member for cooling the heat generated by the LED element and the control circuit, may be attached to the outer surface of the LED substrate 202.

[0146] Also, on the frame (not shown) of the liquid crystal display panel attached to the upper surface of the case of the light source device 13, a liquid crystal display panel 11 attached to the frame, and further, an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11 are attached and configured. That is, the liquid crystal display panel 11, which is an image display element, together with the LED element 201, which is a solid light source, modulates the intensity of transmitted light based on a control signal from a control circuit (image control unit 1160 in FIG. 3) that constitutes an electronic device to generate a display image. At this time, since the generated image light has a narrow diffusion angle and only a specific polarization component, a novel image display device that is close to a surface-emitting laser image source driven by an image signal can be obtained. At present, it is impossible technically and for safety reasons to obtain a laser light beam of the same size as the image obtained by the above-described display device 1 using a laser device. Therefore, in this embodiment, for example, light close to the above-described surface-emitting laser image light is obtained from the light beam from a general light source equipped with an LED element.

[0147] Subsequently, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 together with FIG. 6.

[0148] Since FIGS. 6 and 7 are cross-sectional views, only one of the plurality of LED elements 201 that constitute the light source is shown, and these are converted into substantially collimated light by the shape of the light-receiving end face 203a of the light guide 203. For this reason, the light-receiving part of the light guide end face and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Note that each of these light guides 203 is formed of a light-transmissive resin such as acrylic, for example. And the LED light-receiving surface at the end of this light guide 203 has, for example, an outer peripheral surface in the shape of a conical convex obtained by rotating a parabolic cross-section. At its top, it has a concave portion with a convex portion (i.e., a convex lens surface) formed at its central portion, and at the central portion of its flat portion, it has a convex lens surface protruding outward (or it may be a concave lens surface recessed inward) (not shown). Note that the outer shape of the light-receiving portion of the light guide for mounting the LED element 201 has a parabolic surface shape forming a conical outer peripheral surface, and is set within a range of angles that can totally reflect the light emitted from the LED element in the peripheral direction inside it, or a reflecting surface is formed.

[0150] On the other hand, the LED elements 201 are respectively arranged at predetermined positions on the surface of the LED substrate 202, which is its circuit board. This LED substrate 202 is arranged and fixed with respect to the LED collimator (light-receiving end face 203a) such that the LED elements 201 on its surface are respectively located at the central portions of the aforementioned concave portions.

[0151] According to such a configuration, due to the shape of the light-receiving end face 203a of the light guide 203, the light emitted from the LED element 201 can be taken out as substantially parallel light, and the utilization efficiency of the generated light can be improved.

[0152] As described above, the light source device 13 is configured by attaching a light source unit in which a plurality of LED elements 201, which are light sources, are arranged to the light-receiving end face 203a, which is the light-receiving portion provided at the end face of the light guide 203. The divergent light beam from the LED element 201 is made into substantially parallel light by the lens shape of the light-receiving end face 203a of the light guide end face, and is guided inside the light guide 203 (in a direction parallel to the drawing) as shown by the arrow. Then, by the light beam direction conversion means 204, it is emitted toward the liquid crystal display panel 11 arranged substantially parallel to the light guide 203 (in a direction perpendicular to the front from the drawing). By optimizing the distribution (density) of this light beam direction conversion means 204 according to the shape inside or on the surface of the light guide, the uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled.

[0153] The above-described light beam direction conversion means 204 emits the light beam propagating in the light guide body toward the liquid crystal display panel 11 arranged substantially parallel to the light guide body 203 (in the direction perpendicular to the front from the drawing plane) by the shape of the surface of the light guide body or by providing, for example, portions with different refractive indices inside the light guide body. At this time, if the relative luminance ratio when comparing the luminance of the center and the peripheral portion of the screen in a state where the liquid crystal display panel 11 is facing the center of the screen and the viewpoint is placed at the same position as the diagonal dimension of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it has even better characteristics.

[0154] Note that FIG. 6 is a cross-sectional layout view for explaining the configuration and operation of the light source of the present embodiment that performs polarization conversion in the light source device 13 including the above-described light guide body 203 and the LED element 201. In FIG. 6, the light source device 13 includes, for example, a light guide body 203 provided with light beam direction conversion means 204 on its surface or inside formed of plastic or the like, an LED element 201 as a light source, a reflection sheet 205, a retardation plate 206, a lenticular lens, etc., and a liquid crystal display panel 11 provided with polarizing plates on its upper surface on the light source light incident surface and the video light emission surface.

[0155] Further, a film or sheet-shaped reflective polarizing plate 49 is provided on the light source light incident surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one-sided polarized wave (for example, P wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by the reflection sheet 205 provided on one (lower side in the figure) surface of the light guide body 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflection sheet 205 and the light guide body 203 or between the light guide body 203 and the reflective polarizing plate 49, and the light is reflected by the reflection sheet 205 and passed twice to convert the reflected light beam from P polarization to S polarization, thereby improving the utilization efficiency of the light source light as video light. The video light beam whose light intensity is modulated by the video signal in the liquid crystal display panel 11 (arrow 213 in FIG. 6) enters the retroreflective plate 2. After being reflected by the retroreflective plate 2, a spatial floating image that is a real image can be obtained.

[0156] FIG. 7 is a cross-sectional layout view for explaining the configuration and operation of the light source of this embodiment that performs polarization conversion in the light source device 13 including the light guide 203 and the LED element 201, similar to FIG. 6. Similarly, the light source device 13 is also composed of, for example, a light guide 203 provided with a light beam direction conversion means 204 on its surface or inside formed of plastic or the like, an LED element 201 as a light source, a reflection sheet 205, a retardation plate 206, a lenticular lens, and the like. On the upper surface of the light source device 13, a liquid crystal display panel 11 having polarizing plates on the light source light incident surface and the video light output surface is attached as a video display element.

[0157] Also, on the light source light incident surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, a film or sheet-shaped reflective polarizing plate 49 is provided to selectively reflect one-sided polarization waves (for example, S waves) 211 among the natural light beams 210 emitted from the LED element 201. That is, in the example of FIG. 7, the selective reflection characteristics of the reflective polarizing plate 49 are different from those in FIG. 7. The reflected light is reflected by the reflection sheet 205 provided on one side (the lower side in the figure) of the light guide 203 and then heads toward the liquid crystal display panel 11 again. A retardation plate (λ / 4 plate) is provided between the reflection sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the light is reflected by the reflection sheet 205 and passed through twice to convert the reflected light beam from S polarization to P polarization, thereby improving the utilization efficiency of the light source light as video light. The video light beam intensity-modulated by the video signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7) enters the retroreflective plate 2. After being reflected by the retroreflective plate 2, a spatial floating image that is a real image can be obtained.

[0158] In the light source devices shown in FIGS. 6 and 7, in addition to the action of the polarizing plate provided on the light incident surface of the corresponding liquid crystal display panel 11, since the reflective polarizing plate reflects one-sided polarization components, the theoretically obtained contrast ratio is the reciprocal of the cross transmittance of the reflective polarizing plate multiplied by the reciprocal of the cross transmittance obtained by the two polarizing plates attached to the liquid crystal display panel. As a result, high contrast performance can be obtained. Actually, it was experimentally confirmed that the contrast performance of the displayed image was improved by more than 10 times. As a result, a high-quality video comparable to that of self-emitting organic EL was obtained.

[0159] <Example 2 of the display device> FIG. 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing an LED, a collimator, a composite diffuser block, a light guide, etc. in a case made of, for example, plastic, and a liquid crystal display panel 11 is attached to the upper surface thereof. Also, on one side surface of the case of the light source device 13, LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and an LED substrate on which the control circuit thereof is mounted are attached, and on the outer surface of the LED substrate, a heat sink 103, which is a member for cooling the heat generated by the LED elements and the control circuit, is attached.

[0160] Also, on the liquid crystal display panel frame attached to the upper surface of the case, the liquid crystal display panel 11 attached to the frame and, further, an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11 are attached and configured. That is, the liquid crystal display panel 11, which is a liquid crystal display element, together with the LED elements 14a and 14b, which are solid light sources, modulates the intensity of transmitted light based on a control signal from a control circuit (not shown here) that constitutes an electronic device, thereby generating a display image.

[0161] <Example 3 of the display device> Subsequently, another example (Example 3 of the display device) of the specific configuration of the display device 1 will be described with reference to FIG. 9. The light source device of this display device 1 converts the divergent light beam (in which P-polarized light and S-polarized light are mixed) from the LED into a substantially parallel light beam by the collimator 18 and reflects it toward the liquid crystal display panel 11 by the reflecting surface of the reflective light guide 304. The reflected light enters the reflective polarizing plate 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizing plate 49 transmits light of a specific polarization (for example, P-polarized light) and makes the transmitted polarized light enter the liquid crystal display panel 11. Here, light of other polarizations (for example, S-polarized light) other than the specific polarization is reflected by the reflective polarizing plate 49 and heads back to the reflective light guide 304 again.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so as not to be perpendicular to the principal ray of the light from the reflecting surface of the reflective light guide 304. Then, the principal ray of the light reflected by the reflective polarizing plate 49 is incident on the transmitting surface of the reflective light guide 304. The light incident on the transmitting surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again and passes through the transmitting surface of the reflective light guide 304. The light passing through the transmitting surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, since the light incident on the reflective polarizing plate 49 again has passed through the λ / 4 plate 270 twice, the polarization is converted into a polarization (for example, P polarization) that passes through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted passes through the reflective polarizing plate 49 and is incident on the liquid crystal display panel 11. Regarding the polarization design related to polarization conversion, the polarization directions (reversing S polarization and P polarization) may be configured in the reverse manner from the above description.

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), is incident on the liquid crystal display panel 11, is luminance-modulated according to the video signal, and a video is displayed on the panel surface. A plurality of LEDs constituting the light source are shown as in the above example (however, only one is shown in FIG. 9 due to the longitudinal section), and these are attached to the collimator 18 at predetermined positions.

[0165] Note that each of the collimators 18 is formed of a light-transmissive resin such as acrylic or glass. The collimator 18 may have an outer peripheral surface with a conical convex shape obtained by rotating a parabolic cross-section. Further, a concave portion having a convex portion (i.e., a convex lens surface) may be formed at the central portion of the top of the collimator 18 (the side facing the LED substrate 102). Further, a convex lens surface protruding outward (or a concave lens surface recessed inward may also be acceptable) is provided at the central portion of the flat surface portion of the collimator 18 (the side opposite to the above-mentioned top). Note that the parabolic surface forming the conical outer peripheral surface of the collimator 18 is set within an angle range that enables total internal reflection of the light emitted from the LED in the peripheral direction inside it, or a reflecting surface is formed.

[0166] Note that the LEDs are respectively arranged at predetermined positions on the surface of the LED substrate 102, which is their circuit board. The LED substrate 102 is arranged and fixed with respect to the collimator 18 such that the LEDs on its surface are respectively located at the central portion of the top of the conical convex shape (the concave portion if there is a concave portion at the top).

[0167] According to such a configuration, among the light emitted from the LED by the collimator 18, in particular, the light emitted from the central portion thereof is condensed by the convex lens surface forming the outer shape of the collimator 18 to become parallel light. Further, the light emitted from other portions in the peripheral direction is reflected by the parabolic surface forming the conical outer peripheral surface of the collimator 18 and similarly condensed to become parallel light. In other words, according to the collimator 18 having a convex lens formed at its central portion and a parabolic surface formed at its peripheral portion, almost all of the light generated by the LED can be taken out as parallel light, and the utilization efficiency of the generated light can be improved.

[0168] Furthermore, the light converted into substantially parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Among this light, the light of a specific polarization state passes through the reflective polarizing plate 49 due to the action of the reflective polarizing plate 49, and the light of the other polarization state reflected by the action of the reflective polarizing plate 49 passes through the light guide 304 again. This light is reflected by the reflector 271 located at a position opposite to the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is converted in polarization by passing through the λ / 4 plate 270, which is a retardation plate, twice. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizing plate 49 provided on the opposite surface. Since the incident light has been subjected to polarization conversion, it passes through the reflective polarizing plate 49 and is incident on the liquid crystal display panel 11 with the polarization directions aligned. As a result, all the light from the light source can be utilized, so the geometric optical utilization efficiency of the light is doubled. In addition, since the polarization degree (extinction ratio) of the reflective polarizing plate is also multiplied by the extinction ratio of the entire system, the contrast ratio of the entire display device is significantly improved by using the light source device of this embodiment. By adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271, the light reflection and diffusion angles at each reflective surface can be adjusted. For each design, the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 may be adjusted so that the uniformity of the light incident on the liquid crystal display panel 11 becomes more suitable.

[0169] Note that the λ / 4 plate 270, which is the retardation plate in FIG. 9, does not necessarily need to have a retardation of λ / 4 with respect to the polarization incident perpendicularly to the λ / 4 plate 270. In the configuration of FIG. 9, any retardation plate that changes the phase by 90° (λ / 2) when the polarization passes through it twice is acceptable. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarization.

[0170] <Example 4 of the display device> Furthermore, another example (Example 4 of the display device) of the configuration of the optical system such as the light source device of the display device will be described with reference to FIG. 10. This is a configuration example in the case where a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Example 3 of the display device. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) for converting the diffusion characteristics in the vertical and horizontal directions (not shown in the front-back direction of the drawing) of the drawing are used on the light-emitting side of the light from the collimator 18, and the light from the collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] Note that the above optical sheet may be a single sheet instead of two sheets. In the case of a single-sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes of the front and back surfaces of the single optical sheet. Also, a plurality of diffusion sheets may be used to share the functions. Here, in the example of FIG. 10, regarding the reflection and diffusion characteristics due to the front and back surface shapes of the optical sheet 207A and the optical sheet 207B, it is advisable to optimize the design with the number of LEDs, the divergence angle from the LED substrate (optical element) 102, and the optical specifications of the collimator 18 as design parameters so that the surface density of the light beam emitted from the liquid crystal display panel 11 becomes uniform. That is, the diffusion characteristics are adjusted by the surface shapes of a plurality of diffusion sheets instead of the light guide.

[0172] In the example of FIG. 10, the polarization conversion is performed in the same manner as in Example 3 of the display device described above. That is, in the example of FIG. 10, the reflective polarizing plate 49 may be configured to have the characteristic of reflecting S-polarized light (transmitting P-polarized light). In that case, among the light emitted from the LED which is the light source, the P-polarized light is transmitted, and the transmitted light enters the liquid crystal display panel 11. Among the light emitted from the LED which is the light source, the S-polarized light is reflected, and the reflected light passes through the retardation plate 270 shown in FIG. 10. The light that has passed through the retardation plate 270 is reflected by the reflector 271. The light reflected by the reflector 271 is converted into P-polarized light by passing through the retardation plate 270 again. The polarization-converted light passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11.

[0173] Note that the quarter-wave plate 270 in Fig. 10 does not necessarily need to have a phase difference of λ / 4 with respect to the polarized light incident perpendicularly to the quarter-wave plate 270. In the configuration of Fig. 10, any wave plate that changes the phase by 90° (λ / 2) when the polarized light passes through it twice is acceptable. The thickness of the wave plate may be adjusted according to the incident angle distribution of the polarized light. Also, in Fig. 10, regarding the polarization design related to polarization conversion, the polarization state may be configured in the reverse way (reversing the S polarization and P polarization) from the above description.

[0174] The light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (displayed on the X-axis in Fig. 12(a)) and the vertical direction of the screen (displayed on the Y-axis in Fig. 12(b)) in a general TV application device. In contrast, for the diffusion characteristics of the light beam emitted from the liquid crystal display panel of this embodiment, for example, as shown in Example 1 of Fig. 12, when the viewing angle at which the luminance becomes 50% of the front view (angle 0 degrees) is set to 13 degrees, it becomes 1 / 5 compared to 62 degrees of a general TV application device. Similarly, for the viewing angle in the vertical direction, the reflection angle of the reflective light guide and the area of the reflection surface, etc. are optimized so that the upper viewing angle is suppressed to about 1 / 3 of the lower viewing angle with upper and lower non-uniformity. As a result, compared with a conventional liquid crystal TV, the amount of video light directed toward the monitoring direction is significantly improved, and the luminance becomes 50 times or more.

[0175] Furthermore, for the viewing angle characteristics shown in Example 2 of Fig. 12, when the viewing angle at which the luminance becomes 50% of the front view (angle 0 degrees) is set to 5 degrees, it becomes 1 / 12 compared to 62 degrees of a general TV application device. Similarly, for the viewing angle in the vertical direction, the reflection angle of the reflective light guide and the area of the reflection surface, etc. are optimized so that the viewing angle is suppressed to about 1 / 12 of a general TV application device with upper and lower uniformity. As a result, compared with a conventional liquid crystal TV, the amount of video light directed toward the monitoring direction is significantly improved, and the luminance becomes 100 times or more.

[0176] As described above, by using the viewing angle as the included angle, the amount of light beam directed toward the monitoring direction can be concentrated, so that the light utilization efficiency is greatly improved. As a result, even when using a liquid crystal display panel for general TV use, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant increase in brightness with the same power consumption, and it can be made into a video display device corresponding to an information display system for bright outdoors.

[0177] When using a large liquid crystal display panel, the light around the screen is directed inward so that it heads toward the viewer when the viewer is facing the center of the screen, thereby improving the overall uniformity of the screen brightness. FIG. 11 shows the convergence angles of the long side and short side of the panel when the distance L from the viewer to the panel and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in a vertical orientation, the convergence angle can be set according to the short side. For example, when using a 22-inch panel vertically and the monitoring distance is 0.8 m, if the convergence angle is set to 10 degrees, the video light from the four corners of the screen can be effectively directed toward the viewer.

[0178] Similarly, when monitoring with a 15-inch panel vertically and the monitoring distance is 0.8 m, if the convergence angle is set to 7 degrees, the video light from the four corners of the screen can be effectively directed toward the viewer. As described above, by directing the video light around the screen to the viewer who is in the optimal position for monitoring the center of the screen according to the size of the liquid crystal display panel and whether it is used vertically or horizontally, the overall uniformity of the screen brightness can be improved.

[0179] As a basic configuration, as shown in FIG. 9, a light beam with an included angle directivity characteristic is incident on the liquid crystal display panel 11 by a light source device, and the video information displayed on the screen of the liquid crystal display panel 11 is modulated in brightness according to the video signal, and the spatially floating video obtained by reflecting with a retroreflective plate is displayed outdoors or indoors through a transparent member 100.

[0180] By using the display device and the light source device according to an embodiment of the present invention described above, it is possible to realize a spatially floating video display device with higher light utilization efficiency.

[0181] <Example of video display processing in a spatial floating video display device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to FIG. 13A. In the spatial floating video display device 1000, when the back side of the spatial floating video 3 as viewed by the user is inside the housing of the spatial floating video display device 1000 and is sufficiently dark, the user visually recognizes that the background of the spatial floating video 3 is black.

[0182] Here, an example of displaying the character "Panda" 1525 in the spatial floating video 3 will be described with reference to FIG. 13A. First, the video control unit 1160 in FIG. 3 distinguishes and recognizes the pixel area for drawing the image of the character "Panda" 1525 and the transparent information area 1520 which is the background image, as shown in FIG. 13A(1), for the image including the pixel area for drawing the image of the character "Panda" 1525 and the transparent information area 1520 which is the background image.

[0183] A method of distinguishing and recognizing the character image and the background image is, for example, configured in the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the character image and the background image may be distinguished and recognized according to the superimposition relationship when these layers are combined.

[0184] Here, the video control unit 1160 recognizes the pixels for drawing an object such as a character image as information different from the black and transparent information pixels. However, it is assumed that the luminance of both the black and transparent information pixels of the pixels for drawing the object is 0. In this case, when displaying the floating video 3, there is no difference in luminance between the pixels for drawing black in the image of the character "panda" 1525 and the pixels in the transparent information area 1520 which is the background image. Therefore, in the floating video 3, as shown in Fig. 13A(2), there is no luminance in either the pixels for drawing black in the image of the character "panda" 1525 or the pixels in the transparent information area 1520, and it is visually recognized by the user as an optically identical black space. That is, the portion for drawing black in the image of the object, the character "panda" 1525, blends into the background, and only the portion of the character "panda" 1525 that is not black is recognized as a video floating in the display area of the floating video 3.

[0185] An example of the image processing of this embodiment will be described with reference to Fig. 13B. Fig. 13B is a diagram for explaining an example of image processing that more preferably solves the problem that the black image area of the object melts into the background, which was described in Fig. 13A. In Figs. 13B(1) and (2), the display state of the floating video 3 is shown on the upper side, and the input / output characteristics of the image processing of the object image are shown on the lower side, respectively. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or the memory 1109 in Fig. 3. Alternatively, it may be input from the video signal input unit 1131. Alternatively, it may be acquired via the communication unit 1132.

[0186] Here, in the state of Fig. 13B(1), the input / output characteristics of the image processing of the object image are in a linear state without any particular adjustment. In this case, the display state is the same as that in Fig. 13A(2), and the black image area of the object melts into the background. On the other hand, in Fig. 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing for the image of the object (character "panda" 1525) to be as shown in the input / output characteristics shown in the lower part.

[0187] That is, the video control unit 1160 performs image processing with input-output characteristics that have the property of converting, for the image of an object (the character "panda" 1525), the luminance value of a pixel in a low-luminance region of the pixels of the input image into an output pixel with an increased luminance value. After the image of the object (the character "panda" 1525) has undergone the image processing with the input-output characteristics, a video including the image of the object (the character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B(2), the display state of the spatial floating video 3 is such that the luminance of the pixel region that draws black in the image of the character "panda" 1525 increases. As a result, among the regions that draw the image of the character "panda" 1525, the region that draws black can also be recognized by the user without blending into the black background and being distinguishable, and the object can be more suitably displayed.

[0188] That is, by using the image processing of FIG. 13B(2), the region that displays the image of the character "panda" 1525, which is an object, can be recognized as distinct from the black background inside the housing of the spatial floating video display device 1000 through the window, and the visibility of the object is improved. Therefore, for example, before the image processing (that is, when the image of the object or the corresponding data is read from the storage unit 1170 or the memory 1109 in FIG. 3, or when the image of the object is input from the video signal input unit 1131, or when the data of the object is acquired via the communication unit 1132, etc.), even if the object includes pixels with a luminance value of 0 among the pixels that constitute the object, after being converted by the image processing with the input-output characteristics by the video control unit 1160 into an object with an increased luminance value of the pixels in the low-luminance region, it is displayed on the display device 1 and converted into the spatial floating video 3 by the optical system of the spatial floating video display device 1000.

[0189] That is, the pixels constituting the object after the image processing of the input / output characteristics are converted to a state where pixels with a luminance value of 0 are not included, and then displayed on the display device 1, and are converted into the spatial floating image 3 by the optical system of the spatial floating image display device 1000.

[0190] In the image processing of FIG. 13B(2), as a method of performing the image processing of the input / output characteristics only on the region of the image of the object (character “panda” 1525), for example, in the image processing of the video control unit 1160, the background image layer and the character image layer in front of the background image layer are configured to be processed as separate layers so that the image processing of the input / output characteristics of FIG. 13B(2) can be performed on the character image layer, and the background image layer is not subjected to the image processing.

[0191] After that, if these layers are combined, as shown in FIG. 13B(2), only the character image is subjected to the image processing with the characteristic of raising the low-luminance region of the input image. As another method, after the character image layer and the background image layer are combined, the image processing of the input / output characteristics of FIG. 13B(2) may be configured to be performed only on the region of the character image.

[0192] In addition, the input / output video characteristics used in the video processing for raising the low-luminance region of the input / output characteristics with respect to the input video are not limited to the example of FIG. 13B(2). Any video processing that raises the low luminance may be used, such as so-called brightness adjustment. Alternatively, video processing for improving visibility may be performed by controlling the gain that changes the weighting of the retinex processing as disclosed in International Publication No. 2014 / 162533.

[0193] According to the image processing of FIG. 13B(2) described above, for the region where black is drawn among the regions for drawing images of characters, objects, etc., it is possible to make the user recognize it without melting into the black of the background, and it is possible to realize a more suitable display.

[0194] In the examples of FIGS. 13A and 13B, the problems and more suitable image processing were described by taking as an example a spatial floating image display device in which the background appears black (for example, the spatial floating image display device 1000 in FIGS. 4A to 4G or the spatial floating image display device 1000 in the state where the rear window is shielded from light in FIGS. 4I and 4J). However, the said image processing is also effective in devices other than these spatial floating image display devices.

[0195] Specifically, in the spatial floating image display device 1000 in FIG. 4H or the spatial floating image display device 1000 in the state where the rear window is not shielded from light in FIGS. 4I and 4J, the background of the spatial floating image 3 is not black but the scenery on the rear side of the spatial floating image display device 1000 through the window. Also in this case, the problems described in FIGS. 13A and 13B exist in the same way.

[0196] That is, the portions of the image of the character "panda" 1525 that are drawn in black will blend into the scenery on the rear side of the spatial floating image display device 1000 through the window. Also in this case, by using the image processing in FIG. 13B(2), the portions of the image of the character "panda" 1525 that are drawn in black can be recognized separately from the scenery on the rear side of the spatial floating image display device 1000 through the window, and the visibility of the said object is improved.

[0197] That is, by using the image processing in FIG. 13B(2), the area for displaying the image of the character "panda" 1525 that is the object can be recognized separately from the scenery on the rear side of the spatial floating image display device 1000 through the window, and it becomes possible to more preferably recognize that the character "panda" that is the said object is in front of the said scenery, and the visibility of the said object is improved.

[0198] Also, in the spatial floating image display device 1000 of FIGS. 4K, 4L, and 4M, as described above, when another image (such as the image of the transmissive self-luminous image display device 1650 or the image of the second display device 1680) is displayed at a position with a different depth from the spatial floating image 3, the background of the spatial floating image 3 is not black but becomes the other image. Also in this case, the problems described with reference to FIGS. 13A and 13B similarly exist.

[0199] That is, the portion of the image of the character "panda" 1525 that is drawn in black, which is an object, will blend into the other image that is displayed at a position with a different depth from the spatial floating image 3. Also in this case, by using the image processing of FIG. 13B(2), the portion of the image of the character "panda" 1525 that is drawn in black can be recognized separately from the other image, and the visibility of the object is improved.

[0200] That is, by using the image processing of FIG. 13B(2), the area where the image of the character "panda" 1525, which is an object, is displayed can be recognized separately from the other image, and it becomes possible to more preferably recognize that the character "panda" 1525, which is the object, is in front of the other image, and the visibility of the object is improved.

[0201] An example of the video display process of this embodiment will be described with reference to FIG. 13C. FIG. 13C is an example of video display in which the spatial floating image 3 and a second image 2050, which is another image, are simultaneously displayed among the examples of video display of this embodiment. The second image 2050 may correspond to the display image of the transmissive self-luminous image display device 1650 of FIG. 4K or FIG. 4L. Also, the second image 2050 may correspond to the display image of the second display device 1680 of FIG. 4M.

[0202] That is, an example of the video display in FIG. 13C shows a specific example of the video display example of the spatial floating video display device 1000 in FIGS. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in the spatial floating video 3. The areas other than the bear character in the spatial floating video 3 are black, and the spatial floating video is transparent. Also, the second image 2050 is a background image with a plain, mountains, and the sun drawn on it.

[0203] Here, in FIG. 13C, the spatial floating video 3 and the second image 2050 are displayed at different depths. When the user 230 visually recognizes the two videos of the spatial floating video 3 and the second image 2050 in the line-of-sight direction of the arrow 2040, the user 230 can visually recognize the videos in a state where the two videos overlap. Specifically, the bear character of the spatial floating video 3 can be seen superimposed in front of the background of the plain, mountains, and sun drawn in the second image 2050.

[0204] Here, since the spatial floating video 3 forms a real image in the air, when the user 230 slightly moves the viewing point, the user 230 can recognize the depth of the spatial floating video 3 and the second image 2050 due to the parallax. Therefore, the user 230 can obtain a stronger sense of spatial floating for the spatial floating video 3 while visually recognizing the two videos in an overlapping state.

[0205] An example of the video display process of this embodiment will be described with reference to FIG. 13D. FIG. 13D(1) is a view of the spatial floating video 3 in the example of the video display of this embodiment in FIG. 13C as seen from the line-of-sight direction of the user 230. Here, a bear character is displayed in the spatial floating video 3. The areas other than the bear character in the spatial floating video 3 are black, and the spatial floating video is transparent.

[0206] FIG. 13D(2) is a view of the second image 2050 in the example of the video display of this embodiment in FIG. 13C as seen from the line-of-sight direction of the user 230. In the example of this figure, the second image 2050 is a background image with a plain, mountains, and the sun drawn on it.

[0207] FIG. 13D(3) is a diagram showing a state in which, among the examples of video display of the present embodiment in FIG. 13C, the second image 2050 and the spatial floating video 3 appear to overlap in the line-of-sight direction of the user 230. Specifically, in front of the background of the plain, mountains, and sun drawn in the second image 2050, the bear character of the spatial floating video 3 appears to overlap.

[0208] Here, when the spatial floating video 3 and the second image 2050 are displayed simultaneously, in order to more preferably ensure the visibility of the spatial floating video 3, it is desirable to pay attention to the balance of the brightness of the two videos. If the second image 2050 is too bright compared to the brightness of the spatial floating video 3, the displayed video of the spatial floating video 3 will become transparent, and the second image 2050 as the background will be strongly visible through the transparency.

[0209] Therefore, at least, the brightness per unit area of the spatial floating video 3 at the display position of the spatial floating video 3 should be greater than the brightness per unit area of the video light reaching the display position of the spatial floating video 3 from the second image 2050. The output of the light source of the spatial floating video 3, the display video brightness of the display device 1, the output of the light source of the display device for displaying the second image 2050, and the display video brightness of the said display device may be set.

[0210] Note that since this condition only needs to be satisfied when the spatial floating video 3 and the second image 2050 are displayed simultaneously, when switching from the first display mode in which only the second image 2050 is displayed without displaying the spatial floating video 3 to the second display mode in which the spatial floating video 3 and the second image 2050 are displayed simultaneously, the output of the light source of the display device for displaying the second image 2050 and / or the display video brightness of the said display device may be reduced to perform control to reduce the brightness of the second image 2050. These controls may be realized by the control unit 1110 in FIG. 3 controlling the display device 1 and the display device for displaying the second image 2050 (the transmissive self-emitting video display device 1650 in FIG. 4K or FIG. 4L or the second display device 1680 in FIG. 4M).

[0211] In addition, when performing control to reduce the brightness of the second image 2050 in the switching from the above-described first display mode to the above-described second display mode, the brightness may be uniformly reduced for the entire screen of the second image 2050. Alternatively, without uniformly reducing the brightness for the entire screen of the second image 2050, the portion where the object is displayed in the spatial floating image 3 may be in a state with the highest brightness reduction effect, and the brightness reduction effect may be gradually eased in the periphery thereof. That is, it is sufficient to ensure the visibility of the spatial floating image 3 as long as the brightness reduction of the second image 2050 is realized only for the portion where the spatial floating image 3 is superimposed on the second image 2050 and visually recognized.

[0212] Here, since the spatial floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes the viewing point, due to the parallax, the superimposition position of the spatial floating image 3 with respect to the second image 2050 changes. Therefore, when reducing the brightness non-uniformly for the entire screen of the second image 2050 in the switching from the above-described first display mode to the above-described second display mode, it is not desirable to sharply reduce the brightness based on the outline of the object displayed in the spatial floating image 3. It is desirable to perform gradation processing of the brightness reduction effect in which the brightness reduction effect gradually changes depending on the position as described above.

[0213] In the spatial floating image display device 1000 in which the position of the object displayed in the spatial floating image 3 is substantially at the center of the spatial floating image 3, the position with the highest brightness reduction effect in the gradation processing of the brightness reduction effect may be at the center position of the spatial floating image 3.

[0214] According to the video display process of the present embodiment described above, the user 230 can more suitably view the spatial floating image 3 and the second image 2050.

[0215] When displaying the floating image 3 in space, it may be controlled so as not to display the second image 2050. Since the visibility of the floating image 3 in space is enhanced when the second image 2050 is not displayed, it is suitable for a floating image display device 1000 in a space where the user must surely visually recognize the floating image 3 in space when the floating image 3 in space is displayed.

[0216] <Example 2> As a second embodiment of the present invention, an example of another configuration example of the floating image display device in space will be described. Note that the floating image display device according to the present embodiment is obtained by changing the optical system stored in the floating image display device described in the first embodiment to the optical system shown in FIG. 14(1) or FIG. 14(2). In the present embodiment, the differences from the first embodiment will be described, and the description of the same configuration as that of the first embodiment will be omitted. In the following description of the present embodiment, a predetermined polarization and the other polarization are polarizations of polarization waves having a phase difference of 90° from each other.

[0217] FIG. 14(1) is an example of the optical system and the optical path according to the present embodiment. The optical system shown in FIG. 14(1) is configured by bringing the display device 1 closer to the polarization separation member 101B in the optical system of FIG. 2C, making the entire optical system more compact. In FIG. 14(1), the detailed description of the configuration denoted by the same reference numeral as in FIG. 2C will be omitted.

[0218] In FIG. 14(1), similar to FIG. 2C, the video light of a predetermined polarization (P polarization in the figure) emitted from the display device 1 travels in the vertical direction from the video display surface of the display device 1. Here, the polarization separation member 101B selectively transmits a predetermined polarization (P polarization in the figure) emitted from the display device 1 and reflects the other polarization (S polarization in the figure), similar to FIG. 2C.

[0219] Therefore, the video light of a predetermined polarization (P-polarization in the figure) that travels vertically from the video display surface of the display device 1 passes through the polarization separation member 101B and reaches the retroreflective plate 2 to which the λ / 4 plate 21 is attached. The video light that is retroreflectively reflected by the retroreflective plate 2 and travels again toward the polarization separation member 101B has passed through the λ / 4 plate 21 twice, and thus is converted from the predetermined polarization (P-polarization in the figure) at the time of emission from the display device 1 to the other polarization (S-polarization in the figure). Since the video light that travels again toward the polarization separation member 101B is the other polarization (S-polarization in the figure), it is reflected by the polarization separation member 101B toward the position where the user should be. The traveling direction of the video reflected by the polarization separation member 101B is determined based on the angle at which the polarization separation member 101B is arranged.

[0220] In the example of FIG. 14(1), the video light that travels toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The video light reflected by the polarization separation member 101B forms a spatial floating image 3A. The spatial floating image 3A can be suitably viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of the retroreflective reflection by the retroreflective plate 2, the optical path length from when the video light emitted from the display device 1 reaches the retroreflective plate 2 and the optical path length from when the video light emitted from the retroreflective plate 2 reaches the formation position of the spatial floating image 3A are in an equal relationship. Due to this relationship, the formation position of the spatial floating image 3A in the traveling direction of the video light reflected by the polarization separation member 101B is determined.

[0222] In the example of FIG. 14(1), the display device 1, the polarization separation member 101B, and the retroreflective plate 2 are arranged so as to be closer than in the example of FIG. 2C. Thereby, it is realized that the entire optical system is configured more compactly. However, the amount by which the spatial floating image 3A protrudes from the optical system of FIG. 14(1) is not so large. For example, as an index of the amount by which the spatial floating image 3A protrudes from the optical system, the distance from the position where the light beam at the center portion of the video light is reflected by the polarization separation member 101B to the position where the video light forms the spatial floating image 3A is shown in the figure (L1 in the example of FIG. 14(1)).

[0223] Regarding the polarization design in the optical system of FIG. 14(1), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, a predetermined polarization of the video light emitted from the display device 1 may be set as S-polarized light, and regarding the reflection characteristics of the polarization separation member 101B, the characteristics of P-polarized light and S-polarized light may be interchanged. In this case, although both the P-polarized light and S-polarized light shown in the figure are reversed, the optical design such as the optical path can be realized in exactly the same way.

[0224] Next, FIG. 14(2) shows another example of the optical system and the optical path according to the present embodiment. The optical system of FIG. 14(2) modifies the configuration in the optical system of FIG. 14(1) in order to make the amount of the spatial floating image jumping out from the optical system larger while realizing the same compactness as the optical system of FIG. 14(1). In FIG. 14(2), for the configurations denoted by the same reference numerals as those in FIG. 14(1), repeated detailed descriptions are omitted.

[0225] In FIG. 14(2), similar to FIG. 14(1), the video light of a predetermined polarization (P-polarized light in the figure) emitted from the display device 1 travels in the vertical direction from the video display surface of the display device 1. Here, the polarization characteristics of the polarization separation member 101B are different from those in FIG. 14(1) by 90 degrees in terms of arrangement. The video light of a predetermined polarization (P-polarized light in the figure) that has traveled in the vertical direction from the video display surface of the display device 1 passes through the polarization separation member 101B.

[0226] Here, different from FIG. 14(1), instead of the retroreflector 2 with the λ / 4 plate 21 attached, a specular reflector 4 with the λ / 4 plate 21B attached is arranged at the position where the video light passes through the polarization separation member 101B. Here, the reflection in the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the video light transmitted through the polarization separation member 101B is specularly reflected by the specular reflection plate 4 to which the λ / 4 plate 21B is attached. The video light that is specularly reflected by the specular reflection plate 4 and travels again toward the polarization separation member 101B has been converted from a predetermined polarization (P polarization in the figure) at the time of emission from the display device 1 to the other polarization (S polarization in the figure) by passing through the λ / 4 plate 21 twice. Since the video light traveling toward the polarization separation member 101B again is the other polarization (S polarization in the figure), it is reflected by the polarization separation member 101B.

[0228] Here, since the orientation of the polarization separation member 101B in FIG. 14(2) is different from that in FIG. 14(1), the video light reflected by the polarization separation member 101B travels in the direction opposite to the position where the user should be. At the destination where the video light reflected by the polarization separation member 101B travels, the retroreflective plate 2 to which the λ / 4 plate 21C is attached is arranged. The video light is retroreflectively reflected by the retroreflective plate 2. The video light that is retroreflectively reflected by the retroreflective plate 2 and travels again toward the polarization separation member 101B has been converted from the other polarization (S polarization in the figure) to the predetermined polarization (P polarization in the figure) again by passing through the λ / 4 plate 21C twice.

[0229] The video light traveling toward the polarization separation member 101B again is the predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and travels straight toward the position where the user should be. The video light passing through the polarization separation member 101B forms the spatial floating image 3B. The spatial floating image 3B can be suitably viewed by the user from the direction of arrow A.

[0230] Here, also in FIG. 14(2), similar to FIG. 14(1), due to the characteristics of the retroreflective reflection by the retroreflective plate 2, the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflective plate 2 and the optical path length from the retroreflective plate 2 to the point where the video light emitted from the retroreflective plate 2 reaches the formation position of the spatial floating image 3B are equal. Due to this relationship, the formation position of the spatial floating image 3B in the traveling direction of the video light transmitted through the polarization separation member 101B is determined.

[0231] In Fig. 14(2), the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflector 2 is longer than the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflector 2 in Fig. 14(1). In the optical system of Fig. 14(2), an optical path that reciprocates between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of Fig. 14(1), is added to the optical path length from the display device 1 to the point where the video light reaches the retroreflector 2.

[0232] As a result, in the optical system of Fig. 14(2), the distance (L2 in the example of Fig. 14(2)) from the position where the light rays in the central portion of the video light pass through the polarization separation member 101B to the position where the video light forms the spatial floating image 3B is much longer than the distance (L1 in the example of Fig. 14(1)) from the position where the light rays in the central portion of the video light are reflected by the polarization separation member 101B to the position where the video light forms the spatial floating image 3A in the optical system of Fig. 14(1).

[0233] Regarding the polarization design in the optical system of Fig. 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, a predetermined polarization of the video light emitted from the display device 1 may be set as S-polarized light, and the reflection characteristics of the polarization separation member 101B may have the characteristics of P-polarized light and S-polarized light interchanged. In this case, although both the illustrated P-polarized light and S-polarized light are reversed, the optical design such as the optical path can be realized in exactly the same way.

[0234] According to the optical systems of Figs. 14(1) and 14(2) in the second embodiment of the present invention described above, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to make the amount by which the spatial floating image jumps out from the optical system larger while being a more compact optical system.

[0235] When incorporating the optical system of FIG. 14(1) or FIG. 14(2) into the spatial floating image display device, it can be realized by replacing the optical system in the spatial floating image display device described in Example 1 with the optical system of FIG. 14(1) or FIG. 14(2). Specifically, the optical system of FIG. 14(1) may be replaced with the optical system of the spatial floating image display device in FIGS. 4E, 4F, 4G, 4H, 4I, 4J, 4K, 4L, or 4M. In this case, since the optical system becomes more compact, it is possible to make the housing of the spatial floating image display device in each figure smaller.

[0236] Specifically, the optical system of FIG. 14(2) may also be replaced with the optical system of the spatial floating image display device in FIGS. 4E, 4F, 4G, 4K, or 4L. In this case, it is possible to make the amount by which the spatial floating image protrudes from the optical system larger. Also, since the optical system becomes more compact, it is possible to make the housing of the spatial floating image display device in each figure smaller.

[0237] <Example 3> As Example 3 of the present invention, a configuration example of a spatial floating image display device will be described. The basic configuration of the spatial floating image display device in Example 3 can be applied in the same manner as the configurations in Example 1 and Example 2. The spatial floating image display device in Example 3 can be based on, for example, FIG. 2A in terms of the configuration of the optical system, and based on, for example, FIG. 4A in terms of the configuration of the housing, in other words, the main body. Also, Example 3 is a system having a spatial floating image display device and external devices connected thereto (which may be described as a spatial floating image display system, etc.). The external devices include, for example, portable terminals such as smartphones, tablets, and wearable terminals (such as smartwatches) owned by users. The portable terminals may also be referred to as mobile information processing terminal devices.

[0238] The basic functions and the like realized in the spatial floating image display device and system of Embodiment 3 are as follows. This spatial floating image display system communicatively connects and cooperates a spatial floating image display device and a user's mobile terminal. This system has a function of causing a spatial floating image display device to display, as a spatial floating image 3, an image / video that the user has, for example, an image (which may be referred to as a target image or the like) that is in the mobile terminal and that the user is displaying on the screen of the mobile terminal. For this purpose, the user performs an operation / movement so that the mobile terminal contacts the spatial floating image 3 (a predetermined imaging position and display range corresponding thereto) of the spatial floating image display device. This operation / movement is a predetermined operation / movement and may be described as an insertion operation / contact operation or the like. The spatial floating image display device determines and detects the contact operation using a sensor or the like. In this embodiment, for example, while the mobile terminal issues a display request based on a user input operation, the user performs a contact operation so that the mobile terminal is inserted into and contacts the position of the plane of the spatial floating image 3.

[0239] Based on the contact operation, the spatial floating image display device or the mobile terminal generates a display request / display instruction for causing the target image on the mobile terminal side to be displayed as the spatial floating image 3. In other words, the spatial floating image display device / mobile terminal generates such a display request / display instruction when it determines that a predetermined contact operation that satisfies a predetermined condition regarding the contact between the mobile terminal and the spatial floating image 3 has been performed. In particular, the video control unit 1160 in FIG. 3 of the spatial floating image display device receives the display request and receives / acquires the data of the target image from the mobile terminal side by communication. Then, the spatial floating image display device executes video processing of the display device 1 so that the target image is displayed as the spatial floating image 3.

[0240] This function may be as follows. When the user's mobile terminal makes the above contact operation, that is, immediately before, during, or immediately after the contact, it issues the above display request. The spatial floating image display device receives, permits / enables the display request from the mobile terminal based on the above contact operation, and executes a process of displaying the target image as the spatial floating image 3 in the same manner as above. The display request may be set with the transmission of the data of the target image. Although there is a difference in whether the display request is generated by the spatial floating image display device or the mobile terminal, in either case, the same applies to automatically displaying the target image as the spatial floating image 3 triggered by a predetermined contact operation.

[0241] This function may be as follows. First, the mobile terminal does not issue a display request, and the user touches the mobile terminal to the spatial floating image 3. When the spatial floating image display device determines / detects a predetermined contact operation, it transmits information such as confirmation, that is, information indicating that the image display by this function is possible, to the mobile terminal via communication, stating that it accepts the display request. When the mobile terminal receives the information such as the confirmation, it transmits the data of the target image to the spatial floating image display device. The data of the target image may be accompanied by a display request. When the spatial floating image display device receives the data of the target image, it displays the target image as the spatial floating image 3.

[0242] [Spatial Floating Image Display System] Figure 15 shows the configuration of system 3000, which is the spatial floating video display system of Example 3. This system 3000 is a system in which the spatial floating video display device 1000 and the mobile terminal 2000 of user 230 are connected via a communication network such as LAN 3020 and / or the Internet 3010. In Figure 15, the housing 1190 of the spatial floating video display device 1000, in other words, the main body, and a part of the spatial floating video 3 and the communication unit 1132 (Figure 3) are schematically illustrated. Each component as shown in Figure 3 above is mounted on the housing 1190. Note that the control unit 1110, the communication unit 1132, etc. in Figure 3 may be mounted inside the housing 1190 or outside the housing 1190. Outside the housing 1190, a camera of the imaging unit 1180, an air operation detection sensor 1351, a microphone, a speaker, etc. may be installed.

[0243] This system 3000 communicates between the mobile terminal 2000 and the spatial floating video display device 1000 using an arbitrary communication interface. The mobile terminal 2000 and the spatial floating video display device 1000 may be directly communicatively connected, for example, by short-range wireless communication, or may be communicatively connected with the Internet 3010 or the like intervening between the mobile terminal 2000 and the spatial floating video display device 1000. Note that in this system 3000, since image data and the like are exchanged between the mobile terminal 2000 and the spatial floating video display device 1000, a certain communication speed is required. Therefore, in the example of Figure 15, the mobile terminal 2000 and the spatial floating video display device 1000 are connected by LAN 3020. That is, the mobile terminal 2000 and the spatial floating video display device 1000 are connected to the same network via a router or the like and are set so that they can identify each other using an IP address or the like. This is not the only way, and communication by mirroring or the like is also possible.

[0244] The mobile terminal 2000 has an application 2010 and an image 2020. The application 2010 is any application program that displays, for example, the image 2020 or the like on the screen of the display of the mobile terminal 2000. The application 2010 may be a general OS or application, or may be a dedicated application for cooperating and communicating with the spatial floating image display device 1000 of the system 3000 in the third embodiment. The data of the application 2010 may be distributed from the spatial floating image display device 1000, or may be distributed from a server or the like on the Internet 3010. The image 2020 is image data to be a target image. The image 2020 may be held in a server or the like on the Internet 3010, or may be distributed from the spatial floating image display device 1000. The image 2020 may be a still image or a moving image. The image 2020 may also be data in the form of a program or the like.

[0245] The application 2010 of the mobile terminal 2000 may create a display request 2030 for displaying the target image (image 2020) as the spatial floating image 3, and transmit it to the spatial floating image display device 1000.

[0246] FIG. 16 shows an overview of the functions of the spatial floating image display system according to the third embodiment. (1). First, the user 230 has a target image 1601 or the like on his / her mobile terminal 2000. The target image 1601 is an image to be displayed as the spatial floating image 3.

[0247] (2). The user 230 performs a contact operation 1600 of inserting and contacting the mobile terminal 2000 with respect to the display range 3R of the spatial floating image 3 of the spatial floating image display device 1000. At this time, the mobile terminal 2000 may transmit a display request 2030. The display request 2030 is a request or instruction for displaying the target image 1601 as the spatial floating image 3.

[0248] (3) When the spatial floating image display device 1000 detects the contact operation 1600, it causes the target image 1601 of the mobile terminal 2000 to be displayed as an image 1602 that is a spatial floating image 3. At this time, the spatial floating image display device 1000 may receive the display request 2030 and acquire the target image 1601 of the mobile terminal 2000 based on the display request 2030. The spatial floating image display device 1000 may transmit some response to the display request 2030 to the mobile terminal 2000.

[0249] FIG. 17 shows a configuration example of the mobile terminal 2000. The mobile terminal 2000 includes a control unit 20011, a display panel 20012, an external power input IF (interface) 20013, a power supply 20014, a secondary battery 20015, a storage unit 20016, a video control unit 20017, an attitude sensor 20018, an operation input unit 20019, a communication unit (including an antenna) 20020, an audio output unit (including a speaker) 20021, an audio input unit (including a microphone) 20022, a video signal input unit 20023, an audio signal input unit 20024, an imaging unit 20025, a memory 20026, a non-volatile memory 20027, etc. in a housing 2090. These components are connected by an architecture such as a bus.

[0250] The control unit 20011 includes a processor or the like that controls the entire mobile terminal 2000 and each unit. The control unit 20011 expands data such as programs stored in the non-volatile memory 20027 or the storage unit 20016 into the memory 20026 and executes processing according to the programs. Thereby, various functions are realized. The control unit 20011 may perform arithmetic processing based on information acquired from each unit within the mobile terminal 2000 in cooperation with the programs stored in the memory 20026. The application 2010 in FIG. 15 is executed, for example, by the control unit 20011 or the video control unit 20017.

[0251] The memory 20026 stores video data for display on the display panel 20012, control data for the mobile terminal 2000, and the like. The control unit 20011 may read out programs of various software from the storage unit 20016 and the like, expand and store them in the memory 20026. The non-volatile memory 20027 and the storage unit 20016 hold various data and various information used in the mobile terminal 2000. The data and information stored in the non-volatile memory 20027 and the storage unit 20016 include, for example, various operation data for display on the display panel 20012, display icons, object data for the user 230 to operate, and layout information.

[0252] The display panel 20012 is a display unit, for example, a touch panel, and is a display means and an input means for receiving touch operation inputs. An image or the like is displayed on the screen of the display panel 20012. The display panel 20012 is provided with a touch sensor and receives touch operation inputs by the finger of the user 230 or the like. As the display panel 20012, a liquid crystal panel, an organic EL panel, or the like can be applied.

[0253] The communication unit 20020 is a device on which various communication interfaces are implemented. As the communication interface / communication method, the communication unit 20020 implements, for example, a mobile communication interface such as 4G or 5G, a wireless LAN communication interface such as Wi-Fi (registered trademark), and a short-range communication interface such as Bluetooth (registered trademark) or NFC. The communication unit 20020 of the mobile terminal 2000 can communicate with the communication unit 1132 (FIG. 3) of the spatial floating video display device 1000 using these communication methods. Similarly, from the perspective of the spatial floating video display device 1000, the communication unit 1132 has those communication interfaces / communication methods implemented. The communication unit 20020 can communicate with a communication device connected to the Internet 3010, such as a wireless base station, by any of these communication methods. Thereby, the mobile terminal 2000 can communicate with a server or the like connected to the Internet 3010.

[0254] The power supply 20014 converts the AC current input from the outside through the external power supply input IF (interface) 20013 into a DC current, and supplies the necessary DC current to each part of the mobile terminal 2000. The secondary battery 20015 stores the power supplied from the power supply 20014. The secondary battery 20015 supplies power to each part that requires power when power is not supplied from the outside through the external power supply input IF 20013.

[0255] The video signal input unit 20023 connects to an external video output device and inputs video data. Various digital video input interfaces can be applied to the video signal input unit 20023. For example, a video input interface of the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface of the DVI (Digital Visual Interface) standard, or a video input interface of the DisplayPort standard, etc. can be applied. Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The video signal input unit 20023 may also be various USB interfaces, etc.

[0256] The audio signal input unit 20024 connects to an external audio output device and inputs audio data. An audio input interface of the HDMI standard, an optical digital terminal interface, or a coaxial digital terminal interface, etc. can be applied to the audio signal input unit 20024. The audio signal input unit 20024 may also be various USB interfaces, etc. In the case of an interface of the HDMI standard, the video signal input unit 20023 and the audio signal input unit 20024 may be configured as an integrated interface with integrated terminals and cables.

[0257] The voice output unit 20021 can output voice based on the voice data input to the voice signal input unit 20024 or the voice data stored in the storage unit 20016. The voice output unit 20021 may be composed of a speaker or may have a headphone jack. Also, the voice output unit 20021 may output built-in operation sounds or error warning sounds. The voice output unit 20021 may adopt a configuration that outputs a digital signal to an external device, such as the Audio Return Channel function defined in the HDMI standard.

[0258] The voice input unit 20022 including a microphone picks up the sounds around the mobile terminal 2000 and converts them into signals to generate voice signals. The voice input unit 20022 may record the voice of a person, such as the voice of the user 230, with the microphone to generate a voice signal, and the control unit 20011 or the like may perform voice recognition processing on the voice signal to obtain character information.

[0259] The imaging unit 20025 is, for example, a camera having an image sensor. The mobile terminal 2000 may be provided with a camera (in-camera) on the front side, which is, for example, on the display panel 20012 side in the housing 2090, or may be provided with a camera (out-camera) on the opposite back side. In this embodiment, the imaging unit 20025 has both an in-camera and an out-camera.

[0260] The storage unit 20016 is a storage device that records various types of data such as video data, image data, audio data, and various types of information such as programs. The storage unit 20016 may be composed of, for example, a magnetic recording medium recording device such as a hard disk drive (HDD), or a semiconductor element memory such as a solid state drive (SSD). In the storage unit 20016, for example, various types of data such as video data, image data, audio data, and various types of information such as programs may be recorded in advance at the time of product shipment. Also, the storage unit 20016 may record various types of data and various types of information acquired from an external server or the like via the communication unit 20020. The image data and the like recorded in the storage unit 20016 are output and displayed on the display panel 20012 through the processing by the video control unit 20017. The mobile terminal 2000 may output and transmit various types of data and information recorded in the storage unit 20016 to an external server or the like via the communication unit 20020.

[0261] The video control unit 20017 performs various controls regarding the video signal input to the display panel 20012. The video control unit 20017 may be referred to as a video processing circuit, a video processing unit, an image processing unit, etc. The video control unit 20017 may be composed of hardware such as, for example, an ASIC, an FPGA, a video processor, etc. The video control unit 20017 performs video switching control such as which video signal among the video signals stored in, for example, the memory 20026 or the video signals (video data) input to the video signal input unit 20023 is to be input to and displayed on the display panel 20011. Also, the video control unit 20017 may perform control to perform image processing on the video signals input from the video signal input unit 20023 or the video signals stored in the memory 20026. Examples of the image processing include scaling processing such as enlarging, reducing, and deforming an image, brightness adjustment processing for changing the brightness, contrast adjustment processing for changing the contrast curve of an image, and retinex processing for decomposing an image into light components and changing the weighting for each component. The video control unit 20017 may include a video memory or the like that holds video data for input to the display panel 20012.

[0262] The posture sensor 20018 is a sensor composed of a combination of a gyro sensor, a gravity sensor, an acceleration sensor, a geomagnetic sensor, etc., and can detect the posture of the mobile terminal 2000. The posture can be expressed, for example, by the directions and angles of three orthogonal axes (X, Y, Z) in space. Based on the posture detection result of the posture sensor 20018, the control unit 20011 may control the operations of each part. In addition, the mobile terminal 2000 may be provided with a GPS receiver, a proximity sensor, an illuminance sensor, a distance measurement sensor, etc.

[0263] The operation input unit 20019 is a device for the user 230 to perform operation input, and includes, for example, a power button and a volume button.

[0264] [Video display method] FIG. 18 (FIGS. 18A, etc.) is an explanatory diagram of a video display method by cooperation between the space floating video display device 1000 and the mobile terminal 2000 in the system 3000 of Embodiment 3. In FIGS. 18A, etc., it shows a state where the user 230 makes a display request 2030 from the mobile terminal 2000 to the space floating video display device 1000 by bringing the mobile terminal 2000 into contact with the display range 3R of the space floating video 3 of the space floating video display device 1000. FIG. 18A is a perspective view, FIG. 18B is a top view (XY plane view) seen from above, FIG. 18C is a side view (YZ plane view) seen from the side, and FIG. 18D is a front view (XZ plane view) seen from the front of the device.

[0265] In FIG. 18A, the upper surface of the housing 1190 has an opening 1200, and in the opening 1200, a transparent member 100 (e.g., a glass plate) and a polarization separation member 101 are provided in the same manner as in FIG. 2A, for example. The display range 3R represents a predetermined position and plane within the space where the spatial floating image 3, which is a real image, is formed. In FIG. 18A, the spatial coordinate system is represented by (X, Y, Z). The X-axis and the X direction are the left-right direction (the first horizontal direction) as viewed from the user 230 (a viewpoint not shown at the origin of the direction of arrow A). The Y-axis and the Y direction are the front-rear direction (the depth direction, the second horizontal direction) as viewed from the user 230. The Z-axis and the Z direction are the up-down direction (the vertical direction) as viewed from the user 230. Also, the coordinate system in the spatial floating image 3 is shown by (x, y, z). The x direction is the horizontal direction of the screen, the y direction is the vertical direction of the screen, and the z direction is the depth direction.

[0266] Direction 1801 shows an example of the direction (insertion direction) in which the portable terminal 2000 is inserted into the display range 3R of the spatial floating image 3 during the contact operation by the user 230, and in this example, it is the Y direction. In FIG. 18A, an example of the location where the portable terminal 2000 and the display range 3R come into contact is shown as the contact location 1802. In this example, the contact location 1802 is near the lower right of the display range 3R.

[0267] In the embodiment of FIG. 18, a display request 2030 from the mobile terminal 2000 to the spatial floating image display device 1000 is made by the contact between the mobile terminal 2000 and the display range 3R. In other words, immediately before, during, or immediately after the contact, the display request 2030 is transmitted from the mobile terminal 2000 to the spatial floating image display device 1000. For example, the spatial floating image display device 1000 determines and detects this contact, and when detecting this contact, receives the display request 2030 transmitted from the mobile terminal 2000. That is, the spatial floating image display device 1000 permits the target image to be displayed as the spatial floating image 3 based on the display request 2030. The spatial floating image display device 1000 receives the image data of the target image associated with the display request 2030 from the mobile terminal 2000 by transfer. The spatial floating image display device 1000 controls the display device 1 based on the acquired image data to display the target image as the spatial floating image 3. In other words, the spatial floating image display device 1000 converts the target image specified by the user 230 into a spatial floating image and displays it as the spatial floating image 3.

[0268] The display request 2030 transmitted from the mobile terminal 2000 is possible with the various communication methods described above, and the details are not limited. In this example, it is assumed that the display request 2030 is transmitted by wireless communication in the LAN 3020, but it is not limited to this, and a wired connection may be used, or the mobile terminal 2000 and the spatial floating image display device 1000 may be connected by infrared communication or the like.

[0269] The determination and detection of the contact between the mobile terminal 2000 and the display range 3R will be described later, and examples include those by the air operation detection sensor 1351 etc. (FIG. 3), those by the imaging unit 1180 (camera), etc. In addition, the determination and detection of the contact may be performed using various sensors attached inside and outside the housing 1190, or various sensors included in the mobile terminal 2000, for example, a gyro sensor, or a combination of these.

[0270] As described above, when the contact operation of the mobile terminal 2000 with respect to the display range 3R is determined and detected, the spatial floating image display device 1000 recognizes the display request 2030 from the mobile terminal 2000, and the spatial floating image display device 1000 starts displaying the target image of the spatial floating image. As described above, as a feature in the third embodiment, by simply bringing the mobile terminal 2000 into contact with and placing it within the display range 3R of the spatial floating image 3, the target image that is the spatial floating image can be displayed.

[0271] [Contact operation] FIG. 19 (FIGS. 19A, etc.) shows the direction of posture change and the like when the mobile terminal 2000 is brought into contact with the display range 3R of the spatial floating image 3. The direction of posture change 1901 of the mobile terminal 2000 has various directions as shown in the figure. In FIG. 19A, in a perspective view, it represents the rotation angles of the three axes of the pitch angle α, roll angle β, and yaw angle γ. In this embodiment, basically, during the contact operation, the posture of the mobile terminal 2000 may be arbitrary, that is, the states of these angles (α, β, γ) may be in any state. This simplifies the contact operation by the user 230 and provides high convenience.

[0272] FIG. 19B is a plan view in the XY plane, and shows an example in which when performing the contact operation, the posture of the mobile terminal 2000 is set to a posture rotated within the horizontal plane (XY plane). Here, for the sake of explanation, the representative position and orientation of the mobile terminal 2000 are, as in FIG. 19A, the front surface of the housing, particularly the central position P1 on the display panel 20012, and the orientation R1 along the long side of the front surface (the same axis as the roll angle β). In this example, the orientation R1 of the mobile terminal 2000 at the time of contact and the corresponding insertion direction 1801 are rotated around the Z axis (yaw angle γ) and are inclined obliquely to the left with respect to the Y axis. The direction of posture change 1901 is the rotation direction around the Z axis (yaw angle γ). With respect to the display range 3R, the vicinity of the upper side of the housing of the mobile terminal 2000 is in contact (contact point 1802). The user 230 inserts the mobile terminal 2000 held in the right hand, for example, near the center of the display range 3R.

[0273] FIG. 19C is a YZ plane view, showing an example in which the posture of the mobile terminal 2000 is rotated within the YZ plane during a contact operation. In this example, the orientation R1 of the mobile terminal 2000 at the time of contact and the corresponding insertion direction 1801 are rotated around the X axis (pitch angle α) and are inclined diagonally downward with respect to the Y axis. The posture change direction 1901 is the rotation direction around the X axis (pitch angle α). This is the case where the insertion direction 1801 is perpendicular to the display range 3R. In the display range 3R, the vicinity of the upper side of the housing of the mobile terminal 2000 is in contact (contact point 1802).

[0274] FIG. 19D is an XZ plane view, showing an example in which the posture of the mobile terminal 2000 is rotated within the XZ plane during a contact operation. In this example, the orientation R1 of the mobile terminal 2000 at the time of contact and the corresponding insertion direction 1801 are rotated around the Y axis (roll angle β) and the flat plate of the housing stands vertically. The posture change direction 1901 is the rotation direction around the Y axis (roll angle β).

[0275] FIG. 19E is a YZ plane cross-sectional view, showing another example of the posture of the mobile terminal 2000 during a contact operation. This example shows a case where the screen of the display panel 20012 on the front surface of the mobile terminal 2000 faces downward during the contact operation. FIG. 19E shows an example applied to the configuration of FIG. 4A as the configuration of the spatial floating image display device 1000. Video light is emitted obliquely upward from the opening 1200 of the transparent member 100 on the upper surface (XY plane) of the housing 1190 to form the spatial floating image 3. The plane (xy plane) of the display range 3R of the spatial floating image 3 is arranged at an angle of about 45 degrees from the XY plane. In FIG. 19E, illustration of the air operation detection sensor 1351 and the like is omitted (described later).

[0276] When performing the contact operation, user 230 turns the mobile terminal 2000 so that the screen 1902 faces downward (direction 1903), and inserts it in the Y direction (positive in the depth direction) as the insertion direction 1801 with respect to the display range 3R. FIG. 19E shows a state (mobile terminal 2000b) in which the mobile terminal 2000 has contacted the display range 3R and passed through to the back side. In this state, the screen 1902 of the mobile terminal 2000b faces downward (direction 1903). Such a contact operation may be used.

[0277] FIG. 19F is a YZ plane cross-sectional view, showing another example of the posture of the mobile terminal 2000 during the contact operation. In this example, the screen 1902 of the display panel 20012 on the front surface of the mobile terminal 2000 is oriented along the plane (xy plane) of the display range 3R of the spatial floating image 3 during the contact operation. User 230 operates the mobile terminal 2000 so that the screen 1902 is overlapped with the display range 3R facing each other. In FIG. 19F, the state where the screen 1902 of the mobile terminal 2000 is in contact with the display range 3R is maintained.

[0278] Also, the predetermined contact operation may be determined based on the time (referred to as the contact time) during which the mobile terminal 2000 is in contact with the display range 3R. That is, the spatial floating image display device 1000 counts the contact time, and determines that a predetermined contact operation has been performed when the contact time is equal to or longer than a certain time as a determination condition.

[0279] FIG. 20 (FIG. 20A, etc.) is an explanatory diagram showing that when performing a contact operation to insert the mobile terminal 2000 into the display range 3R, the position of the mobile terminal 2000 and the like may be basically at any position, including before insertion, during insertion, and after insertion. The insertion of the mobile terminal 2000 into the display range 3R only requires the mobile terminal 2000 and the display range 3R to be in contact. In other words, as long as contact determination is possible, it can take any position, insertion direction, or posture angle. Also, the contact between the display range 3R and the mobile terminal 2000 may basically be performed at any location within the display range 3R.

[0280] In this way, by widening the allowable contact operations and actions (corresponding determination conditions), there is an advantage that user operations become easier. Conversely, it is also possible to narrow the allowable contact operations and actions (determination conditions). That is, the position, posture, insertion direction, contact time, contact area within the display range 3R, etc. of the mobile terminal 2000 allowed during contact operations (before insertion, during insertion, after insertion, etc.) may be defined so as to be limited.

[0281] FIG. 20A shows various examples of the position and insertion direction of the mobile terminal 2000 before insertion in the YZ plane. In FIG. 20A, the illustration of the mobile terminal 2000 is omitted, and the position of the mobile terminal 2000 is indicated by dots. The black dots and white dots (such as p1) indicate examples of the position of the mobile terminal 2000 before insertion. As shown in the illustration, if the mobile terminal 2000 contacts the display range 3R, various positions and insertion directions are possible.

[0282] FIG. 20B shows various examples of the position and insertion direction of the mobile terminal 2000 before insertion in the XY plane. Operations can be performed in any insertion direction so as to contact the display range 3R from any position (such as p11) before insertion. Also, in the case of position p14, before insertion, the mobile terminal 2000 is on the right side of the display range 3R and deeper in the Y direction than the display range 3R, and the insertion direction is from the back (+Y) to the front (-Y) with respect to the display range 3R, or the left - right direction. Even in such a case, if the mobile terminal 2000 contacts the display range 3R, it is allowed as a contact operation.

[0283] FIG. 20C shows an example of the state when the mobile terminal 2000 contacts the display range 3R in a YZ - plane view. The contact of the mobile terminal 2000 with the display range 3R may be by contact of any part of the mobile terminal 2000 as long as the contact can be detected and determined. Also, after the mobile terminal 2000 has once contacted the display range 3R (after the contact operation is detected), the mobile terminal 2000 may move out of the display range 3R.

[0284] State 2000c1 shows an example of the state of the mobile terminal 2000 at the time of contact. The contact location (contact position) c1 at that time is near the center on the display range 3R side, and is a part near the upper side of the housing, for example, on the mobile terminal 2000 side. State 2000c2 shows another example of the state of the mobile terminal 2000 at the time of contact. The contact location c2 at that time is at the upper part of the display range 3R, and is a part near the lower side of the housing, for example. State 2000c3 shows another example of the state of the mobile terminal 2000 at the time of contact. The contact location c3 at that time is at the lower part on the display range 3R side. After once contacting, the mobile terminal 2000 has passed to the back side and is not touching the display range 3R. Also, state 2000c4 is a state where the mobile terminal 2000 has been returned to the front side in the Y direction from the contacted state 2000c1.

[0285] [Modification Example: Determination Conditions] In the third embodiment, the posture, position, etc. of the mobile terminal 2000 during the insertion / contact operation to the display range 3R are arbitrary, but it is not limited thereto. In the modification example, the posture, position, etc. of the mobile terminal 2000 during the insertion / contact operation may be limited to a specific state. That is, the spatial floating image display device 1000 may determine / detect a state such as a specific posture during the contact operation of the mobile terminal 2000, and allow the display request 2030 only when detected. As a modification example, when narrowly limiting the allowed predetermined contact operation, for example, the following determination conditions can be cited.

[0286] (1). Only the insertion direction that inserts in the Y direction forward, that is, from the side where the user 230 views the spatial floating image 3 to the back, with respect to the plane of the display range 3R of the spatial floating image 3 is allowed. In other words, it is not allowed to contact from the back or side with respect to the display range 3R. Also, only when the insertion direction is within a predetermined angle range with respect to the plane of the display range 3R of the spatial floating image 3 is allowed.

[0287] (2). The contact location when the mobile terminal 2000 contacts the plane of the display range 3R, that is, the position of the mobile terminal 2000, the area within the display range 3R, is limited to a part. For example, a part in the center, a part in the lower right, a part in the upper left, etc. within the display range 3R.

[0288] (3) When the mobile terminal 2000 contacts the plane of the display range 3R, the posture of the mobile terminal 2000 is limited to some postures. For example, the angle with respect to the plane of the display range 3R is within a predetermined angle range, etc. For example, as shown in FIG. 19C, only when the flat housing of the mobile terminal 2000 is inserted substantially vertically with respect to the plane of the display range 3R, it may be allowed. Also, for example, as shown in FIG. 19F, only when the flat housing of the mobile terminal 2000 contacts substantially parallel to the plane of the display range 3R, it may be allowed.

[0289] (4) After the mobile terminal 2000 contacts the plane of the display range 3R and passes through to the back side, the posture and holding time of the mobile terminal 2000 are limited. For example, as in the example of FIG. 19E, the posture angle with respect to the plane of the display range 3R or the opening 1200 is within a predetermined angle range, etc.

[0290] [Guide Display] FIG. 21A shows an example in which the spatial floating image display system displays guide information regarding the insertion / contact operation for the user 230 when inserting the mobile terminal 2000 into the display range 3R of the spatial floating image 3. In the third embodiment, the spatial floating image display device 1000 displays the guide information, in other words, information for guiding / directing the location and direction of inserting the mobile terminal 2000, and user interface information, on the spatial floating image 3. Thereby, the user 230 can easily perform the contact operation of inserting the mobile terminal 2000 into the display range 3R according to the guide information.

[0291] Examples of the guiding display in this guide information are as follows. (a) Displaying the entire display range 3R (the area that receives the contact operation. A partial area may also be used) with a predetermined color or the like. Alternatively, displaying a frame representing the entire display range 3R with a color or the like. (b) Explaining the insertion location and insertion direction by means of displays such as arrows. (c) Explaining the insertion location and insertion direction in natural language, video / images / animations, etc.

[0292] In the example of FIG. 21A, at the bottom, there is a plan view of the xy plane when the spatial floating image 3 is viewed from the front, showing an example of the display of the guide information. The guide information in FIG. 21A collectively shows several example guiding displays, and at least one of these may be applied. The guiding display example 2101 is an example of displaying a rectangular frame representing the entire display range 3R in color. The guiding display example 2102 is an example of showing the user the area that accepts insertion / contact by displaying an arrow image (which may also be other predetermined symbols or figures, etc.). The arrow image in this example is an image of a plurality of arrows pointing inward from the four sides of the display range 3R. The guiding display example 2103 is an example of conveying to the user the insertion / contact to the area of the display range 3R by displaying a character image in natural language, for example, "Insert here". The guiding display example 2104 is an example of conveying to the user the insertion / contact to the area of the display range 3R by a video / image, for example, an animation, etc.

[0293] As a modification, as will be described later, guide information may similarly be displayed on the screen on the side of the mobile terminal 2000.

[0294] Also, as a modification, a guide image for conveying the contact situation may be displayed on the spatial floating image 3 during the insertion / contact operation of the mobile terminal 2000. In the example of FIG. 21B, only a partial area 2105 of the display range 3R of the spatial floating image 3 is defined as a predetermined area (insertion area / contact area) that accepts the insertion / contact operation.

[0295] The spatial floating image display device 1000 uses a sensor to detect the contact situation of the mobile terminal 2000 with respect to the display range 3R, and displays a guide image 2106 representing the contact situation according to the detected contact location. This guide image 2106 is an image indicating that the mobile terminal 2000 is in contact with the display range 3R at the spatial position corresponding to the guide image 2106. In this example, the guide image 2106 is an effect image like ripples spreading around the contact location. By looking at the guide image 2106, the user 230 can more clearly recognize the contact situation of the mobile terminal 2000 with the display range 3R. Then, the user 230 may move the mobile terminal 2000 to the insertion area, which is a partial area 2105, based on the guide image 2106. The guide information / feedback information for the user 230 is not limited to only the above-described image display, and voice output or the like may be used in combination.

[0296] [Display restriction function] Moreover, this system 3000 may have the following function (referred to as a display restriction function). When the mobile terminal 2000 is inserted into the display range 3R during a contact operation, the system 3000 may make a partial area near the column where the mobile terminal 2000 is inserted in the display range 3R non-display the spatial floating image 3, and then perform another display on the remaining area of the display range 3R. When the user 230 views the display range 3R along with the insertion operation of the mobile terminal 2000, there may be a situation where the spatial floating image 3, which is the display image, becomes invisible due to being blocked by the mobile terminal 2000. This function is a function that controls the display of the spatial floating image 3 in such an invisible area (partial area) to be temporarily non-displayed. In this function, the video control unit 1160 in FIG. 3 of the spatial floating image display device 1000, for example, may realize the above non-display by controlling the light source device 13 and the video display element (liquid crystal display panel 11) of the display device 1.

[0297] In addition, when a partial area of the display range 3R is made non-display in accordance with the contact operation, the spatial floating image display device 1000 may reduce the display size of the display image, change the aspect ratio, etc., so that the remaining display area matches the spatial floating image 3 which is the display image.

[0298] FIG. 22 (FIGS. 22A, etc.) shows an example of this display restriction function. FIG. 22A is a plan view of the XY plane. For example, similar to FIGS. 18A to 18D, it shows a case where the user 230 inserts and contacts the mobile terminal 2000 in the Y direction with respect to a part of the display range 3R. The area 2201 indicates the position and area of the mobile terminal 2000 when it is in contact after being inserted into the display range 3R. FIG. 22B shows an XZ plan view corresponding to the state of FIG. 22A. The area 2202 is an example of a non-visible area caused by the shielding of the image light. Correspondingly, FIG. 22C is a plan view of the YZ plane, showing the state of the shielding of the image light. The position 2203 indicates the position and area of the mobile terminal 2000 when it is in contact after being inserted into the display range 3R.

[0299] The image light component 2204 indicates a component that forms the spatial floating image 3 in the display range 3R by entering the viewpoint (eye) of the user 230 without being shielded by the mobile terminal 2000 (position 2203) among the light beams of the image light emitted out of the opening 1200. The image light component 2205 indicates a component that forms the non-visible area 2202 in the display range 3R by being shielded by the mobile terminal 2000 (position 2203) and not entering the viewpoint (eye) of the user 230 among the light beams of the image light emitted out of the opening 1200.

[0300] FIG. 23 (FIGS. 23A, etc.) shows an operation example of the display restriction function together with an example of the display of the guide image in the spatial floating image 3.

[0301] First, FIG. 23A shows an example in which a guide image 2302 indicating that a display request 2030 is in progress is displayed when a contact operation is detected in a front view (xy plane) of the display range 3R of the spatial floating image 3. In this example, in the display range 3R, there is a frame display 2301 showing the whole, and within the frame display 2301, a guide image 2302 indicating that the display request 2030 is in progress is displayed. The guide image 2302 is, for example, a character image indicating that the display request 2030 is in progress, such as "Requesting..." or an arrow icon. Also, in this state, an invisible region 2202 is generated due to the shielding by the mobile terminal 2000.

[0302] FIG. 23B shows an example in which the spatial floating image display device 1000 turns off a partial region of the display range 3R and controls the display of the remaining region based on the detection of the occurrence of the invisible region 2202 as shown in FIG. 23A. The detection of the occurrence of the invisible region 2202 can be detected by the air operation detection sensor 1351 or the imaging unit 1180.

[0303] For example, the video control unit 1160 of the spatial floating image display device 1000 shown in FIG. 3 determines and detects the invisible region 2202 while determining and detecting the contact operation, and determines a non-display region including the invisible region 2202 in the display range 3R. In this example, based on the detection by the air operation detection sensor 1351 (described later), in the display range 3R, the contact location 2303 in the X direction and the x direction, in other words, the width and the column can be understood. Therefore, the spatial floating image display device 1000 determines that the display column corresponding to the contact location 2303 in the display range 3R is a non-display region 2304 indicated by a diagonal hatching pattern. The non-display region 2304 includes the invisible region 2202. Among the display range 3R, the region other than the non-display region 2304 becomes the display region 2305. In other words, the spatial floating image display device 1000 configures the display region 2305 by reducing the size of the display range 3R by the width of the non-display region 2304 in the X direction and the x direction.

[0304] The spatial floating image display device 1000 controls the display of the display device 1 so that the spatial floating image 3 is not displayed in the non-display area 2304 and is displayed only in the display area 2305. In this example, in the display area 2305 of the display range 3R, from the original frame image 2301 and the guide image 2302 as shown in FIG. 23A, while maintaining the aspect ratio, it is adjusted such as being reduced so as to fit within the width of the display area 2305, and thus is changed to the frame image 2306 and the guide image 2307.

[0305] With the above function, it becomes easier for the user 230 to visually recognize the guide image 2307 etc. in the spatial floating image 3. The above example shows the case where a wider area is set as the non-display area 2304 with respect to the non-visible area 2202, but it is not limited to this. The non-display area may be the same area as the non-visible area, or may be a different area that at least partially overlaps with the non-visible area. Also, the change and adjustment of the guide image can be various, such as changing the aspect ratio, changing the color or brightness, and shifting the display position.

[0306] FIG. 24A shows an example of the display of the guide image of the spatial floating image 3 as feedback such as notification to the user 230 when a predetermined contact operation is detected, in other words, when it is determined that the contact operation is successful and the display request 2030 is received. In this example, a guide image 2401 indicating success is displayed throughout the display range 3R. The guide image 2401 is composed of characters, icons, etc. indicating success. When it becomes successful in this way, it shifts to the phase of displaying the target image as the spatial floating image 3.

[0307] FIG. 24B shows an example of the display of the guide image of the spatial floating image 3 as feedback such as notification to the user 230 when a predetermined contact operation cannot be detected, in other words, when it is determined that the contact operation fails and the display request 2030 is not received. In this example, a guide image 2402 indicating failure is displayed throughout the display range 3R. The guide image 2402 is composed of characters, icons, etc. indicating failure or retry.

[0308] When the user 230 fails to properly insert or touch the mobile terminal 2000 into the display range 3R, or when, even if the operation is properly performed, the system 3000 fails to appropriately detect it due to a sensor defect or the like, the spatial floating image display device 1000 determines that it is a failure or an error as described above, and outputs a guide image 2402 as feedback to the user 230. In this case, the spatial floating image display device 1000 may prompt the user 230 to retry the touch operation. The user 230 starts over from the touch operation.

[0309] Furthermore, FIG. 24C shows an example in which, when a guide image indicating that a display request 2030 from the mobile terminal 2000 to the spatial floating image display device 1000 together with the touch operation of the mobile terminal 2000 to the display range 3R has been properly performed (success) is displayed as the spatial floating image 3, the display area is adjusted according to the invisible area 2202 and then displayed. The non-display area 2404 is an area of the display range 3R that includes the invisible area 2202, and the display area 2405 is an area other than the non-display area 2404.

[0310] (A) on the lower left side of FIG. 24C is an example of a successful case, and in the display area 2405, a guide video 2401b whose size and the like are adjusted based on the guide video 2401 as shown in FIG. 24A is displayed. Similarly, (B) on the lower right side of FIG. 24C is an example of a failed case, and in the display area 2405, a guide video 2402b whose size and the like are adjusted based on the guide video 2402 as shown in FIG. 24B is displayed.

[0311] The feedback to the user 230 is not limited to visual means such as the display of a guide image in the spatial floating image 3 or the emission of light from a lamp, but may also use auditory means such as voice output or tactile means such as a device that generates stimuli such as vibration / air / ultrasonic waves. Such devices, for example, an air ejector or an ultrasonic generator, may be installed inside or outside the housing 1190. Also, each means may be combined and used simultaneously.

[0312] FIG. 24D shows a case where, in the YZ plane, when the contact operation and display request 2030 result in success or failure, the spatial floating image display device 1000 generates voice 2408 indicating success or failure from the voice output unit 1140 in FIG. 3, such as a speaker, to the user 230. (A) is the voice 2408 (voice notification example) in the case of success, for example, "Connected". (B) is the voice 2408 (voice notification example) in the case of failure, for example, "Connection failed. Please try again". The speaker may be installed inside or outside the housing 1190. The speaker may be a super-directional speaker or the like. The above examples are voice output examples in natural language, but are not limited thereto, and may be output of a predetermined short sound, for example, a beep sound, an alert sound, or music.

[0313] [Air operation detection sensor] FIG. 25 (FIG. 25A, etc.) shows an implementation example of the air operation detection sensor 1351 and the imaging unit 1180 (camera) as sensors in the spatial floating image display device 1000 of Example 3. In particular, the air operation detection unit 1350, the control unit 1110, or the video control unit 1160, etc. in FIG. 3 of the spatial floating image display device 1000 of Example 3 uses these sensors to determine and detect the insertion / contact operation of the mobile terminal 2000 with respect to the display range 3R of the spatial floating image 3.

[0314] FIG. 25A is a YZ plane view showing an arrangement example of the air operation detection sensor 1351 and the imaging unit 1180 (camera) in the housing 1190. The configuration example of the optical system in FIG. 25A is the same as that in FIGS. 4A and 18E. In the example of FIG. 25A, the air operation detection sensor 1351 is installed near the transparent member 100 on the upper surface of the housing 1190, on the front side closer to the user 230 in the Y direction. The optical axis of the air operation detection sensor 1351, indicated by the dashed-dotted arrow, is obliquely upward as shown in the figure, that is, it faces the y direction, and its optical axis is set to overlap the display range 3R of the spatial floating image 3. Also, in the example of FIG. 25A, the imaging unit 1180 (camera) is installed in the housing 1190 with the optical axis, indicated by the dashed-dotted arrow, facing upward, that is, the Z direction. The imaging range of the imaging unit 1180 (camera) is set to include the display range 3R and the opening 1200 as shown by the broken line.

[0315] FIG. 25B shows a configuration example of the air operation detection sensor 1351. FIG. 25B shows an xy plane view corresponding to the display range 3R of the spatial floating image 3. In the air operation detection sensor 1351, a plurality of optical elements 1351c are arranged in the x direction (the X direction in FIG. 25A). The optical element 1351c is a pair of a light emitting element 1351a and a light receiving element 1351b. The light emitting element 1351a is composed of, for example, an infrared element or the like. It is assumed that the emission surface of the light emitting element 1351a of the air operation detection sensor 1351 coincides with the upper surface of the transparent member 100 of the housing 1190 in FIG. 25A. The light emitting element 1351a emits light a1, for example, infrared light, in the y direction. If the light a1 is not blocked by an object, it passes through the display range 3R. If the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2. The reflected light a2 is received by the light receiving element 1351b.

[0316] For example, when there is a contact point 2501 by the finger of the user 230 within the xy plane of the display range 3R, light a1 is reflected at the contact point 2501 and returns as reflected light a2. The light receiving element 1351b at a certain position in the x direction detects the reflected light a2. Thereby, the air operation detection unit 1350 can know that there is a contact point 2501 at that position in the x direction. Also, the air operation detection unit 1350 can calculate the distance by the TOF method from the time until the emitted light a1 returns as the reflected light a2. For example, the distance 2502 to the contact point 2501 can be calculated. Thereby, the position coordinates of the contact point 2501 in the xy plane of the display range 3R are also known.

[0317] Also, in FIG. 25B, the case where the mobile terminal 2000b is arranged within the display range 3R by an insertion / contact operation is shown as a contact point 2503. Also in this case, similarly, the air operation detection sensor 1351 and the air operation detection unit 1350 can detect the position coordinates of the contact point 2503. In the example of FIG. 25A, there is nothing above the spatial floating video 3, but a part of the housing 1190 may also be provided above the spatial floating video 3. In that case, the light a1 of the air operation detection sensor 1351 is reflected by a part of the housing 1190 and returns as the reflected light a2.

[0318] The spatial floating video display device 1000 can detect a contact operation when the mobile terminal 2000b contacts the display range 3R as shown in FIG. 25A by using the air operation detection sensor 1351 and the air operation detection unit 1350. Not limited to the above arrangement example, the air operation detection sensor 1351 may be arranged above the xy plane of the display range 3R, may be arranged at a position shifted in the front-back direction, that is, in the z direction or the Y direction, or a plurality of air operation detection sensors may be arranged at a plurality of positions in the front-back direction.

[0319] Also, in the example of FIG. 25A, when the mobile terminal 2000b is placed within the display range 3R, the mobile terminal 2000b is captured in the captured image of the imaging unit 1180 (camera). Therefore, the video control unit 1160 and the like that cooperate with the imaging unit 1180 can detect the mobile terminal 2000b from within the captured image and determine whether the mobile terminal 2000b has come into contact with the display range 3R. Thereby, even when using the imaging unit 1180, the spatial floating video display device 1000 can detect it as a contact operation when the mobile terminal 2000b comes into contact with the display range 3R as shown in FIG. 25A. The spatial floating video display device 1000 may determine and detect a contact operation by using at least one of the above-described air operation detection sensor 1351 and the imaging unit 1180. When using both of them, the accuracy of determination and detection can be made higher.

[0320] Not limited to the example of FIG. 25A, as another arrangement of the imaging unit 1180 (camera), for example, as shown by the broken line as the imaging unit 1180b, it may be arranged outside the housing 1190 at a position deeper in the Y direction with the optical axis facing the display range 3R. The imaging unit 1180b may capture the face of the user 230 or detect the approach of the user 230. Also, not limited to the air operation detection sensor 1351 that senses parallel to the plane of the spatial floating video 3 as shown in FIG. 25B, a distance measurement sensor, a stereo camera, or the like that senses the plane from a direction perpendicular to the plane of the spatial floating video 3 may be used.

[0321] FIG. 25C shows another arrangement example of the air operation detection sensor 1351 and the imaging unit 1180 (camera) in the housing 1190. In the example of FIG. 25C, as the optical system, based on FIG. 2B, the housing 1190 is placed vertically, and the case where the spatial floating image 3 is formed so as to stand in the vertical direction (Z direction) is shown. In the Y direction, the components of the aforementioned optical system are mounted at the rear of the housing 1190, and in the front of the housing 1190, between the upper and lower housing parts, the spatial floating image 3 is formed. In this example, the air operation detection sensor 1351 and the imaging unit 1180 (camera) are installed in the upper housing part. The air operation detection sensor 1351 faces downward so that the optical axis overlaps with the display range 3R. The imaging range of the imaging unit 1180 (camera) is set to include the display range 3R. Even with such a configuration, similarly, when the mobile terminal 2000 is inserted into the display range 3R, at least one of the air operation detection sensor 1351 and the imaging unit 1180 (camera) can be used to detect a contact operation.

[0322] [Regarding the distinction between a finger and a mobile terminal] The spatial floating image display device 1000 may or may not distinguish between the finger of the user 230 and the mobile terminal 2000 as an object that contacts the spatial floating image 3 (display range 3R). When distinguishing them, for example, the following can be cited as technical means.

[0323] (1) When performing arbitrary communication between the mobile terminal 2000 and the spatial floating image display device 1000, the spatial floating image display device 1000 may recognize the mobile terminal 2000 from communication information such as a connection request. When the spatial floating image display device 1000 detects that some object is in contact with the display range 3R by the air operation detection sensor 1351, if there is the above communication information, it may be estimated that the object is the mobile terminal 2000.

[0324] (2) The spatial floating image display device 1000 may recognize and determine that the object in contact with the spatial floating image 3 is the mobile terminal 2000 based on an imaging image by the imaging unit 1180 (camera) or the like through image recognition processing or the like.

[0325] (3). The spatial floating image display device 1000 may make a determination by the air operation detection sensor 1351 and the air operation detection unit 1350 to distinguish whether the object in contact with the spatial floating image 3 is the finger of the user 230 or the mobile terminal 2000. For example, for normal air operations and touch operations, the device is set to accept operations by a single finger (such as the detection at the contact location 2501 in FIG. 25B). When the mobile terminal 2000 is inserted into the display range 3R, the area of the contact location is larger for the mobile terminal 2000 than for a single finger, and the shape of the contact location becomes linear. The spatial floating image display device 1000 detects and determines the area, shape, etc. of the contact location, such as the contact location 2503 in FIG. 25B, by the air operation detection sensor 1351 or the like, and based on this, determines whether the contacting object is a finger or the mobile terminal 2000.

[0326] (4). Among the information exchanged between the spatial floating image display device 1000 and the mobile terminal 2000, information representing the mobile terminal 2000, such as a QR code described later, is provided. The spatial floating image display device 1000 distinguishes whether the object in contact with the spatial floating image 3 is the finger of the user 230 or the mobile terminal 2000 by recognizing and acquiring the information.

[0327] [Application and Screen of Mobile Terminal] FIG. 26A shows a display example of the screen of the display unit 20012 by the application 2010 or the like of the mobile terminal 2000 in Embodiment 3. The mobile terminal 2000 displays the application screen 2601 on the screen of the display unit 20012 by the processing of the OS, the application 2010, etc. based on the input operation of the user 230. In this example, the application 2010 is a dedicated application for enabling the user 230 to conveniently use various functions of the spatial floating image display device 1000 in cooperation with the spatial floating image display device 1000.

[0328] In the application screen 2601 of FIG. 26A, there is a GUI that prompts the user 230 to specify an image 2020 (target image) to be displayed on the spatial floating image 3 of the spatial floating image display device 1000. For example, a message such as "Please specify the image to be displayed on the aerial display" is displayed. The user 230 selects the image 2020 (FIG. 15) to be displayed on the aerial display from the images in the mobile terminal 2000. The selected image is previewed as the display target image 2602. After the user 230 checks the display target image 2602, the user presses the display request button 2603. Thereby, the mobile terminal 2000 transmits a display request 2030 with the display target image 2602 as the target image to the spatial floating image display device 1000 by means of short-range wireless communication or the like.

[0329] FIG. 26B is a modification example regarding the display of the guide image in FIG. 21A described above, and is an example of displaying a guide image on the screen of the mobile terminal 2000 side. In the application screen 2601 of FIG. 26B, a guide image 2604 for prompting the user 230 to perform a contact operation is displayed. The guide image 2604 displays a message such as "Please bring the smartphone into contact with the spatial floating image", and an image representing the operation of inserting and contacting the mobile terminal 2000 within the display range 3R is displayed.

[0330] FIG. 26C is a modification example regarding the display of the guide image in FIG. 23A and the display of the guide image in FIG. 24A described above, and is an example of displaying a guide image on the screen of the mobile terminal 2000 side. In the application screen 2601 of FIG. 26C, depending on the situation, a guide image 2605 for informing the user 230 that the display request 2030 is in progress, a guide image 2606 for informing that the display request 2030 has been received and is successful, etc. are displayed.

[0331] In the third embodiment, as one method, during the contact operation, as described above, the user 230 designates the target image 2020 on the screen of the application 2010 of the mobile terminal 2000, presses the display request button 2603, and the mobile terminal 2000 transmits the display request 2030. After that, the user 230 inserts the mobile terminal 2000 into the display range 3R.

[0332] Not limited to this, in other embodiments, the operation of the user 230 may be minimized. For example, the user 230 displays a desired target image on the screen of the mobile terminal 2000, and in this state, inserts the mobile terminal 2000 into the display range 3R. The spatial floating image display device 1000 determines and detects the contact operation, and if successful, generates a display request 2030. The spatial floating image display device 1000 communicates with the mobile terminal 2000 to acquire the target image associated with the display request 2030 and displays it on the spatial floating image 3. The target image is the image displayed on the screen of the mobile terminal 2000 at that time.

[0333] [Display of Target Image in Spatial Floating Image] The video control unit 1160 (FIG. 3) of the spatial floating image display device 1000 generates image data for display on the screen of the display device 1 (liquid crystal display panel 11) based on the image data of the target image. This image data is, for example, two-dimensional image data generated by processing from the two-dimensional image data when the target image is two-dimensional image data. Also, when the target image is three-dimensional image data (such as a three-dimensional model), it is two-dimensional image data generated by rendering or the like in a virtual three-dimensional space from the three-dimensional image data.

[0334] With the above functions, for example, as shown in FIG. 16, the target image 1601 (an example of a character image) displayed and designated by the user 230 on the mobile terminal 2000 can be displayed as the image 1602 in the spatial floating image 3 in response to the contact operation and the display request 2030. Thereby, the user 230 can enjoy viewing the desired target image 1601 as the image 1602 in the spatial floating image 3.

[0335] Thereafter, the method for canceling the display of the target image in the floating image 3 in space is not particularly limited. For example, it may be automatically canceled after a certain period of time has elapsed since the start of the display of the target image. Also, a cancel display button or the like may be provided on the screen of the floating image 3 in space or the mobile terminal 2000. A cancel display button or the like may be provided on the housing 1190.

[0336] As described above, according to the floating image display system of the third embodiment, by cooperating with the mobile terminal 2000, it is possible to provide the user 230 with a service or the like that allows a desired image to be displayed as the floating image 3 in space and viewed.

[0337] Note that the data / information exchanged between the mobile terminal 2000 and the floating image display device 1000 is not limited to only the target image to be displayed, and it may be possible to exchange, together with the target image to be displayed, some data / information for management / control, such as user information.

[0338] As a modification, a background image, a character image, or the like may be displayed in advance as the floating image 3 in space, and from that state, the target image may be superimposed and displayed on the background image or the like by a contact operation of the mobile terminal 2000 by the user 230 and a display request 2030.

[0339] In the third embodiment, when the mobile terminal 2000 contacts the floating image 3 in space, it is assumed that the display request 2030 is received successfully and the target image is displayed, but it is not limited to this. In a modification, after the user 230 inserts and contacts the mobile terminal 2000 with the floating image 3 in space and the mobile terminal 2000 enters the space on the back side after passing through the floating image 3 and is in a state where it is not in contact with the display range 3R (for example, FIG. 19E), it may be considered a success.

[0340] Also, the following is possible as a modification. The function in the modification is a function of acquiring the image displayed as the spatial floating image 3 on the side of the mobile terminal 2000. In advance, the spatial floating image display device 1000 displays a character image or the like on the spatial floating image 3 (for example, similar to (3) in FIG. 16). When the user 230 wants to acquire the image displayed as the spatial floating image 3 on the side of the mobile terminal 2000, the user 230 performs a contact operation of the mobile terminal 2000 on the spatial floating image 3 in the same manner as described above. When this contact operation is successful, the spatial floating image display device 1000 transmits the image (corresponding image data) displayed as the spatial floating image 3 to the mobile terminal 2000. Thereby, the user 230 can acquire and display the image on the mobile terminal 2000 and enjoy watching it (for example, similar to (1) in FIG. 16).

[0341] In the third embodiment, after causing the mobile terminal 2000 to perform a contact operation on the spatial floating image 3 to display the target image, even if the mobile terminal 2000 is not in contact with the spatial floating image 3, the display of the target image continues. However, the present invention is not limited to this. In the modification, it may be a function of displaying the target image only while the mobile terminal 2000 continues to be in contact with the spatial floating image 3. When the spatial floating image display device 1000 detects that the mobile terminal 2000 has left the display range 3R, the spatial floating image display device 1000 cancels the display of the target image in the spatial floating image 3.

[0342] <Fourth Embodiment> Next, an example using a QR code (two-dimensional code) or the like will be described as another embodiment (referred to as the fourth embodiment) with reference to FIGS. 27A and subsequent figures. The fourth embodiment is a modification of the third embodiment. The mobile terminal 2000 appropriately has a function of displaying a QR code on the screen of the display unit 20012. This QR code may be, for example, a code representing the aforementioned display request 2030.

[0343] In the fourth embodiment, when a display request 2030 is sent from the mobile terminal 2000 to the spatial floating image display device 1000, the following operations and actions may be performed for the purpose of causing the mobile terminal 2000 side to recognize the spatial floating image display device 1000 or causing the spatial floating image display device 1000 side to recognize the mobile terminal 2000. The user 230 performs an operation from an application 2010 or the like on the mobile terminal 2000 side to the effect of performing cooperation or connection with the spatial floating image display device 1000 and of making a display request 2030 (for example, similar to FIG. 26A and the like). Then, the user 230 performs a contact operation of inserting the mobile terminal 2000 into the display range 3R. Alternatively, for example, the user 230 displays a QR code or the like on the screen of the mobile terminal 2000 before insertion, and then performs a contact operation of inserting the mobile terminal 2000 into the display range 3R in such a posture that the QR code or the like of the mobile terminal 2000 is read by the imaging unit 1180 (camera) on the spatial floating image display device 1000 side.

[0344] FIG. 27A is an example in which the mobile terminal 2000 displays a QR code screen 2701 on the screen 1902 of the display unit 20012 based on an input operation by the user 230 and displays image information such as a QR code 2702 on the QR code screen 2701.

[0345] FIG. 27B is a YZ plan view showing an example in which the user 230 sets the mobile terminal 2000a at a position before insertion into the display range 3R to display a QR code on the screen 1902 (FIG. 27A), and then inserts the mobile terminal 2000a from that position into the display range 3R to reach a state of the mobile terminal 2000b at the contact position. In this state of the mobile terminal 2000b, the screen 1902 faces downward and is within the imaging range of the imaging unit 1180 (camera). Therefore, the spatial floating image display device 1000 can detect the mobile terminal 2000b from the captured image and can also detect the QR code 2702 displayed on the screen 1902. The spatial floating image display device 1000 recognizes the QR code and extracts the data and information described in the QR code. For example, the spatial floating image display device 1000 obtains the display request 2030 from the QR code.

[0346] The spatial floating image display device 1000 detects the contact of the mobile terminal 2000b with the display range 3R by the air operation detection sensor 1351 or the imaging unit 1180, and acquires a display request 2030 from the QR code of the mobile terminal 2000b. Based on the success of these operations, the spatial floating image display device 1000 causes the target image associated with the display request 2030 to be displayed as the spatial floating image 3.

[0347] In addition, in FIG. 27B, the mobile terminal 2000b is in contact with the display range 3R, but it is not limited to this. As described above, after the mobile terminal 2000 comes into contact with the display range 3R, it may enter a non-contact state by penetrating deeper than the display range 3R. The posture of the mobile terminal 2000 only needs to be a posture in which the QR code on the screen 1902 can be detected by the imaging unit 1180 (camera). For example, in the case of the imaging unit 1180b in FIG. 25A, the posture of the mobile terminal 2000 may be a posture in which the screen 1902 faces the back side in the Y direction (for example, similar to FIG. 19E).

[0348] In the fourth embodiment, as described above, the condition for recognizing the QR code is added to the determination condition for the predetermined contact operation in the third embodiment. In other words, in the fourth embodiment, the spatial floating image display device 1000 receives and permits the display request 2030 and displays the target image on the spatial floating image 3 when the AND condition that a predetermined contact operation is detected and the QR code is recognized is satisfied.

[0349] In the fourth embodiment (FIG. 27B), the user 230 may insert the mobile terminal 2000 in a horizontal direction parallel to the upper surface of the housing 1190 with the screen 1902 facing down with respect to the display range 3R (similar to FIG. 19E). Also, in a modified example, after the user 230 inserts the mobile terminal 2000 with the screen 1902 facing down with respect to the display range 3R, the screen 1902 may be placed in contact with the upper surface of the transparent member 100 of the housing 1190. Even in these cases, the QR code can be recognized.

[0350] [QR code] The data and information described in the codes such as the above QR code / barcode are not limited to the examples of the above display requirement 2030, and examples include the following. These data and information may be used in combination.

[0351] (1). Connection information: In the communication network of FIG. 15, information for the mobile terminal 2000 and the spatial floating image display device 1000 to communicate and connect. For example, information such as IP address, ID, and password. The communication network here refers to the Internet 3010, LAN 3020, etc., or direct communication between the mobile terminal 2000 and the spatial floating image display device 1000 may also be used.

[0352] (2). Terminal ID / User ID: Information such as the ID of the mobile terminal 2000. Or information such as the ID of the user 230.

[0353] (3). Image ID: Information such as the ID or URL of the target image that the user 230 wants to display.

[0354] When the above connection information is a QR code, the mobile terminal 2000 conveys the connection information to the spatial floating image display device 1000 by presenting the QR code, and a wireless connection is established between the communication unit 20020 of the mobile terminal 2000 and the communication unit 1132 of the spatial floating image display device 1000 using the connection information.

[0355] When the above terminal ID / user ID, etc. is a QR code, the spatial floating image display device 1000 can recognize a specific mobile terminal 2000 / user 230. The spatial floating image display device 1000 may be configured to permit only a specific mobile terminal 2000 / user 230 associated with a specific terminal ID / user ID as the service provision target of this function.

[0356] When the above image ID is a QR code, the target image is not limited to the image held by the mobile terminal 2000 and can be used.

[0357] As described above, according to the spatial floating image display system of Example 4, by using the QR code, a predetermined contact operation between the mobile terminal 2000 and the spatial floating image display device 1000 is defined, and various controls using the QR code are possible.

[0358] Note that, even in the case of the form of Example 3 or the like that does not use the QR code, the above-described data and information may be exchanged and used between the mobile terminal 2000 and the spatial floating image display device 1000 together with the display request 2030.

[0359] As a modification of Example 4, the spatial floating image display device 1000 may transmit the information of the QR code to a nearby mobile terminal 2000, and the mobile terminal 2000 that has received the information may display the QR code.

[0360] In Examples 3 and 4, the data of the target image to be displayed on the spatial floating image 3 is assumed to be held in the mobile terminal 2000 in advance, but it is not limited to this. In a modification, the data of the target image may be held on the side of the spatial floating image display device 1000, and the user 230 or the mobile terminal 2000 side may select and specify the target image and issue a display request 2030. In another modification, the data of the target image may be held in a server or the like on the communication network, and the user 230 or the mobile terminal 2000 side may select and specify the target image and issue a display request 2030.

[0361] Also, as a modification of Example 4, the spatial floating image display device 1000 may display the information of the QR code on the spatial floating image 3, and the user 230 may recognize the information of the QR code displayed as the spatial floating image 3 by using the camera of the mobile terminal 2000.

[0362] FIG. 28 (FIG. 28A, etc.) shows this modification example. FIG. 28A is an example in which the spatial floating image display device 1000 displays a QR code 2801 which is the spatial floating image 3. FIG. 28B is a plan view of the YZ plane, showing an example in which the mobile terminal 2000 captures and reads the QR code 2801 of the spatial floating image 3 in FIG. 28A with the camera of the imaging unit 20025, for example, an out-camera. The mobile terminal 2000 obtains predetermined information from the recognized QR code 2801. After recognizing the QR code 2801, the user 230 brings the mobile terminal 2000 into contact with the display range 3R.

[0363] FIG. 28C is a further modification example, showing an example in which a QR code 2802 is displayed on the screen of the second display device 1680 of the spatial floating image display device 1000. The spatial floating image display device 1000 uses, for example, a second display device 1680 arranged to overlap the spatial floating image 3 on the back side, similar to FIG. 4M, and displays the QR code 2802 on the screen of the second display device 1680. Also, the spatial floating image display device 1000 may display a QR code on the screen of the transmissive self-emitting image display device 1650 using the transmissive self-emitting image display device 1650, similar to FIG. 4L.

[0364] The user 230 captures the QR code 2802 displayed on the screen of the second display device 1680 with the camera of the imaging unit 20025 of the mobile terminal 2000, and the mobile terminal 2000 recognizes the QR code and obtains information. After recognizing the QR code 2802, the user 230 brings the mobile terminal 2000 into contact with the display range 3R.

[0365] <Example 5> As Example 5 of the present invention, a configuration example of the spatial floating image display device will be described. As the basic configuration of the spatial floating image display device of Example 5, the configurations of Examples 1 to 4 can be similarly applied. Also, Example 5 can be a system (spatial floating image display system) having a spatial floating image display device and external devices connected thereto, similar to Examples 3 and 4. The external device includes, for example, a mobile terminal (mobile information processing terminal device) such as a smartphone, tablet, or wearable terminal (smartwatch, etc.) owned by the user.

[0366] [Problems and Solutions Regarding Example 5] As a problem, the space floating video display device 1000 includes various sensors such as an air operation detection sensor 1351 and an attitude sensor 1113 (Fig. 3) as the sensors it is equipped with. However, there are cases where the mobile terminal 2000 also has the same type of sensors (touch sensors and attitude sensors). When the mobile terminal 2000 is communicatively connected to the space floating video display device 1000 as in Examples 3 and 4, various possibilities can be considered regarding which device's operations and detections (sensors) to enable and use. If it is unclear which side's operations and detections to enable, depending on the application, function, situation, etc., there may be undesirable operations and effects. Therefore, it is necessary to clarify which side's operations and detections to enable according to the application, function, situation, etc.

[0367] Therefore, the space floating video display system and display method in Example 5 have the following as functions, etc. The system in Example 5 utilizes both a sensor for detecting user operations, etc. on the mobile terminal 2000 side and a sensor for detecting user operations, etc. on the space floating video display device 1000 side. This system deals with a sensor for detecting touch operations and rotation operations on the image displayed on the screen of the mobile terminal 2000 side and a sensor for detecting touch operations and rotation operations on the image displayed on the space floating video 3 of the space floating video display device 1000 side. When images / videos are displayed on both the screen of the mobile terminal 2000 side and the space floating video 3 side (dual display described later), this system selects, sets, and controls whether to enable / disable the sensors of which device.

[0368] [Example of Problem and Solution 1] Figure 37 shows Example 1 of a problem and its solution. First, an image 3701 (e.g., Image B) is displayed on the screen 2005 of the mobile terminal 2000. Based on this Image B, a similar image 3702 (e.g., Image B) is also displayed on the spatial floating video 3, which is the screen of the spatial floating video display device 1000 (dual display function 2910 in FIG. 29). As one of the problems, in such a state, when the user 230 performs a touch operation on Image B on the screen 2005 of the mobile terminal 2000, it has been ambiguous in the past as to what happens to those two images 3701 and 3702.

[0369] In this embodiment, as one of the solutions, it is done as follows. In response to detecting, for example, a touch operation on the image 3701 (Image B) on the screen 2005 of the mobile terminal 2000, display updates and the like are performed as a predetermined process associated with the image 3701, and synchronization is performed so that the display updates and the like are also reflected in the image 3702 (Image B) on the side of the spatial floating video 3 (synchronization control function 2920 in FIG. 29). The Image B on the side of the mobile terminal 2000 changes from Image B to Image Bb as a display update, and the Image B on the side of the spatial floating video 3 changes from Image B to Image Bb as a display update.

[0370] Conversely, as shown in the lower part of FIG. 37, when a touch operation is performed on the image 3702 (Image B) on the side of the spatial floating video 3, the system does not perform display updates or the like on the image 3702 and does not reflect them on the image 3701 (Image B) on the side of the mobile terminal 2000. In other words, synchronization is not performed on the side of the mobile terminal 2000.

[0371] Such synchronization control as in this example is performed based on a setting that enables detection of touch operations on the side of the mobile terminal 2000 and disables detection of touch operations on the side of the spatial floating video 3 (operation detection control function 2930 in FIG. 29). The system makes settings regarding the state after such connection when the two devices are communicatively connected. Each device controls operations, detections, and displays according to the settings after the connection. Thereby, the effects as shown in FIG. 37 are realized.

[0372] [Example 2 of a problem and its solution] FIG. 38 shows Example 2 of problems and solutions. In the example of FIG. 38, an image 3802 (Image A) is displayed on the spatial floating image 3 which is the screen of the spatial floating image display device 1000, and based on that Image A, a similar image 3801 (Image A) is also displayed on the screen of the mobile terminal 2000 (the dual display function 2910 in FIG. 29). As one of the problems, in such a state, when the user 230 performs a touch operation on, for example, Image A of the spatial floating image 3, it has been ambiguous conventionally as to what happens to those two images 3801 and 3802.

[0373] In this embodiment, as one of the solutions, it is done as follows. In response to detection of, for example, a touch operation on the image 3802 (Image A) of the spatial floating image 3, display updates and the like are performed as predetermined processing associated with the image 3802, and synchronization is performed so that the display updates and the like are also reflected in the image 3801 (Image A) on the mobile terminal 2000 side (the synchronization control function 2920 in FIG. 29). The Image A on the spatial floating image 3 side changes to Image Ab as a display update, and the Image A on the mobile terminal 2000 side changes to Image Ab as a display update.

[0374] Conversely, as shown in the lower part of FIG. 38, when a touch operation is performed on the image 3801 (Image A) on the mobile terminal 2000 side, the system does not perform display updates and the like on the image 3801 and does not reflect them on the image 3802 (Image A) on the spatial floating image 3 side either.

[0375] Such synchronization control as in this example is performed based on a setting that disables detection of touch operations on the mobile terminal 2000 side and enables detection of touch operations on the spatial floating image 3 side (the operation detection control function 2930 in FIG. 29). The system performs settings regarding the state after such connection when the two devices are communicatively connected. Each device controls operations, detections, and displays according to the settings after the connection. Thereby, the effects as shown in FIG. 38 are realized.

[0376] [Example 3 of Problems and Solutions] FIG. 39 shows Example 3 of the problem and its solution. In the example of FIG. 39, first, similar to FIG. 37, an image 3901 (e.g., image B) is displayed on the screen of the mobile terminal 2000, and based on that image B, a similar image 3902 (e.g., image B) is also displayed on the spatial floating video 3 which is the screen of the spatial floating video display device 1000 (the dual display function 2910 of FIG. 29). As one of the problems, in such a state, when the user 230 performs a rotation operation on the image B on the screen 2005 of the mobile terminal 2000, for example, it has been ambiguous in the past as to what happens to those two images 3901 and 3902.

[0377] In this embodiment, as one of the solutions, it is done as follows. The system detects a rotation operation on the image 3901 (image B) on the screen 2005 of the mobile terminal 2000, for example, the rotation of the image 3901 due to the rotation of the housing. The image B in the screen 2005 as seen by the user 230 is, for example, an image Bc in a state rotated by 90 degrees. In response to this detection, the system synchronizes the image 3902 (image B) on the spatial floating video 3 side so that the image rotation is reflected (the synchronization control function 2920 of FIG. 29). The image B of the spatial floating video 3 as seen by the user 230 is an image Bc in a state rotated by 90 degrees.

[0378] Alternatively, as shown in the lower part of FIG. 39, the system detects a rotation operation on the image 3901 (image B) on the screen 2005 of the mobile terminal 2000, for example, the image rotation by a touch operation. The image B in the screen 2005 as seen by the user 230 is, for example, an image Bc in a state rotated by 90 degrees. In response to this detection, the system synchronizes the image 3902 (image B) on the spatial floating video 3 side so that the image rotation is reflected (the synchronization control function 2920 of FIG. 29).

[0379] Synchronization control such as in this example is performed based on the detection of a rotation operation on the mobile terminal 2000 side, enabling image rotation, and the detection of a rotation operation on the spatial floating image 3 side, disabling image rotation (operation detection control function 2930 in FIG. 29). When the two devices are communicatively connected, this system performs settings regarding the state after such connection. Each device controls operations, detections, and displays according to the settings after the connection. Thereby, the operational effects as shown in FIG. 39 are realized.

[0380] [Example of Problem and Solution 4] FIG. 40 shows an example of Problem and Solution 4. In the example of FIG. 40, first, similar to FIG. 39, an image 4001 (e.g., image B) is displayed on the screen 2005 of the mobile terminal 2000, and based on that image B, a similar image 4002 (e.g., image B) is also displayed on the spatial floating image 3 which is the screen of the spatial floating image display device 1000 (dual display function 2910 in FIG. 29). As one of the problems, in such a state, when the user 230 performs a zoom-in / zoom-out operation, particularly a pinch operation, on the image B on the screen 2005 of the mobile terminal 2000, it has been ambiguous conventionally what would happen to those two images 4001 and 4002.

[0381] In this embodiment, as one of the solutions, it is as follows. This system detects a touch operation, such as a pinch-out operation, on the image 4001 (image B) on the screen 2005 of the mobile terminal 2000, and an enlarged display of the image 4001 corresponding to the pinch-out operation is performed. Image B has become an enlarged image Bd. In response to this detection, this system synchronizes so that the enlarged image display is also reflected in the image 4002 (image B) on the spatial floating image 3 side (synchronization control function 2920 in FIG. 29). As shown at the bottom of FIG. 40, the same applies to the case of a reduced display by a pinch-in operation.

[0382] Synchronization control such as in this example is performed based on a setting that enables detection of touch operations and zoom display on the mobile terminal 2000 side and disables detection of touch operations and zoom display on the spatial floating image 3 side (operation detection control function 2930 in FIG. 29). When the two devices are communicatively connected, this system performs settings regarding the state after such connection. Each device controls operations, detections, and displays according to the settings after the connection. Thereby, the operational effects as shown in FIG. 40 are realized.

[0383] As in each of the above-described examples, there is a case (dual display) where the mobile terminal 2000 and the spatial floating image display device 1000 are connected and the target image / video is displayed on both the screen of the mobile terminal 2000 and the screen of the spatial floating image display device 1000 (spatial floating image 3). In this case, there are cases where the user 230 performs a touch operation or the like on the displayed image on the mobile terminal 2000 side and cases where the user 230 performs a touch operation or the like on the displayed image on the spatial floating image 3 side. When a user operation is performed on one device, it has been unclear conventionally whether the operation should be reflected in the displayed image on the device where the operation was not performed, and various possibilities have been conceivable as countermeasures. In the state after the communication connection, particularly in the dual display state, it is necessary to appropriately set and control the operations and detections of the hardware and software of one or both of the devices in terms of the operations and detections on the mobile terminal 2000 side and the operations and detections on the spatial floating image 3 side. By appropriately setting and controlling according to the desires of the user 230, a clear user operation on the displayed image on the mobile terminal 2000 side and a reaction (display control process) corresponding to the operation can be realized. In one example, by setting the mobile terminal 2000 side to be enabled and the spatial floating image 3 side to be disabled, for example, when the user 230 rotates the displayed image on the mobile terminal 2000 side as shown in FIG. 39, the displayed image on the spatial floating image 3 side can be rotated in synchronization with the operation. In this way, user operations can be clarified and the convenience of the user 230 and the like can be enhanced.

[0384] As described above, in the state after communication connection, in a certain time period and mode, only the operation detection on the mobile terminal 2000 side is effective, and in another time period and mode, only the operation detection on the spatial floating image display device 1000 side is effective. In each mode, a predetermined function is realized. When the operation detection on one device is effective, unnecessary operation detection on the other device can be prevented. Depending on the predetermined mode, the operation detection on both devices may be effective.

[0385] Also, when the communication connection between the mobile terminal 2000 and the spatial floating image display device 1000 is released, the present system returns the settings regarding the operation detection of each device as described above to the state of the settings before connection, the normal settings.

[0386] [Spatial Floating Image Display System and Display Method] FIG. 29 is an explanatory diagram regarding the spatial floating image display system and display method in Example 5. The system 2900 in FIG. 29 is a spatial floating image display system in which the spatial floating image display device 1000 and the mobile terminal 2000 are communicatively connected. The outline of Example 5 will be described with reference to FIG. 29.

[0387] State A shows a case where, after the communication connection between the spatial floating image display device 1000 and the mobile terminal 2000, for example, the image 2901 on the screen 2005 of the mobile terminal 2000 is also displayed as the image 2902 in the spatial floating image 3 (display range 3R) of the spatial floating image display device 1000. State A is a state of double display of the image 2901 on the mobile terminal 2000 side and the image 2902 on the spatial floating image 3 side. This double display is a state where, based on the same image data, the images 2901 and 2902 in each aspect are displayed on the screens of the respective devices.

[0388] State B indicates a case where, for example, user 230 performs a touch operation on image 2901 on screen 2005 of mobile terminal 2000, starting from State A. In the example of FIG. 29, it is assumed that this system is set such that, as a post-connection setting (a certain mode described later), operation detection on the mobile terminal 2000 side is enabled and operation detection on the spatial floating image 3 side is disabled. Image 2901 is an object that accepts touch operations, and a predetermined process associated with the image 2901, such as display update, is executed. In this example, since the touch operation and detection of image 2901 are determined to be valid, image 2901 changes to image 2903 due to the display update.

[0389] Next, based on the detection of the touch operation on image 2901 on the mobile terminal 2000 side and the display update, this system communicates and cooperates with the spatial floating image display device 1000 side, and as synchronization control, the process corresponding to the touch operation is also reflected in image 2903 on the spatial floating image 3 side. That is, as a result of the touch operation, display update is also performed on image 2902 on the spatial floating image 3 side. Due to the display update, image 2902 changes to image 2904. In this way, when the setting on the mobile terminal 2000 side is enabled, if user 230 performs a user operation on the mobile terminal 2000 side, the same user operation can be reflected on the spatial floating image 3 side.

[0390] On the other hand, when user 230 performs a touch operation on image 2902 displayed on the spatial floating image 3 side, since the operation detection on the spatial floating image 3 side is set to be disabled, this system does not reflect the touch operation on image 2902 on the spatial floating image 3 side and image 2901 on the mobile terminal 2000 side. As a result, since display update is not performed for both, the states of image 2902 on the spatial floating image 3 side and image 2901 on the mobile terminal 2000 side remain as they are.

[0391] System 2900 in FIG. 29 includes a dual display function 2910, a synchronization control function 2920, and an operation detection control function 2930 as functions for performing the settings and controls as in the above example.

[0392] The dual display function 2910 is a function that, in response to a communication connection, displays an image / video being displayed on the screen by the spatial floating video 3 of the spatial floating video display device 1000 on the screen of the mobile terminal 2000, and / or displays an image / video being displayed on the screen of the mobile terminal 2000 on the screen by the spatial floating video 3 of the spatial floating video display device 1000. The synchronization control function 2920 is a function that synchronizes, via communication, a predetermined process (in other words, a display control process, a system process) that is executed in response to an operation such as a touch operation by the user 230 on the screen of one device, with respect to the display images of the screens of each device by dual display, so as to be reflected also in the display image of the screen of the other device. Note that the synchronization control function 2920 may be regarded as integrated as part of the dual display function 2910.

[0393] Furthermore, the operation detection control function 2930 is a function that sets and controls whether to enable / disable either or both of the operation, detection, and display control on the spatial floating video display device 1000 side and the operation, detection, and display control on the mobile terminal 2000 side, regarding an operation such as a touch operation on the display image of the screen of each device and a predetermined process executed in response to the operation.

[0394] [Dual display function] FIG. 30 is an explanatory diagram regarding the dual display function 2910 of FIG. 29. State A shows, as a first example of dual display, a case where an image A being displayed on the spatial floating video 3 of the spatial floating video display device 1000 is also displayed as image A on the screen 2005 of the mobile terminal 2000. First, image A is being displayed as image 3002 on the spatial floating video 3 of the spatial floating video display device 1000. The user 230 communicatively connects the mobile terminal 2000 to the spatial floating video display device 1000. The spatial floating video display device 1000 transmits, via communication, image data corresponding to image A to the mobile terminal 2000, and the mobile terminal 2000 acquires, via communication, the image data corresponding to image A from the spatial floating video display device 1000 and displays image A as image 3001 on the screen 2005. The user 230 can view image A of the spatial floating video 3 and can also view the corresponding image A on the screen 2005 of the mobile terminal 2000.

[0395] As the second example of the dual display, state B shows a case where the image B displayed on the screen 2005 of the mobile terminal 2000 is also displayed as the image B in the spatial floating image 3 of the spatial floating image display device 1000. First, the image B is displayed as the image 3003 on the screen 2005 of the mobile terminal 2000. The user 230 communicatively connects the mobile terminal 2000 to the spatial floating image display device 1000. The mobile terminal 2000 transmits, via communication, the image data corresponding to the image B to the spatial floating image display device 1000, and the spatial floating image display device 1000 acquires the image data corresponding to the image B from the mobile terminal 2000 and displays the image B as the image 3004 in the spatial floating image 3. The user 230 can view the image B on the screen 2005 of the mobile terminal 2000 and can also view the corresponding image B in the spatial floating image 3.

[0396] [Modification Example Regarding Acquisition of Image Data] As shown in FIG. 30 and FIG. 32B described later, it is not limited to transmitting the target image data in the direction from the spatial floating image display device 1000 to the mobile terminal 2000 or transmitting the target image data in the reverse direction. In a modification example, for example, the spatial floating image display device 1000 transmits the ID information of the target image to the mobile terminal 2000. The mobile terminal 2000 specifies the target image data held in the memory of the mobile terminal 2000 from the received ID information and displays the target image on the screen 2005 based on the specified image data. Alternatively, the mobile terminal 2000 refers to an external server or the like from the received ID information (which may be a URL or the like), specifies the target image data held in the external server or the like, acquires the specified image data from the external server or the like, and displays the target image on the screen 2005 based on the image data.

[0397] [Basic Control Flow] Figure 31 shows the basic control flow in the spatial floating video display system (system 2900 in FIG. 29) and the display method in Example 5. The main bodies of each device, namely the spatial floating video display device 1000 and the mobile terminal 2000, particularly the control unit of the mobile terminal 2000 and the video processing unit of the spatial floating video display device 1000, perform the processes related to the control in this embodiment. Unless otherwise specified hereinafter, the main bodies are the same.

[0398] Step S10 is a state where the spatial floating video display device 1000 and the mobile terminal 2000 are not in a communication connection state (in other words, a non-connected state). Each device is in an arbitrary usage state. In step S10, there may be a selection / setting of a mode described later as a system setting or a user setting in advance.

[0399] From the state before connection in step S10, in step S11, the mobile terminal 2000 and the spatial floating video display device 1000 are communicatively connected. For example, based on the operations / actions of user 230 who holds the mobile terminal 2000, the mobile terminal 2000 is communicatively connected to the spatial floating video display device 1000. The communication connection here is a connection in any manner regardless of the details including the communication interface. This communication connection may be a connection in a manner where the mobile terminal 2000 transmits a connection request to the spatial floating video display device 1000 and establishes a connection therebetween, like a conventional general method. This communication connection may also be a communication co...

Claims

1. An airborne floating image display device, a video processing unit that performs video processing, a display unit that displays the video processed by the video processing unit, an optical system that generates an airborne floating image based on the video displayed by the display unit, a user operation detection mechanism that detects an operation by a user with respect to the display range of the airborne floating image, a communication unit that communicates with the user's mobile terminal, comprising: Based on the communication connection between the user and the mobile terminal, the first image displayed in the airborne floating image is also displayed as the first image on the screen of the mobile terminal, and / or the second image displayed on the screen of the mobile terminal is also displayed as the second image in the airborne floating image, Based on the detection and determination of the user's operation on the first image displayed in the airborne floating image, a predetermined first process associated with the first image of the airborne floating image is executed, and the first process is also reflected on the first image on the screen of the mobile terminal, and / or based on the detection and determination of the user's operation on the second image displayed on the screen of the mobile terminal on the mobile terminal side, a predetermined second process associated with the second image on the screen of the mobile terminal is executed, and the second process is also reflected on the second image of the airborne floating image. An airborne floating image display device.

2. In the airborne floating image display device according to Claim 1, As a setting after the communication connection, the validity / invalidity of the user's operation and detection on the display image of the airborne floating image on the side of the airborne floating image display device and the user's operation and detection on the display image of the screen on the side of the mobile terminal are set, and the operation, detection, display, and processing are controlled according to the setting. An airborne floating image display device.

3. In the airborne floating image display device according to Claim 2, At the time of the communication connection, the validity / invalidity setting after the communication connection is performed, At the time of cancellation of the communication connection, the validity / invalidity setting is restored to the setting before the communication connection. An airborne floating image display device.

4. In the airborne floating image display device according to Claim 2, The user's operation on the display image of the airborne floating image or the screen of the mobile terminal includes a touch operation and a rotation operation. Regarding the touch operation, it is detected using the touch sensor among the user operation detection mechanisms provided in the airborne floating image display device or the touch sensor provided in the mobile terminal. Regarding the rotation operation, it is detected using the attitude sensor provided in the airborne floating image display device or the attitude sensor provided in the mobile terminal. Airborne floating image display device.

5. In the airborne floating image display device according to claim 2, In a certain mode, the user's operations and detections on the display image of the airborne floating image on the side of the airborne floating image display device are effectively set, and the user's operations and detections on the display image of the screen on the side of the mobile terminal are invalidated. Airborne floating image display device.

6. In the airborne floating image display device according to claim 2, In a certain mode, the user's operations and detections on the display image of the airborne floating image on the side of the airborne floating image display device are invalidated, and the user's operations and detections on the display image of the screen on the side of the mobile terminal are effectively set. Airborne floating image display device.

7. In the airborne floating image display device according to claim 2, In a certain mode, the user's operations and detections on the display image of the airborne floating image on the side of the airborne floating image display device are effectively set, and the user's operations and detections on the display image of the screen on the side of the mobile terminal are also effectively set. Airborne floating image display device.

8. In the airborne floating image display device according to claim 2, On the airborne floating image of the airborne floating image display device and / or on the screen of the mobile terminal, display options for modes corresponding to the effective / invalid settings for the user's operations and detections on the display image of the airborne floating image on the side of the airborne floating image display device and the user's operations and detections on the display image of the screen on the side of the mobile terminal, and set the effective / invalid according to the user's selection of the mode. Airborne floating image display device.

9. In the airborne floating image display device according to claim 4, As the touch sensor provided in the airborne floating image display device, it has an air operation detection sensor or a camera that detects the position of the user's finger or operating object with respect to the display range of the airborne floating image. Airborne floating image display device.

10. In the airborne floating image display device according to claim 4, the rotation operation includes the case of rotating the housing of the airborne floating image display device or the housing of the portable terminal, and the predetermined process according to the determination of the rotation operation includes displaying the display image in the airborne floating image or the display image on the screen of the portable terminal in a rotated state as viewed by the user. Airborne floating image display device.

11. In the airborne floating image display device according to claim 1, at the time of the communication connection, based on the detection of contact of the portable terminal with the display range of the airborne floating image, the first image displayed on the airborne floating image is also displayed as the first image on the screen of the portable terminal, and / or the second image displayed on the screen of the portable terminal is also displayed as the second image on the airborne floating image. Airborne floating image display device.

12. A display method in a system in which an airborne floating image display device and a user's portable terminal are communicatively connected, wherein the airborne floating image display device includes an image processing unit that performs image processing, a display unit that displays the image processed by the image processing unit, an optical system that generates an airborne floating image based on the image displayed by the display unit, a user operation detection mechanism that detects an operation by the user on the display range of the airborne floating image, and a communication unit that communicates with the user's portable terminal, and comprises: a step of establishing a communication connection between the airborne floating image display device and the user's portable terminal; a step of, based on the communication connection, displaying the first image displayed on the airborne floating image also as the first image on the screen of the portable terminal and / or displaying the second image displayed on the screen of the portable terminal also as the second image on the airborne floating image; a step of, based on the detection and determination of the user's operation on the first image displayed on the airborne floating image, executing a predetermined first process associated with the first image of the airborne floating image and controlling to reflect the first process also on the first image on the screen of the portable terminal, and / or, based on the detection and determination of the user's operation on the second image displayed on the screen of the portable terminal on the portable terminal side, causing the portable terminal to execute a predetermined second process associated with the second image on the screen of the portable terminal and controlling to reflect the second process also on the second image of the airborne floating image; A display method having the above steps.

13. A mobile terminal in a system in which an airborne floating image display device and a user's mobile terminal are communicatively connected, wherein the airborne floating image display device includes: an image processing unit that performs image processing; a display unit that displays the image processed by the image processing unit; an optical system that generates an airborne floating image based on the image displayed by the display unit; a user operation detection mechanism that detects an operation by the user with respect to the display range of the airborne floating image; a communication unit that communicates with the user's mobile terminal; and the mobile terminal establishes a communication connection with the airborne floating image display device, and based on the communication connection, displays the first image displayed in the airborne floating image also as the first image on the screen of the mobile terminal, and / or causes the second image displayed on the screen of the mobile terminal to be also displayed as the second image in the airborne floating image, and based on the detection and determination of the user's operation on the first image displayed in the airborne floating image, causes the predetermined first process associated with the first image in the airborne floating image to be executed, and controls so as to reflect the first process also on the first image on the screen of the mobile terminal, and / or based on the detection and determination of the user's operation on the second image displayed on the screen of the mobile terminal on the mobile terminal side, executes the predetermined second process associated with the second image on the screen of the mobile terminal, and controls so as to reflect the second process also on the second image in the airborne floating image. Mobile terminal.

Images