Aerial floating video display apparatus

Abstract

To provide a more preferable aerial floating video display apparatus.SOLUTION: An aerial floating video display apparatus according to the present invention has a video processing unit, a display unit for displaying a video subjected to video processing by the video processing unit, an optical system for generating an aerial floating video based on the video displayed by the display unit, and a detection mechanism for detecting an aerial operation by a user with respect to the aerial floating video. Based on individual identification of a user, an aerial operation for each identified user is registered, and in the aerial floating video, the aerial operation for the user is received.SELECTED DRAWING: Figure 46

Description

Technical Field

[0001] The present invention relates to an airborne video display device.

Background Art

[0002] Regarding airborne information display technology, for example, it is disclosed in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the disclosure of Patent Document 1, the consideration regarding the configuration for obtaining practical brightness and quality of the airborne video, and the configuration for allowing the user to visually recognize the airborne video more enjoyably, etc. was not sufficient.

[0005] An object of the present invention is to provide a more suitable airborne video display device.

Means for Solving the Problems

[0006] In order to solve the above problems, for example, the configuration described in the claims is adopted. This application includes a plurality of means for solving the above problems. If an example is given, the airborne video display device includes a video processing unit, a display unit that displays the video processed by the video processing unit, an optical system that generates an airborne video based on the video displayed by the display unit, and a detection mechanism that detects the user's airborne operation on the airborne video. Based on the identification of the user's individual, for each identified individual, the user's airborne operation is registered, and in the airborne video, the user's airborne operation is received.

Effects of the Invention

[0007] According to the present invention, a more suitable airborne floating video display device can be realized. Other problems, configurations, and effects will be clarified in the following description of the embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 2I

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 4F

Figure 4G

Figure 4H

Figure 4I

Figure 4J

Figure 4K

Figure 4L

Figure 4M

Figure 4N

Figure 4O

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21A

Figure 21B

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29A

Figure 29B

Figure 30A

Figure 30B

Figure 30C

Figure 30D

Figure 30E

Figure 31A

Figure 31B

Figure 32A

Figure 32B

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41A

Figure 41B

Figure 42A

Figure 42B

Figure 43

Figure 44A

Figure 44B

Figure 45

Figure 46

Figure 47A

Figure 47B

Figure 48

Figure 49A

Figure 49B

Figure 50A

Figure 50B

Figure 50C

Figure 51

Figure 52

Figure 53A

Figure 53B

Figure 53C

Figure 54

Figure 55

Figure 56

Figure 57A

Figure 57B

Figure 57C

Figure 57D

Figure 57E

Figure 58

Figure 59

Figure 60

Figure 61

Figure 62A

Figure 62B

Figure 63

Figure 64

Figure 65

Figure 66

Figure 67A

Figure 67B

Figure 68

Figure 69

Figure 70

Figure 71

Figure 72A

Figure 72B

Figure 73

Figure 74

Figure 75

Figure 76

Figure 77

Figure 78

Figure 79

Figure 80

Figure 81

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the examples, and various changes and modifications can be made by those skilled in the art within the scope of the technical idea disclosed in this specification. Also, in all the drawings for explaining the present invention, those having the same function are given the same reference numerals, and the repeated description thereof may be omitted.

[0010] The following examples relate to a video display device capable of transmitting a video by video light from a video light source through a transparent member that partitions a space such as glass and displaying it as a space-floating video outside the transparent member. In the description of the following examples, a video floating in space is expressed by the term "space-floating video". Instead of this term, it may be expressed as "aerial image", "space image", "aerial floating video", "space-floating optical image of the display video", "aerial floating optical image of the display video", etc. The term "space-floating video" mainly used in the description of the examples is used as a representative example of these terms.

[0011] According to the following embodiments, for example, a video display device suitable for use in bank ATMs, station ticket vending machines, digital signage, etc. can be realized. For example, currently, touch panels are usually used in bank ATMs, station ticket vending machines, etc., but by using a transparent glass surface or a light-transmissive plate material, high-resolution video information can be displayed in a state of floating in space on this glass surface or light-transmissive plate material. At this time, by making the divergence angle of the emitted video light small, that is, an acute angle, and further aligning it with a specific polarization, only the regular reflected light can be efficiently reflected to the retroreflective plate, so the light utilization efficiency is high, and the ghost image that occurs in addition to the main space floating image, which has been a problem in the conventional retroreflective method, can be suppressed, and a clear space floating video can be obtained. In addition, by means of the device including the light source of this embodiment, a novel and highly usable space floating video display device (space floating video display system) capable of significantly reducing power consumption can be provided. Also, for example, a vehicle-use space floating video display device capable of so-called unidirectional space floating video display that can be visually recognized inside and / or outside the vehicle can be provided. <Example 1>

[0012] <An example of the usage form of the space floating video display device> FIG. 1 is a diagram showing an example of a usage form of a spatial floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the spatial floating image display device according to the present embodiment. The specific configuration of the spatial floating image display device will be described in detail with reference to FIG. 2 and the like. Light of a specific polarization with a sandwiching angle directivity characteristic is emitted as an image light beam from the image display device 1, once enters the retroreflective plate 2 after being reflected, etc. in the optical system in the spatial floating image display device, is retroreflected, passes through a transparent member 100 (such as glass), and forms a real image, an aerial image (spatial floating image 3), outside the glass surface. In the following embodiments, the retroreflective plate 2 (retroreflective plate) will be used as an example of the retroreflective member. However, the retroreflective plate 2 of the present invention is not limited to a planar plate, and is used as an example of a concept including a sheet-like retroreflective member attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflective member is attached to a planar or non-planar member. Further, since the light beam after reflection by the retroreflective plate 2 has an optical characteristic of forming an image, the retroreflective plate 2 may be expressed as an imaging optical member or an imaging optical plate.

[0013] Also, in a store or the like, a space is partitioned by a show window (also referred to as "window glass") 105 which is a translucent member such as glass. According to the spatial floating image display device of the present embodiment, it is possible to display a floating image in one direction with respect to the outside and / or the inside of the store (space) through such a transparent member.

[0014] In FIG. 1, the inside (inside the store) of the window glass 105 is shown in the depth direction and the outside (for example, a sidewalk) is shown in the front. On the other hand, by providing means for reflecting a specific polarization wave on the window glass 105, it is also possible to reflect and form an aerial image at a desired position inside the store.

[0015] <Configuration example of the optical system of the spatial floating image display device> FIG. 2A is a diagram showing an example of the configuration of an optical system of a spatial floating image display device according to an embodiment of the present invention. Using FIG. 2A, the configuration of the spatial floating image display device will be described more specifically. As shown in FIG. 2A(1), in an oblique direction of a transparent member 100 such as glass, a display device 1 that diverges image light of a specific polarization at a sandwiching angle is provided. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having a sandwiching diffusion characteristic.

[0016] The image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 having a film that selectively reflects the image light of a specific polarization provided on the transparent member 100 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100), and is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light is polarization-converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, when entering and exiting the retroreflector 2. Here, since the polarization separation member 101 that selectively reflects the image light of a specific polarization has the property of transmitting the polarization of the other polarization that has been polarization-converted, the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a spatial floating image 3, which is a real image, outside the transparent member 100. Note that, in FIG. 2A, an example is described in which the principal ray of the image light incident on the retroreflector 2 is incident at 90° with respect to the retroreflector 2. However, the incident angle of the principal ray of the image light with respect to the retroreflector 2 is not limited to 90°, and for example, it can also be used at 90° ± 15°.

[0017] Here, a first example of the polarization design in the optical system of FIG. 2A will be described. For example, the configuration may be such that S-polarized video light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized video light that has reached the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from S-polarized light to P-polarized light. The video light converted to P-polarized light travels toward the polarization separation member 101 again. Here, since the polarization separation member 101 has the property of reflecting S-polarized light and transmitting P-polarized light, the P-polarized video light passes through the polarization separation member 101 and then through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a mirror relationship with the display image of the display device 1 with respect to the polarization separation member 101. Such a polarization design can preferably form the spatial floating image 3.

[0018] Next, a second example of the polarization design in the optical system of FIG. 2A will be described. For example, the configuration may be such that P-polarized video light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized video light that has reached the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from P-polarized light to S-polarized light. The video light converted to S-polarized light travels toward the polarization separation member 101 again. Here, since the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light, the S-polarized video light passes through the polarization separation member 101 and then through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a mirror image relationship with the display image of the display device 1 with respect to the polarization separation member 101. Such a polarization design can preferably form the spatial floating image 3.

[0019] Note that the light forming the spatial floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the spatial floating image 3, and these light rays continue to travel straight even after passing through the optical image of the spatial floating image 3. Therefore, unlike the diffused video light formed on a screen by a general projector or the like, the spatial floating image 3 is an image with high directivity. Thus, in the configuration of FIG. 2A, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in a system that displays an image that requires high security or an image with high confidentiality that needs to be concealed from a person facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axes of the image light after reflection may become uneven. Also, the reflection angles may become uneven. Such uneven light may not maintain the polarization state and propagation angle assumed in the design. For example, light with a polarization state and propagation angle outside the design assumption may directly re-enter the image display surface side of the liquid crystal display panel 11 from the position of the retroreflector 2 without passing through the polarization separation member. Light with a polarization state and propagation angle outside the design assumption may also re-enter the image display surface side of the liquid crystal display panel 11 after being reflected by components within the spatial floating image display device. The light that re-enters the image display surface side of such a liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and potentially degrading the image quality of the spatial floating image. Therefore, in this embodiment, an absorption-type polarizing plate 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorption-type polarizing plate 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorption-type polarizing plate 12, thereby suppressing the above re-reflection. Thereby, it is possible to prevent the degradation of the image quality due to the ghost image of the spatial floating image. Specifically, if the configuration is such that S-polarized image light is emitted from the display device 1 to the polarization separation member 101, the absorption-type polarizing plate 12 may be a polarizing plate that absorbs P-polarization. Also, if the configuration is such that P-polarized image light is emitted from the display device 1 to the polarization separation member 101, the absorption-type polarizing plate 12 may be a polarizing plate that absorbs S-polarization.

[0021] The above-described polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects a specific polarization wave.

[0022] Next, FIG. 2A(2) shows an example of the surface shape of a representative retroreflector 2 as the retroreflector. Light rays incident inside regularly arranged hexagonal prisms are reflected by the wall surfaces and bottom surfaces of the hexagonal prisms and are emitted as retroreflected light in a direction corresponding to the incident light, displaying a spatial floating image that is a real image based on the image displayed on the display device 1.

[0023] The resolution of this floating image in space depends greatly on the outer shape D and pitch P of the retroreflective portion of the retroreflector 2 shown in Fig. 2A(2), in addition to the resolution of the liquid crystal display panel 11. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, for example, one pixel of the floating image in space corresponds to 300 μm. Therefore, the effective resolution of the floating video image is reduced to about one-third.

[0024] Therefore, in order to make the resolution of the floating video image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion closer to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moiré due to the pixels of the retroreflector and the liquid crystal display panel, it is advisable to design with the pitch ratio of each removed from an integer multiple of one pixel. Also, the shape should be arranged so that none of the sides of the retroreflective portion overlap any of the sides of one pixel of the liquid crystal display panel.

[0025] Note that the surface shape of the retroreflector according to this embodiment is not limited to the above example. It may have various surface shapes that realize retroreflection. Specifically, a retroreflective element in which triangular pyramid prisms, hexagonal pyramid prisms, other polygonal prisms, or combinations thereof are periodically arranged may be provided on the surface of the retroreflector of this embodiment. Alternatively, a retroreflective element that forms a cube corner by periodically arranging these prisms may be provided on the surface of the retroreflector of this embodiment. These can also be expressed as a corner reflector array or a polyhedral reflector array. Alternatively, a capsule lens type retroreflective element in which glass beads are periodically arranged may be provided on the surface of the retroreflector of this embodiment. Since the detailed configuration of these retroreflective elements may use existing technologies, detailed description is omitted. Specifically, the technologies disclosed in Japanese Patent Application Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, 2009-229942, etc. may be used.

[0026] <Another Configuration Example 1 of the Optical System of the Spatial Floating Image Display Device> Another configuration example of the optical system of the spatial floating image display device will be described with reference to FIG. 2B. In FIG. 2B, components labeled with the same reference numerals as those in FIG. 2A have the same functions and configurations as in FIG. 2A. For the sake of simplicity, repeated descriptions of such components will be omitted.

[0027] In the optical system of FIG. 2B, similar to FIG. 2A, image light of a specific polarization is output from the display device 1. The image light of the specific polarization output from the display device 1 is input to the polarization separation member 101B. The polarization separation member 101B is a member that selectively transmits the image light of the specific polarization. Different from the polarization separation member 101 in FIG. 2A, the polarization separation member 101B is not integrated with the transparent member 100 and has an independent plate-like shape. Therefore, the polarization separation member 101B may be referred to as a polarization separation plate. The polarization separation member 101B may be configured, for example, as a reflective polarizing plate formed by attaching a polarization separation sheet to a transparent member. Alternatively, it may be formed of a metal multilayer film or the like that selectively transmits a specific polarization of light through the transparent member and reflects the polarization of other specific polarizations. In FIG. 2B, the polarization separation member 101B is configured to transmit the image light of the specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarization-converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when entering the retroreflector and once when exiting. Here, since the polarization separation member 101B has the property of reflecting the polarization of the other polarization that has been polarization-converted by the λ / 4 plate 21, the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a spatial floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of the polarization design in the optical system of FIG. 2B will be described. For example, the configuration may be such that P-polarized video light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized video light that has reached the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from P-polarized light to S-polarized light. The video light converted to S-polarized light travels toward the polarization separation member 101B again. Here, since the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light, the S-polarized video light is reflected by the polarization separation member 101 and passes through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a mirror relationship with the display image of the display device 1 with respect to the polarization separation member 101B. Such a polarization design can preferably form the spatial floating image 3.

[0030] Next, a second example of the polarization design in the optical system of FIG. 2B will be described. For example, the configuration may be such that the S-polarized video light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized video light that has reached the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and travels toward the retroreflector 2. When the video light is reflected by the retroreflector 2, since it passes through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2 twice, the video light is converted from S-polarized light to P-polarized light. The video light converted to P-polarized light travels toward the polarization separation member 101B again. Here, since the polarization separation member 101B has the property of reflecting P-polarized light and transmitting S-polarized light, the P-polarized video light is reflected by the polarization separation member 101 and passes through the transparent member 100. Since the video light that has passed through the transparent member 100 is the light generated by the retroreflector 2, a spatial floating image 3, which is an optical image of the display image of the display device 1, is formed at a position that has a mirror relationship with the display image of the display device 1 with respect to the polarization separation member 101B. With such a polarization design, the spatial floating image 3 can be preferably formed.

[0031] In Fig. 2B, the video display surface of the display device 1 and the surface of the retroreflective plate 2 are arranged in parallel. The polarization separation member 101B is arranged at an angle α (e.g., 30°) with respect to the video display surface of the display device 1 and the surface of the retroreflective plate 2. Then, in the reflection of the polarization separation member 101B, the traveling direction of the video light reflected by the polarization separation member 101B (the direction of the principal ray of the video light) is different from the traveling direction of the video light incident from the retroreflective plate 2 (the direction of the principal ray of the video light) by an angle β (e.g., 60°). By configuring in this way, in the optical system of Fig. 2B, the video light is output at a predetermined angle shown toward the outside of the transparent member 100, and the spatial floating image 3 which is a real image is formed. In the configuration of Fig. 2B, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable when adopted in a system for displaying an image that requires high security or an image with high secrecy that needs to be concealed from a person facing the user.

[0032] As described above, in the optical system of Fig. 2B, although it is an optical system with a configuration different from that of the optical system of Fig. 2A, a suitable spatial floating image can be formed in the same manner as the optical system of Fig. 2A.

[0033] An absorption type polarizing plate may be provided on the surface of the transparent member 100 on the side of the polarization separation member 101B. The absorption type polarizing plate may be an absorption type polarizing plate that transmits the polarization wave of the video light from the polarization separation member 101B and absorbs the polarization wave whose phase is 90° different from the polarization wave of the video light from the polarization separation member 101B. In this way, while allowing sufficient transmission of the video light for forming the spatial floating image 3, the external light incident from the side of the spatial floating image 3 of the transparent member 100 can be reduced by about 50%. Thereby, the stray light in the optical system of Fig. 2B based on the external light incident from the side of the spatial floating image 3 of the transparent member 100 can be reduced.

[0034] <Another configuration example 2 of the optical system of the spatial floating image display device> Another configuration example of the optical system of the spatial floating image display device will be described with reference to FIG. 2C. In FIG. 2C, components labeled with the same reference numerals as in FIG. 2B have the same functions and configurations as those in FIG. 2B. For such components, repetitive descriptions will be omitted for the sake of simplicity.

[0035] The difference between the optical system of FIG. 2C and the optical system of FIG. 2B lies only in the arrangement angle of the polarization beam splitter member 101B with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. Since all other configurations are the same as those of the optical system of FIG. 2B, repetitive descriptions will be omitted. Since the polarization design of the optical system of FIG. 2C is the same as that of the optical system of FIG. 2B, repetitive descriptions will be omitted.

[0036] In the optical system of FIG. 2C, the polarization beam splitter member 101B is arranged at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C, the angle α is 45°. With this configuration, in the reflection of the polarization beam splitter member 101B, the angle β formed between the traveling direction of the image light incident from the retroreflector 2 (the direction of the principal ray of the image light) and the traveling direction of the image light reflected by the polarization beam splitter member 101B (the direction of the principal ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are in a right-angle relationship with the traveling direction of the image light reflected by the polarization beam splitter member 101B, and the angular relationship of the surfaces constituting the optical system can be simplified. If the surface of the transparent member 100 is arranged perpendicular to the traveling direction of the image light reflected by the polarization beam splitter member 101B, the angular relationship of the surfaces constituting the optical system can be further simplified. In the configuration of FIG. 2C, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in systems that display images requiring high security or highly confidential images that need to be concealed from people facing the user.

[0037] As described above, the optical system in FIG. 2C is an optical system with a configuration different from those of the optical systems in FIGS. 2A and 2B, but can form a suitable aerial image in the same manner as the optical systems in FIGS. 2A and 2B. Also, the angles of the surfaces constituting the optical system can be made simpler.

[0038] Note that an absorption-type polarizing plate may be provided on the surface of the transparent member 100 on the side of the polarization separation member 101B. The absorption-type polarizing plate may be an absorption-type polarizing plate that transmits the polarization wave of the video light from the polarization separation member 101B and absorbs the polarization wave having a phase difference of 90° from the polarization wave of the video light from the polarization separation member 101B. In this way, while allowing sufficient transmission of the video light for forming the aerial image 3, it is possible to reduce by about 50% the external light incident from the side of the aerial image 3 of the transparent member 100. Thereby, it is possible to reduce stray light in the optical system of FIG. 2C based on the external light incident from the side of the aerial image 3 of the transparent member 100.

[0039] <Another Configuration Example 3 of the Optical System of the Aerial Image Display Device> Another configuration example of the optical system of the aerial image display device will be described with reference to FIG. 2D. The optical system of FIG. 2D is an optical system using a retroreflector 5, which is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, with reference to FIGS. 2D to 2I, another configuration example 3 of the optical system will be described more specifically. In FIG. 2D, the components denoted by the same reference numerals as in FIGS. 2A to 2C have the same functions and configurations as in FIGS. 2A to 2C. For such components, repetitive description will not be given in order to simplify the explanation.

[0040] FIG. 2D is a diagram showing an example of the main configuration and the retroreflective portion configuration of an aerial image display device according to an embodiment of the present invention. In an oblique direction of a transparent member 100 such as glass, a display device 10 that emits video light is provided. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] The chief ray 9020, which represents the light beam emitted from the display device 10, travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and it can also be used, for example, within the range of 45° ± 15°.

[0042] The retroreflector 5 is an optical member having the optical property of retroreflecting at least part of the light rays in some directions. Also, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may be referred to as an imaging optical member or an imaging optical plate.

[0043] Regarding the specific configuration of the retroreflector 5, it will be described in detail with reference to FIGS. 2E, 2F, etc. Due to the retroreflector 5, the chief ray 9020 is retroreflected in the x and y directions while traveling in the z direction. As a result, the reflected ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is specularly symmetric with respect to the chief ray 9020 with reference to the retroreflector 5, passes through the transparent member 100, and forms a spatial floating image 3 as a real image on the imaging surface.

[0044] The light beam forming the spatial floating image 3 is a collection of light rays converging from the retroreflector 5 to the optical image of the spatial floating image 3, and these light rays continue to travel straight even after passing through the optical image of the spatial floating image 3. Therefore, unlike the diffused image formed on a screen by a general projector or the like, the spatial floating image 3 is an image with high directivity. Thus, in the configuration of FIG. 2, when viewed by the user from the direction of arrow A, the spatial floating image 3 is viewed as a bright image. However, when viewed by another person from the direction of arrow B, the spatial floating image 3 cannot be viewed as an image at all. This characteristic is suitable for use in systems that display images requiring high security or highly confidential images that need to be concealed from people facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described with reference to FIGS. 2E and 2F. The retroreflector 5 has a configuration in which a plurality of corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may be referred to as a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail with reference to FIGS. 2G, 2H, and 2I. The light rays 9111, 9112, 9113, and 9114 emitted from the light source 9110 are reflected twice by the two mirror surfaces 9041 and 9042 of the corner reflector 9040 to become reflected light rays 9121, 9122, 9123, and 9124. This double reflection results in a retroreflective reflection that folds back in the same direction as the incident direction (advances in the direction rotated by 180°) with respect to the x and y directions, and a regular reflection in which the incident angle and the reflection angle coincide due to total reflection with respect to the z direction.

[0046] That is, the light rays 9111 to 9114 generate the reflected light rays 9121 to 9124 on a straight line symmetric with respect to the z direction with respect to the corner reflector 9040, and form an aerial image 9120. The light rays 9111 to 9114 emitted from the light source 9110 are four light rays representing the diffused light from the light source 9110. Depending on the diffusion characteristics of the light source 9110, the light rays incident on the retroreflector 5 are not limited to these, but any incident light ray causes a similar reflection and forms an aerial image 9120. For ease of viewing the drawing, the positions of the light source 9110 and the aerial image 9120 in the x direction are shifted in the drawing, but actually, the positions of the light source 9110 and the aerial image 9120 in the x direction are the same, and they overlap when viewed from the z direction.

[0047] Next, the configuration and effect of the corner reflector 9040 that constitutes the retroreflector 5 will be described with reference to FIGS. 2G, 2H, and 2I. The corner reflector 9040 is a rectangular parallelepiped in which only two specific surfaces are the mirror surfaces 9041 and 9042, and the other four surfaces are formed of a transparent member. The retroreflector 5 has a configuration in which the corner reflectors 9040 are arranged in an array such that their corresponding mirror surfaces face the same direction.

[0048] When viewed from above (+z direction), the light ray 9111 emitted from the light source 9110 is incident on the mirror surface 9041 (or mirror surface 9042) at a specific incident angle, and after total reflection at the reflection point 9130, it is totally reflected again at the reflection point 9132 on the mirror surface 9042 (or mirror surface 9041).

[0049] Assuming that the incident angle of the light ray 9111 with respect to the mirror surface 9041 (or mirror surface 9042) is θ, the incident angle of the first reflected light ray 9131 reflected by the mirror surface 9041 (or mirror surface 9042) with respect to the mirror surface 9042 (or mirror surface 9041) can be expressed as 90° - θ. Therefore, for the light ray 9111, the second reflected light ray 9121 obtains a rotation of 2θ by the first reflection and 2×(90° - θ) by the second reflection, resulting in a total reverse optical path of 180°. On the other hand, when viewed from the side (the intermediate direction between -x and -y), total reflection with respect to the z direction occurs only once. Therefore, assuming that the incident angle with respect to the mirror surface 9041 or mirror surface 9042 is φ, for the light ray 9111, the reflected light ray 9121 obtains a rotation of 2×φ by one reflection.

[0050] From the above, the light rays incident on the corner reflector 9040 produce a recursive reflection with a reverse optical path in the x and y directions and a regular reflection by total reflection in the z direction. Considering the retroreflective plate 5, similar reflections are caused in each optical path, so that an image is formed at a point symmetric with respect to the z-axis direction by a convergent reverse optical path with respect to the x and y directions.

[0051] Here, in the optical systems of FIGS. 2A to 2C, the retroreflective plate 2 has recursive reflection characteristics in three axial directions. Thereby, when a diffusive incident light beam is incident on the retroreflective plate 2, a convergent reflected light beam travels toward the side where the light source of the incident light ray exists with respect to the retroreflective plate 2. The main light ray of the convergent reflected light beam reflected from the retroreflective plate 2 is in the reverse direction of the main light ray of the diffusive incident light beam incident on the retroreflective plate 2.

[0052] On the other hand, in the optical system of FIG. 2D, the retroreflective plate 5 has retroreflective characteristics in two axial directions and specularly reflects in the other one axial direction. Thus, when a diffusive incident light beam is incident on the retroreflective plate 5, the convergent reflected light beam reflected by the corner reflector array travels toward the side opposite to the side where the light source of the incident light ray exists with respect to the retroreflective plate 5. The convergent reflected light beam forms a spatial floating image 3 by imaging in the air.

[0053] The traveling direction of the principal ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflective plate 5 does not become the reverse direction of the traveling direction of the principal ray of the diffusive incident light beam incident on the retroreflective plate 5. The normal direction component of the plate-shaped surface of the retroreflective plate 5 with respect to the traveling direction of the principal ray of the diffusive incident light beam incident on the retroreflective plate 5 and the normal direction component of the plate-shaped surface of the retroreflective plate 5 with respect to the traveling direction of the principal ray after the light beam is reflected by the retroreflective plate 5 and becomes a convergent reflected light beam travel straight without changing before and after reflection by the corner reflector array.

[0054] That is, due to the reflection in the retroreflective plate 5, the diffusive incident light beam is converted into a convergent reflected light beam, but in the normal direction of the plate-shaped surface of the retroreflective plate 5, the light beam travels so as to pass through the retroreflective plate 5. Here, the diffusive incident light beam incident on the retroreflective plate 5 and the convergent reflected light beam emitted from the retroreflective plate 5 are geometrically plane-symmetric with respect to the plate-shaped surface of the retroreflective plate 5.

[0055] The resolution of the spatial floating image formed by the light ray from the video output unit 10 depends greatly on the diameter D and pitch P (not shown) of the retroreflective portion of the retroreflective plate 5 shown in FIGS. 2E and 2F in addition to the resolution of the liquid crystal display panel 11. For example, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch P is 300 μm, for example, one pixel of the spatial floating image corresponds to 300 μm. Therefore, the effective resolution of the spatial floating video is reduced to about 1 / 3.

[0056] Therefore, in order to make the resolution of the floating image in space equivalent to that of the display device 10, it is desirable to make the diameter D and the pitch P of the retroreflective portion approach one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moiré due to the pixels of the retroreflective plate and the liquid crystal display panel, it is advisable to design by deviating the respective pitch ratios from an integral multiple of one pixel. Also, the shape should be arranged so that none of the sides of the retroreflective portion overlap any of the sides of one pixel of the liquid crystal display panel.

[0057] Note that the shape of the retroreflective plate (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that realize retroreflectivity. Specifically, various cubic corner bodies, corner reflector arrays may be used, or a slit mirror array, a two-sided corner reflector array, a multi-sided reflector array, or a shape in which combinations of its reflecting surfaces are periodically arranged may be used. Alternatively, a capsule lens type retroreflective element in which glass beads are periodically arranged may be provided on the surface of the retroreflective plate of this embodiment. Since the detailed configuration of these retroreflective elements may use existing technologies, detailed description is omitted. Specifically, the technologies disclosed in JP-A-2017-33005, JP-A-2019-133110, JP-A-2017-67933, WO2009 / 131128, etc. may be used.

[0058] In the optical system of FIG. 2D, the video light emitted from the display device 10 may be in any polarization state. There is no problem whether it is S-polarized light or P-polarized light.

[0059] As described above, in the optical system of FIG. 2D, although it is an optical system using a retroreflective plate different from the optical systems of FIGS. 2A to 2C, a more suitable floating image in space can be formed in the same manner as the optical systems of FIGS. 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, a brighter and higher-quality floating image in space can be provided.

[0061] <<Block Diagram of the Internal Configuration of the Spatial Floating Image Display Device>>

[0062] Next, a block diagram of the internal configuration of the spatial floating image display device 1000 will be described. FIG. 3 is a block diagram showing an example of the internal configuration of the spatial floating image display device 1000.

[0063] The spatial floating image display device 1000 includes a retroreflective unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power input interface 1111, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an air operation detection sensor 1351, an air operation detection unit 1350, an audio output unit 1140, a microphone 1139, a video control unit 1160, a storage unit 1170, an imaging unit 1180, etc. Note that a removable media interface 1134, an attitude sensor 1113, a transmissive self-emitting image display device 1650, a second display device 1680, or a secondary battery 1112, etc. may also be provided.

[0064] Each component of the spatial floating image display device 1000 is arranged in a housing 1190. Note that the imaging unit 1180 and the air operation detection sensor 1351 shown in FIG. 3 may be provided outside the housing 1190.

[0065] The retroreflective unit 1101 in FIG. 3 corresponds to the retroreflective plate 2 in FIGS. 2A, 2B, and 2C. The retroreflective unit 1101 retroreflects the light modulated by the image display unit 1102. Among the reflected light from the retroreflective unit 1101, the spatial floating image 3 is formed by the light output to the outside of the spatial floating image display device 1000.

[0066] The image display unit 1102 in FIG. 3 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. The light source 1105 in FIG. 3 corresponds to the light source device 13 in FIGS. 2A, 2B, and 2C. And the image display unit 1102, the light guide 1104, and the light source 1105 in FIG. 3 correspond to the display device 1 in FIGS. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating the transmitted light based on the input video signal under the control of the video control unit 1160 described later. The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. As the video display unit 1102, for example, a transmissive liquid crystal panel is used. Also, as the video display unit 1102, for example, a reflective liquid crystal panel or a DMD (Digital Micromirror Device: registered trademark) panel that modulates the reflected light may be used.

[0068] The light source 1105 generates light for the video display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power supply 1106 converts the AC current input from the outside through the external power supply input interface 1111 into a DC current and supplies power to the light source 1105. Also, the power supply 1106 supplies the necessary DC current to each part in the spatial floating video display device 1000. The secondary battery 1112 stores the power supplied from the power supply 1106. Also, when no power is supplied from the outside through the external power supply input interface 1111, the secondary battery 1112 supplies power to the light source 1105 and other components that require power. That is, when the spatial floating video display device 1000 is equipped with the secondary battery 1112, the user can use the spatial floating video display device 1000 even when no power is supplied from the outside.

[0069] The light guide 1104 guides the light generated by the light source 1105 and irradiates the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be referred to as the backlight of the video display unit 1102. The light guide 1104 may mainly be configured using glass. The light guide 1104 may mainly be configured using plastic. The light guide 1104 may be configured using a mirror. Various methods are conceivable for the combination of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The air operation detection sensor 1351 is a sensor that detects operations on the spatially floating image 3 by the finger of the user 230. The air operation detection sensor 1351 senses, for example, a range that overlaps with the entire display range of the spatially floating image 3. Note that the air operation detection sensor 1351 may sense only a range that overlaps with at least a part of the display range of the spatially floating image 3.

[0071] Specific examples of the air operation detection sensor 1351 include distance sensors using invisible light such as infrared rays, invisible light lasers, ultrasonic waves, etc. Further, the air operation detection sensor 1351 may be configured by combining a plurality of sensors in a plurality of ways so as to be able to detect coordinates in a two-dimensional plane. Further, the air operation detection sensor 1351 may be composed of a ToF (Time of Flight) type LiDAR (Light Detection and Ranging) or an image sensor.

[0072] The air operation detection sensor 1351 only needs to be able to sense, for example, a touch operation by the user's finger on an object displayed as the spatially floating image 3. Such sensing can be performed using existing technologies.

[0073] The air operation detection unit 1350 acquires a sensing signal from the air operation detection sensor 1351, and based on the sensing signal, determines the presence or absence of contact of the object of the spatially floating image 3 by the finger of the user 230, calculates the position (contact position) where the finger of the user 230 contacts the object, and so on. The air operation detection unit 1350 is composed of a circuit such as an FPGA (Field Programmable Gate Array), for example. Further, some functions of the air operation detection unit 1350 may be realized by software by a space operation detection program executed by the control unit 1110, for example.

[0074] The air operation detection sensor 1351 and the air operation detection unit 1350 may be incorporated into the spatial floating image display device 1000, or may be provided externally as a separate unit from the spatial floating image display device 1000. When provided as a separate unit from the spatial floating image display device 1000, the air operation detection sensor 1351 and the air operation detection unit 1350 are configured to be able to transmit information and signals to the spatial floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Also, the air operation detection sensor 1351 and the air operation detection unit 1350 may be provided as separate units. Thereby, it is possible to construct a system in which the spatial floating image display device 1000 without an air operation detection function is used as the main body and only the air operation detection function can be added as an option. Also, only the air operation detection sensor 1351 may be a separate unit, and the air operation detection unit 1350 may be incorporated into the spatial floating image display device 1000. When it is desired to arrange the air operation detection sensor 1351 more freely with respect to the installation position of the spatial floating image display device 1000, etc., there is an advantage in a configuration in which only the air operation detection sensor 1351 is a separate unit.

[0076] The imaging unit 1180 is a camera having an image sensor, and images the space near the spatial floating image 3 and / or the face, arm, finger, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180 or by using an imaging unit with a depth sensor, the air operation detection unit 1350 can be assisted during the detection process of a touch operation on the spatial floating image 3 by the user 230. The imaging unit 1180 may be provided as a separate unit from the spatial floating image display device 1000. When the imaging unit 1180 is provided as a separate unit from the spatial floating image display device 1000, it may be configured to be able to transmit an imaging signal to the spatial floating image display device 1000 via a wired or wireless communication connection path or the like.

[0077] For example, when the air operation detection sensor 1351 is configured as an object intrusion sensor that detects the presence or absence of an object entering a plane (intrusion detection plane) including the display surface of the floating image 3 in space, the air operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., the user's finger) that has not entered the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, by using information such as the object depth calculation information based on the captured images of the plurality of imaging units 1180 and the object depth information from the depth sensor, the distance between the object and the intrusion detection plane can be calculated. Then, these information and various information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating image 3 in space.

[0079] Alternatively, without using the air operation detection sensor 1351, the air operation detection unit 1350 may detect a touch operation on the floating image 3 in space by the user 230 based on the captured image of the imaging unit 1180.

[0080] Further, the imaging unit 1180 may capture the face of the user 230 who operates the floating image 3 in space, and the control unit 1110 may perform identification processing of the user 230. Also, in order to determine whether there is someone standing around or behind the user 230 who operates the floating image 3 in space and whether the person is peeping at the operation of the user 230 on the floating image 3, the imaging unit 1180 may capture a range including the user 230 who operates the floating image 3 in space and the surrounding area of the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal reception unit such as a remote controller, or an infrared light receiving unit, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. Separately from the aforementioned user 230 who performs a touch operation on the floating image 3 in space, the operation input unit 1107 may be used, for example, by an administrator to operate the floating image display device 1000.

[0082] The video signal input unit 1131 connects to an external video output device and inputs video data. Various digital video input interfaces can be considered for the video signal input unit 1131. For example, it may be configured with a video input interface compliant with the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface compliant with the DVI (Digital Visual Interface) standard, or a video input interface compliant with the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured with an audio input interface compliant with the HDMI standard, an optical digital terminal interface, or a coaxial digital terminal interface, etc. In the case of an interface compliant with the HDMI standard, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an integrated interface with integrated terminals and cables. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be composed of speakers.

[0084] Also, the audio output unit 1140 may output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output digital signals to external devices, such as the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that picks up sounds around the spatial floating video display device 1000, converts them into signals, and generates audio signals. It may be configured such that the microphone records the voice of a person, such as the user's voice, and the control unit 1110 described later performs speech recognition processing on the generated audio signal to obtain character information from the audio signal.

[0085] The non-volatile memory 1108 stores various data used in the airborne image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed in the airborne image 3, display icons, data of objects to be operated by the user's operations, layout information, and the like. The memory 1109 stores video data to be displayed as the airborne image 3, control data of the device, and the like.

[0086] The control unit 1110 controls the operations of each connected unit. Further, the control unit 1110 may perform arithmetic processing based on the information acquired from each unit within the airborne image display device 1000 in cooperation with the program stored in the memory 1109.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. When the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, as a LAN interface conforming to the Ethernet standard. When the communication unit 1132 has a wireless communication interface, it may be configured, for example, as a communication interface of the Wi-Fi system, a communication interface of the Bluetooth system, a mobile communication interface such as 4G or 5G. Through the communication via the communication unit 1132, various data such as video data, image data, and audio data are transmitted and received.

[0088] Also, the removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor element memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as various data including video data, image data, audio data, etc. recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the spatial floating image 3 via the video display unit 1102 and the retroreflective unit 1101.

[0089] The storage unit 1170 is a storage device for recording various information such as various data including video data, image data, audio data, etc. The storage unit 1170 may be composed of a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor element memory such as a solid state drive (SSD). For example, various information such as various data including video data, image data, audio data, etc. may be recorded in the storage unit 1170 in advance when the product is shipped. Also, the storage unit 1170 may record various information such as various data including video data, image data, audio data, etc. acquired from an external device or an external server via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the spatial floating image 3 via the video display unit 1102 and the retroreflective unit 1101. The video data, image data, etc. such as display icons and objects for the user to operate, which are displayed as the spatial floating image 3, are also recorded in the storage unit 1170.

[0091] The layout information of display icons, objects, etc. and various metadata information related to the objects, which are displayed as the spatial floating image 3, are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from, for example, the audio output unit 1140.

[0092] The video control unit 1160 performs various controls on the video signal input to the video display unit 1102. The video control unit 1160 may also be referred to as a video processing circuit and may be composed of hardware such as an ASIC, an FPGA, or a video processor. Note that the video control unit 1160 may also be referred to as a video processing unit or an image processing unit. The video control unit 1160 performs controls such as video switching, for example, which video signal among the video signals stored in the memory 1109 and the video signals (video data) input to the video signal input unit 1131 is to be input to the video display unit 1102.

[0093] Also, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal stored in the memory 1109 and the video signal input from the video signal input unit 1131, and perform control to form a composite video as the spatial floating video 3 by inputting the superimposed video signal to the video display unit 1102.

[0094] Also, the video control unit 1160 may perform control to perform image processing on the video signal input from the video signal input unit 1131, the video signal stored in the memory 1109, etc. Examples of image processing include scaling processing such as enlarging, reducing, and deforming an image, brightness adjustment processing for changing brightness, contrast adjustment processing for changing the contrast curve of an image, and Retinex processing for decomposing an image into light components and changing the weighting for each component.

[0095] Also, the video control unit 1160 may perform special effect video processing or the like to assist the air operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed, for example, based on the detection result of the touch operation of the user 230 by the air operation detection unit 1350 and the captured image of the user 230 by the imaging unit 1180.

[0096] The posture sensor 1113 is a sensor composed of a gravity sensor, an acceleration sensor, or a combination thereof, and can detect the posture in which the spatial floating video display device 1000 is installed. Based on the posture detection result of the posture sensor 1113, the control unit 1110 may control the operations of the connected components. For example, when an unfavorable posture as the user's usage state is detected, the display of the video being displayed on the video display unit 1102 may be stopped, and control may be performed to display an error message to the user. Or, when the posture sensor 1113 detects that the installation posture of the spatial floating video display device 1000 has changed, control may be performed to rotate the display direction of the video being displayed on the video display unit 1102.

[0097] As described so far, the spatial floating video display device 1000 is equipped with various functions. However, the spatial floating video display device 1000 does not necessarily need to have all of these functions, and any configuration may be used as long as it has the function of forming the spatial floating video 3.

[0098] <Configuration Example of Spatial Floating Video Display Device> Next, a configuration example of the spatial floating video display device will be described. The layout of the components of the spatial floating video display device according to this embodiment can have various layouts depending on the usage form. Hereinafter, each layout of FIGS. 4A to 4M will be described. In any of the examples of FIGS. 4A to 4M, the thick line surrounding the spatial floating video display device 1000 shows an example of the housing structure of the spatial floating video display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system of FIG. 2A. In the spatial floating image display device 1000 shown in FIG. 4A, it is installed horizontally so that the surface on which the spatial floating image 3 is formed faces upward. That is, in FIG. 4A, in the spatial floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected. Here, the x-direction is the left-right direction as seen from the user, the y-direction is the front-rear direction (depth direction) as seen from the user, and the z-direction is the up-down direction (vertical direction). Hereinafter, since the definitions of the x-direction, y-direction, and z-direction are the same in each figure of FIG. 4, repeated explanations are omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The spatial floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the spatial floating image 3 is formed faces the front (the direction of the user 230) of the spatial floating image display device 1000. That is, in FIG. 4B, in the spatial floating image display device, a transparent member 100 is installed on the front (the direction of the user 230) of the device. The spatial floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected. Here, as shown in FIG. 4B, the air operation detection sensor 1351 can use the reflection of the sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, since the nail has a higher reflectivity than the fingertip, the accuracy of touch detection can be improved by configuring it in this way.

[0101] FIG. 4C is a diagram showing an example of the configuration of the airborne floating image display device. The airborne floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. In the airborne floating image display device 1000 shown in FIG. 4C, it is installed horizontally so that the surface on which the airborne floating image 3 is formed faces upward. That is, in FIG. 4C, in the airborne floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The airborne floating image 3 is formed above the surface of the transparent member 100 of the airborne floating image display device 1000. The light of the airborne floating image 3 travels in an obliquely upward direction. When the airborne operation detection sensor 1351 is provided as shown in the figure, the operation of the airborne floating image 3 by the finger of the user 230 can be detected.

[0102] FIG. 4D is a diagram showing an example of the configuration of the airborne floating image display device. The airborne floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The airborne floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the airborne floating image 3 is formed faces the front (the direction of the user 230) of the airborne floating image display device 1000. That is, in FIG. 4D, in the airborne floating image display device 1000, a transparent member 100 is installed on the front (the direction of the user 230) of the device. The airborne floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the airborne floating image display device 1000. The light of the airborne floating image 3 travels in an obliquely upward direction. When the airborne operation detection sensor 1351 is provided as shown in the figure, the operation of the airborne floating image 3 by the finger of the user 230 can be detected. Here, as shown in FIG. 4D, the airborne operation detection sensor 1351 can use the reflection of the sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, since the nail has a higher reflectivity than the fingertip, the accuracy of touch detection can be improved by configuring it in this way.

[0103] FIG. 4E is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. In the spatial floating image display device 1000 shown in FIG. 4E, it is installed horizontally so that the surface on which the spatial floating image 3 is formed faces upward. That is, in FIG. 4E, in the spatial floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in the directly upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0104] FIG. 4F is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The spatial floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the spatial floating image 3 is formed faces the front (the direction of the user 230) of the spatial floating image display device 1000. That is, in FIG. 4F, in the spatial floating image display device 1000, a transparent member 100 is installed on the front (the direction of the user 230) of the device. The spatial floating image 3 is formed on the user 230 side with respect to the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in the direction in front of the user's hand. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0105] FIG. 4G is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system of FIG. 2C. In the optical systems of the spatial floating image display devices from FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, within the optical systems of the spatial floating image display devices from FIGS. 4A to 4F, the image light traveled in the front-rear direction and the up-down direction as viewed by the user. In contrast, in the optical system of the spatial floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, within the optical system of the spatial floating image display device shown in FIG. 4G, the image light travels in the left-right direction and the front-rear direction as viewed by the user. In the spatial floating image display device 1000 shown in FIG. 4G, the surface on which the spatial floating image 3 is formed is installed so as to face the front of the device (the direction of the user 230). That is, in FIG. 4G, in the spatial floating image display device 1000, the transparent member 100 is installed on the front of the device (the direction of the user 230). The spatial floating image 3 is formed on the user side with respect to the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in the direction in front of the user. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0106] FIG. 4H is a diagram showing an example of the configuration of the airborne video display device. The airborne video display device 1000 in FIG. 4H is different from the airborne video display device in FIG. 4G in that it has a window with a transparent plate 100B such as glass or plastic on the back of the device (the side opposite to the position where the user 230 views the airborne video 3, that is, the side opposite to the traveling direction of the video light of the airborne video 3 toward the user 230). Since the other configurations are the same as those of the airborne video display device in FIG. 4G, repeated explanations are omitted. The airborne video display device 1000 in FIG. 4H includes a window with a transparent plate 100B at a position on the opposite side of the traveling direction of the video light of the airborne video 3 with respect to the airborne video 3. Therefore, when the user 230 views the airborne video 3, the scenery behind the airborne video display device 1000 can be recognized as the background of the airborne video 3. Therefore, the user 230 can recognize that the airborne video 3 is floating in the air in front of the scenery behind the airborne video display device 1000. Thereby, the floating feeling of the airborne video 3 in the air can be emphasized more.

[0107] Note that depending on the polarization distribution of the video light output from the display device 1 and the performance of the polarization separation member 101B, a part of the video light output from the display device 1 may be reflected by the polarization separation member 101B and may go toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again on the surface of the transparent plate 100B and be visually recognized by the user as stray light. Therefore, in order to prevent the stray light, the window on the back of the airborne video display device 1000 may be configured not to be provided with the transparent plate 100B.

[0108] FIG. 4I is a diagram showing an example of the configuration of the airborne video display device. The airborne video display device 1000 in FIG. 4I is different from the airborne video display device in FIG. 4H in that an opening / closing door 1410 for light shielding is provided in the window of the transparent plate 100B arranged on the back of the device (the side opposite to the position where the user 230 views the airborne video 3). Since the other configurations are the same as those of the airborne video display device in FIG. 4H, repeated explanations are omitted.

[0109] The opening / closing door 1410 of the spatial floating image display device 1000 in Fig. 4I has, for example, a light-shielding plate, and is provided with a mechanism for moving (sliding), rotating, or attaching / detaching the light-shielding plate, so that the window (rear-side window) of the transparent plate 100B located on the back side of the spatial floating image display device 1000 can be switched between an open state and a light-shielded state. The movement (sliding) or rotation of the light-shielding plate by the opening / closing door 1410 may be an electric type driven by a motor (not shown). The motor may be controlled by the control unit 1110 in Fig. 3. In the example of Fig. 4I, an example of the number of light-shielding plates of the opening / closing door 1410 is two. In contrast, the number of light-shielding plates of the opening / closing door 1410 may be one.

[0110] For example, when the scenery visible through the window of the transparent plate 100B of the spatial floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the spatial floating image 3 may become too bright, and the visibility of the spatial floating image 3 for the user 230 may decrease. In such a case, if the rear-side window is shielded from light by moving (sliding), rotating, or attaching the light-shielding plate of the opening / closing door 1410, the background of the spatial floating image 3 will become darker, so that the visibility of the spatial floating image 3 can be relatively increased. Such a shielding operation by the light-shielding plate of the opening / closing door 1410 may be directly performed by the force of the user 230's hand. The control unit 1110 may control a motor (not shown) according to an operation input via the operation input unit 1107 in Fig. 3 to perform a shielding operation by the light-shielding plate of the opening / closing door 1410.

[0111] In addition, an illuminance sensor may be provided on the back side (opposite side of the user 230) of the spatial floating image display device 1000, such as near the rear-side window, to measure the brightness of the space in front of the rear-side window. In this case, the control unit 1110 in Fig. 3 may control a motor (not shown) according to the detection result of the illuminance sensor to perform an opening / closing operation by the light-shielding plate of the opening / closing door 1410. By controlling the opening / closing operation by the light-shielding plate of the opening / closing door 1410 in this way, it is possible to more suitably maintain the visibility of the spatial floating image 3 without the user 230 manually operating the opening / closing operation of the light-shielding plate of the opening / closing door 1410.

[0112] In addition, the light-shielding plate by the opening / closing door 1410 may be manually detachable. Depending on the usage and installation environment of the spatial floating image display device 1000, the user can select whether to keep the rear window in an open state or a light-shielded state. If it is planned to use the rear window in a light-shielded state for a long time, the detachable light-shielding plate can be fixed in the light-shielded state. Also, if it is planned to use the rear window in an open state for a long time, it can be used with the detachable light-shielding plate removed. For the attachment and detachment of the light-shielding plate, screws, a hook structure, or a fitting structure may be used.

[0113] In the example of the spatial floating image display device 1000 in FIG. 4I as well, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a part of the image light output from the display device 1 may be reflected by the polarization separation member 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again on the surface of the transparent plate 100B and be visually recognized by the user as stray light. Therefore, in order to prevent such stray light, the window on the back of the spatial floating image display device 1000 may be configured not to be provided with the transparent plate 100B. The above-described opening / closing door 1410 may be provided in the window without the transparent plate 100B. In order to prevent the stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 has a coating or material with a low light reflectance.

[0114] FIG. 4J is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 in FIG. 4J is different in that an electronically controlled variable transmittance device 1620 is arranged in the rear window of the spatial floating image display device in FIG. 4H instead of arranging a transparent plate 100B made of glass or plastic. Since the other configurations are the same as those of the spatial floating image display device in FIG. 4H, repeated description is omitted. An example of the electronically controlled variable transmittance device 1620 is a liquid crystal shutter or the like.

[0115] That is, the liquid crystal shutter can control the transmitted light by voltage-controlling a liquid crystal element sandwiched between two polarizing plates. Therefore, if the liquid crystal shutter is controlled to increase the transmittance, the background of the spatial floating image 3 will be in a state where the scenery through the rear window can be seen through. Also, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be made invisible as the background of the spatial floating image 3. Moreover, since the liquid crystal shutter can be controlled in terms of intermediate length, it can also be set to a state such as a transmittance of 50%. For example, the control unit 1110 may control the transmittance of the electronically controlled variable transmittance device 1620 according to an operation input via the operation input unit 1107 in FIG. 3. With such a configuration, when one wants to see the scenery through the rear window as the background of the spatial floating image 3, but the scenery through the rear window as the background is too bright and the visibility of the spatial floating image 3 is reduced, the visibility of the spatial floating image 3 can be adjusted by adjusting the transmittance of the electronically controlled variable transmittance device 1620.

[0116] An illuminance sensor may be provided on the rear side (opposite side of the user 230) of the spatial floating image display device 1000, such as near the rear window, to measure the brightness of the space in front of the rear window. In this case, the control unit 1110 in FIG. 3 may control the transmittance of the electronically controlled variable transmittance device 1620 according to the detection result of the illuminance sensor. By doing so, even if the user 230 does not perform an operation input via the operation input unit 1107 in FIG. 3, the transmittance of the electronically controlled variable transmittance device 1620 can be adjusted according to the brightness of the space in front of the rear window, so that the visibility of the spatial floating image 3 can be maintained more suitably.

[0117] Also, in the above example, an example of a liquid crystal shutter was described as the electronically controlled variable transmittance device 1620. In contrast, as another example of the electronically controlled variable transmittance device 1620, electronic paper may be used. Even when using electronic paper, the same effects as described above can be obtained. Moreover, the power consumption for maintaining the intermediate tone state of electronic paper is very small. Therefore, a spatial floating image display device with low power consumption can be realized compared to the case of adopting a liquid crystal shutter.

[0118] Figure 4K is a diagram showing an example of the configuration of a spatial floating image display device. The spatial floating image display device 1000 in Figure 4K is different from the spatial floating image display device in Figure 4G in that it has a transmissive self-emitting image display device 1650 instead of the transparent member 100. Since the other configurations are the same as those of the spatial floating image display device in Figure 4G, repeated explanations are omitted.

[0119] In the spatial floating image display device 1000 of Figure 4K, after the image light beam passes through the display surface of the transmissive self-emitting image display device 1650, the spatial floating image 3 is formed outside the spatial floating image display device 1000. That is, when an image is being displayed on the transmissive self-emitting image display device 1650, which is a two-dimensional plane display, the spatial floating image 3 can be displayed as an image that pops out further on the user's front side of the image of the transmissive self-emitting image display device 1650. At this time, the user 230 can simultaneously view two images with different depth positions. The transmissive self-emitting image display device 1650 may be configured using existing technologies such as transmissive organic EL panels disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although not shown in Figure 3, the transmissive self-emitting image display device 1650 may be configured to be connected to other processing units such as the control unit 1110 as a component of the spatial floating image display device 1000 in Figure 3.

[0120] Here, if an effect such as moving only the object such as a character to the spatial floating image 3 on the front side after displaying both the background and the object such as a character on the transmissive self-emitting image display device 1650 is performed, an effective surprise effect video experience can be provided to the user 230.

[0121] Also, if the interior of the airborne video display device 1000 is in a light-shielded state, the background of the transmissive self-luminous video display device 1650 will become sufficiently dark. Therefore, when no video is displayed on the display device 1 or the light source of the display device 1 is turned off and video is displayed only on the transmissive self-luminous video display device 1650, to the user 230, the transmissive self-luminous video display device 1650 appears to be a normal two-dimensional flat display rather than a transmissive display. (Since the airborne video 3 in the embodiment of the present invention is displayed as an optical image of a real image in a space without a screen, if the light source of the display device 1 is turned off, the planned display position of the airborne video 3 will be a space with nothing.) Therefore, when the transmissive self-luminous video display device 1650 is used to display video as if it were a general two-dimensional flat display, by suddenly displaying characters, objects, etc. in the air as the airborne video 3, an effective surprise production video experience can be provided to the user 230.

[0122] Note that the darker the interior of the airborne video display device 1000 is made, the more the transmissive self-luminous video display device 1650 appears as a two-dimensional flat display. Therefore, an absorption-type polarizing plate (not shown) that transmits the polarization of the video light reflected by the polarization separation member 101B and absorbs a polarization having a phase difference of about 90° from the polarization may be provided on the inner side surface of the transmissive self-luminous video display device 1650 with respect to the airborne video display device 1000 (the incident surface of the video light reflected by the polarization separation member 101B to the transmissive self-luminous video display device 1650, that is, the surface of the transmissive self-luminous video display device 1650 opposite to the airborne video 3). In this way, the influence on the video light forming the airborne video 3 is not so great, but the light incident from the outside into the interior of the airborne video display device 1000 through the transmissive self-luminous video display device 1650 can be significantly reduced, the interior of the airborne video display device 1000 can be made darker, which is preferable.

[0123] FIG. 4L is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4L is a modified example of the spatial floating image display device in FIG. 4K. The orientation of the configuration arrangement in the spatial floating image display device 1000 is different from that of the spatial floating image display device in FIG. 4K and is similar to the arrangement of the spatial floating image display device in FIG. 4F. Since the functions, operations, etc. of each configuration are the same as those of the spatial floating image display device in FIG. 4K, repeated explanations are omitted.

[0124] In the spatial floating image display device of FIG. 4L as well, after the light beam of the image light passes through the transmissive self-luminous image display device 1650, the spatial floating image 3 is formed on the user 230 side rather than the transmissive self-luminous image display device 1650.

[0125] In the example of the spatial floating image display device in FIG. 4K as well as in the example of the spatial floating image display device in FIG. 4L, from the user 230, the spatial floating image 3 is displayed overlapping in front of the image of the transmissive self-luminous image display device 1650. Here, the position of the spatial floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves the head (the position of the viewpoint), the depth of the two images can be recognized due to the parallax. Therefore, by displaying two images with different depth positions, a three-dimensional visual experience can be more suitably provided to the user without the need for a stereoscopic glasses or the like, with the naked eye.

[0126] FIG. 4M is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4M is provided with a second display device 1680 on the back side as viewed from the user with respect to the polarization separation member 101B of the spatial floating image display device in FIG. 4G. Since the other configurations are the same as those of the spatial floating image display device in FIG. 4G, repeated explanations are omitted.

[0127] In the configuration example shown in FIG. 4M, the second display device 1680 is provided on the back side of the display position of the spatial floating image 3, and the image display surface is directed toward the spatial floating image 3. With such a configuration, as viewed from the user 230, the images displayed at two positions with different depths, namely, the image of the second display device 1680 and the spatial floating image 3, can be visually recognized in an overlapping manner. That is, it can be said that the second display device 1680 is arranged in a direction in which the image is displayed in the direction of the user 230 who views the spatial floating image 3. Although the second display device 1680 is not shown in FIG. 3, it may be configured to be connected to other processing units such as the control unit 1110 as a component of the spatial floating image display device 1000 in FIG. 3.

[0128] Note that the image light of the second display device 1680 of the spatial floating image display device 1000 in FIG. 4M is visually recognized by the user 230 after passing through the polarization separation member 101B. Therefore, in order for the image light of the second display device 1680 to more preferably pass through the polarization separation member 101B, it is desirable that the image light output from the second display device 1680 be polarized light with a polarization direction in which the polarization separation member 101B more preferably passes through. That is, it is desirable that the polarization direction be the same as the polarization direction of the image light output from the display device 1. For example, when the image light output from the display device 1 is S-polarized light, it is desirable that the image light output from the second display device 1680 is also S-polarized light. Also, when the image light output from the display device 1 is P-polarized light, it is desirable that the image light output from the second display device 1680 is also P-polarized light.

[0129] The example of the spatial floating image display device in FIG. 4M also has the same effect as the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L in that a second image is displayed behind the spatial floating image 3. However, unlike the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L, in the example of the spatial floating image display device in FIG. 4M, the light beam of the image light for forming the spatial floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device and may be a liquid crystal display which is a two-dimensional plane display. The second display device 1680 may also be an organic EL display. Therefore, in the example of the spatial floating image display device in FIG. 4M, it is possible to realize the spatial floating image display device 1000 at a lower cost than the example of the spatial floating image display device in FIG. 4K and the example of the spatial floating image display device in FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a part of the image light output from the display device 1 may be reflected by the polarization separation member 101B and may head toward the second display device 1680. This light (a part of the image light) may be reflected again on the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, in order to prevent the stray light, an absorption type polarizing plate may be provided on the surface of the second display device 1680. In this case, the absorption type polarizing plate may be an absorption type polarizing plate that transmits the polarization wave of the image light output from the second display device 1680 and absorbs the polarization wave whose phase is 90° different from the polarization wave of the image light output from the second display device 1680. When the second display device 1680 is a liquid crystal display, there is also an absorption type polarizing plate on the image emission side inside the liquid crystal display. However, when there is a cover glass (cover glass on the image display surface side) on the further emission surface of the absorption type polarizing plate on the image emission side inside the liquid crystal display, it is not possible to prevent the stray light generated by the reflection of the cover glass due to the light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-described absorption type polarizing plate on the surface of the cover glass.

[0132] When the second display device 1680, which is a two-dimensional plane display, is displaying an image, the spatial floating image 3 can be displayed as an image further on the user side in front of the image of the second display device 1680. At this time, the user 230 can simultaneously view two images with different depth positions. By displaying a character on the spatial floating image 3 and displaying a background on the second display device 1680, it is possible to provide an effect as if the user 230 is three-dimensionally viewing the space where the character exists.

[0133] In addition, after displaying both an object such as a background and a character on the second display device 1680, if an effect such as moving only the object such as a character to the spatial floating image 3 on the front side is performed, it is possible to provide the user 230 with an effective surprise effect video experience.

[0134] Next, FIG. 4N is a diagram showing an example of the configuration of the spatial floating image display device. The spatial floating image display device 1000 in FIG. 4N is a spatial floating image display device that employs the optical system of FIG. 2D. Similar to the example of the spatial floating image display device that employs the optical systems of FIGS. 2A to 2C, the image light that has passed through the transparent member 100 forms an image in the air as the spatial floating image 3. Also, using the sensing light of the air operation detection sensor 1351 disposed on the back side of the transparent member 100 as viewed from the user, the operation of the spatial floating image 3 by the user's finger 9004 can be detected.

[0135] In the example of the spatial floating image display device that employs the optical systems of FIGS. 2A to 2C and in the example of the spatial floating image display device that employs the optical system of FIG. 2D, the spatial floating image 3 forms an image in front of the transparent member 100, and using the sensing light of the air operation detection sensor 1351 disposed on the back side of the transparent member 100 as viewed from the user, the operation of the spatial floating image 3 by the user's finger can be detected. Therefore, the spatial floating image display device that employs the optical system of FIG. 2D has a different optical system from the spatial floating image display device in which the optical systems of FIGS. 2A to 2C are disposed on the back side of the transparent member 100 as viewed from the user.

[0136] However, the usability of the spatial floating image display device employing the optical system of FIG. 2D as viewed by the user is almost the same as that of the spatial floating image display device employing the optical systems of FIGS. 2A to 2C.

[0137] Next, FIG. 4O is a diagram showing an example of the configuration of the spatial floating image display device. FIG. 4O is a diagram showing the configuration of the internal optical system visible in the spatial floating image display device 1000 of FIG. 4N. The spatial floating image display device 1000 shown in FIG. 4O is equipped with an optical system corresponding to the optical system of FIG. 2D. In the spatial floating image display device 1000 shown in FIG. 4O, it is installed horizontally so that the surface on the side where the spatial floating image 3 is formed faces upward.

[0138] That is, in FIG. 4O, for the spatial floating image display device 1000, a transparent member 100 is installed on the upper surface of the device. The spatial floating image 3 is formed above the surface of the transparent member 100 of the spatial floating image display device 1000. The light of the spatial floating image 3 travels in an obliquely upward direction. When the air operation detection sensor 1351 is provided as shown in the figure, the operation of the spatial floating image 3 by the finger of the user 230 can be detected.

[0139] Here, the configuration of FIG. 4O is compared with the configuration of FIG. 4A to confirm the differences. In FIG. 4A, the display device 1 and the spatial floating image 3 are in a plane-symmetric relationship with respect to the surface of the polarization separation member 101. In contrast, in FIG. 4O, the display device 1 and the spatial floating image 3 are in a plane-symmetric relationship with respect to the surface of the retroreflective plate 5. Also, in the configuration of FIG. 4A, there are a retroreflective plate 2 and a λ / 4 plate 21, but these do not exist in FIG. 4O. Further, in FIG. 4A, it is more preferable to have an absorption type polarizing plate 12, but in FIG. 4O, the absorption type polarizing plate 12 is not particularly necessary.

[0140] That is, in order to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and replace it with the configuration of FIG. 4O, the following steps can be taken. That is, the polarization beam splitter member 101 in the configuration of FIG. 4A can be replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 can be removed from the configuration of FIG. 4A. The absorption type polarizing plate 12 may or may not be present. By performing the replacement based on this idea, the optical systems of FIGS. 2A to 2C mounted in the configuration of the spatial floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the spatial floating image display device equipped with the optical system of FIG. 2D can be replaced. At this time, in FIGS. 4A and 4B, the polarization beam splitter member 101 may be replaced with the retroreflector 5, and in FIGS. 4C to 4G, the polarization beam splitter member 101B may be replaced with the retroreflector 5.

[0141] In this way, a spatial floating image display device in which the optical system in the configuration of the spatial floating image display device of FIGS. 4A to 4G is replaced with the optical system of FIG. 2D can be realized. Even in these spatial floating image display devices replaced with the optical system of FIG. 2D, a spatial floating image display device with almost the same usability as the spatial floating image display device of FIGS. 4A to 4G can be realized.

[0142] <Display device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes a light source device 13 that constitutes its light source together with an image display element 11 (liquid crystal display panel). In FIG. 5, the light source device 13 is shown as a developed perspective view together with the liquid crystal display panel.

[0143] This liquid crystal display panel (image display element 11) receives, as shown by the arrow 30 in FIG. 5, an illumination light beam from the light source device 13, which is a backlight device, and has a diffusion characteristic with a sandwiching angle, that is, a characteristic similar to a laser beam with strong directivity (linear propagation) and the polarization planes aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam according to the input video signal. The modulated video light is reflected by the retroreflector 2, transmitted through the transparent member 100, and forms a spatial floating image that is a real image (see FIG. 1).

[0144] Also, in FIG. 5, the liquid crystal display panel 11 constituting the display device 1 further includes a light direction conversion panel 54 that controls the directivity characteristics of the emitted light beam from the light source device 13, and an included angle diffusion plate (not shown) as necessary. That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and video light of a specific polarization modulates the intensity of light according to the video signal and emits it (see arrow 30 in FIG. 5). As a result, a desired video is projected as light of a specific polarization with high directivity (linear propagation) through the light direction conversion panel 54 toward the retroreflective plate 2. After being reflected by the retroreflective plate 2, it passes through and forms a spatial floating video 3 toward the eyes of a monitor outside the store (space). A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the above-described light direction conversion panel 54.

[0145] <Example 1 of Display Device> FIG. 6 shows an example of the specific configuration of the display device 1. In FIG. 6, the liquid crystal display panel 11 and the light direction conversion panel 54 are arranged on the light source device 13 of FIG. 5. This light source device 13 is formed of, for example, plastic on the case shown in FIG. 5, and houses an LED element 201 and a light guide 203 inside. On the end face of the light guide 203, as also shown in FIG. 5 and the like, in order to convert the emitted light from each LED element 201 into a substantially parallel light beam, it has a shape in which the cross-sectional area gradually increases toward the opposite side of the light receiving part, and has a lens shape that has the effect of gradually reducing the divergence angle by total internal reflection multiple times when propagating inside. The liquid crystal display panel 11 constituting such a display device 1 is attached to the upper surface of the display device 1. Also, on one side surface (the left end face in this example) of the case of the light source device 13, an LED (Light Emitting Diode) element 201, which is a semiconductor light source, and an LED substrate 202 on which its control circuit is mounted are attached. A heat sink, which is a member for cooling the heat generated by the LED element and the control circuit, may be attached to the outer surface of the LED substrate 202.

[0146] In addition, on a frame (not shown) of a liquid crystal display panel attached to the upper surface of the case of the light source device 13, a liquid crystal display panel 11 attached to the frame, and further, an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11 are attached and configured. That is, the liquid crystal display panel 11, which is an image display element, together with the LED element 201, which is a solid light source, modulates the intensity of transmitted light based on a control signal from a control circuit (image control unit 1160 in FIG. 3) that constitutes the electronic device to generate a display image. At this time, since the generated image light has a narrow diffusion angle and only a specific polarization component, a novel image display device close to a surface-emitting laser image source driven by an image signal can be obtained. Currently, it is impossible technically and from a safety perspective to obtain a laser light beam of the same size as the image obtained by the above-described display device 1 using a laser device. Therefore, in this embodiment, for example, light close to the above-described surface-emitting laser image light is obtained from a light beam from a general light source including an LED element.

[0147] Subsequently, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 together with FIG. 6.

[0148] Since FIGS. 6 and 7 are cross-sectional views, only one of the plurality of LED elements 201 that constitute the light source is shown, and these are converted into substantially collimated light by the shape of the light-receiving end face 203a of the light guide 203. Therefore, the light-receiving part of the light guide end face and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Note that each of these light guides 203 is formed of a light-transmissive resin such as acrylic. The LED light-receiving surface at the end of the light guide 203 has, for example, an outer peripheral surface in the shape of a conical convex obtained by rotating a parabolic cross-section. At its top, it has a concave portion with a convex portion (i.e., a convex lens surface) formed at its central portion, and at the central portion of its flat portion, it has a convex lens surface protruding outward (or it may be a concave lens surface recessed inward) (not shown). Note that the outer shape of the light-receiving portion of the light guide to which the LED element 201 is attached has a parabolic surface shape that forms a conical outer peripheral surface, and is set within a range of angles that enables total internal reflection of the light emitted from the LED element in the peripheral direction inside it, or a reflecting surface is formed.

[0150] On the other hand, the LED elements 201 are respectively arranged at predetermined positions on the surface of the LED substrate 202, which is their circuit board. The LED substrate 202 is arranged and fixed with respect to the LED collimator (light-receiving end face 203a) such that the LED elements 201 on its surface are respectively positioned at the central portions of the aforementioned concave portions.

[0151] According to such a configuration, due to the shape of the light-receiving end face 203a of the light guide 203, the light emitted from the LED element 201 can be taken out as substantially parallel light, and it becomes possible to improve the utilization efficiency of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit in which a plurality of LED elements 201, which are light sources, are arranged to the light-receiving end face 203a, which is a light-receiving portion provided on the end face of the light guide 203. The divergent light beam from the LED element 201 is made into substantially parallel light by the lens shape of the light-receiving end face 203a of the light guide end face, and as shown by the arrow, it is guided inside the light guide 203 (in a direction parallel to the drawing), and is emitted toward the liquid crystal display panel 11 arranged substantially parallel to the light guide 203 (in a direction perpendicular to the front from the drawing) by the light beam direction conversion means 204. By optimizing the distribution (density) of the light beam direction conversion means 204 according to the shape inside or on the surface of the light guide, the uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled.

[0153] The above-described light beam direction conversion means 204 emits the light beam propagating in the light guide body toward the liquid crystal display panel 11 arranged substantially parallel to the light guide body 203 (in the direction perpendicular to the front from the drawing plane) by the shape of the surface of the light guide body or by providing, for example, portions having different refractive indexes inside the light guide body. At this time, if the relative luminance ratio when comparing the luminance of the center of the screen and the peripheral portion of the screen in a state where the liquid crystal display panel 11 is directly facing the center of the screen and the viewpoint is placed at the same position as the diagonal dimension of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it has more excellent characteristics.

[0154] Note that FIG. 6 is a cross-sectional layout view for explaining the configuration and operation of the light source of the present embodiment that performs polarization conversion in the light source device 13 including the above-described light guide body 203 and the LED element 201. In FIG. 6, the light source device 13 includes, for example, a light guide body 203 provided with light beam direction conversion means 204 on its surface or inside formed of plastic or the like, an LED element 201 as a light source, a reflection sheet 205, a retardation plate 206, a lenticular lens, etc. A liquid crystal display panel 11 having polarizing plates on its upper surface on the light source light incident surface and the video light emission surface is attached.

[0155] Further, a film or sheet-shaped reflective polarizing plate 49 is provided on the light source light incident surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one-sided polarization wave (for example, P wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflection sheet 205 provided on one (lower side in the figure) surface of the light guide body 203 so as to be directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflection sheet 205 and the light guide body 203 or between the light guide body 203 and the reflective polarizing plate 49, and the light is reflected by the reflection sheet 205 and passed twice to convert the reflected light beam from P polarization to S polarization, thereby improving the utilization efficiency of the light source light as video light. The video light beam whose light intensity is modulated by the video signal in the liquid crystal display panel 11 (arrow 213 in FIG. 6) enters the retroreflective plate 2. After being reflected by the retroreflective plate 2, a spatial floating image that is a real image can be obtained.

[0156] FIG. 7 is a cross-sectional layout view for explaining the configuration and operation of the light source of this embodiment that performs polarization conversion in the light source device 13 including the light guide 203 and the LED element 201, similar to FIG. 6. Similarly, the light source device 13 is also composed of, for example, a light guide 203 provided with a light beam direction conversion means 204 on its surface or inside formed of plastic or the like, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, and the like. On the upper surface of the light source device 13, a liquid crystal display panel 11 having polarizing plates on the light source light incident surface and the video light emitting surface is attached as a video display element.

[0157] Also, a film or sheet-shaped reflective polarizing plate 49 is provided on the light source light incident surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13 to selectively reflect one-sided polarized waves (for example, S waves) 211 among the natural light beams 210 emitted from the LED element 201. That is, in the example of FIG. 7, the selective reflection characteristics of the reflective polarizing plate 49 are different from those in FIG. 7. The reflected light is reflected by the reflective sheet 205 provided on one side (the lower side in the figure) of the light guide 203 and then travels back toward the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49 and reflected by the reflective sheet 205 and passed twice to convert the reflected light beam from S polarization to P polarization, improving the utilization efficiency of the light source light as video light. The video light beam intensity-modulated by the video signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7) enters the retroreflective plate 2. After being reflected by the retroreflective plate 2, a spatial floating image that is a real image can be obtained.

[0158] In the light source devices shown in FIGS. 6 and 7, in addition to the action of the polarizing plate provided on the light incident surface of the corresponding liquid crystal display panel 11, since one-sided polarization components are reflected by the reflective polarizing plate, the theoretically obtained contrast ratio is the reciprocal of the cross transmittance of the reflective polarizing plate multiplied by the reciprocal of the cross transmittance obtained by the two polarizing plates attached to the liquid crystal display panel. Thereby, high contrast performance can be obtained. Actually, it was experimentally confirmed that the contrast performance of the displayed image was improved by more than 10 times. As a result, a high-quality video comparable to that of the self-luminous organic EL was obtained.

[0159] <Example 2 of the display device> FIG. 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing an LED, a collimator, a synthetic diffusion block, a light guide, etc. in a case made of, for example, plastic, and a liquid crystal display panel 11 is attached to the upper surface thereof. Further, on one side surface of the case of the light source device 13, LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and an LED substrate on which the control circuit thereof is mounted are attached, and on the outer surface of the LED substrate, a heat sink 103, which is a member for cooling the heat generated by the LED element and the control circuit, is attached.

[0160] Also, on the liquid crystal display panel frame attached to the upper surface of the case, the liquid crystal display panel 11 attached to the frame and, further, an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11 are attached and configured. That is, the liquid crystal display panel 11, which is a liquid crystal display element, together with the LED elements 14a and 14b, which are solid light sources, modulates the intensity of transmitted light based on a control signal from a control circuit (not shown here) that constitutes an electronic device, thereby generating a display image.

[0161] <Example 3 of the display device> Subsequently, another example (Example 3 of the display device) of the specific configuration of the display device 1 will be described with reference to FIG. 9. The light source device of this display device 1 converts the divergent light beam (in which P-polarized light and S-polarized light are mixed) from the LED into a substantially parallel light beam by the collimator 18 and reflects it toward the liquid crystal display panel 11 by the reflecting surface of the reflection type light guide 304. The reflected light enters the reflection type polarizing plate 49 disposed between the liquid crystal display panel 11 and the reflection type light guide 304. The reflection type polarizing plate 49 transmits light of a specific polarization (for example, P-polarized light) and makes the transmitted polarized light enter the liquid crystal display panel 11. Here, light of other polarizations (for example, S-polarized light) other than the specific polarization is reflected by the reflection type polarizing plate 49 and heads back to the reflection type light guide 304 again.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so as not to be perpendicular to the principal ray of light from the reflecting surface of the reflective light guide 304. Then, the principal ray of light reflected by the reflective polarizing plate 49 enters the transmissive surface of the reflective light guide 304. The light that has entered the transmissive surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again and passes through the transmissive surface of the reflective light guide 304. The light that has passed through the transmissive surface of the reflective light guide 304 enters the reflective polarizing plate 49 again.

[0163] At this time, since the light that enters the reflective polarizing plate 49 again has passed through the λ / 4 plate 270 twice, the polarization is converted into a polarization (for example, P-polarization) that passes through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11. Note that regarding the polarization design related to polarization conversion, the polarization directions (reversing S-polarization and P-polarization) may be configured in the reverse from the above description.

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P-polarization), enters the liquid crystal display panel 11, is luminance-modulated according to the video signal, and a video is displayed on the panel surface. A plurality of LEDs constituting the light source are shown as in the above example (however, only one is shown in FIG. 9 due to the longitudinal section), and these are attached to the collimator 18 at predetermined positions.

[0165] Note that each of the collimators 18 is formed of a light-transmissive resin such as acrylic or glass. The collimator 18 may have an outer peripheral surface with a conical convex shape obtained by rotating a parabolic cross-section. Further, a concave portion having a convex portion (i.e., a convex lens surface) may be formed at the central portion of the top of the collimator 18 (the side facing the LED substrate 102). Further, a convex lens surface protruding outward (or a concave lens surface recessed inward may also be acceptable) is provided at the central portion of the flat surface portion of the collimator 18 (the side opposite to the above-mentioned top). Note that the parabolic surface forming the conical outer peripheral surface of the collimator 18 is set within an angle range capable of total reflection of the light emitted from the LED in the peripheral direction inside the collimator 18, or a reflective surface is formed.

[0166] Note that the LEDs are respectively arranged at predetermined positions on the surface of the LED substrate 102 which is their circuit board. The LED substrate 102 is arranged and fixed with respect to the collimator 18 such that the LEDs on its surface are respectively positioned at the central portion of the top of the conical convex shape (the concave portion if there is a concave portion at the top).

[0167] According to such a configuration, among the light emitted from the LED, particularly the light emitted from the central portion thereof is condensed by the convex lens surface forming the outer shape of the collimator 18 to become parallel light. Further, the light emitted from other portions in the peripheral direction is reflected by the parabolic surface forming the conical outer peripheral surface of the collimator 18 and similarly condensed to become parallel light. In other words, according to the collimator 18 having a convex lens formed at its central portion and a parabolic surface formed at its peripheral portion, almost all of the light generated by the LED can be taken out as parallel light, and the utilization efficiency of the generated light can be improved.

[0168] Furthermore, the light converted into substantially parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Among this light, the light of a specific polarization state passes through the reflective polarizing plate 49 due to the action of the reflective polarizing plate 49, and the light of the other polarization state reflected by the action of the reflective polarizing plate 49 passes through the light guide 304 again. This light is reflected by a reflector 271 that is located at a position opposite to the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarization-converted by passing through the λ / 4 plate 270, which is a retardation plate, twice. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizing plate 49 provided on the opposite surface. Since the incident light has been polarization-converted, it passes through the reflective polarizing plate 49 and is incident on the liquid crystal display panel 11 with its polarization directions aligned. As a result, all the light from the light source can be utilized, so the geometric optical utilization efficiency of the light is doubled. Also, since the polarization degree (extinction ratio) of the reflective polarizing plate is multiplied by the extinction ratio of the entire system, the contrast ratio of the entire display device is significantly improved by using the light source device of this embodiment. Note that by adjusting the surface roughness of the reflection surface of the reflective light guide 304 and the surface roughness of the reflector 271, the light reflection diffusion angle at each reflection surface can be adjusted. The surface roughness of the reflection surface of the reflective light guide 304 and the surface roughness of the reflector 271 may be adjusted for each design so that the uniformity of the light incident on the liquid crystal display panel 11 becomes more suitable.

[0169] Note that the λ / 4 plate 270, which is the retardation plate in FIG. 9, does not necessarily need to have a retardation of λ / 4 with respect to the polarization light incident perpendicularly to the λ / 4 plate 270. In the configuration of FIG. 9, any retardation plate that changes the phase by 90° (λ / 2) when the polarization light passes through it twice is acceptable. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarization light.

[0170] <Example 4 of the display device> Furthermore, another example (Example 4 of the display device) of the configuration of the optical system such as the light source device of the display device will be described with reference to FIG. 10. This is a configuration example in the case where a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Example 3 of the display device. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) for converting the diffusion characteristics in the vertical and horizontal directions (not shown in the front-rear direction of the figure) of the drawing are used on the light-emitting side of the light from the collimator 18, and the light from the collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] Note that the above optical sheets may be configured as one sheet instead of two sheets. In the case of a single-sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes of the front and back surfaces of the single optical sheet. Also, multiple diffusion sheets may be used to share the functions. Here, in the example of FIG. 10, regarding the specular diffusion characteristics due to the front surface shape and the back surface shape of the optical sheet 207A and the optical sheet 207B, it is advisable to perform optimal design using the number of LEDs, the divergence angle from the LED substrate (optical element) 102, and the optical specifications of the collimator 18 as design parameters so that the surface density of the light beam emitted from the liquid crystal display panel 11 becomes uniform. That is, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of the light guide.

[0172] In the example of FIG. 10, the polarization conversion is performed in the same manner as in Example 3 of the display device described above. That is, in the example of FIG. 10, the reflective polarizing plate 49 may be configured to have the characteristic of reflecting S-polarized light (transmitting P-polarized light). In that case, among the light emitted from the LED which is the light source, the P-polarized light is transmitted, and the transmitted light enters the liquid crystal display panel 11. Among the light emitted from the LED which is the light source, the S-polarized light is reflected, and the reflected light passes through the retardation plate 270 shown in FIG. 10. The light that has passed through the retardation plate 270 is reflected by the reflector 271. The light reflected by the reflector 271 is converted into P-polarized light by passing through the retardation plate 270 again. The polarization-converted light passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11.

[0173] Note that the quarter-wave plate 270, which is the phase difference plate in Fig. 10, does not necessarily need to have a phase difference of λ / 4 with respect to the polarized light incident perpendicularly to the quarter-wave plate 270. In the configuration of Fig. 10, any phase difference plate that changes the phase by 90° (λ / 2) when the polarized light passes through it twice is acceptable. The thickness of the phase difference plate may be adjusted according to the incident angle distribution of the polarized light. Also, in Fig. 10, regarding the polarization design related to polarization conversion, the polarization state may be configured in the reverse manner (reversing the S polarization and P polarization) from the above description.

[0174] The light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (displayed on the X-axis in Fig. 12(a)) and the vertical direction of the screen (displayed on the Y-axis in Fig. 12(b)) in a general TV application device. In contrast, for the diffusion characteristics of the light beam emitted from the liquid crystal display panel of this embodiment, for example, as shown in Example 1 of Fig. 12, when the viewing angle at which the luminance becomes 50% of the front view (angle 0 degrees) is set to 13 degrees, it becomes 1 / 5 compared to 62 degrees of a general TV application device. Similarly, for the viewing angle in the vertical direction, the reflection angle of the reflective light guide and the area of the reflection surface, etc. are optimized so that the upper viewing angle is suppressed to about 1 / 3 of the lower viewing angle with upper and lower non-uniformity. As a result, compared with a conventional liquid crystal TV, the amount of video light directed toward the monitoring direction is significantly improved, and the luminance becomes 50 times or more.

[0175] Furthermore, for the viewing angle characteristics shown in Example 2 of Fig. 12, when the viewing angle at which the luminance becomes 50% of the front view (angle 0 degrees) is set to 5 degrees, it becomes 1 / 12 compared to 62 degrees of a general TV application device. Similarly, for the viewing angle in the vertical direction, the reflection angle of the reflective light guide and the area of the reflection surface, etc. are optimized so that the viewing angle is suppressed to about 1 / 12 of that of a general TV application device with upper and lower uniformity. As a result, compared with a conventional liquid crystal TV, the amount of video light directed toward the monitoring direction is significantly improved, and the luminance becomes 100 times or more.

[0176] As described above, by using the viewing angle as the included angle, the amount of light flux directed toward the monitoring direction can be concentrated, so the light utilization efficiency is greatly improved. As a result, even when using a liquid crystal display panel for general TV use, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant increase in brightness with the same power consumption, and it can be made into a video display device corresponding to an information display system for a bright outdoor environment.

[0177] When using a large liquid crystal display panel, the light around the screen is directed inward so that it heads toward the viewer when the viewer is facing the center of the screen, thereby improving the overall uniformity of the screen brightness. FIG. 11 shows the convergence angles of the long side and the short side of the panel with the distance L from the panel to the viewer and the panel size (screen ratio 16:10) as parameters. When monitoring with the screen in a portrait orientation, the convergence angle can be set according to the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, if the convergence angle is set to 10 degrees, the video light from the four corners of the screen can be effectively directed toward the viewer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation and the monitoring distance is 0.8 m, if the convergence angle is set to 7 degrees, the video light from the four corners of the screen can be effectively directed toward the viewer. As described above, by directing the video light around the screen to the viewer who is in the optimal position for monitoring the center of the screen according to the size of the liquid crystal display panel and whether it is used in portrait or landscape orientation, the overall uniformity of the screen brightness can be improved.

[0179] As a basic configuration, as shown in FIG. 9, a light flux with an included angle directivity characteristic is made incident on the liquid crystal display panel 11 by the light source device, and the brightness is modulated according to the video signal, so that the video information displayed on the screen of the liquid crystal display panel 11 and the spatially floating image obtained by reflecting with the retroreflective plate are displayed outdoors or indoors through the transparent member 100.

[0180] By using the display device and the light source device according to an embodiment of the present invention described above, it is possible to realize a spatially floating image display device with higher light utilization efficiency.

[0181] <Example of video display processing in a spatial floating video display device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to FIG. 13A. In the spatial floating video display device 1000, when the back side of the spatial floating video 3 as viewed by the user is inside the housing of the spatial floating video display device 1000 and is sufficiently dark, the user visually recognizes that the background of the spatial floating video 3 is black.

[0182] Here, an example of displaying the character "Panda" 1525 in the spatial floating video 3 will be described with reference to FIG. 13A. First, the video control unit 1160 in FIG. 3 distinguishes and recognizes the pixel area for drawing the image of the character "Panda" 1525 and the transparent information area 1520 which is the background image in the image including the pixel area for drawing the image of the character "Panda" 1525 as shown in FIG. 13A(1).

[0183] A method of distinguishing and recognizing the character image and the background image is, for example, configured in the image processing of the video control unit 1160 such that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the character image and the background image may be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the video control unit 1160 recognizes the black and transparent information pixels of the pixels for drawing an object such as a character image as different information. However, it is assumed that the luminance of both the black and transparent information pixels of the pixels for drawing the object is 0. In this case, when displaying the spatial floating video 3, there is no difference in luminance between the pixels for drawing black in the image of the character "panda" 1525 and the pixels in the transparent information area 1520 which is the background image. Therefore, in the spatial floating video 3, as shown in Fig. 13A(2), there is no luminance in either the pixels for drawing black in the image of the character "panda" 1525 or the pixels in the transparent information area 1520, and it is visually recognized by the user as an optically identical black space. That is, the part for drawing black in the image of the object, the character "panda" 1525, melts into the background, and only the part that is not black in the character "panda" 1525 is recognized as a video floating in the display area of the spatial floating video 3.

[0185] An example of the image processing in this embodiment will be described with reference to Fig. 13B. Fig. 13B is a diagram for explaining an example of image processing that more preferably resolves the problem that the black image area of the object melts into the background, which was described in Fig. 13A. In Fig. 13B(1) and (2), the display state of the spatial floating video 3 is shown on the upper side, and the input / output characteristics of the image processing of the object image are shown on the lower side, respectively. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or the memory 1109 in Fig. 3. Alternatively, it may be input from the video signal input unit 1131. Alternatively, it may be acquired via the communication unit 1132.

[0186] Here, in the state of Fig. 13B(1), the input / output characteristics of the image processing of the object image are in a linear state without any particular adjustment. In this case, it is the same display state as Fig. 13A(2), and the black image area of the object melts into the background. On the other hand, in Fig. 13B(2), the video control unit 1160 in this embodiment adjusts the input / output characteristics of the image processing for the image of the object (character "panda" 1525) according to the input / output characteristics shown in the lower part.

[0187] That is, the video control unit 1160 performs image processing of input / output characteristics, which has the characteristic of converting the pixels of the input image into output pixels with increased luminance values for the pixels in the low-luminance region of the image of the object (character "panda" 1525). After the image of the object (character "panda" 1525) has undergone the image processing of the input / output characteristics, the video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B(2), the display state of the floating video 3 in space is such that the luminance of the pixel region for drawing black in the image of the character "panda" 1525 increases. As a result, among the regions for drawing the image of the character "panda" 1525, the region for drawing black can also be recognized by the user without blending into the black of the background, and the object can be more suitably displayed.

[0188] That is, by using the image processing of FIG. 13B(2), the region for displaying the image of the character "panda" 1525, which is an object, can be recognized separately from the black background inside the housing of the floating video display device 1000 through the window, and the visibility of the object is improved. Therefore, for example, before the image processing (that is, when the image of the object and the corresponding data are read from the storage unit 1170 or the memory 1109 in FIG. 3, or when the image of the object is input from the video signal input unit 1131, or when the data of the object is acquired via the communication unit 1132, etc.), even if the object includes pixels with a luminance value of 0 among the pixels constituting the object, after being converted into an object with increased luminance values of the pixels in the low-luminance region by the image processing of the input / output characteristics by the video control unit 1160, it is displayed on the display device 1 and converted into the floating video 3 in space by the optical system of the floating video display device 1000.

[0189] That is, the pixels constituting the object after the image processing of the input / output characteristics are converted to a state where pixels with a luminance value of 0 are not included, and then displayed on the display device 1, and are converted into the spatial floating image 3 by the optical system of the spatial floating image display device 1000.

[0190] In the image processing of FIG. 13B(2), as a method of performing the image processing of the input / output characteristics only on the image area of the object (character "panda" 1525), for example, in the image processing of the video control unit 1160, the background image layer and the character image layer in front of the background image layer are configured to be processed as separate layers so that the image processing of the input / output characteristics of FIG. 13B(2) can be performed on the character image layer, and the background image layer is not subjected to the image processing.

[0191] After that, by synthesizing these layers, as shown in FIG. 13B(2), only the character image is subjected to the image processing with the characteristic of raising the low-luminance area of the input image. As another method, after the character image layer and the background image layer are synthesized, the image processing of the input / output characteristics of FIG. 13B(2) may be configured to be performed only on the area of the character image.

[0192] In addition, the input / output video characteristics used in the video processing for raising the low-luminance area of the input / output characteristics with respect to the input video are not limited to the example of FIG. 13B(2). Any video processing that raises the low luminance may be used, such as so-called brightness adjustment. Alternatively, video processing for improving visibility may be performed by controlling the gain that changes the weighting of the retinex processing as disclosed in International Publication No. 2014 / 162533.

[0193] According to the image processing of FIG. 13B(2) described above, regarding the area where black is drawn among the areas for drawing images such as characters and objects, it is possible to make the user recognize it without melting into the black of the background, and a more suitable display can be realized.

[0194] In the examples of FIGS. 13A and 13B, the problems and more suitable image processing are described by taking as an example a spatial floating image display device in which the background appears black (for example, the spatial floating image display device 1000 in FIGS. 4A to 4G, or the spatial floating image display device 1000 in the state where the rear window is shielded from light in FIGS. 4I and 4J). However, the said image processing is also effective in devices other than these spatial floating image display devices.

[0195] Specifically, in the spatial floating image display device 1000 in FIG. 4H, or in the spatial floating image display device 1000 in the state where the rear window is not shielded from light in FIGS. 4I and 4J, the background of the spatial floating image 3 is not black, but is the scenery behind the spatial floating image display device 1000 through the window. Also in this case, the problems described in FIGS. 13A and 13B exist in the same way.

[0196] That is, the portions of the image of the character "panda" 1525 that are drawn in black will blend into the scenery behind the spatial floating image display device 1000 through the window. Also in this case, by using the image processing in FIG. 13B(2), the portions of the image of the character "panda" 1525 that are drawn in black can be recognized separately from the scenery behind the spatial floating image display device 1000 through the window, and the visibility of the object is improved.

[0197] That is, by using the image processing in FIG. 13B(2), the area where the image of the character "panda" 1525, which is an object, is displayed can be recognized separately from the scenery behind the spatial floating image display device 1000 through the window, and it becomes possible to more preferably recognize that the character "panda" 1525, which is the object, is in front of the said scenery, and the visibility of the object is improved.

[0198] Also, in the spatial floating image display device 1000 of FIGS. 4K, 4L, and 4M, as described above, when another image (such as the image of the transmissive self-luminous image display device 1650 or the image of the second display device 1680) is displayed at a position with a different depth from the spatial floating image 3, the background of the spatial floating image 3 is not black but becomes the other image. Also in this case, the problems described with reference to FIGS. 13A and 13B similarly exist.

[0199] That is, the portions of the image of the character "panda" 1525 that are drawn in black, which is the object, will blend into the other image displayed at a position with a different depth from the spatial floating image 3. Also in this case, by using the image processing of FIG. 13B(2), the portions of the image of the character "panda" 1525 that are drawn in black can be recognized separately from the other image, and the visibility of the object is improved.

[0200] That is, by using the image processing of FIG. 13B(2), the area where the image of the character "panda" 1525, which is the object, is displayed can be recognized separately from the other image, and it becomes possible to more preferably recognize that the character "panda" 1525, which is the object, is in front of the other image, and the visibility of the object is improved.

[0201] An example of the video display process of this embodiment will be described with reference to FIG. 13C. FIG. 13C is an example of video display in which the spatial floating image 3 and a second image 2050, which is another image, are simultaneously displayed among the examples of video display of this embodiment. The second image 2050 may correspond to the display image of the transmissive self-luminous image display device 1650 of FIG. 4K or FIG. 4L. Also, the second image 2050 may correspond to the display image of the second display device 1680 of FIG. 4M.

[0202] That is, an example of the video display in FIG. 13C shows a specific example of the video display example of the spatial floating video display device 1000 in FIGS. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in the spatial floating video 3. The areas other than the bear character in the spatial floating video 3 are displayed in black and become transparent as the spatial floating video. Also, the second image 2050 is a background image on which a plain, mountains, and the sun are drawn.

[0203] Here, in FIG. 13C, the spatial floating video 3 and the second image 2050 are displayed at different depths. When the user 230 visually recognizes the two videos of the spatial floating video 3 and the second image 2050 in the line-of-sight direction of the arrow 2040, the user 230 can visually recognize the videos in a state where the two videos overlap. Specifically, the bear character of the spatial floating video 3 can be seen superimposed in front of the background of the plain, mountains, and sun drawn in the second image 2050.

[0204] Here, since the spatial floating video 3 forms a real image in the air, when the user 230 slightly moves the viewing point, the user 230 can recognize the depths of the spatial floating video 3 and the second image 2050 due to the parallax. Therefore, the user 230 can obtain a stronger sense of spatial floating for the spatial floating video 3 while visually recognizing the two videos in an overlapping state.

[0205] An example of the video display process of this embodiment will be described with reference to FIG. 13D. FIG. 13D(1) is a view of the spatial floating video 3 in the example of the video display of this embodiment in FIG. 13C as seen from the line-of-sight direction of the user 230. Here, a bear character is displayed in the spatial floating video 3. The areas other than the bear character in the spatial floating video 3 are displayed in black and become transparent as the spatial floating video.

[0206] FIG. 13D(2) is a view of the second image 2050 in the example of the video display of this embodiment in FIG. 13C as seen from the line-of-sight direction of the user 230. In the example of this figure, the second image 2050 is a background image on which a plain, mountains, and the sun are drawn.

[0207] FIG. 13D(3) is a diagram showing a state in which, among the examples of video display of the present embodiment in FIG. 13C, the second image 2050 and the spatial floating video 3 appear to overlap in the line-of-sight direction of the user 230. Specifically, in front of the background of the plain, mountains, and sun drawn in the second image 2050, the bear character of the spatial floating video 3 appears to overlap.

[0208] Here, when the spatial floating video 3 and the second image 2050 are displayed simultaneously, in order to more preferably ensure the visibility of the spatial floating video 3, it is desirable to pay attention to the balance of the brightness of the two videos. If the second image 2050 is too bright compared to the brightness of the spatial floating video 3, the displayed video of the spatial floating video 3 will become transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, at least, the brightness per unit area of the spatial floating video 3 at the display position of the spatial floating video 3 is made larger than the brightness per unit area of the video light reaching the display position of the spatial floating video 3 from the second image 2050. Then, the output of the light source of the spatial floating video 3, the display video brightness of the display device 1, the output of the light source of the display device for displaying the second image 2050, and the display video brightness of the said display device may be set.

[0210] Note that since it is only necessary to satisfy this condition when the spatial floating video 3 and the second image 2050 are displayed simultaneously, when switching from the first display mode in which only the second image 2050 is displayed without displaying the spatial floating video 3 to the second display mode in which the spatial floating video 3 and the second image 2050 are displayed simultaneously, the brightness of the second image 2050 may be reduced by lowering the output of the light source of the display device for displaying the second image 2050 and / or the display video brightness of the said display device. These controls may be realized by the control unit 1110 in FIG. 3 controlling the display device 1 and the display device for displaying the second image 2050 (the transmissive self-emitting video display device 1650 in FIG. 4K or FIG. 4L or the second display device 1680 in FIG. 4M).

[0211] In addition, when performing control to reduce the brightness of the second image 2050 in the switching from the above-described first display mode to the above-described second display mode, the brightness may be uniformly reduced for the entire screen of the second image 2050. Alternatively, without uniformly reducing the brightness for the entire screen of the second image 2050, the portion where the object is displayed in the spatial floating image 3 may be in the state with the highest brightness reduction effect, and the periphery thereof may gradually reduce the brightness reduction effect step by step. That is, only for the portion where the spatial floating image 3 is superimposed on and visually recognized in the second image 2050, ensuring the brightness reduction of the second image 2050 is sufficient to ensure the visibility of the spatial floating image 3.

[0212] Here, since the spatial floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes the viewing point, due to the parallax, the superimposition position of the spatial floating image 3 with respect to the second image 2050 changes. Therefore, in the switching from the above-described first display mode to the above-described second display mode, when reducing the brightness non-uniformly for the entire screen of the second image 2050, it is not desirable to sharply reduce the brightness based on the outline of the object displayed in the spatial floating image 3. Instead, it is desirable to perform a gradation process of the brightness reduction effect, where the brightness reduction effect changes step by step according to the position as described above.

[0213] In the spatial floating image display device 1000 where the position of the object displayed in the spatial floating image 3 is approximately at the center of the spatial floating image 3, the position with the highest brightness reduction effect in the gradation process of the brightness reduction effect may be set at the center position of the spatial floating image 3.

[0214] According to the video display process of the present embodiment described above, the user 230 can more suitably visually recognize the spatial floating image 3 and the second image 2050.

[0215] When displaying the floating image 3 in space, it may be controlled so as not to display the second image 2050. Since the visibility of the floating image 3 in space is increased when the second image 2050 is not displayed, it is suitable for a floating image display device 1000 in space for applications where the user must surely visually recognize the floating image 3 in space when the floating image 3 in space is displayed.

[0216] <Example 2> As a second embodiment of the present invention, an example of another configuration example of the floating image display device in space will be described. The floating image display device according to the present embodiment is obtained by changing the optical system stored in the floating image display device described in the first embodiment to the optical system shown in FIG. 14(1) or FIG. 14(2). In this embodiment, the differences from the first embodiment will be described, and repeated descriptions of the same configurations as those in the first embodiment will be omitted. In the following description of this embodiment, a predetermined polarization and the other polarization are polarizations of polarization waves having a phase difference of 90° from each other.

[0217] FIG. 14(1) is an example of the optical system and the optical path according to the present embodiment. The optical system shown in FIG. 14(1) is configured by bringing the display device 1 closer to the polarization separation member 101B in the optical system of FIG. 2C to make the entire optical system more compact. In FIG. 14(1), repeated detailed descriptions of the configurations denoted by the same reference numerals as those in FIG. 2C are omitted.

[0218] In FIG. 14(1), similar to FIG. 2C, the video light of a predetermined polarization (P polarization in the figure) emitted from the display device 1 travels in the vertical direction from the video display surface of the display device 1. Here, the polarization separation member 101B selectively transmits a predetermined polarization (P polarization in the figure) emitted from the display device 1 and reflects the other polarization (S polarization in the figure), as in FIG. 2C.

[0219] Therefore, the video light of a predetermined polarization (P-polarization in the figure) that travels vertically from the video display surface of the display device 1 passes through the polarization separation member 101B and reaches the retroreflective plate 2 to which the λ / 4 plate 21 is attached. The video light that is retroreflectively reflected by the retroreflective plate 2 and travels again toward the polarization separation member 101B has passed through the λ / 4 plate 21 twice, and thus is converted from the predetermined polarization (P-polarization in the figure) at the time of emission from the display device 1 to the other polarization (S-polarization in the figure). Since the video light that travels again toward the polarization separation member 101B is the other polarization (S-polarization in the figure), it is reflected by the polarization separation member 101B toward the position where the user should be. The traveling direction of the video reflected by the polarization separation member 101B is determined based on the angle at which the polarization separation member 101B is arranged.

[0220] In the example of FIG. 14(1), the video light that travels toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The video light reflected by the polarization separation member 101B forms a spatial floating image 3A. The spatial floating image 3A can be suitably visually recognized by the user from the direction of arrow A.

[0221] Here, due to the characteristics of the retroreflective reflection by the retroreflective plate 2, the optical path length from when the video light emitted from the display device 1 reaches the retroreflective plate 2 and the optical path length from when the video light emitted from the retroreflective plate 2 reaches the formation position of the spatial floating image 3A are in an equal relationship. Due to this relationship, the formation position of the spatial floating image 3A in the traveling direction of the video light reflected by the polarization separation member 101B is determined.

[0222] In the example of FIG. 14(1), the display device 1, the polarization separation member 101B, and the retroreflective plate 2 are arranged so as to be closer than in the example of FIG. 2C. Thereby, it is realized that the entire optical system is configured more compactly. However, the amount by which the spatial floating image 3A protrudes from the optical system of FIG. 14(1) is not so large. For example, as one index of the amount by which the spatial floating image 3A protrudes from the optical system, the distance from the position where the light beam at the center portion of the video light is reflected by the polarization separation member 101B to the position where the video light forms the spatial floating image 3A is shown in the figure (L1 in the example of FIG. 14(1)).

[0223] Regarding the polarization design in the optical system of FIG. 14(1), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, a predetermined polarization of the video light emitted from the display device 1 is set as S-polarized light, and regarding the reflection characteristics of the polarization separation member 101B, the characteristics of P-polarized light and S-polarized light may be interchanged. In this case, although both the P-polarized light and S-polarized light shown in the figure are reversed, the optical design such as the optical path can be realized in exactly the same way.

[0224] Next, FIG. 14(2) shows another example of the optical system and the optical path according to this embodiment. The optical system of FIG. 14(2) modifies the configuration in the optical system of FIG. 14(1) in order to make the amount of the spatial floating image jumping out from the optical system larger while realizing the same compactness as the optical system of FIG. 14(1). In FIG. 14(2), for the configurations denoted by the same reference numerals as in FIG. 14(1), repeated detailed descriptions are omitted.

[0225] In FIG. 14(2), similar to FIG. 14(1), the video light of a predetermined polarization (P-polarized light in the figure) emitted from the display device 1 travels in the vertical direction from the video display surface of the display device 1. Here, the polarization characteristics of the polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). The video light of a predetermined polarization (P-polarized light in the figure) that has traveled in the vertical direction from the video display surface of the display device 1 passes through the polarization separation member 101B.

[0226] Here, different from FIG. 14(1), instead of the retroreflector 2 with the λ / 4 plate 21 attached, a specular reflector 4 with the λ / 4 plate 21B attached is arranged at the position where the video light passes through the polarization separation member 101B. Here, the reflection in the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the video light transmitted through the polarization separation member 101B is specularly reflected by the specular reflection plate 4 to which the λ / 4 plate 21B is attached. The video light that is specularly reflected by the specular reflection plate 4 and travels again toward the polarization separation member 101B has been converted from a predetermined polarization (P polarization in the figure) at the time of emission from the display device 1 to the other polarization (S polarization in the figure) by passing through the λ / 4 plate 21 twice. Since the video light that has traveled again toward the polarization separation member 101B is the other polarization (S polarization in the figure), it is reflected by the polarization separation member 101B.

[0228] Here, since the orientation of the polarization separation member 101B in Fig. 14(2) is different from that in Fig. 14(1), the video light reflected by the polarization separation member 101B travels in the direction opposite to the position where the user should be. At the destination where the video light reflected by the polarization separation member 101B travels, the retroreflector 2 to which the λ / 4 plate 21C is attached is arranged. The video light is retroreflected by the retroreflector 2. The video light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has been converted from the other polarization (S polarization in the figure) to the predetermined polarization (P polarization in the figure) again by passing through the λ / 4 plate 21C twice.

[0229] The video light that has traveled again toward the polarization separation member 101B is the predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and travels directly toward the position where the user should be. The video light that has passed through the polarization separation member 101B forms the spatial floating image 3B. The spatial floating image 3B can be preferably visually recognized by the user from the direction of arrow A.

[0230] Here, also in Fig. 14(2), similar to Fig. 14(1), due to the characteristics of the retroreflection by the retroreflector 2, the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflector 2 and the optical path length from the point where the video light emitted from the retroreflector 2 reaches the formation position of the spatial floating image 3B are in an equal relationship. Based on this relationship, the formation position of the spatial floating image 3B in the traveling direction of the video light transmitted through the polarization separation member 101B is determined.

[0231] In Fig. 14(2), the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflector 2 is longer than the optical path length from the display device 1 to the point where the video light emitted from the display device 1 reaches the retroreflector 2 in Fig. 14(1). In the optical system of Fig. 14(2), an optical path that reciprocates between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of Fig. 14(1), is added to the optical path length from the display device 1 to the point where the video light reaches the retroreflector 2.

[0232] As a result, in the optical system of Fig. 14(2), the distance (L2 in the example of Fig. 14(2)) from the position where the light rays in the central part of the video light pass through the polarization separation member 101B to the position where the video light forms the spatial floating image 3B is much longer than the distance (L1 in the example of Fig. 14(1)) from the position where the light rays in the central part of the video light are reflected by the polarization separation member 101B to the position where the video light forms the spatial floating image 3A in the optical system of Fig. 14(1).

[0233] Regarding the polarization design in the optical system of Fig. 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, a predetermined polarization of the video light emitted from the display device 1 may be set as S-polarized light, and the reflection characteristics of the polarization separation member 101B may have the characteristics of P-polarized light and S-polarized light interchanged. In this case, although both the illustrated P-polarized light and S-polarized light are reversed, the optical design such as the optical path can be realized in exactly the same way.

[0234] According to the optical systems of Figs. 14(1) and 14(2) in the second embodiment of the present invention described above, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to make the amount by which the spatial floating image protrudes from the optical system larger while having a more compact optical system.

[0235] When incorporating the optical system of FIG. 14(1) or FIG. 14(2) into the spatial floating image display device, it can be realized by replacing the optical system in the spatial floating image display device described in Example 1 with the optical system of FIG. 14(1) or FIG. 14(2). Specifically, the optical system of FIG. 14(1) may be replaced with the optical system of the spatial floating image display device in FIGS. 4E, 4F, 4G, 4H, 4I, 4J, 4K, 4L, or 4M. In this case, since the optical system becomes more compact, it is possible to make the housing of the spatial floating image display device in each figure smaller.

[0236] Specifically, the optical system of FIG. 14(2) may also be replaced with the optical system of the spatial floating image display device in FIGS. 4E, 4F, 4G, 4K, or 4L. In this case, it is possible to make the amount by which the spatial floating image protrudes from the optical system larger. Also, since the optical system becomes more compact, it is possible to make the housing of the spatial floating image display device in each figure smaller.

[0237] [Comparative Example] FIG. 15 shows a case where there is a deviation between the display range 1501 of the spatial floating image 3 and the aerial operation detection range 1502 by the aerial operation detection sensor 1351 in the spatial floating image display device 1500 of the comparative example. In the comparative example of FIG. 15, in the horizontally placed spatial floating image display device 1500, there is a display range 1501 of the spatial floating image 3 at a predetermined position, and from that position, there is an aerial operation detection range 1502 by the aerial operation detection sensor 1351 at a predetermined position on the back side in the Y direction. There is a deviation 1503 in the spatial position between the display range 1501 and the aerial operation detection range 1502.

[0238] The problems in the comparative example are as follows. In the spatial floating image display device 1500, the spatial floating image 3 can be preferably visually recognized from a predetermined viewpoint position of the user 230, but the air operation detection range 1502 is invisible. Therefore, for the user 230, it is unclear where and how to perform an air operation within the space with respect to the spatial floating image display device 1500. Even when the air operation detection range 1502 is taught to the user 230 in advance, since the air operation detection range 1502 is invisible, it may be difficult to perform an air operation on the air operation detection range 1502. Therefore, in the comparative example, the air operation by the user 230 on the spatial floating image 3 may deviate from the air operation detection range 1502. In this case, it may be impossible to detect the user's air operation at all, or it may be difficult to detect it with high accuracy. Even when the air operation detection range 1502 is constituted by a camera or the like of the imaging unit 1180 as another sensor, there are similar problems.

[0239] As a solution to the above problems, as in the spatial floating image display device 1000 of an embodiment shown in FIG. 16 etc., within a range where the air operation by the user 230 can be preferably detected, for example, within the air operation detection range 1602 by the air operation detection sensor 1351, the display range 1601 of the spatial floating image 3 may be formed. In this configuration, there is no deviation between the display range 1601 and the air operation detection range 1602.

[0240] Thereby, even if the user 230 does not know the mechanism of air operation detection or the spatial position of the air operation detection range 1602, the user 230 will perform an air operation on the visually recognizable spatial floating image 3. If the user 230 operates so as to touch the finger to the plane of the spatial floating image 3, it can be detected as an air operation by the air operation detection range 1602 provided so as to overlap the display range 1601 of the spatial floating image 3. Therefore, with such a configuration, the user operation can be preferably detected, and the user 230 is less likely to be confused about the air operation. Further, if the display content of the spatial floating image 3 is an image that prompts an air operation as described later, it is even better.

[0241] <Example 3> As an example of Example 3 of the present invention, an example of the spatial floating image display device 1000 will be described. As the basic configuration of the spatial floating image display device 1000 of Example 3, any of the spatial floating image display devices 1000 in the respective figures of Example 1 or Example 2 can be applied. In this Example 3, the differences from Example 1 or Example 2 will be mainly described, and repeated descriptions of the same configurations will be omitted.

[0242] [Overview of Example 3] The airborne floating image display device of Example 3 is premised on a configuration in which a display range of the spatial floating image 3 is formed within the air operation detection range as shown in FIG. 16. Moreover, the airborne floating image display device of Example 3 includes two types of modes: a mode (which may be described as a 2D operation mode) for accepting two-dimensional (abbreviated as 2D) operations on the spatial floating image 3, and a mode (which may be described as a 3D operation mode) for accepting three-dimensional (abbreviated as 3D) operations on the spatial floating image 3, and is configured to be able to appropriately select and switch between these modes. The 2D operation mode is a mode for accepting 2D operations that do not include operations in the depth direction on the spatial floating image 3, in other words, planar operations. The 3D operation mode is a mode for accepting not only 2D operations but also 3D operations that include operations in the depth direction on the spatial floating image 3.

[0243] In the 2D operation mode, the spatial floating image display device 1000 of Example 3 controls the display of the spatial floating image 3 so that when displaying images such as GUI and content on the spatial floating image 3, it displays that 2D operations are accepted, or that operations in the depth direction are not accepted, or displays to prompt 2D operations. In the 3D operation mode, the spatial floating image display device 1000 controls the display of the spatial floating image 3 so that when displaying images such as GUI and content on the spatial floating image 3, it displays that 3D operations are accepted, or displays to prompt 3D operations.

[0244] Note that the control and processing corresponding to the characteristic functions in Example 3 and the like below can be performed by, for example, the control unit 1110, the video control unit 1160, or the air operation detection unit 1350 in FIG. 3, and these may also be referred to as a video processing unit.

[0245] [System Capable of Accepting 2D Operations and 3D Operations] Example 3 provides a system capable of accepting both 2D operations and 3D operations as operations on the floating-in-air video 3 by the user 230. The 2D operation is an operation of moving an operating object such as a finger in a planar direction along the plane of the floating-in-air video 3. The 3D operation is an operation of moving an operating object such as a finger in a three-dimensional space with an added depth dimension in the front-back direction, which is the depth direction with respect to the plane of the floating-in-air video 3. The operation in the depth direction may be described as a depth operation.

[0246] FIG. 17 shows a configuration example of the floating-in-air video display device 1000 according to Example 3, and is a YZ cross-sectional view of the vertically oriented floating-in-air video display device 1000 and the user 230 as seen from the side. In the configuration example of FIG. 17, the floating-in-air video display device 1000 includes components such as the aforementioned display device 1, polarization separation member 101B, and retroreflective member 2 with a λ / 4 plate 21 inside the rear part of the housing 1190. The video light reflected from the polarization separation member 101B travels forward in the Y direction, that is, toward the user 230 side, passes through the transparent member 100, and forms the floating-in-air video 3, which is a real image, at a predetermined external position. This floating-in-air video 3 is formed so as to stand in the Z direction, which is the vertical direction, and has an X-Z plane, which is a two-dimensional plane, as the display range 3R.

[0247] In front of the transparent member 100 of the housing 1190 in the Y direction, the lower part 1190A and the upper part 1190B of the housing 1190 protrude. A user operation detection mechanism 1701 is provided in the lower part 1190A. The user operation detection mechanism 1701 is a mechanism capable of detecting at least the aerial operations in the left-right direction and the up-down direction, which are the X-Z directions of the floating-in-air video 3, in other words, the position and movement of a finger. As the user operation detection mechanism 1701, for example, the aforementioned aerial operation detection sensor 1351 can be applied. The aerial operation detection sensor 1351, which is the user operation detection mechanism 1701, is installed corresponding to the position of the floating-in-air video 3 in the Y direction, and for example, emits light upward in the Z direction and senses the plane of the floating-in-air video 3. The detection optical axis is indicated by a dashed-dotted arrow.

[0248] In addition, a user operation detection mechanism 1702 is provided in the upper part 1190B. The user operation detection mechanism 1702 is a mechanism capable of detecting at least an in-depth direction operation in the Y direction with respect to the spatial floating image 3 (in other words, the position and movement of a finger). As the user operation detection mechanism 1702, for example, the camera as the aforementioned imaging unit 1180 (FIG. 3) can be applied. The imaging unit 1180 which is the user operation detection mechanism 1702 is installed, for example, at a position near the spatial floating image 3 in the Y direction, the optical axis of imaging is downward in the Z direction, and the display range 3R of the spatial floating image 3 is included in the imaging range. In the configuration example of FIG. 17, the user operation detection mechanism 1702 is a mechanism capable of detecting the position and movement of a finger in the Y direction as an in-depth operation.

[0249] By combining the detection result of the user operation detection mechanism 1701 and the detection result of the user operation detection mechanism 1702, a 3D operation can be detected. Not limited to this, the user operation detection mechanism 1701 may be a mechanism capable of detecting an operation in the Y direction, that is, a mechanism capable of detecting a 3D operation alone. Also, the user operation detection mechanism 1702 may be a mechanism capable of detecting an operation in the X direction or the Z direction, that is, a mechanism capable of detecting a 3D operation alone, by arranging it at other positions and orientations, or by using other types of sensors. For example, as the user operation detection mechanism 1702, if the imaging unit 1180 is installed near the transparent member 100 as shown by the imaging unit 1180b and the optical axis of imaging is directed in the plane of the spatial floating image 3 in the Y direction, an operation in the X-Z direction can also be detected.

[0250] The detection of the in-depth direction air operation is performed by the user operation detection mechanism 1701 and the user operation detection mechanism 1702 provided inside and outside the spatial floating image display device 1000. These user operation detection mechanisms are not limited to the air operation detection sensor 1351 and the camera which is the imaging unit 1180, and LiDAR (Light Detection And Ranging), a distance measurement sensor using the TOF (Time of Flight) method, etc. may be used. The details of the user operation detection mechanism are not limited.

[0251] In the third embodiment, a configuration may be adopted that includes a 3D operation for the floating image 3 in space, that is, a user operation detection mechanism capable of detecting the position and movement of fingers in each three-dimensional direction. The configuration is not limited to the example of the combination of the user operation detection mechanism 1701 and the user operation detection mechanism 1702. For example, a configuration integrated into one user operation detection mechanism may be used, or a configuration by a combination of three or more sensors may be used. A plurality of air operation detection sensors 1351 may be provided at a plurality of positions in the depth direction. Further, not limited to the vertical type as shown in FIG. 17, in various configuration examples for forming the floating image 3 in space, similarly, a UI corresponding to 2D operations and 3D operations and display control are provided.

[0252] The user 230 operates the floating image 3 in space, for example, using a finger as an operating object. The finger position 1721 illustrated by the solid line is an example of the position of the user 230's finger during an operation in the X-Z direction, which is the plane direction corresponding to the plane of the floating image 3 in space. For example, a single finger touches the plane of the floating image 3, or in other words, is arranged within the plane, and as an example of an operation, the single finger moves in the X direction or the Z direction along the plane. Such an operation may be described as a plane operation or a 2D operation.

[0253] Also, the finger position 1722 illustrated by the dashed line is an example of the position of the user 230's finger during an operation in the Y direction, which is the depth direction with respect to the plane of the floating image 3 in space. For example, a single finger moves from the front side with respect to the plane of the floating image 3, touches the plane, and then moves so as to penetrate to the back side, and the single finger is arranged at the position on the back side. Such an operation may be described as a depth operation. Further, a three-dimensional operation constituted by a combination of a 2D operation and a depth operation may be described as a 3D operation.

[0254] In the configurations of FIGS. 16 and 17, the air operation detection sensor 1351 is installed and mounted so as to cover an air operation detection range that senses air operations with respect to the entire plane of the spatial floating image 3. As a result, first, the detection of 2D operations can be covered. Further, in FIGS. 17 and the implementation examples described later, for example, the user operation detection mechanism 1702 is installed and mounted so as to cover an air operation detection range that senses a three-dimensional space including the depth direction of the spatial floating image 3.

[0255] In the configurations of FIGS. 16 and 17, the spatial floating image 3 is displayed within an air operation detection range, which is a range in which air operations can be suitably detected by the user operation detection mechanism. In other words, in these configurations, the air operation detection range of the user operation detection mechanism, for example, the air operation detection sensor 1351, is designed to cover the sensing of the plane in accordance with the formation position of the plane of the spatial floating image 3 (display range 3R in FIG. 17).

[0256] [2D Operations and 3D Operations] FIG. 18 is an arrangement of the concepts of 2D operations and 3D operations. Although it may be difficult to understand because the concept of 2D operations is included in the concept of 3D operations, note that these are different concepts. The 2D operation mode described later is a mode capable of detecting 2D operations. The 3D operation mode described later is a mode capable of detecting 3D operations, and since the 2D operations are also included in the 3D operations, it can be said that it is also a mode capable of detecting 2D operations. In other words, the 3D operation mode described later is a mode capable of detecting both plane operations and depth operations.

[0257] As an overview of Example 3, as shown in FIG. 17, the spatial floating image display device 1000 includes, in addition to the user operation detection mechanism 1701 that can detect an air operation in the X-Z direction, which is the planar direction, a user operation detection mechanism 1702 that can detect an air operation in the Y direction, which is the depth direction. With these user operation detection mechanisms, the spatial floating image display device 1000 can detect both 2D operations and 3D operations on the spatial floating image 3 by the user 230. In other words, the spatial floating image display device 1000 in FIG. 17 can accept each air operation of 2D operation and 3D operation, and execute display control processing according to the accepted air operation.

[0258] Also, the spatial floating image display device 1000 of Example 3 provides a user interface (UI), or a man-machine interface, corresponding to the acceptance of these 2D operations and 3D operations. This UI includes the provision of a graphical user interface (GUI) in the spatial floating image 3. At the same time, the spatial floating image display device 1000 of Example 3 controls to change the behavior of the spatial floating image 3 to be displayed according to the 2D operation or 3D operation of the user 230. Specifically, as will be described later, in the case of a 2D operation, the spatial floating image display device 1000 provides a GUI and content suitable for the 2D operation and a display prompting the 2D operation, and in the case of a 3D operation, provides a GUI and content suitable for the 3D operation and a display prompting the 3D operation.

[0259] [Air operation detection sensor] As a supplement, as the configuration of the air operation detection sensor 1351, which is the user operation detection mechanism 1701, for example, the configuration of FIG. 19 can be applied. FIG. 19 shows an implementation example of the air operation detection sensor 1351 applicable in Example 3 and the like. The air operation detection unit 1350, the control unit 1110, or the video control unit 1160, etc. in FIG. 3, in other words, the video processing unit, determines and detects an air operation on the display range of the spatial floating image 3 using the detection signal of the air operation detection sensor 1351.

[0260] FIG. 19 shows an x-y plan view corresponding to the display range 3R of the spatial floating image 3. In the air operation detection sensor 1351, a plurality of optical elements 1351c are arranged in the x direction. The optical element 1351c is a pair of a light emitting element 1351a and a light receiving element 1351b. The light emitting element 1351a is composed of, for example, an infrared element or the like. The emission surface of the light emitting element 1351a of the air operation detection sensor 1351 coincides with, for example, the upper surface of the housing 1190. The light emitting element 1351a emits light a1, for example, infrared light, in the y direction. When the light a1 is not blocked by an object, it passes through the display range 3R. When the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2. The reflected light a2 is received by the light receiving element 1351b.

[0261] For example, when there is a contact point 1901 by the finger of the user 230 in the x-y plane of the display range 3R, the light a1 is reflected at the contact point 1901 and returns as the reflected light a2. The light receiving element 1351b at a certain position in the x direction detects the reflected light a2. Thereby, the air operation detection unit 1350 can know that there is a contact point 1901 at that position in the x direction. Further, the air operation detection unit 1350 can calculate the distance by the TOF method from the time until the emitted light a1 returns as the reflected light a2. For example, the distance 1902 to the contact point 1901 can be calculated. Thereby, the position coordinates of the contact point 1901 in the x-y plane of the display range 3R can also be known.

[0262] In the example of FIG. 19, there is nothing above the spatial floating image 3, but a part of the housing 1190 may also be provided above the spatial floating image 3. In that case, the light a1 of the air operation detection sensor 1351 is reflected by a part of the housing 1190 and returns as the reflected light a2. Not limited to this example, the air operation detection sensor 1351 may be arranged above, in the left-right direction, or at a position shifted in the front-rear direction, that is, the z direction, with respect to the x-y plane of the display range 3R. A plurality of air operation detection sensors 1351 may be arranged at a plurality of positions in the front-rear direction.

[0263] [Modification Example of Air Operation Detection Sensor] FIG. 20 shows a modified example of the air operation detection sensor 1351 which is the user operation detection mechanism 1701. This modified example is based on the same configuration as FIG. 17, and in the depth direction, i.e., the Y direction, a plurality of air operation detection sensors 1351, for example, three air operation detection sensors 1351, are provided as A1, A2, and A3 from the front side. The individual air operation detection sensors 1351 can apply, for example, the configuration of FIG. 19. In this example, the air operation detection sensors 1351 are provided on the upper part 1190B of the housing 1190 and emit light downward. The air operation detection sensor A2 is provided so as to cover the detection of the display range 3R corresponding to the position in the Y direction of the spatial floating image 3. The air operation detection sensor A1 is provided on the front side with respect to the position in the Y direction of the display range 3R, and can detect the position even when the finger of the user 230 is placed on the front side of the display range 3R. The air operation detection sensor A3 is provided on the back side with respect to the position in the Y direction of the display range 3R, and can detect the position when the finger of the user 230 penetrates and is placed on the back side of the display range 3R. Further, the camera as the imaging unit 1180 is provided so as to face the front side in the Y direction and can image the display range 3R and the user 230. Even with such a configuration, 3D operations can be detected.

[0264] [Control of the state of the user operation detection mechanism] For example, in the configuration of FIG. 17, for the two user operation detection mechanisms, i.e., the user operation detection mechanism 1701 and the user operation detection mechanism 1702, it is possible to switch between an effective state (also described as the ON state) in which operations can be received and an ineffective state (also described as the OFF state) in which operations cannot be received. The spatial floating image display device 1000 controls the ON / OFF of these user operation detection functions. The switching control example will be described later. The detectable operations differ depending on the combination of the states of each user detection mechanism. In the third embodiment, an operation mode corresponding to the detectable operation is provided.

[0265] Note that the invalid state (OFF state) of the air operation detection sensor 1351 is, for example, as follows. Generally speaking, this invalid state means not reacting to an air operation by the finger of the user 230, in other words, not reacting to an operation input. Specifically, this invalid state can be in the following states, and any of them is acceptable.

[0266] (1) When a finger is placed within the air operation detection range of the air operation detection sensor 1351, the air operation detection sensor 1351 does not perform detection itself. For example, this is the stop of the operation of the air operation detection sensor 1351 itself, such as the stop of the emission of light from the light emitting element of the air operation detection sensor 1351.

[0267] (2) Detection is performed by the air operation detection sensor 1351 or the detection process is performed by the air operation detection unit 1350, but the detection result is not output. For example, this is obtaining a detection signal from the air operation detection sensor 1351 but not outputting the detection signal, or outputting all detection signals as off values (for example, 0 values). Alternatively, it is also acceptable that the air operation detection unit 1350 does not output a detection result based on the detection signal of the air operation detection sensor 1351, or outputs an off - value detection result. Alternatively, it is also acceptable that the video control unit 1160 etc. does not input the detection result from the air operation detection unit 1350, or does not use it even if it is input.

[0268] (3) Detection is performed by the in-air operation detection sensor 1351 and detection processing is performed by the in-air operation detection unit 1350, and the detection result of the in-air operation is output. However, in the video control unit 1160 or the like, part or all of the video control processing, which is the processing normally executed for the in-air operation, is not executed. For example, assume that the video control unit 1160 receives, as a detection result, a value representing a touch operation on an object of the spatially floating video 3. Normally, the video control unit 1160 executes predetermined display control processing associated with the determination that the object has been touched. The display control processing is, for example, selection or execution of the object. In contrast, in this embodiment, the video control unit 1160 makes a determination that the object has not been touched and does not execute the predetermined display control processing. In other words, this control disables the in-air operation on the object of the spatially floating video 3 or disables the display control processing of the object.

[0269] In any of the above controls, from the perspective of the user 230, the spatially floating video display device 1000 side including the in-air operation detection sensor 1351 does not react to the in-air operation on the spatially floating video 3.

[0270] [Operation Mode According to the State of the User Operation Detection Mechanism] The table in FIG. 21A summarizes some possible states regarding the reception and detection of in-air operations according to combinations of the invalid state (OFF state) and the valid state (ON state) of each user operation detection mechanism in the case of the configuration example having the user operation detection mechanism 1701 and the user operation detection mechanism 1702 in FIG. 17. Such a state may be described as a mode. In the table in FIG. 21A, as rows, it has the user operation detection mechanism 1701 capable of detecting operations in the X-Z direction, the user operation detection mechanism 1702 capable of detecting operations in the Y direction, operation detection and mode, and GUI display in the spatially floating video 3. As columns, for example, it has four types of states from state 1 to state 4. There may be other states, which will be described later.

[0271] First, state 1 is the case where both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the OFF state. In this state 1, neither the air operation of 2D operation nor 3D operation is detected and accepted. For the sake of explanation, this state 1 is set as mode 1.

[0272] State 2 is the case where the user operation detection mechanism 1701 is in the ON state and the user operation detection mechanism 1702 is in the OFF state. In this state 2, only the plane operation in the X-Z direction, that is, the 2D operation, is detected and accepted. State 2 is set as mode 2. Mode 2 is also described as the 2D operation mode.

[0273] State 3 is the case where the user operation detection mechanism 1701 is in the OFF state and the user operation detection mechanism 1702 is in the ON state. State 3 can detect and accept only the depth operation. State 3 is set as mode 3. In this state 3, for example, when a finger penetrates from the front to the back with respect to the plane of the floating image 3 in space, it can detect whether the finger is in front of the plane, within the plane, or behind the plane, etc. In other words, this state 3 can detect the position of the finger in the Y direction, but cannot detect the position of the finger in the X direction or the Y direction within the plane.

[0274] State 4 is the case where both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the ON state. State 4 can detect both the plane operation in the X-Z direction and the depth operation in the Y direction. State 4 is set as mode 4. That is, mode 4 is a mode that can accept both 2D operations and 3D operations. Mode 4 is also described as the 3D operation mode. In mode 4, when a plane operation is detected, it is accepted as a 2D operation, and the corresponding display control process is executed. In mode 4, when a depth operation is detected, it is accepted as a 3D operation, and the corresponding display control process is executed.

[0275] Furthermore, as other modes of states, the following are also possible. In the table of FIG. 21B, states 5 and 6 are similarly shown. State 5 is a case where both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the ON state, similar to state 4, and both planar operations and depth operations can be detected. Set state 5 as mode 5. In state 5, during processing, both planar operations and depth operations can be detected, that is, 3D operations can be determined and detected. However, as valid air operations, only planar operations are accepted as 2D operations. In mode 5, even if a depth operation is detected, it is not accepted as a 3D operation, and the corresponding display control process is not executed either. For example, when a planar operation is detected for a button object that accepts planar operations, the display control process is executed. In state 5, the user operation detection mechanism 1702 is used as an auxiliary for processing.

[0276] State 6 is a case where both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the ON state, similar to state 4, and both planar operations and depth operations can be detected. Set state 6 as mode 6. In state 6, during processing, both planar operations and depth operations can be detected, that is, 3D operations can be determined and detected. However, as valid air operations, only depth operations are accepted as 3D operations. In mode 6, even if a planar operation is detected, it is not accepted as a 2D operation, and the corresponding display control process is not executed either. For example, when a depth operation is detected for a button object that accepts depth operations, the display control process is executed. In state 6, the user operation detection mechanism 1701 is used as an auxiliary for processing.

[0277] The spatial floating image display device 1000 of Example 3 implements at least the 2D operation mode which is the above mode 2 and the 3D operation mode which is the above mode 4. Not limited to this, in a modification, other modes may be implemented.

[0278] The spatial floating video display device 1000 of Example 3 further controls the display of the spatial floating video 3 so as to provide respective GUI displays corresponding to the respective modes, as shown in the fourth row of the table. Not only the GUI but also the content and the like can be controlled in the same way. Examples of the GUI displays for each mode are as follows. As the GUI display for Mode 1, a display indicating that aerial operations are not accepted is provided. As the GUI display for Mode 2, a display prompting 2D operations (planar operations) is provided. Also, for example, a display indicating the 2D operation mode is provided. As the GUI display for Mode 3, a display prompting depth operations is provided. As the GUI display for Mode 4, a display indicating acceptance of both 2D and 3D operations, or a display prompting 3D operations, is provided. Also, for example, a display indicating the 3D operation mode is provided. As the GUI display for Mode 5, a display prompting planar operations (2D operations) is provided. As the GUI display for Mode 6, a display prompting depth operations (3D operations) is provided.

[0279] [Reasons for Providing Two Types of Modes: 2D Operation Mode and 3D Operation Mode] In the third embodiment, two types of modes, namely the above-described 2D operation mode and 3D operation mode, are provided separately and can be switched appropriately for use. The reasons for such a configuration are as follows. Each of the above modes has its own advantages. When in the 3D operation mode for detecting and processing 3D operations, generally, the operating load on the hardware and software of the spatial floating image display device 1000 is greater than when in the 2D operation mode for detecting and processing 2D operations. Therefore, although it depends on the implementation, in the 3D operation mode, the power consumption may be larger, it may be difficult to improve the detection accuracy, and the detection time may also become longer. In other words, in the 2D operation mode, the power consumption may be smaller, the detection accuracy may be higher, and the detection time may also be shorter. That is, although the 2D operation mode has limited effective air operations available to the user 230, it has the advantage that the response to user operations is faster. Therefore, when the 2D operation suffices as the UI, the 2D operation mode can be used to improve the operability of the user 230 and reduce power consumption and the like. Conversely, when the 3D operation is suitable as the UI, the 3D operation mode can be used to enable the acceptance of more complex 3D operations. These can be used properly according to the application and situation.

[0280] [3-2.2D Mode and 3D Mode Display Changes] As described above, in the third embodiment, in a system capable of accepting both 2D operations and 3D operations, the display of the GUI of the spatial floating image 3 and the like is changed in each of the 2D operation mode and the 3D operation mode.

[0281] As described above (Fig. 18 etc.), the in-air operation as a user operation can be broadly classified into a 2D operation and a 3D operation. The 2D operation is a planar operation, a two-dimensional operation, which is performed within the in-air operation detection range in the planar direction corresponding to the plane of the display range of the spatially floating image 3. The 3D operation is a three-dimensional operation including a depth operation, which is performed within the in-air operation detection range including the planar direction and the depth direction corresponding to the plane of the display range of the spatially floating image 3. Furthermore, the spatially floating image display device 1000 changes the display content of the spatially floating image 3 in each of a state in which only 2D operations are accepted (Mode 2 in Fig. 21A, 2D operation mode) and a state in which both 2D operations and 3D operations are accepted (Mode 4 in Fig. 21A, 3D operation mode).

[0282] The following problems may occur. In a spatially floating image display system capable of 2D and 3D operations, the user may be confused about the in-air operation with respect to the spatially floating image, and the user may not be able to perform the in-air operation expected by the system side. The user may not know whether the spatially floating image accepts 2D operations, 3D operations, etc., and may not be able to judge whether to perform a 2D operation, a 3D operation, etc. For example, accepting a 3D operation when the spatially floating image is displaying a 2D image may be difficult for the user to understand. Also, for example, accepting a 2D operation when the spatially floating image 3 is displaying a 3D image may be difficult for the user to understand.

[0283] In Example 3, the solution has the following. When accepting a 2D operation, for example, in the 2D operation mode, the GUI, content, etc. of the spatially floating image 3 are displayed so as to induce a planar operation by the user 230, that is, a 2D operation. This display includes, for example, a display that does not give a sense of depth / gives a weak sense of depth, such as a browser screen or a planar figure. Or, this display includes a display subjected to image processing or the like so as not to give a sense of depth / give a weak sense of depth. This display may typically be a 2D image display, but is not limited thereto, and any display such as an image that does not easily give a sense of depth so as to prompt a 2D operation may be used.

[0284] Similarly, when accepting a 3D operation, for example, in the 3D operation mode, a GUI of the floating image 3 in space, content, etc. that induces a 3D operation including an operation in the depth direction by the user 230 is displayed. This display includes, for example, a display that gives a sense of depth, such as a three-dimensional figure, or a display that is subjected to image processing or the like so as to easily give a sense of depth. This display is typically a 3D video display, but is not limited thereto, and may be a display of a video or the like that easily gives a sense of depth so as to prompt a 3D operation.

[0285] Thus, in the third embodiment, control is performed such as changing the display of the GUI and content of the floating image 3 in space according to the 2D operation / 3D operation. When accepting a 2D operation, for example, when it is desired to receive an input by a 2D operation on an object, the object is displayed in a manner that prompts a 2D operation. This makes it difficult for the user to be conscious of the 3D operation and prompts the 2D operation. Further, when accepting a 3D operation, for example, when it is desired to receive an input by a 3D operation including a depth operation on an object, the object is displayed in a manner that prompts a 3D operation. This makes it difficult for the user to be conscious of the 2D operation and prompts the 3D operation.

[0286] This can reduce the user's hesitation in performing an air operation between the 2D operation and the 3D operation. Further, the floating image display system in space can obtain an air operation with less disturbance and high detectability as the air operation expected by the user. Specific examples of the display that prompts a 2D operation and the display that prompts a 3D operation in the floating image 3 in space will be described later.

[0287] [Implementation Examples Regarding Two Types of Operation Modes] Examples of the implementation of the spatial floating image display device 1000 related to the 2D operation mode and the 3D operation mode as the above two types of operation modes include the following. That is, there are a system that implements only the 2D operation mode, a system that implements only the 3D operation mode, and a system that implements both the 2D operation mode and the 3D operation mode and can switch between them. Any implementation example is possible. Each implementation example will be described in order below. FIG. 22 shows an implementation example that includes only the 2D operation mode. FIG. 23 shows an implementation example that includes only the 3D operation mode. FIG. 24 shows an implementation example that includes both the 2D operation mode and the 3D operation mode.

[0288] [Implementation Example 1. 2D Operation Mode] FIG. 22 shows, as Implementation Example 1, a Y-Z cross-sectional view of the spatial floating image display device 1000 in which the 2D operation mode, which is a mode that accepts only 2D operations, is implemented, and shows the state of the 2D operation, etc., as an in-air operation of the user 230 in the 2D operation mode. In the example of FIG. 22, components similar to those in FIG. 16 described above are implemented in the horizontally placed housing 1190. The video light from the retroreflective member 2 travels in an obliquely upward direction, passes through the polarization separation member 101 and the transparent member 100, and forms a spatial floating image 3, which is a real image, in the display range 2203 at a predetermined position outside. This spatial floating image 3 is a spatial floating image for 2D operations. In the housing 1190, for example, an air operation detection sensor 1351 is implemented as the 2D operation detection device 2201, which is the user operation detection mechanism 2201. The configuration of, for example, FIG. 19 can be applied to the air operation detection sensor 1351.

[0289] The air operation detection sensor 1351 is installed near the transparent member 100 on the upper surface of the housing 1190, on the front side close to the user 230 in the Y direction. The optical axis of the detection of this air operation detection sensor 1351 is directed obliquely upward, that is, in the y direction, as indicated by the dashed-dotted arrow, and overlaps with the display range 2203 of the spatially floating image 3 for 2D operations. The air operation detection sensor 1351 forms a 2D operation detection range 2202. The 2D operation detection range 2202 is a range that includes the display range 2203 of the spatially floating image 3 for 2D operations. In other words, within the 2D operation detection range 2202, the display range 2203 of the spatially floating image 3 for 2D operations is provided.

[0290] Note that the coordinate system of the spatially floating image 3 is represented by (x, y, z). The x direction is the horizontal direction or the left-right direction on the screen in the spatially floating image 3, the y direction is the vertical direction or the up-down direction on the screen in the spatially floating image 3, and the z direction is the depth direction or the front-back direction with respect to the spatially floating image 3. That is, the 2D operation detection range 2202 is a range along the x-y plane, which is the display range 2203 of the spatially floating image 3. The air operation detection sensor 1351, which is the 2D operation detection device 2201, senses the 2D operation detection range 2202 including the x-y plane of the spatially floating image 3. Thereby, a 2D operation is detected as a plane operation in the x-y plane of the spatially floating image 3. The user 230 can preferably visually recognize the spatially floating image 3 in the z direction.

[0291] The finger position 2211 of the user 230 is an example in the case where a finger, for example, a single finger, is arranged within the x-y plane of the spatially floating image 3. The user 230 performs, as a 2D operation, for example, an operation of touching or tapping a finger on the x-y plane of the spatially floating image 3, or an operation of moving a finger in the x direction or the y direction within the x-y plane of the spatially floating image 3. Note that the operations of touching and tapping are basic operations that can be determined by whether a finger is arranged within the x-y plane and are included in the concept of 2D operations. The spatially floating image display device 1000 in the 2D operation mode uses the 2D operation detection device 2201 to detect such 2D operations and executes display control processing as processing corresponding to the detected 2D operations.

[0292] The 2D operation mode's spatial floating image display device 1000 displays images such as the GUI and content for 2D operations on the spatial floating image 3 for 2D operations. At the bottom of FIG. 22, as an example of the spatial floating image 3 for 2D operations, an example of 2D video display prompting 2D operations is shown. In this example, a character image 2231 and a button object 2232 are displayed. The character image 2231 is, for example, a character image guiding a slide operation as "Please Slide!". The button object 2232 is a GUI that accepts slide operations in the x-y direction. These images are planar images that are not likely to give a sense of depth.

[0293] [Implementation Example 2. 3D Operation Mode] FIG. 23 shows, as Implementation Example 2, a Y-Z cross-sectional view of the spatial floating image display device 1000 in which a 3D operation mode, which is a mode that accepts both 2D and 3D operations simultaneously, is implemented, and also shows the state of 3D operations, etc. as the user 230's in-air operations in that 3D operation mode.

[0294] In the example of FIG. 23, similar to FIG. 22, the aforementioned components are implemented in the horizontally placed housing 1190. A spatial floating image 3 that is a real image is formed in the display range 2203 at a predetermined position outside the housing 1190. This spatial floating image 3 is a spatial floating image for 3D operations. In the housing 1190, similar to FIG. 22, an air operation detection sensor 1351 is implemented as the 2D operation detection device 2201, for example. The 2D operation detection device 2201 has the same 2D operation detection range 2202 as in FIG. 22 and senses the x-y plane of the display range 2203 of the spatial floating image 3. In addition, a 3D operation detection device 2301 is implemented in the housing 1190. In this example, the 3D operation detection device 2301 is installed at a position deeper in the Y direction than the 2D operation detection device 2201 near the transparent member 100, which is the upper surface of the housing 1190.

[0295] The 3D operation detection range 2302 is the detection range of the air operation by the 3D operation detection device 2301, covering the spatial region in the depth direction (z direction) with respect to the spatially floating image 3 for 3D operations and the 2D operation detection range 2202 of the 2D operation detection device 2201. In this example, the 3D operation detection range 2302 covers the spatial region on the back side from the spatially floating image 3, but it is not limited to this, and it may also cover the spatial region on the front side (the user 230 side) in the depth direction (z direction) with respect to the spatially floating image 3. The 3D operation detection device 2301 is a mechanism capable of detecting the position and movement of fingers in the z direction. The 3D operation detection device 2301 can apply, for example, the camera of the imaging unit 1180 or other sensors. It is not limited to this, and the 3D operation detection device 2301 may be installed at other positions.

[0296] By combining the 2D operation detection range 2202 of the 2D operation detection device 2201 and the 3D operation detection range 2302 of the 3D operation detection device 2301, a three-dimensional detection range 2303 in the 3D operation mode is configured.

[0297] As a modification regarding FIG. 23, the 2D operation detection device 2201 or the 3D operation detection device 2301 may be installed at a position as shown by the detection device 2350, and sensing in the x-y direction or the z direction may be performed from the detection device 2350 toward the spatially floating image 3 on the front side in the Y direction.

[0298] The finger position 2311 of the user 230 is an example of a position when a finger, for example, the palm, is placed within the 3D operation detection range 2302 including the x-y plane of the spatial floating image 3. The user 230 performs, as 3D operations, for example, an operation of touching the palm on the x-y plane of the spatial floating image 3 and moving the palm, or an operation of moving the palm in the z direction, which is the depth direction, from the x-y plane of the spatial floating image 3 to the back side. Also, as another example, the finger position 2312 is an example of a position in the case of a pushing operation of inserting a single finger in the z direction, which is the depth direction, from the x-y plane of the spatial floating image 3 to the back side. The spatial floating image display device 1000 in the 3D operation mode uses the 2D operation detection device 2201 and the 3D operation detection device 2301 to detect such 2D operations or 3D operations, and executes display control processing as processing corresponding to the detected in-air operations.

[0299] The spatial floating image display device 1000 in the 3D operation mode displays images such as a GUI for 3D operations and content on the spatial floating image 3 for 3D operations. At the bottom of FIG. 23, as an example of the spatial floating image 3 for 3D operations, an example of 3D video display prompting a 3D operation is shown. In this example, a character image 2331 and a button object 2332 are displayed. The character image 2331 is a character image that guides a pushing operation, for example, as "Please Push!". The button object 2332 is a GUI that accepts a pushing operation in the z direction. These images are three-dimensional images that are easy to give a sense of depth.

[0300] [Implementation Example 3. 2D Operation Mode and 3D Operation Mode] FIG. 24 shows a Y-Z cross-sectional view of a spatial floating image display device 1000 in which two modes, a 2D operation mode and a 3D operation mode, are implemented as Implementation Example 3 and selection / switching between them is possible, and also shows the state of operation of a user 230 in the selected mode (for example, the 3D operation mode). The configuration of FIG. 24 is the same as that of FIG. 23, but the difference is that the spatial floating image display device 1000 has a control function for appropriately switching between the 2D operation mode and the 3D operation mode, and this control function can be realized by a video processing unit such as the control unit 1110 and the video control unit 1160 in FIG. 3. Also, in Implementation Example 3, as shown in the figure, an indicator for displaying the current mode and other states, and an operation button 2309 (operation input unit 1107 in FIG. 3) for making the mode selectable may be provided on the housing 1190.

[0301] In the 2D operation mode that accepts only 2D operations, the spatial floating image display device 1000 uses the 2D operation detection device 2201 to detect a plane operation in the x-y plane of the spatial floating image 3 and executes processing corresponding to the detected plane operation. In the 3D operation mode that accepts both 2D operations and 3D operations simultaneously, the spatial floating image display device 1000 uses the 2D operation detection device 2201 and the 3D operation detection device 2301 to detect a plane operation and a depth operation with respect to the x-y plane of the spatial floating image 3 and executes processing corresponding to the detected 2D operation or 3D operation.

[0302] The spatial floating image display device 1000 in the 2D operation mode displays an image such as a GUI or content for 2D operations on the spatial floating image 3 for 2D operations. The spatial floating image display device 1000 in the 3D operation mode displays an image such as a GUI or content for 3D operations on the spatial floating image 3 for 3D operations. Examples of the spatial floating image 3 for 2D operations in the 2D operation mode and examples of the spatial floating image 3 for 3D operations in the 3D operation mode are shown at the bottom of FIG. 24. The spatial floating image display device 1000 switches these displays according to the mode.

[0303] Not limited to the above implementation example, a system implementing other modes in FIG. 21A may also be used. For example, Mode 5 may be implemented as the 2D operation mode, or Mode 6 may be implemented as the 3D operation mode.

[0304] [Example of switching control of user operation detection mechanism] In the third embodiment, regarding the control of the user operation detection mechanism, the following may be adopted. The spatial floating image display device 1000 sets the user operation detection mechanism in the planar direction, for example, the user operation detection mechanism 1701 in FIG. 17, to an effective state (ON state) in which operations can always be received, and sets the user operation detection mechanism in the depth direction, for example, the user operation detection mechanism 1702 in FIG. 17, to an ineffective state (OFF state) in which operations cannot always be received. Moreover, the spatial floating image display device 1000 switches the user operation detection mechanism 1702 in the depth direction to an effective state (ON state) in which operations can be received at a predetermined timing. The predetermined timing includes, for example, when the proximity of the user 230 is detected by a sensor such as the imaging unit 1180, when an operation in the planar direction is detected by the user operation detection mechanism 1701, or when a sensor such as the imaging unit 1180 detects that the user 230 is about to perform not only an operation in the planar direction but also an operation in the depth direction.

[0305] [Example of switching control of operation mode] For example, in a configuration for switching between two types of operation modes, namely the 2D operation mode and the 3D operation mode as shown in FIG. 24, examples of the control of the switching of the operation mode are as follows.

[0306] The spatial floating image display device 1000 initially starts in, for example, the 2D operation mode. "Initially" can refer to, for example, when the spatial floating image display device 1000 is powered on, when an application is launched, or when the proximity of the user 230 is detected by a sensor such as the imaging unit 1180. Since the 2D operation mode has a simpler mechanism than the 3D operation mode, it has advantages in terms of processing efficiency and power consumption. When the spatial floating image display device 1000 is in the 2D operation mode, it turns on the 2D operation detection device 2201 and turns off the 3D operation detection device 2301. Thereby, the 2D operation can be detected. In the 2D operation mode, the spatial floating image display device 1000 displays a spatial floating image 3 for 2D operations, for example, an image prompting a 2D operation. When the spatial floating image display device 1000 receives a 2D operation, it executes the corresponding display control process.

[0307] The spatial floating image display device 1000 appropriately switches from the 2D operation mode to the 3D operation mode based on a predetermined trigger or judgment. When the spatial floating image display device 1000 is in the 3D operation mode, it turns on the 2D operation detection device 2201 and the 3D operation detection device 2301. Thereby, both 2D operations and 3D operations can be detected. In the 3D operation mode, the spatial floating image display device 1000 displays a spatial floating image 3 for 3D operations, for example, an image prompting a 3D operation. When the spatial floating image display device 1000 receives a 2D operation or a 3D operation, it executes the corresponding display control process respectively.

[0308] Subsequently, the spatial floating image display device 1000 appropriately switches from the 3D operation mode to the 2D operation mode based on a predetermined trigger or judgment. When the spatial floating image display device 1000 is in the 2D operation mode, it turns on the 2D operation detection device 2201 and turns off the 3D operation detection device 2301. Thereby, the 2D operation can be detected. In the 2D operation mode, the spatial floating image display device 1000 displays a spatial floating image 3 for 2D operations, for example, an image prompting a 2D operation. When the spatial floating image display device 1000 receives a 2D operation, it executes the corresponding display control process.

[0309] [Triggers for Mode Determination] There are various triggers and judgment conditions for selection, switching, and determination regarding modes such as the above 2D operation mode and 3D operation mode. FIG. 25 shows a table summarizing the triggers for mode determination.

[0310] (1) Mode setting in the spatial floating image display device 1000. In advance, the mode to be used is set for each spatial floating image display device 1000. A person such as an operator selects and sets the mode to be used during device installation or the like. The spatial floating image display device 1000 also provides a setting screen or the like for mode setting (for example, similar to FIG. 26).

[0311] (2) Mode setting in the application of the spatial floating image display device 1000. In advance, the mode to be used is set for each application. For example, the 2D operation mode for a window guidance application, the 3D operation mode for an application for conversation with a character, etc.

[0312] (3) Mode specification operation input by the user 230. The spatial floating image display device 1000 may be provided with an operation input device such as a physical button for mode selection and switching. Alternatively, in the display of the spatial floating image 3 or the like, a display such as a button for mode selection and switching may be provided. For example, FIG. 26.

[0313] (4). Mode for each individual video to be displayed. For example, for each GUI or content, in other words, for each video type or ID, the mode to be used is selected. For example, in the attribute information attached to the video data of the video to be displayed, an attribute of 2D or 3D, or a mode specification may be described, and the space floating video display device 1000 may determine and decide the mode to be used based on the description (described later). The producer or provider of the video data of the video to be displayed may set at least one of the information on 2D / 3D display and 2D / 3D operation. For example, if a certain GUI object accepts 2D operations, a 2D operation mode may be specified. In the space floating video display device 1000, the correspondence between the video type or ID, 2D display / 3D display, and 2D operation / 3D operation may be set in a correspondence table or the like.

[0314] (5). Automatic determination by the space floating video display device 1000. For example, the space floating video display device 1000 may determine whether to use the 2D operation mode or the 3D operation mode based on the situation of the in-air operation of the user 230 on the space floating video 3. This will be described in Example 4 in detail.

[0315] [Example of mode selection by user] FIG. 26 shows an example of an operation input device and an example of the display of the space floating video 3 when the user 230 can select and specify the operation mode to be used from two types of operation modes in a configuration such as that shown in FIG. 24. As an example of the indicator and operation button 2309 of the housing 1190, a physical 2D button 2601 and a 3D button 2602 are provided. When the user 230 wants to set the 2D operation mode, the user presses the 2D button 2601, and when the user 230 wants to set the 3D operation mode, the user presses the 3D button 2602. The space floating video display device 1000 switches to the operation mode corresponding to the pressed button.

[0316] In this example, in the floating virtual image 3, a guidance message "Please select a mode" is displayed within the display of the outer frame representing the display range, and a GUI is displayed that can be selected between a 2D operation mode and a 3D operation mode by an air operation. If the user 230 wants to set the 2D operation mode, the user touches the "2D operation mode" area, and if the user wants to set the 3D operation mode, the user touches the "3D operation mode" area. The floating virtual image display device 1000 detects the touch operation and switches to the operation mode corresponding to the pressed area.

[0317] [Control of the correspondence between air operation detection and floating virtual image display] As described above, starting from whether to accept 2D operations or 3D operations, that is, whether to use the 2D operation mode or the 3D operation mode, the display content of the floating virtual image 3 is determined to be 2D display (display for 2D operations, such as a display prompting 2D operations) or 3D display (display for 3D operations, such as a display prompting 3D operations). The control of the correspondence between air operation detection and floating virtual image display is not limited to this example and is possible in other ways.

[0318] Figure 27 shows a table summarizing the control examples of the correspondence between air operation detection and the display of the floating virtual image 3. In the third embodiment, the floating virtual image display device 1000 may be controlled to determine whether the display of the floating virtual image 3 is a 2D display or a 3D display starting from whether the air operation and its detection are 2D operations or 3D operations, as described above. Once it is determined whether it is a 2D operation or a 3D operation, the display of the floating virtual image 3 being a 2D display or a 3D display is determined according to the corresponding relationship indicated by the arrow. The rows #1 and #2 in the table show such control examples. #1 is the control to determine a 2D display in response to a 2D operation, and #2 is the control to determine a 3D display in response to a 3D operation. Here, whether it is a 2D display or a 3D display is also described as the 2D display mode / 3D display mode. The control examples of #1 and #2 correspond to, for example, the opportunities of #1, #2, and #3 in Figure 25, which is to select the display mode according to the setting or specification of the operation mode.

[0319] Conversely, the control may be such that the aerial operation and its detection are determined to be 2D operation or 3D operation starting from whether the display of the floating image 3 is 2D display or 3D display. First, whether the display of the floating image 3 is 2D display or 3D display is determined, and whether the detection of the aerial operation is 2D operation or 3D operation is determined so as to show a corresponding response with an arrow. The rows #3 and #4 in the table show such control examples. #3 is control for determining 2D operation according to 2D display, and #4 is control for determining 3D operation according to 3D display. The control examples of #3 and #4 correspond to, for example, selecting an operation mode according to whether the display target image is 2D or 3D as an opportunity for #4 in FIG. 25.

[0320] The floating image display device 1000 may apply any of the above control examples. In the above description, the operation mode and the display mode are defined separately, but those obtained by integrating the operation mode and the display mode may be defined as a mode. When specifying a mode as in the example of FIG. 26, it may be possible to specify the display mode instead of the operation mode.

[0321] [Video data and mode information] FIG. 28 shows a configuration example of a floating image display system when information regarding the mode is attached to video data corresponding to the example of the opportunity for #4 in FIG. 25. Here, the mode is an example of the above-described operation mode or display mode, in other words, it is an aerial operation method, an aerial operation detection method, a display method, and the like.

[0322] In the system of FIG. 28, an external device 2800 is connected to the spatial floating image display device 1000. The external device 2800 is a video provider regarding the video to be displayed on the spatial floating image 3. The external device 2800 creates video data 2811 of the video to be displayed and provides it to the spatial floating image display device 1000 through communication or the like. The spatial floating image display device 1000 acquires the video data 2811 through communication or the like and stores it in the memory. The video processing unit 2801 performs necessary image processing such as image processing based on the video data 2811 to create display data (not shown), supplies it to the display unit 2802, and controls the display unit 2802. The display unit 2802 displays a video on a screen such as a liquid crystal display panel based on the display data. The video light of the video displayed on the screen is emitted via the optical system 2803, and the spatial floating image 3 is formed. Further, the video processing unit 2801 controls the user operation detection mechanism 2804 to detect an air operation by the user 230 on the spatial floating image 3 by the user operation detection mechanism 2804. The video processing unit 2801 executes a display control process according to the detected air operation on the object of the video or the like.

[0323] Furthermore, in this embodiment, the external device 2800 creates or sets a 2D or 3D GUI or content in the video data 2811, and designates and sets a 2D / 3D operation or mode regarding the air operation on the video in the attribute information 2812 attached to the video data 2811. The spatial floating image display device 1000 acquires such video data 2811 and attribute information 2812 from the external device 2800, and determines the aforementioned 2D operation mode or 3D operation mode according to the designation of the operation or mode in the attribute information 2812. The video processing unit 2801 controls the display of the display unit 2802 and the detection of the user operation detection mechanism 2804 so as to perform 2D / 3D air operation detection and display of the 2D / 3D spatial floating image 3 corresponding to the operation mode.

[0324] The callout at the bottom of FIG. 28 shows an example description of the attribute information 2812. As Example 1, the video type of the video data 2811 is video type 1, the video is created as a 2D display (in other words, a 2D video), and the operation is specified as a 2D operation. Instead of specifying a 2D operation, a 2D operation mode may be specified. As Example 2, the video type of the video data 2811 is video type 2, the video is created as a 3D display (in other words, a 3D video), and the operation is specified as a 3D operation. Instead of specifying a 3D operation, a 3D operation mode may be specified.

[0325] In other embodiments, instead of the external device 2800, the spatial floating video display device 1000 may assign the attribute information 2812 to the video data 2811. For example, the video processing unit 2801 may determine the correspondence between the 2D display / 3D display mode of the video data 2811 of the video to be displayed and the 2D operation / 3D operation of the air operation, and describe the correspondence in the attribute information 2812.

[0326] For example, when the modes of the air operations of the 2D operation and the 3D operation are determined for the video data 2811 of the video to be displayed, the video processing unit 2801 may determine the display modes of the 2D display and the 3D display so as to match the mode of the air operation, and select and determine the operation mode corresponding to the mode of the air operation and the display mode corresponding to the display mode. Also, for example, when the display modes of the 2D display and the 3D display are determined for the video data 2811 of the video to be displayed, the video processing unit 2801 may determine the modes of the air operations of the 2D operation and the 3D operation so as to match the display mode, and select and determine the operation mode corresponding to the mode of the air operation and the display mode corresponding to the display mode.

[0327] The video processing unit 2801 may perform image processing or the like for 2D display or 3D display in the floating video 3 on the original video data 2811 to create display data. Further, the video processing unit 2801 may create video data 2811 by itself and determine the mode of 2D operation / 3D operation according to the mode of 2D display / 3D display in the created video data 2811.

[0328] [3-3.2D Operations and 3D Operations Examples] Next, specific examples of the above 2D operations and 3D operations are given below. The table in Fig. 29A shows examples of 2D operations in the left column and examples of 3D operations in the right column. The floating video display device 1000 of Embodiment 3 is configured to detect and accept such 2D operations and 3D operations (the above-described implementation example).

[0329] Examples of 2D operations are as follows. The numbers in parentheses are identifiers for explanation. (1-1) Tap. Place a finger on the plane of the floating video 3 and then move it away towards the front side. (1-2) Swipe. In other words, slide. Place a finger on the plane of the floating video 3 and move it in the plane direction. (1-3) Flick. Place a finger on the plane of the floating video 3 and quickly move it in the plane direction. (1-4) Multi-swipe. Place multiple locations of a finger on the plane of the floating video 3 and move them in the plane direction. (1-5) Hold. Place a finger on the plane of the floating video 3 for a certain period of time or more. (1-6) Circular rotation after hold. Rotation within the plane of the floating video 3. (1-7) Pinch-in / pinch-out. Place multiple locations of a finger on the plane of the floating video 3 and change the distance between those multiple locations.

[0330] Examples of 3D operations are as follows. (2-1) Push. Move a finger three-dimensionally including the depth direction with respect to the plane of the floating video 3, pass through the plane, and move the finger to the back side of the plane. (2-2) Grab. Grab an object displayed on the plane of the floating-in-air image 3 with a finger or the palm of the hand. (2-3) Grab and rotate. Three-dimensional rotation. (2-4) Grab and move. Three-dimensional movement. (2-5) Wave the hand. For example, move the palm of the hand in the left-right direction. (2-6) Pull. For example, move the palm of the hand that has grabbed an object forward in the depth direction.

[0331] FIG. 29B shows an example of display control processing that is associated with and executed for an in-air operation as shown in FIG. 29A. Each row corresponds to an example of a 2D operation or a 3D operation in FIG. 29A. The left column shows examples related to the 2D GUI and content of the floating-in-air image 3, and the right column shows examples related to the 3D GUI and content.

[0332] For example, for the tap operation of (1-1) as a 2D operation, object selection / execution is associated as an example of control processing for 2D GUI / content. That is, when the video processing unit of the floating-in-air video display device 1000, for example, the video control unit 1160 in FIG. 3, detects a tap operation on an object displayed as GUI or content by a 2D video in the floating-in-air video 3, it executes a process of controlling the display of the floating-in-air video 3 such as that object so that the object is in a selected or executed state. Further, when the video processing unit detects a tap operation on an object displayed as GUI or content by a 3D video in the floating-in-air video 3, it executes display control processing so that the object is in a selected or executed state.

[0333] Also, for example, when the video processing unit detects a swipe operation of (1-2) as a 2D operation on a 2D video object, it executes display control processing to move the object, such as sliding, within the plane of the floating 3D video 3, or to transition the screen, such as a page, by the floating 3D video 3. Also, when the video processing unit detects a swipe operation of (1-2) as a 2D operation on a 3D video object, it executes display control processing to slide or transition the screen of the object. Also, in this case, as an example of display control processing, rotating the 3D object can also be mentioned.

[0334] Also, for example, when the video processing unit detects a hold operation of (1-5) as a 2D operation on a 2D video object, it executes display control processing to fix, in other words, to make the object stationary. Also, when the video processing unit detects a hold operation of (1-5) as a 2D operation on a 3D video object, it similarly executes display control processing to fix the object.

[0335] Also, when a push-in operation such as (2-1) as a 3D operation is performed on a 2D video object by the video processing unit, it is associated that nothing is done as an example of display control processing. That is, when a push-in operation is performed on a 2D video object, the video processing unit does not accept the push-in operation and controls to maintain the original display of the object.

[0336] Also, when a push-in operation of (2-1) as a 3D operation is performed on a 3D video object by the video processing unit, it executes display control processing to move the object deeper in the depth direction with respect to the plane of the floating 3D video 3.

[0337] As described above, as an example of the control of the in-air operation detection and the display of the spatial floating image 3 in the third embodiment, for a 2D display image, a 2D operation is accepted and a two-dimensional display change is allowed, but a 3D operation including a depth operation is not accepted and a three-dimensional display change including depth is not allowed. Also, for a 3D display image, both 2D operations and 3D operations are accepted, and two-dimensional and three-dimensional display changes are allowed. As a result, in the in-air operation detection and the display of the spatial floating image 3, the correspondence between 2D and 3D becomes easy to understand, and it becomes difficult for the user 230 to be confused by the in-air operation.

[0338] Regarding the correspondence between 2D / 3D in-air operations and 2D / 3D displays as in the above example, the operator of the spatial floating image display device 1000 may design and set it in advance. It is preferable to set the correspondence according to the spatial floating image display device 1000, the application, etc. For example, in the case of a window guidance application, it is defined whether the GUI, content, etc. images displayed on the spatial floating image 3 are to be 2D displays or 3D displays, whether the in-air operations accepted for the images are to be 2D operations or 3D operations, and the associated display control processes are defined.

[0339] [3-4. Examples of 2D Display and 3D Display] Next, specific examples of 2D displays and 3D displays for prompting the aforementioned 2D operations and 3D operations in images such as the GUI and content of the spatial floating image 3 will be described. The table in FIG. 30A shows examples of 2D displays for prompting 2D operations in the left column and examples of 3D displays for prompting 3D operations in the right column. The numbers in parentheses are identifiers.

[0340] The spatial floating image display device 1000 controls 2D displays and 3D displays as appropriate displays of the spatial floating image 3 in each of a 2D operation mode that accepts only 2D operations and a 3D operation mode that accepts 3D operations in addition to 2D operations.

[0341] In FIG. 30A, examples of 2D displays for prompting 2D operations are as follows. (2D-1) No shadow of the object, shadow removal. (2D-2) Applying a shallow shadow to an object. (2D-3) Applying a frame to an object. (2D-4) Representing the movement of an object in the planar direction. (2D-5) The planar figure of an object.

[0342] Examples of 3D displays that encourage 3D operations are as follows. (3D-1) Applying a deep shadow to an object. (3D-2) Representing the movement of an object in a direction other than the planar direction (especially the depth direction). (3D-3) Processing the depth of an object. (3D-4) The three-dimensional model of an object. For example, an avatar, a character, etc. (3D-5) The three-dimensional figure of an object. (3D-6) Processing the gloss of an object.

[0343] Figure 30B shows a first example as a 2D / 3D display example of the spatial floating image 3 corresponding to Figure 30A. The spatial floating image 3 on the left is the 2D display spatial floating image 3001, which contains the 2D display object 3003. The 2D display object 3003 is an example of the character image "Please Push!", and is an example of a GUI that encourages a push operation.

[0344] The spatial floating image 3 on the right is the 3D display spatial floating image 3002, which contains the 3D display object 3004. The 3D display object 3004 is an example in which the character image "Please Push!" is shaded. This shadow is also in the z direction, which is the depth direction, and is a kind of pseudo-3D representation.

[0345] The image processing unit may create video data or display data corresponding to the spatially floating image 3002 for 3D display by performing a shading process on the video data or display data corresponding to the spatially floating image 3001 for 2D display. The image processing unit may create video data or display data corresponding to the spatially floating image 3001 for 2D display by performing a non-shaded display or a shading removal process on the video data or display data corresponding to the spatially floating image 3002 for 3D display. Note that in FIG. 30B, for illustrative purposes, the conversion between the display examples of the spatially floating image 3 is shown by arrows, but the actual conversion is performed between the video data and the display data as described in FIG. 28.

[0346] FIG. 30C shows a second example as a 2D / 3D display example of the spatially floating image 3 corresponding to FIG. 30A. The spatially floating image 3 on the left is the spatially floating image 3011 for 2D display and includes the 2D display object 3013. The 2D display object 3013 is an example of a circular image as a planar figure. The spatially floating image 3 on the right is the spatially floating image 3012 for 3D display and includes the 3D display object 3014. The 3D display object 3014 is an example of a spherical image as a three-dimensional figure. A shadow is applied near the lower right of the object 3014, and a gloss is applied near the upper left. Similarly, the object 3015 on the left is an example of a concentric pattern as a planar figure. Correspondingly, the object 3016 on the right is an example of a pattern subjected to a depth process, for example, being darker toward the center and brighter toward the outer periphery.

[0347] Figure 30D shows a third example of the 2D / 3D display of the spatial floating image 3 corresponding to Figure 30A. The spatial floating image 3 on the left is the 2D-displayed spatial floating image 3021, which includes the 2D-displayed object 3023. The object 3023 is a triangular image as an example of a planar figure. When, for example, a swipe operation or the like is detected as a planar operation (2D operation) on this object 3023, as a display control process associated therewith, for example, a movement representation 3023a in the planar direction is executed, and the display changes like the object 3023b after the movement. Also, when, for example, an operation such as the circular rotation after holding as described above is detected as a planar operation (2D operation) on this object 3023, as a display control process associated therewith, a rotation representation 3023c in the planar direction that rotates the object 3023 within the x-y plane is executed.

[0348] On the other hand, the spatial floating image 3 on the right is the 3D-displayed spatial floating image 3022, which includes the 3D-displayed object 3024. When, for example, a swipe operation as the aforementioned 2D operation or an operation of grasping and rotating as a 3D operation is detected on the 3D-displayed object 3024, a three-dimensional rotation representation 3024a (in this example, a rotation toward the back around the y-axis) including the depth direction is executed, and the display changes like the object 3024b after the rotation.

[0349] Figure 30E shows a fourth example of the 2D / 3D display of the spatial floating image 3 corresponding to Figure 30A. The spatial floating image 3 in (1) shows the 3D-displayed or pseudo-3D-displayed spatial floating image 3031A, which includes the 3D-displayed or pseudo-3D-displayed object 3032A. The object 3032A is an example of an avatar / character image and is an animated image generated based on a 3D solid model. Note that it is also possible to obtain a 2D-displayed / pseudo-2D-displayed image by cutting out a still image from such a 3D-displayed / pseudo-3D-displayed image through an image processing process.

[0350] (2)'s floating image 3 in space shows an example of the display state of the floating image 3031B in space that includes an object 3032B such as an avatar. The object 3032B moves with a movement that includes a three-dimensional rotation 3033 with respect to the x-y plane.

[0351] (3)'s floating image 3 in space shows an example of the display state of the floating image 3031C in space that includes an object 3032C such as an avatar. The object 3032C moves with a movement that includes a three-dimensional movement expression 3034 that includes a depth direction with respect to the x-y plane.

[0352] As shown in FIG. 30A and the like, the floating image display device 1000 controls to appropriately perform a 2D display that prompts a 2D operation and a 3D display that prompts a 3D operation in the display of the floating image 3 so that the user 230 can intuitively operate on the floating image 3. Thereby, it becomes difficult for the user 230 to be confused between the 2D operation and the 3D operation. In this embodiment, an example of control in which the object is set to a 2D display or a pseudo-2D display when prompting a 2D operation of the object, and the object is set to a 3D display or a pseudo-3D display when prompting a 3D operation of the object has been described, but it is not limited to this. Depending on the object, a 3D operation may be accepted in the case of a 2D display, or a 2D operation may be accepted in the case of a 3D display.

[0353] Note that when switching the display content of the floating image 3 along with the mode switch, for example, when switching between a 2D video display and a 3D video display, in addition to making a sharp switch with a temporary deletion in between, it may also be a display control that continuously transitions and changes from one display to the other over a predetermined time.

[0354] Regarding the above 2D display or 3D display, it is basically 2D, but it is also possible to create something that appears pseudo-3D. Such a display is called a pseudo-3D display. Also, it is possible to create something that is basically 3D but appears pseudo-2D. Such a display is called a pseudo-2D display. The 2D display includes the concept of such a pseudo-2D display, and the 3D display includes the concept of such a pseudo-3D display. For example, as shown in FIG. 30B, for the 2D object 3003, the object 3004 with a shadow can be said to be a pseudo-3D display. Also, as shown in FIG. 30C, the object 3016 generated by depth processing from the planar object 3015 appears as a pseudo-solid, so it can be said to be a pseudo-3D display. Also, videos such as animations generated based on a 3D model as shown in FIG. 30E, especially those with movement in the depth direction, can be said to be 3D displays or pseudo-3D displays.

[0355] [Regarding the Generation of 2D and 3D Displays] Regarding FIGS. 30A to 30E, etc., when displaying the spatial floating video 3 of the 2D display or 3D display, the video processing unit adds rendering processing, and shadow application processing / shadow removal processing, etc. to the video of the object of the original video data to generate display data for the video of the 2D display or 3D display. The 2D display includes the concept of a pseudo-2D display. The 3D display includes the concept of a pseudo-3D display. The pseudo-3D display includes, for example, displays such as shaded objects and pseudo-solids. The shadow application processing may generate a two-dimensional shadow by two-dimensional image processing or may generate a three-dimensional shadow by three-dimensional image processing.

[0356] As also shown in FIG. 28, the video processing unit of the spatial floating video display device 1000 performs a process of creating display data for displaying the spatial floating video 3 based on the original video data. The display data is, for example, data for displaying a video on the screens of the display device 1 in FIG. 24, the liquid crystal display panel 11 in FIG. 4A, the video display unit 1102 in FIG. 3, and the display unit 2802 in FIG. 28. Based on the 2D or 3D video in the original video data, the video processing unit creates 2D or 3D display data as the display data by performing rendering in a virtual three-dimensional space, in other words, rendering-based processing.

[0357] In this rendering, all of the following conversions are possible based on known techniques. · Conversion from a 2D original video to a 2D spatial floating video 3. · Conversion from a 3D original video to a 3D spatial floating video 3. · Conversion from a 2D original video to a 3D spatial floating video 3. · Conversion from a 3D original video to a 2D spatial floating video 3.

[0358] The outline of the rendering-based processing is as follows. The video processing unit performs video processing on the video generated by rendering a virtual 3D space in which objects are arranged. The rendering of the virtual 3D space is performed by taking a perspective projection image with a virtual 3D space camera set in the virtual 3D space. The video generated by the rendering is an image obtained by setting the angle of view of the virtual 3D space camera so as to satisfy predetermined conditions based on the focal length of the lens of the virtual 3D space camera, the distance at which the user 230 views the screen corresponding to the display range of the spatial floating video 3, the diagonal length of the screen, etc., and taking a picture of the virtual 3D space in which the objects are arranged and rendering it.

[0359] Figure 31A is a table summarizing an example of conversion from original video data to display data for the floating-in-air video 3. The table in Figure 31A shows, as columns, the original video data, the conversion, and the display data for the floating-in-air video 3. Each row of (1) to (4) shows various conversion patterns as described above. There are various possibilities for the process of generating the display data for forming the floating-in-air video 3. For example, as the pattern of (1), when there is 2D display video data as the original video data, based on the 2D display video data, 2D display data for the floating-in-air video 3 is generated by rendering processing. Also, for example, as the pattern of (2), when there is 3D display video data such as a 3D model as the original video data, based on the data, 3D display data for the floating-in-air video 3 is generated by rendering processing. Also, conversion from one display data to another display data is also possible.

[0360] Figure 31B shows examples of each pattern of (1) to (4) corresponding to Figure 31A. The left side is the original video data and the right side is the display data. Here, as an example of the object of the video, the case of the character image "Pleas Push!" is shown, and as an example of a 3D or pseudo-3D expression, the case where the character is shaded is shown. The display data is a video adjusted according to the size of the display range of the floating-in-air video 3 and the like.

[0361] [3-5. Method for Judging and Detecting Aerial Operations] Next, an example of a method for judging and detecting an aerial operation as a user operation will be described.

[0362] The spatial floating image display device 1000 of Embodiment 3 prepares an algorithm or table for determining and detecting an air operation according to the 2D operation mode and an algorithm or table for determining and detecting an air operation according to the 3D operation mode, and applies the algorithm or table according to the operation mode. This algorithm or table corresponds to software, data, etc. for determining the validity or invalidity of a 2D operation or 3D operation as an air operation, in other words, the air operation determination and detection method. The processor performs processing based on the cooperation of hardware such as a memory and sensors and the software. Since there are advantages corresponding to each mode, in Embodiment 3 (for example, FIG. 24), it is configured to appropriately switch between two types of operation modes and the algorithms or tables corresponding to the operation modes. By changing and applying the algorithm or table according to the state of the operation mode, 2D / 3D air operations can be preferably detected.

[0363] As a problem, for example, when in a state of accepting a 3D operation, for example, when switched to the 3D operation mode, if an algorithm that can only determine a 2D operation is applied, the detection accuracy of the user operation will decrease.

[0364] In this embodiment, as a solution, for example, there are the following. (1). The spatial floating image display device 1000 includes, for example, two types of air operation determination and detection algorithms, namely, an algorithm for 2D operations and an algorithm for 3D operations. The video processing unit of the spatial floating image display device 1000 appropriately switches between these two types of algorithms and applies them according to the aforementioned operation mode.

[0365] (2). The spatial floating image display device 1000 is equipped with a common algorithm for air operation judgment and detection, which is not distinguished by 2D operation or 3D operation. Additionally, the spatial floating image display device 1000 is equipped with two types of tables, such as a 2D operation determination table and a 3D operation determination table. When the video processing unit of the spatial floating image display device 1000 receives a 2D operation (e.g., a planar operation), it refers to the 2D operation determination table, and when it receives a 3D operation (e.g., a depth operation), it refers to the 3D operation determination table.

[0366] As described above, by applying an appropriate algorithm or table according to 2D or 3D operations to judge and detect air operations, the detection accuracy of air operations can be improved.

[0367] FIG. 32A shows an example of a 2D operation determination table as an example of a method for judging an air operation that is a user operation in the 2D operation mode. In this example, the case corresponding to an operation with up to two fingers is shown. For example, x1 and y1 are the x - coordinate and y - coordinate as the position coordinates of the location where the first finger is placed on the plane of the spatial floating image 3 (here, the x - y plane). Similarly, x2 and y2 are the position coordinates of the location where the second finger is placed. "Hold" indicates whether there is a holding of the placement at the position coordinates. "Move" is the direction and amount of movement from the position coordinates. "Operation determination" is a determination regarding the received air operation as shown in FIG. 29A above. "Processing corresponding to the operation" is the display control processing described above, for example, object selection, screen scrolling, zooming in / out, etc.

[0368] For example, in the first row, it shows the determination regarding the tap operation described above (FIG. 29A). The position coordinates of the location of one placement are (10, 15), and there is no hold or move. Based on such detection, it is determined as a tap operation. For example, in the second row, it shows the determination regarding the swipe operation described above. The position coordinates of the location of one placement are (8, 12), and the amount of movement in the y - direction (downward) is 5. Based on such detection, it is determined as a swipe operation, particularly a downward swipe operation.

[0369] As in the example of FIG. 32A, the video processing unit refers to the display data of the spatial floating video 3, the detection result by the above-described user operation detection mechanism, and the operation determination table, and determines and detects a finger operation within the plane of the spatial floating video 3. That is, the detected operation, in other words, the accepted operation, and the display control process associated with the object and the like are executed.

[0370] FIG. 32B is similarly an example of a 3D operation determination table. In this example, only the portion of the 3D operation including the z direction, which is the depth direction, in the 3D operation determination table is shown, but 2D operations not including the depth direction can also be determined in the 3D operation determination table. "X initial" and "X after movement" are the coordinates before and after movement in the x direction. The same applies to the y coordinate and the z coordinate. "Shape" is the shape of the finger used as the operating object.

[0371] For example, the first row shows the determination regarding the pushing operation (FIG. 29A) described above. For example, regarding the z direction, which is the depth direction, the amount of movement in the direction from the front to the back as the z+ direction is 5. The shape is one finger, and there is a hold. Based on such detection, it is determined that a pushing operation has been performed, and in response to the determination of the pushing, as the process associated therewith, the determination of the object is executed.

[0372] In the second row, the shape is a palm, and the movement amount is 6 in the left direction as the x - direction. Based on such detection, it is determined as an operation of "swiping left", and correspondingly, the process of "moving forward / backward to the left" is executed. In the third row, the shape is a palm, there is a hold, and the movement amount is 8 in the direction from the back to the front as the z - direction. Based on such detection, it is determined as an operation of "pulling forward", and correspondingly, the process of "zooming in / out" is executed. In the fourth row, the shape is a palm, there is a hold, and there is no movement. Based on such detection, it is determined as an operation of "grasping", and correspondingly, the process of "ending" is executed. In the fifth row, the shape is a palm, there is a hold, and there are rotation as the movement, a movement amount of 5 in the x - direction, and a movement amount of 5 in the y + direction. Based on such detection, it is determined as an operation of "grabbing and releasing", and correspondingly, the process of "display update" is executed.

[0373] As in the above example, for example, when determining and detecting operations with an algorithm / table for 2D operations in 2D operation mode, since the determination of the depth direction is not included, it is possible to determine and detect with lower load and higher speed than in 3D operation mode.

[0374] [3 - 6. Processing flow for switching between multiple modes] Next, FIG. 33 shows the processing flow of the spatial floating image display device 1000 when, for example, with a configuration as shown in FIG. 24, switching between a 2D operation mode and a 3D operation mode as the above - mentioned multiple modes and performing control to change the display of the spatial floating image 3 for each operation mode.

[0375] In step S1, the video processing unit of the spatial floating image display device 1000 checks the setting of the current operation mode. If it is the aforementioned 2D operation mode, it proceeds to step S2; if it is the 3D operation mode, it proceeds to step S3. In step S1, in order to determine the mode to be used first, the setting of the operation mode is checked. At step S1, various triggers and conditions as shown in FIG. 25 above may be determined to determine the mode to be used.

[0376] In step S2, the video processing unit displays the floating virtual image 3 for 2D operations in correspondence with the 2D operation mode. That is, for example, it is a display that prompts a 2D operation as the floating virtual image 3 in the aforementioned 2D display.

[0377] In step S3, the video processing unit displays the floating virtual image 3 for 3D operations in correspondence with the 3D operation mode. That is, for example, it is a display that prompts a 3D operation as the floating virtual image 3 in the aforementioned 3D display.

[0378] After step S2 or step S3, in step S4, the video processing unit uses the aforementioned user operation detection mechanism to determine and detect the in-air operation of the user 230 on the floating virtual image 3. For example, in the case of the 2D operation mode, the video processing unit uses an algorithm / table corresponding to the 2D operation mode to determine and detect the 2D operation. In the case of the 3D operation mode, the video processing unit uses an algorithm / table corresponding to the 3D operation mode to determine and detect a 2D operation or a 3D operation as the in-air operation that is a user operation. If the result of this determination and detection is a valid operation, in other words, an operation that is accepted, an operation that is permitted, the process proceeds to step S6. If it is an invalid operation, in other words, an operation that is not accepted, an operation that is not permitted, the process proceeds to step S5.

[0379] In step S5, the video processing unit outputs feedback to the user 230 indicating that it is an invalid operation. This may be, for example, feedback display on the screen of the floating virtual image 3 or feedback audio output. The process returns from step S5 to step S4.

[0380] In step S6, the video processing unit executes processing corresponding to a valid user operation, that is, the display control processing set in association with the above-described air operation and object. For example, if it is a pushing operation on a button object as a 3D operation, the display of the floating image 3 in space is updated so that the button object is in a state of being pushed to a position deeper in the depth direction (described later). As in the flow of steps S4 and S6, it is basic that the processing corresponding to the user operation is executed immediately.

[0381] Also, in step S7, the video processing unit outputs feedback to the user 230 indicating that the operation is valid. For example, it may be a feedback display on the screen of the floating image 3 in space, or a feedback voice output.

[0382] The feedback output in step S7 or step S5 may be an output defined for each object or air operation. Examples of the feedback output include an effect display indicating that a valid air operation has been received and an effect display indicating that an invalid air operation has not been received (described later). Note that the processing in steps S6 and S7 is preferably executed promptly after it is determined in step S4 that a valid user operation has been performed. In this processing flow, step S7 is executed after step S6, but this is just an example, and the order of steps S6 and S7 may be reversed when giving priority to the feedback output to the user 230.

[0383] In step S8, while the 2D or 3D floating image 3 in space is being displayed, the video processing unit checks and determines whether to switch the operation mode or change the setting of the operation mode. If it is determined that the operation mode should be changed, the process proceeds to step S9, and if it is determined not to change, the process proceeds to step S11.

[0384] Examples of this check and determination include the following items. 1. Similar to #3 in FIG. 25, a mode designation operation input by the user 230 (described later). 2. Similar to #5 in FIG. 25, an automatic determination by the spatial floating image display device 1000. For example, a determination according to the user operation status (Example 4 described later).

[0385] In step S9, the video processing unit changes the setting of the operation mode. That is, in the case of the flow from step S2, it is a change from the 2D operation mode to the 3D operation mode, and in the case of the flow from step S3, it is a change from the 3D operation mode to the 2D operation mode.

[0386] In the subsequent step S10, the video processing unit displays the spatial floating image 3 for 2D operation or 3D operation as the display mode corresponding to the changed operation mode. In step S11, it waits for the next user operation, and the same is repeated thereafter.

[0387] [Example of 2D operation and 2D display] FIG. 34 shows an example of a swipe operation as a planar operation for a 2D operation on an object of the spatial floating image 3 and an example of a corresponding 2D display. (1A) is a front view of the x-y plane of the spatial floating image 3, and a button object 3401 is displayed at the central position (the x coordinate is x1 and the y coordinate is y1). This button object 3401 is an object that accepts a swipe operation, in other words, a slide operation. For the sake of clarity in explanation, the case where the character image "Slide" is displayed on the button object 3401 is shown, but there may be no such guide display.

[0388] (1B) is a y-z plane view of the state of (1A) seen from the side. The user 230 touches the location 3403 where the button object 3401 in the x-y plane is located with the finger 3402, particularly a single finger. At this time, the finger 3402 is arranged at the position (x1, y1, z1), for example. The case where there are user operation detection mechanisms 1701 and 1702 similar to FIG. 17 is illustrated.

[0389] (2A) shows the state when the user 230 swipes the button object 3401, for example, to the right (x+ direction) from the state of (1A). (2B) is a y-z plane view of the state of (2A) seen from the side. The user 230 is moving the finger 3402 to the right (x+ direction). The finger 3402b is arranged at a position (x2, y1, z1) corresponding to, for example, the location 3404. Note that the finger may move to some extent in the z direction during the swipe operation. The video processing unit, in response to the detection of the above-described planar operation, which is a swipe operation, using the user detection mechanism 1701, moves the button object 3401 to the right and displays it in the state of the button object 3401b.

[0390] When the video processing unit receives a swipe operation regarding the button object 3401 as described above, since the swipe operation is a 2D operation, it sets the 2D operation mode and displays the button object 3401 in a 2D display mode as shown in the figure. Thereby, the user 230 can be prompted to perform a 2D operation on the button object 3401. Also, in the 2D operation mode, as described above, high-speed detection is possible using the user detection mechanism 1701.

[0391] [Examples of 3D operations and 3D displays] FIG. 35 shows an example of a push operation as a 3D operation on the object of the floating video 3 in space and an example of the corresponding 3D display. (1A) is a front view of the x-y plane of the floating video 3 in space, and a button object 3501 is displayed at the central position (with the x coordinate being x1 and the y coordinate being y1). This button object 3501 is an object that receives a push operation. For the sake of easy understanding in the description, the case where the character image "Push" is displayed on the button object 3501 is shown, but there may be cases where such a guide display is absent.

[0392] (1B) is a y-z plane view of the state of (1A) seen from the side. User 230 is touching a location 3503 where there is a button object 3501 in the x-y plane with finger 3502, particularly with a single finger. At this time, finger 3502 is arranged at, for example, position (x1, y1, z1). A case where there are user operation detection mechanisms 1701 and 1702 similar to those in FIG. 17 is illustrated.

[0393] (2A) shows the state when user 230 performs an operation of pushing the button object 3501 deeper in the z direction from the state of (1A). User 230 is moving finger 3502, particularly a single finger, in the deeper z+ direction. As a result, the moved finger 3502b is arranged at, for example, position (x1, y1, z2) corresponding to location 3503. Note that the finger may move to some extent in the x-y direction during the pushing operation.

[0394] The video processing unit uses the user detection mechanisms 1701 and 1702 to detect a 3D operation including the depth operation which is the above-mentioned pushing operation. In response to the detection, the video processing unit displays the button object 3501 in a state of a button object 3501b as if it has visually moved deeper from user 230's perspective. For example, the button object 3501b has a reduced size compared to the button object 3501 in (1A), and its color / brightness has been changed. Thereby, from user 230's perspective, a visual effect as if the button object 3501 has moved deeper can be obtained.

[0395] When the video processing unit receives a pushing operation regarding the button object 3501 as de...

Claims

1. An airborne floating image display device, a video processing unit, a display unit that displays the video processed by the video processing unit, an optical system that generates an airborne floating image based on the video displayed by the display unit, a detection mechanism that detects a user's airborne operation on the airborne floating image, comprising: registering the user's airborne operation for each identified individual based on the identification of the user; receiving the user's airborne operation in the airborne floating image; An airborne floating image display device.

2. In the airborne floating image display device according to Claim 1, when the user's airborne operation is performed on the video area for receiving the user's airborne operation, system processing corresponding to the video area is executed, registering the user's airborne operation in association with each system process; An airborne floating image display device.

3. In the airborne floating image display device according to Claim 1, the data / information representing the user's airborne operation to be registered includes gesture data representing the gesture of the airborne operation and / or data of an algorithm or table for detecting and determining the airborne operation. An airborne floating image display device.

4. In the airborne floating image display device according to Claim 3, as characteristics for each user in the user's airborne operation, at least one parameter value among the finger used by the user, the position, area, direction, and time of contact with respect to the plane of the airborne floating image, the direction and amount of movement in the plane, and the distance and time of pushing into the back side from the plane is used to set the conditions for detection and determination in the data / information representing the user's airborne operation to be registered. An airborne floating image display device.

5. In the airborne floating image display device according to Claim 1, registering the user's airborne operation based on the detection of the gesture of the user's free airborne operation on the airborne floating image. An airborne floating image display device.

6. In the airborne floating image display device according to Claim 1, in advance, as a default setting, an airborne operation for common application to all users is set as a default operation, registering the user's airborne operation as customization for each user based on the default operation. An airborne floating image display device.

7. In the airborne floating image display device according to Claim 1, having a reset button configured based on predetermined hardware and / or software, resetting the registration / setting of the user's air operation based on the user's operation on the reset button, or based on a voice input representing reset, or based on detection of a predetermined gesture representing reset, an air floating video display device.

8. In the air floating video display device according to claim 1, receiving practice of the user's air operation by the user on the air floating video for the user's air operation registered by the user, and outputting an evaluation result of the practiced air operation, an air floating video display device.

9. In the air floating video display device according to claim 1, having a special air operation constituted by sequentially performing a plurality of air operations, when determining the special air operation by the user, providing display or processing of the special air floating video associated with the special air operation, an air floating video display device.

10. In the air floating video display device according to claim 1, when obtaining, as a voice input by the user, a voice representing the ambiguous air operation or system processing associated with the air operation in the same or a time period before or after detecting the gesture of the ambiguous air operation by the user, determining it as the air operation or system processing indicated by the voice, an air floating video display device.

11. In the air floating video display device according to claim 1, having the user perform the gesture of the user's air operation a plurality of times, and determining whether the plurality of gestures are substantially the same based on the gesture data of the detected plurality of gestures, and determining the user's air operation to be registered, an air floating video display device.

12. In the air floating video display device according to claim 5, computer resources for a registration frame for registering the user's air operation are provided for each user or for all users, and registration is made possible within the range of the registration frame, an air floating video display device.

13. In the air floating video display device according to claim 6, when applying the user's air operation as customization for each user, enabling the setting of the user's air operation and disabling the setting of the default operation, When the aerial operation for the user is not applied, the setting of the aerial operation for the user is invalidated and the setting of the default operation is validated. Aerial floating image display device.

14. In the aerial floating image display device according to Claim 9, The aerial operation for the user is set for one or more aerial operations constituting the special aerial operation. Aerial floating image display device.

15. In the aerial floating image display device according to Claim 10, When a gesture of an ambiguous aerial operation by the user is detected and cannot be specified as one aerial operation, the user is confirmed for an aerial operation or system process estimated as a candidate in the display or audio output of the aerial floating image, and then, when a voice representing the ambiguous aerial operation or the system process associated with the aerial operation is obtained as voice input by the user, it is determined as the aerial operation or system process indicated by the voice. Aerial floating image display device.

16. In the aerial floating image display device according to Claim 1, The data / information representing the aerial operation for the user to be registered has gesture data representing the gesture of the aerial operation. The gesture data detected for the gesture of the aerial operation for the user is compared with the registered gesture data, and the aerial operation for the user is determined based on the result of judging the difference or degree of similarity. Aerial floating image display device.

17. In the aerial floating image display device according to Claim 1, Video information for confirming the aerial operation currently applied to the user is displayed on the aerial floating image. Aerial floating image display device.

18. In the aerial floating image display device according to Claim 1, The aerial operation for the user is set as a shortcut operation for a system process that normally requires multiple aerial operations in the aerial floating image. Aerial floating image display device.

19. In the aerial floating image display device according to Claim 1, The user is asked to perform the gesture of the aerial operation for the user multiple times, and statistical processing is performed based on the gesture data of the detected multiple gestures, and the data / information representing the aerial operation for the user to be registered is set according to the characteristics of the user's gesture. Aerial floating image display device.

Images