Display system and display device

Abstract

The invention provides a better display device. According to the invention, it is possible to contribute to "3 good health and welfare", "9 industry, innovation and infrastructure", "11 sustainable cities and communities" of sustainable development objectives (SDGs). In a display system, a display device (e.g., a spatial floating image display device) receives image data of a camera image, which is an image captured by a camera, displays the camera image on a display screen on the basis of the image data, and displays the camera image on the display screen on the basis of an operation input by a user. A target camera, viewpoint, or subject is selected from among a plurality of cameras, a plurality of viewpoints, or subjects of a camera, image data of a camera image corresponding to the target camera, viewpoint, or subject is received in accordance with the selection, and an image is displayed on a display screen on the basis of the received image data.

Description

Display systems and display devices Technical Field

[0001] This invention relates to an aerial levitation image display device. Background Technology

[0002] Regarding the technology of displaying information suspended in the air, for example, it is disclosed in Patent Document 1.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-128722 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] However, the content disclosed in Patent Document 1 does not fully consider how to obtain practical brightness and quality of aerial levitation images, or how to make users enjoy watching aerial levitation images more pleasantly.

[0008] The purpose of this invention is to provide a better display device, such as an aerial levitation image display device.

[0009] Technical means to solve the problem

[0010] To solve the above problems, for example, the structure described in the claimed technical solution can be adopted. This application includes various means to solve the above problems, one example of which can be configured as follows. A display system including a display device, characterized in that: the display device receives image data of an image captured by a camera, i.e., a camera image, and displays the camera image on a display screen based on the image data, wherein, based on operation input performed by a user of the display device, a camera, viewpoint, or subject is selected as the target from multiple cameras, multiple viewpoints, or the camera's subject, and image data of the camera image corresponding to the target camera, viewpoint, or subject is received according to the selection, and the image is displayed on the display screen based on the received image data.

[0011] Invention Effects

[0012] The present invention enables the realization of a better aerial levitation image display device. Other technical problems, technical features, and technical effects will become clear in the following description of the embodiments. Attached Figure Description

[0013] Figure 1 is a diagram showing an example of the usage mode of a spatial levitation image display device according to an embodiment of the present invention.

[0014] Figure 2A is a diagram showing an example of the main structure and retroreflective part structure of a spatial levitation image display device according to an embodiment of the present invention.

[0015] Figure 2B is a diagram showing an example of the main structure and retroreflective part structure of a spatial levitation image display device according to an embodiment of the present invention.

[0016] Figure 2C is a diagram showing an example of the main structure and retroreflective part structure of a spatial levitation image display device according to an embodiment of the present invention.

[0017] Figure 2D is a diagram showing an example of the main structure and retroreflective part structure of an aerial levitation image display device according to an embodiment of the present invention.

[0018] Figure 2E is a projection diagram of the retroreflector constituting the aerial suspended image display device in one embodiment of the present invention.

[0019] Figure 2F is a top view of the retroreflector constituting the aerial suspended image display device in one embodiment of the present invention.

[0020] Figure 2G is a perspective view of the corner reflector included in the retroreflector constituting the aerial suspended image display device in one embodiment of the present invention.

[0021] Figure 2H is a top view showing the corner reflector included in the retroreflector constituting the aerial levitation image display device in one embodiment of the present invention.

[0022] Figure 2I is a side view showing the corner reflector included in the retroreflector constituting the aerial levitation image display device in one embodiment of the present invention.

[0023] Figure 3 is a diagram showing a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0024] Figure 4A is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0025] Figure 4B is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0026] Figure 4C is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0027] Figure 4D is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0028] Figure 4E is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0029] Figure 4F is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0030] Figure 4G is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0031] Figure 4H is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0032] Figure 4I is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0033] Figure 4J is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0034] Figure 4K is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0035] Figure 4L is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0036] Figure 4M is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0037] Figure 4N is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0038] Figure 40 is a diagram illustrating an example of the structure of a spatial levitation image display device according to an embodiment of the present invention.

[0039] Figure 5 is a cross-sectional view showing an example of the specific structure of a light source device according to an embodiment of the present invention.

[0040] Figure 6 is a cross-sectional view showing an example of the specific structure of a light source device according to an embodiment of the present invention.

[0041] Figure 7 is a cross-sectional view showing an example of the specific structure of a light source device according to an embodiment of the present invention.

[0042] Figure 8 is a configuration diagram showing the main parts of a spatial levitation image display device according to an embodiment of the present invention.

[0043] Figure 9 is a cross-sectional view showing the structure of a display device according to an embodiment of the present invention.

[0044] Figure 10 is a cross-sectional view showing the structure of a display device according to an embodiment of the present invention.

[0045] Figure 11 is an explanatory diagram illustrating the light source diffusion characteristics of an image display device according to an embodiment of the present invention.

[0046] Figure 12 is an explanatory diagram illustrating the diffusion characteristics of an image display device according to an embodiment of the present invention.

[0047] Figure 13A is an illustrative diagram illustrating an example of the technical problem to be solved by image processing according to an embodiment of the present invention.

[0048] Figure 13B is an illustrative diagram illustrating an example of image processing according to an embodiment of the present invention.

[0049] Figure 13C is an illustrative diagram illustrating an example of image display processing according to an embodiment of the present invention.

[0050] Figure 13D is an illustrative diagram illustrating an example of image display processing according to an embodiment of the present invention.

[0051] Figure 14 is a diagram showing an example of the main structure and retroreflective part structure of a spatial levitation image display device according to an embodiment of the present invention.

[0052] Figure 15 is an explanatory diagram of a display system including a spatial levitation image display device according to an embodiment of the present invention.

[0053] Figure 16 is an explanatory diagram of a camera configuration example according to an embodiment of the present invention.

[0054] Figure 17 is an explanatory diagram of a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0055] Figure 18A is an explanatory diagram of a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0056] Figure 18B is an explanatory diagram illustrating a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0057] Figure 18C is an explanatory diagram of a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0058] Figure 18D is an explanatory diagram illustrating a structural example of a spatial levitation image display device according to an embodiment of the present invention.

[0059] Figure 19 is an explanatory diagram of a control example according to an embodiment of the present invention.

[0060] Figure 20 is an illustrative diagram of an example GUI of an embodiment of the present invention.

[0061] Figure 21 is an explanatory diagram of a structural example of a display system according to an embodiment of the present invention.

[0062] Figure 22A is an explanatory diagram of an example of a selection information table according to an embodiment of the present invention.

[0063] Figure 22B is an explanatory diagram of an example of a selection information table according to an embodiment of the present invention.

[0064] Figure 23 is an explanatory diagram of a structural example of a television set, i.e., a display device, according to an embodiment of the present invention.

[0065] Figure 24 is an explanatory diagram of a server structure example according to an embodiment of the present invention.

[0066] Figure 25 is an explanatory diagram of data transmission and reception between a server and a display device according to an embodiment of the present invention.

[0067] Figure 26 is an explanatory diagram of a structural example of a display system according to an embodiment of the present invention.

[0068] Figure 27 is an explanatory diagram of a virtual camera in a virtual space according to an embodiment of the present invention.

[0069] Figure 28 is an illustrative diagram of an example GUI of an embodiment of the present invention.

[0070] Figure 29 is an explanatory diagram illustrating an example of the structure of an air operation detection sensor according to an embodiment of the present invention.

[0071] Figure 30 is an explanatory diagram of a structural example of a display system according to an embodiment of the present invention.

[0072] Figure 31 is an explanatory diagram of motion information according to an embodiment of the present invention.

[0073] Figure 32 is an explanatory diagram of the structure of a display system according to an embodiment of the present invention.

[0074] Figure 33 is an explanatory diagram of the structure of a display system according to an embodiment of the present invention.

[0075] Figure 34A is an illustrative diagram of a specific example of data distribution according to an embodiment of the present invention.

[0076] Figure 34B is an illustrative diagram of data distribution in a display system according to an embodiment of the present invention.

[0077] Figure 35A is an explanatory diagram of data transmission in a transmitting-side system according to an embodiment of the present invention.

[0078] Figure 35B is an explanatory diagram of data reception in a receiving-side system according to an embodiment of the present invention.

[0079] Figure 36 is an explanatory diagram of a display system including a spatial levitation image display device according to an embodiment of the present invention.

[0080] Figure 37 is an explanatory diagram of an example of the use of a display system according to an embodiment of the present invention.

[0081] Figure 38 is an explanatory diagram of an example of the use of a display system according to an embodiment of the present invention.

[0082] Figure 39 is an explanatory diagram of an example of camera image switching according to an embodiment of the present invention.

[0083] Figure 40 is an explanatory diagram of an example of camera image display ON / OFF according to an embodiment of the present invention.

[0084] Figure 41 is an explanatory diagram of a structural example of AI used in a display system according to an embodiment of the present invention.

[0085] Figure 42 is an illustrative diagram illustrating an example of using AI to select and distribute images in one embodiment of the present invention.

[0086] Figure 43 is an illustrative diagram illustrating an example of using AI to select different distributed images for each user in one embodiment of the present invention.

[0087] Figure 44 is an explanatory diagram of an example of image-related content distribution in one embodiment of the present invention.

[0088] Figure 45 is an illustrative diagram of an example of a GUI using AI in one embodiment of the present invention.

[0089] Figure 46 is an illustrative diagram of an example of evaluating an input image using AI in one embodiment of the present invention.

[0090] Figure 47 is an illustrative diagram of an example of evaluating an input image without using AI in one embodiment of the present invention.

[0091] Figure 48 is an illustrative diagram of an example of an evaluation input image in an embodiment of the present invention, in which the selection of AI is evaluated.

[0092] Figure 49 is an explanatory diagram of an example of the use of evaluation information according to an embodiment of the present invention.

[0093] Figure 50 is an illustrative diagram illustrating an example of using AI to select and distribute images in a video-on-demand distribution scenario according to one embodiment of the present invention.

[0094] Figure 51 is an illustrative diagram of an example of a recorded image using a recording medium in one embodiment of the present invention.

[0095] Figure 52 is an illustrative diagram of an example of recorded video using AI and a recording medium in one embodiment of the present invention. Detailed Implementation

[0096] The embodiments of the present invention are described in detail below based on the accompanying drawings. However, the present invention is not limited to the description of the embodiments, and those skilled in the art can make various changes and modifications within the scope of the technical concept disclosed in this specification. Furthermore, in all the drawings used to illustrate the present invention, parts having the same function are labeled with the same reference numerals, and sometimes repeated descriptions are omitted.

[0097] The following embodiments relate to an image display device that enables an image formed by image light from an image light source to be transmitted through a transparent component such as glass used to separate space, and displayed outside the transparent component as a spatially suspended image. Furthermore, in the following description of the embodiments, the term "spatially suspended image" is used to describe an image suspended in space. Alternatively, it can be expressed as "aerial image," "spatial image," "aerially suspended image," "spatially suspended optical image displaying an image," "aerially suspended optical image displaying an image," etc. The term "spatially suspended image" primarily used in the description of the embodiments is taken as a representative example of these terms.

[0098] According to the following embodiments, a good image display device can be implemented in, for example, bank ATMs, station ticket machines, digital signage, etc. For example, while touch panels are commonly used in bank ATMs and station ticket machines, transparent glass or light-transmitting panels can also be used to display high-resolution image information in a spatially suspended state. In this case, by reducing the divergence angle of the emitted image light to an acute angle, and then unifying it to a specific polarization, only the light normally reflected by the retroreflector can be efficiently reflected. Therefore, light utilization efficiency is high, and ghosting, which is a problem in existing retroreflection methods besides the main spatially suspended image, can be suppressed, resulting in a clear spatially suspended image. Furthermore, by using a device including the light source of this embodiment, a novel and highly usable spatially suspended image display device (spatially suspended image display system) that significantly reduces power consumption can be provided. Additionally, for example, a vehicle-use spatially suspended image display device capable of displaying so-called one-way spatially suspended images can be provided, which can be viewed from inside and / or outside the vehicle.

[0099] <Example 1>

[0100] <An example of the usage of a spatial levitation image display device>

[0101] Figure 1 is a diagram showing an example of the usage mode of the spatial levitation image display device according to an embodiment of the present invention, and is a diagram showing the overall structure of the spatial levitation image display device of this embodiment. The specific structure of the spatial levitation image display device will be described in detail using Figure 2, etc., in which light with narrow-angle pointing characteristics and specific polarization emitted from the image display device 1 serves as an image beam. After reflection by the optical system within the spatial levitation image display device, it first enters the retroreflector plate 2, and after retroreflection, it passes through the transparent component 100 (glass, etc.), forming a real aerial image (spatial levitation image 3) on the outer side of the glass surface. In addition, in the following embodiments, the retroreflector plate 2 (retroreflective reflector) is used as an example of a retroreflective component. However, the retroreflector plate 2 of the present invention is not limited to a planar plate, and is used as an example. Its concept includes a sheet-like retroreflective component that can be attached to a planar or non-planar component, and an entire assembly obtained by attaching a sheet-like retroreflective component to a planar or non-planar component. In addition, the light reflected by the retroreflector plate 2 has optical characteristics that enable imaging, so the retroreflector plate 2 can also be expressed as an imaging optical component or an imaging optical plate.

[0102] Furthermore, in shops and similar establishments, display windows (also known as "window glass") 105, constructed from translucent components such as glass, divide the space. The spatial levitation image display device according to this embodiment can display levitation images unidirectionally to the exterior and / or interior of the shop (space) through these transparent components.

[0103] In Figure 1, the inner side of the window glass 105 (inside the shop) is shown as the depth direction, and the outer side (e.g., the sidewalk) is shown as the near side. On the other hand, light can also be reflected by providing a mechanism on the window glass 105 that performs specific polarization reflection, forming an aerial image at a desired location inside the shop.

[0104] <Example of the structure of the optical system of a spatial levitation image display device>

[0105] Figure 2A is a diagram illustrating an example of the structure of the optical system of a spatial levitation image display device according to an embodiment of the present invention. The structure of the spatial levitation image display device will be described in more detail using Figure 2A. As shown in (1) of Figure 2A, a display device 1 is provided in the oblique direction of a transparent component 100 such as glass, which diffuses image light of a specific polarization in a narrow angle. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 for generating light of a specific polarization having narrow-angle diffusion characteristics.

[0106] Image light of a specific polarization from display device 1 is reflected by a polarization separating member 101 (in the figure, the polarization separating member 101 is formed as a sheet and attached to the transparent member 100) which has a film that selectively reflects the image light of the specific polarization and is disposed on the transparent member 100, and then incident on the retroreflector 2. A λ / 4 waveplate 21 is disposed on the image light incident surface of the retroreflector 2. The image light passes through the λ / 4 waveplate 21 twice, once when it is incident on the retroreflector 2 and once when it is emitted, thereby changing from a specific polarization to another polarization. Here, the polarization separating member 101, which selectively reflects the image light of the specific polarization, has the property of transmitting light of the other polarization after polarization change, so the image light of the specific polarization after polarization change passes through the polarization separating member 101. The image light after passing through the polarization separating member 101 forms a spatially suspended image 3 with a real image on the outside of the transparent member 100. In addition, regarding the principal ray of the image light incident on the retroreflector 2, an example of incident at 90° relative to the retroreflector 2 is shown in Figure 2A. However, the incident angle of the main ray of the image light relative to the retroreflector 2 is not limited to 90°; for example, 90°±15° can also be used.

[0107] Here, a first example of polarization design in the optical system of FIG. 2A is described. For example, it can be configured such that S-polarized image light is emitted from display device 1 to polarization separation member 101, which has the characteristic of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching polarization separation member 101 from display device 1 is reflected by polarization separation member 101 and goes to retroreflector 2. When the image light is reflected on retroreflector 2, it passes twice through λ / 4 waveplate 21 disposed on the incident surface of retroreflector 2, so the image light is converted from S-polarized light to P-polarized light. The image light converted to P-polarized light goes to polarization separation member 101 again. Here, since polarization separation member 101 has the characteristic of reflecting S-polarized light and transmitting P-polarized light, P-polarized image light passes through polarization separation member 101 and through transparent member 100. The image light passing through the transparent component 100 is generated by the retroreflector 2. Therefore, at the position where the polarization separation component 101 and the displayed image of the display device 1 are in a mirror relationship, an optical image of the displayed image of the display device 1, namely the spatially suspended image 3, is formed. This polarization design enables the spatially suspended image 3 to be formed well.

[0108] Next, a second example of the polarization design in the optical system of FIG. 2A will be described. For example, it can be configured such that P-polarized image light is emitted from the display device 1 to the polarization separation member 101, which has the characteristic of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light reaching the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and goes to the retroreflector 2. When the image light is reflected on the retroreflector 2, it passes twice through the λ / 4 waveplate 21 disposed on the incident surface of the retroreflector 2, so the image light is converted from P-polarized light to S-polarized light. The image light after being converted to S-polarized light goes to the polarization separation member 101 again. Here, since the polarization separation member 101 has the characteristic of reflecting P-polarized light and transmitting S-polarized light, the S-polarized image light passes through the polarization separation member 101 and through the transparent member 100. The image light passing through the transparent component 100 is generated by the retroreflector 2. Therefore, at the position where the polarization separation component 101 and the displayed image of the display device 1 are in a mirror relationship, an optical image of the displayed image of the display device 1, namely the spatially suspended image 3, is formed. This polarization design enables the spatially suspended image 3 to be formed well.

[0109] Furthermore, the light forming the spatial levitation image 3 is a collection of rays converged from the retroreflector 2 to the optical image of the spatial levitation image 3, and these rays continue to travel in a straight line after passing through the optical image of the spatial levitation image 3. Therefore, the spatial levitation image 3, unlike the diffused image light formed on a screen using a conventional projector, is a highly directional image. Thus, in the structure of Figure 2A, when a user views from the direction of arrow A, the spatial levitation image 3 can be seen as a bright image. However, when other people view from the direction of arrow B, the spatial levitation image 3 cannot be seen at all. This characteristic is very suitable for systems displaying images requiring high security, and systems displaying highly confidential images that need to be kept secret from people directly facing the user.

[0110] Furthermore, depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may not be uniform. Additionally, the reflection angle may also be inconsistent. Such inconsistent light may not maintain the intended polarization state and travel angle. For example, light deviating from the intended polarization state and travel angle may directly re-enter the image display surface of the liquid crystal display panel 11 from the position of the retroreflector 2 without passing through the polarization separation component. After being reflected by components within the spatial levitation image display device, this light deviating from the intended polarization state and travel angle may re-enter the image display surface of the liquid crystal display panel 11. This re-entry of the image display surface of the liquid crystal display panel 11 is reflected again on the image display surface constituting the display device 1, potentially causing ghosting and reducing the image quality of the spatial levitation image. Therefore, in this embodiment, an absorptive polarizer 12 can be provided on the image display surface of the display device 1. By allowing the image light emitted from the display device 1 to pass through the absorptive polarizer 12, and by using the absorptive polarizer 12 to absorb the reflected light returning from the polarization separation member 101, the aforementioned re-reflection can be suppressed. This prevents image quality degradation caused by ghosting of spatially suspended images. Specifically, if a structure is adopted in which S-polarized image light is emitted from the display device 1 to the polarization separation member 101, then the absorptive polarizer 12 can be a polarizer that absorbs P-polarized light. Alternatively, if a structure is adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101, then the absorptive polarizer 12 can be a polarizer that absorbs S-polarized light.

[0111] The polarization separation component 101 described above can be formed, for example, by a reflective polarizer or a multilayer metal film that reflects a specific polarization.

[0112] Next, in Figure 2A (2), a representative retroreflector 2 shows an example of the surface shape of the retroreflector. It is composed of regularly arranged hexagonal prisms. Light incident inside is reflected on the walls and bottom of the hexagonal prisms to become retroreflected light, which is emitted in the direction corresponding to the incident light, and displays a spatial levitation image of the real image based on the image displayed on the display device 1.

[0113] The resolution of the spatial levitation image depends not only on the resolution of the liquid crystal display panel 11, but also significantly on the shape D and spacing P of the retroreflective portion of the retroreflective plate 2 shown in (2) of FIG2A. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the spacing is 300 μm, then one pixel of the spatial levitation image is equivalent to 300 μm. Therefore, the effective resolution of the spatial levitation image is reduced to about 1 / 3.

[0114] Therefore, in order to make the resolution of the spatially suspended image the same as that of the display device 1, it is preferable to make the diameter and spacing of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress moiré patterns caused by the pixels of the retroreflective plate and the liquid crystal display panel, the spacing ratio of each can be designed to deviate from one pixel by an integer multiple. In addition, the shape can be configured such that neither side of the retroreflective portion coincides with either side of one pixel of the liquid crystal display panel.

[0115] Furthermore, the surface shape of the retroreflector in this embodiment is not limited to the examples described above. Various surface shapes can be used to achieve retroreflective properties. Specifically, a retroreflective element obtained by periodically arranging triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or combinations thereof on the surface of the retroreflector in this embodiment can be provided. Alternatively, a retroreflective element formed by periodically arranging these prisms and creating cubic angles can also be provided on the surface of the retroreflector in this embodiment. This can also be expressed as a corner reflector array or a multifaceted reflector array. Alternatively, a capsule lens-type retroreflective element obtained by periodically arranging glass microspheres on the surface of the retroreflector in this embodiment can also be provided. Detailed structures of these retroreflective elements can be derived using existing technology, so detailed descriptions are omitted. Specifically, the technology disclosed in Japanese Patent Application Publication No. 2001-33609, Japanese Patent Application Publication No. 2001-264525, Japanese Patent Application Publication No. 2005-181555, Japanese Patent Application Publication No. 2008-70898, and Japanese Patent Application Publication No. 2009-229942 may be used.

[0116] <Example 1 of other structural features of the optical system of a spatial levitation image display device>

[0117] Other structural examples of the optical system for the spatial levitation image display device will be described using FIG2B. Furthermore, structures in FIG2B labeled with the same reference numerals as those in FIG2A have the same function and structure as those in FIG2A. For the sake of simplicity, repeated descriptions of such structures are omitted.

[0118] Similar to that in Figure 2A, the optical system in Figure 2B outputs image light with a specific polarization from the display device 1. This image light with a specific polarization output from the display device 1 is input to the polarization separation component 101B. The polarization separation component 101B is a component that selectively transmits image light with a specific polarization. Unlike the polarization separation component 101 in Figure 2A, the polarization separation component 101B is not integrally formed with the transparent component 100 but is independently plate-shaped. Therefore, the polarization separation component 101B can also be described as a polarization separation plate. For example, the polarization separation component 101B can be configured as a reflective polarizer formed by attaching a polarization separation sheet to the transparent component. Alternatively, it can be formed on the transparent component using a multilayer metal film that selectively transmits light with a specific polarization and reflects other specific polarizations. In Figure 2B, the polarization separation component 101B is configured to transmit image light with a specific polarization output from the display device 1.

[0119] Image light passing through polarization separation component 101B is incident on retroreflector 2. A λ / 4 waveplate 21 is provided on the image light incident surface of retroreflector. The image light passes through the λ / 4 waveplate 21 twice, once when it is incident on the retroreflector and once when it is emitted, thereby undergoing polarization transformation from a specific polarization to another polarization. Here, polarization separation component 101B has the property of reflecting the polarized light of the other polarization after polarization transformation by λ / 4 waveplate 21, so the polarization-transformed image light is reflected on polarization separation component 101B. The image light reflected on polarization separation component 101B passes through transparent component 100, forming a spatially suspended real image 3 on the outside of transparent component 100.

[0120] Here, a first example of polarization design in the optical system of FIG. 2B is explained. For example, it can be configured such that P-polarized image light is emitted from display device 1 to polarization separation member 101B, which has the characteristic of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light reaching polarization separation member 101B from display device 1 passes through polarization separation member 101B and goes to retroreflector 2. When the image light is reflected on retroreflector 2, it passes twice through λ / 4 waveplate 21 disposed on the incident surface of retroreflector 2, so the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light goes to polarization separation member 101B again. Here, since polarization separation member 101B has the characteristic of reflecting S-polarized light and transmitting P-polarized light, S-polarized image light is reflected on polarization separation member 101B and passes through transparent member 100. The image light passing through the transparent component 100 is generated by the retroreflector 2. Therefore, at the position where the polarization separation component 101B and the displayed image of the display device 1 are in a mirror relationship, an optical image of the displayed image of the display device 1, namely the spatially suspended image 3, is formed. This polarization design enables the spatially suspended image 3 to be formed well.

[0121] Next, a second example of the polarization design in the optical system of Figure 2B will be described. For example, it can be configured such that S-polarized image light is emitted from the display device 1 to the polarization separation member 101B, which has the characteristic of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and goes to the retroreflector 2. When the image light is reflected on the retroreflector 2, it passes twice through the λ / 4 waveplate 21 disposed on the incident surface of the retroreflector 2, so the image light is converted from S-polarized light to P-polarized light. The image light converted to P-polarized light goes to the polarization separation member 101B again. Here, since the polarization separation member 101B has the characteristic of reflecting P-polarized light and transmitting S-polarized light, the P-polarized image light is reflected on the polarization separation member 101B and passes through the transparent member 100. The image light passing through the transparent component 100 is generated by the retroreflector 2. Therefore, at the position where the polarization separation component 101B and the displayed image of the display device 1 are in a mirror relationship, an optical image of the displayed image of the display device 1, namely the spatially suspended image 3, is formed. This polarization design enables the spatially suspended image 3 to be formed well.

[0122] Furthermore, in Figure 2B, the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separation member 101B is configured to be tilted at an angle α (e.g., 30°) relative to the image display surface of the display device 1 and the surface of the retroreflector 2. Thus, when reflected at the polarization separation member 101B, the direction of travel of the image light reflected at the polarization separation member 101B (the direction of the principal ray of the image light) differs from the direction of travel of the image light incident from the retroreflector 2 by an angle β (e.g., 60°). By employing this structure, in the optical system of Figure 2B, image light is output to the outside of the transparent member 100 at the specified angle shown in the figure, forming a spatially suspended image 3 of real image. In the structure of Figure 2B, when a user views from the direction of arrow A, the spatially suspended image 3 can be seen as a bright image. However, when other people view from the direction of arrow B, the spatially suspended image 3 cannot be seen as an image at all. This feature is ideal for use in systems that display images requiring high security, and in systems that display highly confidential images that need to be kept secret from people facing the user.

[0123] As explained above, the optical system in Figure 2B has a different structure than the optical system in Figure 2A, but it can form a good spatial levitation image in the same way as the optical system in Figure 2A.

[0124] Additionally, an absorptive polarizer can be provided on the surface of the transparent component 100 on the side of the polarization separation member 101B. This absorptive polarizer can be an absorptive polarizer that allows the polarized transmission of image light from the polarization separation member 101B and absorbs polarizations that are 90° out of phase with the polarization of the image light from the polarization separation member 101B. In this way, the image light used to form the spatial levitation image 3 can be fully transmitted, while reducing the external light incident from the spatial levitation image 3 side of the transparent component 100 by about 50%. As a result, stray light in the optical system of FIG. 2B generated by external light incident from the spatial levitation image 3 side of the transparent component 100 can be reduced.

[0125] <Example 2 of other structural features of the optical system of a spatial levitation image display device>

[0126] Other structural examples of the optical system for the spatial levitation image display device will be described using FIG2C. Furthermore, structures in FIG2C labeled with the same reference numerals as those in FIG2B have the same function and structure as those in FIG2B. For the sake of simplicity, repeated descriptions of such structures are omitted.

[0127] The optical system in Figure 2C differs from that in Figure 2B only in the arrangement angle of the polarization separation component 101B relative to the image display surface of the display device 1 and the surface of the retroreflector 2. All other structures are the same as the optical system in Figure 2B, so repeated descriptions are omitted. The polarization design of the optical system in Figure 2C is also the same as that of the optical system in Figure 2B, so repeated descriptions are omitted.

[0128] In the optical system of Figure 2C, the polarization separating member 101B is configured to be tilted at an angle α relative to the image display surface of the display device 1 and the surface of the retroreflector 2. In Figure 2C, this angle α is 45°. With this structure, when reflected at the polarization separating member 101B, the angle β between the direction of travel of the image light incident from the retroreflector 2 (the direction of the principal ray of the image light) and the direction of travel of the image light reflected at the polarization separating member 101B (the direction of the principal ray of the image light) is 90°. With this structure, the surfaces of the image display surface of the display device 1 and the retroreflector 2 are perpendicular to the direction of travel of the image light reflected at the polarization separating member 101B, which simplifies the angular relationships of the surfaces constituting the optical system. If the surface of the transparent member 100 is configured to be orthogonal to the direction of travel of the image light reflected at the polarization separating member 101B, the angular relationships of the surfaces constituting the optical system can be further simplified. In the structure shown in Figure 2C, when a user views from the direction of arrow A, the spatially suspended image 3 can be seen as a bright image. However, when other people view from the direction of arrow B, the spatially suspended image 3 cannot be seen as an image at all. This feature is very suitable for systems that display images requiring high security, and systems that display highly confidential images that need to be kept secret from people facing the user.

[0129] As explained above, the optical system in Figure 2C has a different structure than the optical systems in Figures 2A and 2B, but it can form a good spatial levitation image in the same way as the optical systems in Figures 2A and 2B. In addition, it simplifies the angles of the surfaces constituting the optical system.

[0130] Alternatively, an absorptive polarizer can be provided on the surface of the transparent component 100 on the side of the polarization separation member 101B. This absorptive polarizer can be an absorptive polarizer that allows the polarized transmission of image light from the polarization separation member 101B and absorbs polarizations that are 90° out of phase with the polarization of the image light from the polarization separation member 101B. In this way, the image light used to form the spatial levitation image 3 can be fully transmitted, while reducing the external light incident from the spatial levitation image 3 side of the transparent component 100 by about 50%. As a result, stray light in the optical system of FIG. 2C generated by external light incident from the spatial levitation image 3 side of the transparent component 100 can be reduced.

[0131] <Example 3 of other structural examples of the optical system of a spatial levitation image display device>

[0132] Other structural examples of the optical system for the spatial levitation image display device will be described using FIG2D. The optical system of FIG2D uses a retroreflector 5, which is different from the retroreflector 2 used in FIG2A-2C. Hereinafter, other structural examples 3 of the optical system will be described in more detail using FIG2D-2I. The structures in FIG2D labeled with the same reference numerals as those in FIG2A-C have the same function and structure as those in FIG2A-2C. For the sake of simplicity, repeated descriptions of such structures will be omitted.

[0133] Figure 2D is a diagram showing an example of the main structure and retroreflective section structure of a spatial levitation image display device according to an embodiment of the present invention. A display device 10 for emitting image light is provided in the oblique direction of a transparent component 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 for generating light.

[0134] The main ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α can be, for example, 45°. However, the incident angle α is not limited to 45°, and for example, 45°±15° can be used.

[0135] The retroreflector 5 is an optical component that has the optical property of causing retroreflective reflection of light in at least a portion of its directions. Furthermore, the reflected light possesses optical properties capable of forming an image, so the retroreflector 5 can also be described as an imaging optical component or an imaging optical plate.

[0136] The specific structure of the retroreflector 5 will be described in detail using Figures 2E and 2F. Under the action of the retroreflector 5, the principal ray 9020 travels in the z direction while undergoing retroreflection in the x and y directions. As a result, the reflected ray 9021 travels away from the retroreflector 5 in a mirror-symmetric optical path relative to the principal ray 9020 with the retroreflector 5 as a reference, and passes through the transparent component 100 to form a spatially suspended image 3 as a real image on the imaging surface.

[0137] The light beam that forms the spatial levitation image 3 is a collection of light rays converged from the retroreflector 5 to the optical image of the spatial levitation image 3. These light rays continue to travel in a straight line after passing through the optical image of the spatial levitation image 3. Therefore, the spatial levitation image 3, unlike diffused images formed on a screen using a conventional projector, is a highly directional image. Thus, in the structure shown in Figure 2, when a user views from the direction of arrow A, the spatial levitation image 3 can be seen as a bright image. However, when other people view from the direction of arrow B, the spatial levitation image 3 cannot be seen at all. This characteristic is suitable for systems displaying images requiring high security, and systems displaying highly confidential images that need to be kept secret from people directly facing the user.

[0138] Figures 2E and 2F illustrate an example of the structure of the retroreflector 5. The retroreflector 5 is a structure in which multiple corner reflectors 9040 are arranged in an array on the surface of the transparent component 50. It can also be referred to as a corner reflector array or a multi-faceted reflector array. The specific structure of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Rays 9111, 9112, 9113, and 9114 emitted from the light source 9110 are reflected twice by the two mirrors 9041 and 9042 of the corner reflector 9040, becoming reflected rays 9121, 9122, 9123, and 9124. These two reflections are retroreflections in the x and y directions, returning in the same direction as the incident direction (traveling in a direction rotated by 180°), and in the z direction, they are orthographic reflections due to total internal reflection, where the angle of incidence and the angle of reflection are the same.

[0139] That is, rays 9111-9114 produce reflected rays 9121-9124 along a straight line symmetrical about the corner reflector 9040 in the z-direction, forming a real image 9120 in the air. Furthermore, rays 9111-9114 emanating from the light source 9110 represent four rays of diffused light from the light source 9110. Depending on the diffusion characteristics of the light source 9110, the rays incident on the retroreflector 5 are not limited to these, but all incident rays undergo the same reflection, forming a real image 9120 in the air. Additionally, for ease of viewing, the positions of the light source 9110 and the real image 9120 in the x-direction are shown offset, but in reality, the positions of the light source 9110 and the real image 9120 in the x-direction are the same, coinciding when viewed from the z-direction.

[0140] Next, the structure and effect of the corner reflector 9040 constituting the retroreflector 5 will be explained using Figures 2G, 2H, and 2I. The corner reflector 9040 is a cuboid with only two specific surfaces being mirrored surfaces 9041 and 9042, and the other four surfaces being transparent components. The retroreflector 5 is configured such that the corner reflectors 9040 are arranged in an array with the corresponding mirrored surfaces facing the same direction.

[0141] When viewed from the top surface (+z direction), the light ray 9111 emitted from the light source 9110 is incident on the mirror 9041 (or mirror 9042) at a specific angle of incidence. After total internal reflection occurs at the reflection point 9130, total internal reflection occurs again at the reflection point 9132 on the mirror 9042 (or mirror 9041).

[0142] Let θ be the angle of incidence of ray 9111 on mirror 9041 (or mirror 9042). Then, the angle of incidence of the first reflected ray 9131, after reflection on mirror 9041 (or mirror 9042), relative to mirror 9042 (or mirror 9041), can be expressed as 90°-θ. Therefore, relative to ray 9111, the second reflected ray 9121 rotates by 2θ due to the first reflection and by 2×(90°-θ) due to the second reflection, resulting in a total reversal of 180°. On the other hand, when viewed from the side (the midway between -x and -y), total internal reflection in the z-direction occurs only once. Therefore, if φ is the angle of incidence on mirror 9041 or mirror 9042, then the reflected ray 9121 rotates by 2×φ relative to ray 9111 due to the first reflection.

[0143] As described above, the light incident on the corner reflector 9040 undergoes retrograde reflection in the x and y directions, resulting in a reversed light path, and in the z direction, it undergoes orthogonal reflection caused by total internal reflection. Considering the retrograde reflector 5, since all light paths undergo the same reflection, a point image is formed symmetrically in the z-axis direction due to the converging reversed light paths in the x and y directions.

[0144] Here, in the optical system of Figures 2A-2C, the retroreflector 2 has retroreflective characteristics in three axes. Therefore, when a diffusing incident beam is incident on the retroreflector 2, the converging reflected beam travels relative to the retroreflector 2 towards the side where the incident light source is located. This converging reflected beam forms an image in the air, creating a spatially suspended image 3. The direction of travel of the principal ray of the converging reflected beam reflected from the retroreflector 2 is the opposite of the direction of travel of the principal ray of the diffusing incident beam incident on the retroreflector 2.

[0145] In contrast, in the optical system of Figure 2D, the retroreflector 5 exhibits retroreflective characteristics along two axes and orthographic reflection along the other axis. Thus, when a diffusing incident beam is incident on the retroreflector 5, the converging reflected beam, after being reflected by the corner reflector array, travels relative to the retroreflector 5 on the opposite side from the side where the incident light source is located. This converging reflected beam forms an image in the air, creating a spatially suspended image 3.

[0146] The direction of travel of the principal ray of the converging reflected beam after being reflected by the corner reflector array of the retroreflector 5 is not the opposite direction of travel of the principal ray of the diffusing incident beam incident on the retroreflector 5. The component of the normal direction of the principal ray of the diffusing incident beam incident on the retroreflector 5 and the component of the normal direction of the surface of the plate shape of the retroreflector 5 of the direction of travel of the principal ray after being reflected on the retroreflector 5 to become a converging reflected beam do not change before and after reflection by the corner reflector array, but travel in a straight line.

[0147] That is, through reflection on the retroreflector 5, the diffuse incident beam is transformed into a converging reflected beam, but in the direction normal to the surface of the retroreflector 5, the beam travels through the retroreflector 5. Here, the diffuse incident beam incident on the retroreflector 5 and the converging reflected beam exiting the retroreflector 5 have a geometrically symmetrical relationship with respect to the surface of the retroreflector 5.

[0148] Regarding the spatial levitation image obtained by imaging the light from the image output unit 10, its resolution depends not only on the resolution of the liquid crystal display panel 11, but also significantly on the diameter D and spacing P (not shown) of the retroreflective portion of the retroreflective plate 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the spacing P is 300 μm, then one pixel of the spatial levitation image is equivalent to 300 μm. Therefore, the effective resolution of the spatial levitation image is reduced to about one-third.

[0149] Therefore, in order to make the resolution of the spatially suspended image the same as that of the display device 10, it is preferable to make the diameter D and the spacing P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress moiré patterns caused by the pixels of the retroreflective plate and the liquid crystal display panel, the spacing ratio of each can be designed to deviate from one pixel by an integer multiple. In addition, the shape can be configured such that neither side of the retroreflective portion coincides with either side of one pixel of the liquid crystal display panel.

[0150] Furthermore, the shape of the retroreflective plate (imaging optical plate) in this embodiment is not limited to the examples described above. It can have various shapes to achieve retroreflection. Specifically, it can be various cubic corner reflectors, corner reflector arrays, slit mirror arrays, dihedral corner reflector arrays, multifaceted reflector arrays, or combinations of these reflective surfaces arranged periodically. Alternatively, a capsule lens-type retroreflective element obtained by periodically arranging glass microspheres on the surface of the retroreflective plate in this embodiment can also be used. Detailed structures of these retroreflective elements can be derived using existing technology, so detailed descriptions are omitted. Specifically, technologies disclosed in Japanese Patent Application Publication No. 2017-33005, Japanese Patent Application Publication No. 2019-133110, Japanese Patent Application Publication No. 2017-67933, and WO2009 / 131128 can be used.

[0151] Furthermore, in the optical system of Figure 2D, the image light emitted from the display device 10 can be of any polarization state. Whether it is S-polarization or P-polarization is not a problem.

[0152] As explained above, although the optical system in Figure 2D uses a different retroreflector than the optical systems in Figures 2A to 2C, it is able to form a better spatial levitation image, just like the optical systems in Figures 2A to 2C.

[0153] The optical systems described in Figures 2A, 2B, 2C, and 2D above can provide brighter, higher-quality spatial levitation images.

[0154] <<Block diagram of the internal structure of the spatial levitation image display device>>

[0155] Next, a block diagram of the internal structure of the spatial levitation image display device 1000 will be described. Figure 3 is a block diagram showing an example of the internal structure of the spatial levitation image display device 1000.

[0156] The spatial levitation image display device 1000 includes a retroreflective unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power input interface 1111, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an air operation detection sensor 1351, an air operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, and a camera unit 1180. Additionally, it may also include a movable media interface 1134, an attitude sensor 1113, a transmissive self-emissive image display device 1650, a second display device 1680, or a secondary battery 1112.

[0157] The various components of the spatial levitation image display device 1000 are arranged in the housing 1190. In addition, the camera unit 1180 and the air operation detection sensor 1351 shown in FIG3 can also be arranged on the outside of the housing 1190.

[0158] The retroreflective section 1101 in Figure 3 corresponds to the retroreflective plate 2 in Figures 2A, 2B, and 2C. The retroreflective section 1101 causes retroreflection of the light modulated by the image display section 1102. The light reflected from the retroreflective section 1101 and output to the outside of the spatial levitation image display device 1000 is used to form a spatial levitation image 3.

[0159] The image display unit 1102 in Figure 3 corresponds to the liquid crystal display panel 11 in Figures 2A, 2B, and 2C. The light source 1105 in Figure 3 corresponds to the light source device 13 in Figures 2A, 2B, and 2C. The image display unit 1102, light guide 1104, and light source 1105 in Figure 3 correspond to the display device 1 in Figures 2A, 2B, and 2C.

[0160] The image display unit 1102 is a display unit that generates an image by modulating transmitted light based on an image signal input under the control of the image control unit 1160 (described later). The image display unit 1102 corresponds to the liquid crystal display panel 11 in Figures 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the image display unit 1102. Alternatively, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device) panel can be used as the image display unit 1102.

[0161] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power supply 1106 converts AC current input from the outside via the external power input interface 1111 into DC current to power the light source 1105. Additionally, the power supply 1106 supplies necessary DC current to various parts within the spatial levitation image display device 1000. The secondary battery 1112 stores the power supplied by the power supply 1106. Furthermore, when not powered externally via the external power input interface 1111, the secondary battery 1112 powers the light source 1105 and other structures requiring power. In other words, with the secondary battery 1112 in place, the spatial levitation image display device 1000 can be used by the user even without external power.

[0162] The light guide 1104 guides the light generated by the light source 1105, directing it towards the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be referred to as the backlight of the image display unit 1102. The light guide 1104 can be constructed primarily of glass. Alternatively, it can be constructed primarily of plastic. It can also be constructed using a reflector. Various combinations of the light guide 1104 and the light source 1105 are possible. Specific structural examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0163] The air-to-air operation detection sensor 1351 is a sensor that detects the user 230's finger operation on the spatial levitation image 3. The air-to-air operation detection sensor 1351 can, for example, sense the area overlapping with the entire display area of ​​the spatial levitation image 3. Alternatively, the air-to-air operation detection sensor 1351 can also sense only the area overlapping with at least a portion of the display area of ​​the spatial levitation image 3.

[0164] As a specific example of the airborne operation detection sensor 1351, a distance sensor using non-visible light such as infrared, non-visible light laser, or ultrasound can be cited. Alternatively, the airborne operation detection sensor 1351 can also combine multiple sensors to form a system capable of detecting coordinates in a two-dimensional plane. Furthermore, the airborne operation detection sensor 1351 can be constructed from a ToF (Time of Flight) type LiDAR (Light Detection and Ranging) sensor or an image sensor.

[0165] The airborne operation detection sensor 1351 only needs to be able to sense and detect touch operations performed by the user's finger on an object displayed as a spatial levitation image 3. Such sensing can also be performed using existing technologies.

[0166] The airborne operation detection unit 1350 acquires sensing signals from the airborne operation detection sensor 1351, determines whether the user 230's finger has made contact with an object in the space-suspended image 3 based on the sensing signals, and calculates the position of contact between the user 230's finger and the object (contact position), etc. The airborne operation detection unit 1350 is constructed, for example, by a circuit such as an FPGA (Field Programmable Gate Array). In addition, some functions of the airborne operation detection unit 1350 can also be implemented in software, for example, by a spaceborne operation detection program executed by the control unit 1110.

[0167] The airborne operation detection sensor 1351 and the airborne operation detection unit 1350 can be built into the space-based levitation image display device 1000, or they can be installed separately from the space-based levitation image display device 1000. When installed separately from the space-based levitation image display device 1000, the airborne operation detection sensor 1351 and the airborne operation detection unit 1350 are configured to transmit information and signals to the space-based levitation image display device 1000 via wired or wireless communication connection paths or image signal transmission paths.

[0168] Furthermore, the airborne operation detection sensor 1351 and the airborne operation detection unit 1350 can also be installed separately. This allows for the construction of a system that can optionally add only the airborne operation detection function, using the space-based levitation image display device 1000, which does not have airborne operation detection functionality, as the main body. Alternatively, only the airborne operation detection sensor 1351 can be separated, and the airborne operation detection unit 1350 can be integrated into the space-based levitation image display device 1000. In cases where more flexibility in the placement of the airborne operation detection sensor 1351 relative to the installation location of the space-based levitation image display device 1000 is desired, the structure that separates only the airborne operation detection sensor 1351 has advantages.

[0169] The camera unit 1180 is, for example, a camera with an image sensor, that captures images of the space near the levitation image 3 and / or the user 230's face, arms, fingers, etc. Multiple camera units 1180 can be provided. By using multiple camera units 1180, or by using a camera unit with a depth sensor, the air operation detection unit 1350 can be assisted when detecting touch operations by the user 230 on the levitation image 3. The camera unit 1180 can also be provided separately from the levitation image display device 1000. When the camera unit 1180 and the levitation image display device 1000 are provided separately, it can be configured to transmit camera signals to the levitation image display device 1000 via a wired or wireless communication connection path.

[0170] For example, if the airborne operation detection sensor 1351 is configured to detect whether an object has invaded the intrusion detection plane by targeting the plane (intrusion detection plane) that includes the display surface of the spatial levitation image 3, there may be situations where the airborne operation detection sensor 1351 cannot detect how far away an object (such as a user's finger) that has not invaded the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0171] In this case, by using depth calculation information of the object obtained from images captured by multiple camera units 1180 and depth information of the object obtained by a depth sensor, the distance between the object and the intrusion detection plane can be calculated. Then, this information and various other information, such as the distance between the object and the intrusion detection plane, are used for various display controls of the spatial levitation image 3.

[0172] Alternatively, instead of using the air operation detection sensor 1351, the air operation detection unit 1350 can detect the user 230's touch operation on the spatial levitation image 3 based on the image captured by the camera unit 1180.

[0173] Alternatively, the camera unit 1180 can capture the face of the user 230 operating the floating image 3, and the control unit 1110 can perform user 230 recognition processing. Furthermore, to determine whether someone is standing around or behind the user 230 operating the floating image 3, or whether someone is spying on the user 230's operation of the floating image 3, the camera unit 1180 can also capture images of the area including the user 230 operating the floating image 3 and the area surrounding the user 230.

[0174] The operation input unit 1107 is, for example, a signal receiving unit or an infrared receiving unit such as an operation button or a remote control, which inputs signals for operations different from the air operation (touch operation) performed by the user 230. In addition to the user 230 who performs touch operation on the spatial levitation image 3, the operation input unit 1107 can also be used for an administrator to operate the spatial levitation image display device 1000.

[0175] The image signal input unit 1131 is connected to an external image output device to input image data. The image signal input unit 1131 can be various digital image input interfaces. For example, it can be a video input interface of the HDMI (High-Definition Multimedia Interface) standard, a video input interface of the DVI (Digital Visual Interface) standard, or a video input interface of the DisplayPort standard.

[0176] Alternatively, analog video input interfaces such as analog RGB and component video can be provided. The audio signal input unit 1133 connects to an external audio output device to input audio data. The audio signal input unit 1133 can be configured as an HDMI standard audio input interface, an optical digital terminal interface, or a coaxial digital terminal interface. When using an HDMI standard interface, the video signal input unit 1131 and the audio signal input unit 1133 can be configured as an interface integrating terminals and cables. The audio output unit 1140 can output sound based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 can be configured as a speaker.

[0177] Additionally, the audio output unit 1140 can also output built-in operation tones and error warning tones. Alternatively, the audio output unit 1140 can be configured to output digital signals to external devices, similar to the Audio Return Channel function specified in the HDMI standard. The microphone 1139 is a microphone that collects sound from the vicinity of the spatial floating image display device 1000 and converts it into a signal to generate an audio signal. Alternatively, the microphone can be configured to collect the voice of a user or other person (speech), and the generated audio signal can be processed by the control unit 1110 (described later) for voice recognition processing (speech recognition processing) to obtain text information from the audio signal.

[0178] The non-volatile memory 1108 stores various data used in the spatial levitation image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, various operational data for displaying the spatial levitation image 3, display icons, data on objects for user operation, and layout information. The main memory 1109 stores image data displayed as the spatial levitation image 3 and control data for the device.

[0179] The control unit 1110 controls the operation of each connected component. In addition, the control unit 1110 can cooperate with the program stored in the memory 1109 to perform calculations based on information obtained from each component in the spatial levitation image display device 1000.

[0180] The communication unit 1132 communicates with external devices, external servers, etc., via a wired or wireless communication interface. When the communication unit 1132 has a wired communication interface, this interface can be configured as, for example, a LAN interface using an Ethernet standard. When the communication unit 1132 has a wireless communication interface, it can be configured as, for example, a Wi-Fi communication interface, a Bluetooth communication interface, or a 4G or 5G mobile communication interface. Through communication via the communication unit 1132, various data such as video data, image data, and audio data are sent and received.

[0181] Additionally, the removable media interface 1134 is an interface for connecting a removable recording medium (removable medium). The removable recording medium (removable medium) can be composed of semiconductor memory such as a solid-state drive (SSD), magnetic recording media such as a hard disk drive (HDD), or optical recording media such as an optical disc. The removable media interface 1134 can read various data and information, such as image data, audio data, etc., recorded on the removable recording medium. The image data and audio data recorded on the removable recording medium are output as a spatial levitation image 3 via the image display unit 1102 and the retroreflective unit 1101.

[0182] Storage unit 1170 is a storage device that records various types of data and information, such as image data, audio data, etc. Storage unit 1170 can be composed of a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor device memory such as a solid-state drive (SSD). In storage unit 1170, for example, various types of data and information, such as image data, audio data, etc., can be pre-recorded at the time of product shipment. Furthermore, storage unit 1170 can also record various types of data and information, such as image data, audio data, etc., obtained from external devices and external servers via communication unit 1132.

[0183] The image data and other data recorded in the storage unit 1170 are output as a spatial levitation image 3 via the image display unit 1102 and the retroreflection unit 1101. The image data and other data of the display icons and objects for user operation displayed as the spatial levitation image 3 are also recorded in the storage unit 1170.

[0184] The layout information of the display icons, objects, etc., displayed as the spatial levitation image 3, as well as various metadata information about the objects, are also recorded in the storage unit 1170. The sound data recorded in the storage unit 1170 is output as sound, for example, from the sound output unit 1140.

[0185] The image control unit 1160 performs various controls on the image signals input to the image display unit 1102. The image control unit 1160 can also be called an image processing circuit, and may be constructed from hardware such as an ASIC, FPGA, or video processor. Alternatively, the image control unit 1160 can also be called an image processing unit or an image processing unit. For example, the image control unit 1160 controls image switching, switching which image signal from the image signal stored in the memory 1109 or the image signal (image data) input to the image signal input unit 1131 is input to the image display unit 1102.

[0186] In addition, the image control unit 1160 can also be controlled to generate a superimposed image signal obtained by superimposing the image signal stored in the memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby forming a composite image as a spatial levitation image 3.

[0187] Additionally, the image control unit 1160 can also perform control, and perform image processing on the image signals input from the image signal input unit 1131 and the image signals stored in the memory 1109. Examples of image processing include scaling (enlarging, reducing, and distorting the image), brightness adjustment (changing brightness), contrast adjustment (changing the image's contrast curve), and Retinex processing (decomposing the image into light components and changing the weights of each component).

[0188] In addition, the image control unit 1160 can also perform special effects image processing on the image signal input to the image display unit 1102 to assist the user 230 in-flight operation (touch operation). The special effects image processing is performed, for example, based on the detection results of the touch operation of the user 230 by the in-flight operation detection unit 1350 and the image captured by the camera unit 1180 of the user 230.

[0189] The attitude sensor 1113 is a sensor composed of a gravity sensor, an acceleration sensor, or a combination thereof, capable of detecting the setting attitude of the spatial levitation image display device 1000. The control unit 1110 can control the operation of each connected component based on the attitude detection results of the attitude sensor 1113. For example, if the attitude is detected to be unsuitable for the user, control can be implemented to stop the display of the image on the image display unit 1102 and display an error message to the user. Alternatively, if the attitude sensor 1113 detects a change in the setting attitude of the spatial levitation image display device 1000, control can be implemented to rotate the display direction of the image displayed on the image display unit 1102.

[0190] As explained above, the spatial levitation image display device 1000 is equipped with various functions. However, the spatial levitation image display device 1000 does not need to have all of these functions; it can have any structure as long as it has the function of forming the spatial levitation image 3.

[0191] <Structural Example of a Spatial Suspended Image Display Device>

[0192] Next, a structural example of the spatial levitation image display device will be described. Regarding the layout of the constituent elements of the spatial levitation image display device of this embodiment, various layouts may exist depending on the usage configuration. Hereinafter, the layouts of Figures 4A to 4M will be described. Furthermore, in any of the examples in Figures 4A to 4M, the thick lines surrounding the spatial levitation image display device 1000 represent an example of the housing structure of the spatial levitation image display device 1000.

[0193] Figure 4A is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4A is equipped with an optical system corresponding to the optical system in Figure 2A. The spatial levitation image display device 1000 shown in Figure 4A is horizontally arranged with the side forming the spatial levitation image 3 facing upwards. That is, in Figure 4A, the transparent component 100 of the spatial levitation image display device 1000 is provided on the top surface of the device. The spatial levitation image 3 is formed above the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels obliquely upwards. When the air operation detection sensor 1351 is provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3. Furthermore, the x-direction is the left-right direction from the user's perspective, the y-direction is the front-back direction (depth direction) from the user's perspective, and the z-direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x-direction, y-direction, and z-direction in Figures 4A to 4M are the same, so repeated explanations are omitted.

[0194] Figure 4B is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4B is equipped with an optical system corresponding to the optical system in Figure 2A. The spatial levitation image display device 1000 shown in Figure 4B is arranged longitudinally with the side forming the spatial levitation image 3 facing the front of the spatial levitation image display device 1000 (in the direction of the user 230). That is, in Figure 4B, the transparent component 100 of the spatial levitation image display device is provided on the front of the device (in the direction of the user 230). The spatial levitation image 3 is formed on the side of the user 230, compared to the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels obliquely upwards. With the air operation detection sensor 1351 provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3. Here, as shown in Figure 4B, the air operation detection sensor 1351 senses the user 230's finger from above and can use the reflection of the sensing light from the user's fingernail for touch detection. Generally speaking, fingernails have a higher reflectivity than fingertips, so this structure can improve the accuracy of touch detection.

[0195] Figure 4C is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4C is equipped with an optical system corresponding to the optical system in Figure 2B. The spatial levitation image display device 1000 shown in Figure 4C is horizontally arranged with the side forming the spatial levitation image 3 facing upwards. That is, in Figure 4C, the transparent component 100 of the spatial levitation image display device 1000 is provided on the top surface of the device. The spatial levitation image 3 is formed above the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels obliquely upwards. When the air operation detection sensor 1351 is provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3.

[0196] Figure 4D is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4D is equipped with an optical system corresponding to the optical system in Figure 2B. The spatial levitation image display device 1000 shown in Figure 4D is arranged longitudinally with the side forming the spatial levitation image 3 facing the front of the spatial levitation image display device 1000 (in the direction of the user 230). That is, in Figure 4D, the transparent component 100 of the spatial levitation image display device 1000 is provided on the front of the device (in the direction of the user 230). The spatial levitation image 3 is formed on the side of the user 230, compared to the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels obliquely upwards. With the air operation detection sensor 1351 provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3. Here, as shown in Figure 4D, the air operation detection sensor 1351 senses the user 230's finger from above and can use the reflection of the sensing light from the user's fingernail for touch detection. Generally speaking, fingernails have a higher reflectivity than fingertips, so this structure can improve the accuracy of touch detection.

[0197] Figure 4E is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4E is equipped with an optical system corresponding to the optical system in Figure 2C. The spatial levitation image display device 1000 shown in Figure 4E is horizontally arranged with the side forming the spatial levitation image 3 facing upwards. That is, in Figure 4E, the transparent component 100 of the spatial levitation image display device 1000 is provided on the top surface of the device. The spatial levitation image 3 is formed above the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels directly upwards. When the air operation detection sensor 1351 is provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3.

[0198] Figure 4F is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4F is equipped with an optical system corresponding to the optical system in Figure 2C. The spatial levitation image display device 1000 shown in Figure 4F is arranged longitudinally with the side forming the spatial levitation image 3 facing the front of the spatial levitation image display device 1000 (in the direction of the user 230). That is, in Figure 4F, the transparent component 100 in the spatial levitation image display device 1000 is provided on the front of the device (in the direction of the user 230). The spatial levitation image 3 is formed on the side of the user 230, compared to the surface of the transparent component 100 of the spatial levitation image display device 1000. The light from the spatial levitation image 3 travels in the direction in front of the user. With the air operation detection sensor 1351 provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3.

[0199] Figure 4G is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 shown in Figure 4G is equipped with an optical system corresponding to the optical system in Figure 2C. In the optical systems of the spatial levitation image display devices in Figures 4A to 4F, the light path of the center of the image light emitted from the display device 1 is located on the yz plane. That is, in the optical systems of the spatial levitation image display devices in Figures 4A to 4F, the image light travels in the front-back direction and the up-down direction from the user's perspective. In contrast, in the optical system of the spatial levitation image display device shown in Figure 4G, the light path of the center of the image light emitted from the display device 1 is located on the xy plane. That is, in the optical system of the spatial levitation image display device shown in Figure 4G, the image light travels in the left-right direction and the front-back direction from the user's perspective. The spatial levitation image display device 1000 shown in Figure 4G is arranged such that the side forming the spatial levitation image 3 faces the front of the device (the direction of the user 230). That is, in Figure 4G, the transparent component 100 in the spatial levitation image display device 1000 is provided on the front of the device (the direction of the user 230). Compared to the surface of the transparent component 100 of the spatial levitation image display device 1000, the spatial levitation image 3 is formed on the user's side. The light from the spatial levitation image 3 travels in the direction in front of the user. When the air operation detection sensor 1351 is set as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3.

[0200] Figure 4H is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 in Figure 4H has a window made of a transparent plate 100B (such as glass or plastic) on the back of the device (the side opposite to the position where the user 230 views the spatial levitation image 3, i.e., the side opposite to the direction of travel of the image light of the spatial levitation image 3 towards the user 230), which differs from the spatial levitation image display device in Figure 4G. Other structures are the same as those in Figure 4G, so repeated descriptions are omitted. In the spatial levitation image display device 1000 of Figure 4H, for the spatial levitation image 3, a window with a transparent plate 100B is provided on the side opposite to the direction of travel of the image light of the spatial levitation image 3. Therefore, when the user 230 views the spatial levitation image 3, the scenery behind the spatial levitation image display device 1000 can be identified as the background of the spatial levitation image 3. Thus, the user 230 can perceive that the spatial levitation image 3 is suspended in the air in front of the scenery behind the spatial levitation image display device 1000. Therefore, the sense of floating in the air in the spatial levitation image 3 can be emphasized more.

[0201] Furthermore, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, there is a possibility that a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and go to the transparent plate 100B. And depending on the coating performance of the surface of the transparent plate 100B, there is a possibility that this light may be reflected again on the surface of the transparent plate 100B as stray light and be seen by the user. Therefore, to prevent this stray light, the transparent plate 100B may not be provided on the aforementioned window on the back of the spatially suspended image display device 1000.

[0202] Figure 4I is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 in Figure 4I differs from that in Figure 4H in that a light-blocking door 1410 is provided at the window of the transparent plate 100B located on the back of the device (on the opposite side from where the user 230 views the spatial levitation image 3). The other structures are the same as those in the spatial levitation image display device of Figure 4H, so repeated descriptions are omitted.

[0203] The opening / closing door 1410 of the spatial levitation image display device 1000 in Figure 4I has, for example, a light-shielding plate, including a mechanism for moving (sliding) the light-shielding plate, a rotation mechanism, or a mechanism for making the light-shielding plate detachable. This allows the window (rear side window) of the transparent plate 100B located at the depth side of the spatial levitation image display device 1000 to switch between an open state and a light-shielding state. The opening / closing door 1410 can be driven by a motor (not shown) to electrically move (slide) and rotate the light-shielding plate. This motor can be controlled by the control unit 1110 in Figure 3. Furthermore, the example in Figure 4I discloses an example where the opening / closing door 1410 has two light-shielding plates. However, the opening / closing door 1410 can also have only one light-shielding plate.

[0204] For example, when the view visible from the depths of the window of the transparent plate 100B of the spatial levitation image display device 1000 is outdoors, the brightness of sunlight varies with the weather. When the outdoor sunlight is strong, the background of the spatial levitation image 3 becomes too bright, reducing the user 230's ability to distinguish the spatial levitation image 3. In such cases, if the back window is shaded by moving (sliding), rotating, or installing the light-shielding plate of the opening / closing door 1410, the background of the spatial levitation image 3 will darken, thus relatively improving the distinguishability of the spatial levitation image 3. The light-shielding action of the light-shielding plate of the opening / closing door 1410 can also be performed manually by the user 230. Alternatively, the light-shielding action of the light-shielding plate of the opening / closing door 1410 can be performed by the control unit 1110 controlling a motor (not shown) according to the operation input unit 1107 in FIG3.

[0205] Alternatively, an illuminance sensor can be installed on the back side of the spatial levitation image display device 1000 (the side opposite to the user 230) – for example, near the back side window – to measure the brightness of the space outside the back side window. In this case, based on the detection result of the illuminance sensor, the control unit 1110 of FIG3 can control a motor (not shown) to open and close the light shield of the opening and closing door 1410. By controlling the opening and closing of the light shield of the opening and closing door 1410 in this way, the recognizability of the spatial levitation image 3 can be better maintained even if the user 230 does not manually open and close the light shield of the opening and closing door 1410.

[0206] Furthermore, the light shield of the opening / closing door 1410 can also be manually detachable. Depending on the intended use and environment of the spatial floating image display device 1000, the user can choose whether the rear window is open or blocked. If the rear window is intended to be kept blocked for an extended period, the detachable light shield can be fixed in the blocked state. Alternatively, if the rear window is intended to be kept open for an extended period, the detachable light shield can be removed. The light shield can be attached or removed using screws, hooks, or an embedded structure.

[0207] Furthermore, the example of the spatial levitation image display device 1000 in Figure 4I is similar. Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation component 101B, there is a possibility that a portion of the image light output from the display device 1 may be reflected on the polarization separation component 101B and go to the transparent plate 100B. Depending on the coating properties of the surface of the transparent plate 100B, there is a possibility that this light may be reflected again on the surface of the transparent plate 100B as stray light and be seen by the user. Therefore, to prevent this stray light, the transparent plate 100B may not be provided on the aforementioned window on the back of the spatial levitation image display device 1000. The aforementioned opening / closing door 1410 may be provided on the window without the transparent plate 100B. To prevent this stray light, the inner surface of the housing of the light-shielding plate of the opening / closing door 1410 preferably has a coating or material with low light reflectivity.

[0208] Figure 4J is a diagram illustrating an example of the structure of a spatial levitation image display device. The difference between the spatial levitation image display device 1000 in Figure 4J and the spatial levitation image display device in Figure 4H is that, instead of a transparent glass or plastic panel 100B, an electrically controlled variable transmittance device 1620 is used in the rear side window. The other structures are the same as those in the spatial levitation image display device in Figure 4H, so repeated descriptions are omitted. An example of the electrically controlled variable transmittance device 1620 is a liquid crystal shutter, etc.

[0209] In other words, the liquid crystal shutter controls the transmission of light by voltage-controlled liquid crystal elements sandwiched between two polarizers. Therefore, if the transmittance of the liquid crystal shutter is increased, the background of the suspended image 3 becomes transparent, allowing the view through the back side window to be seen. Conversely, if the transmittance of the liquid crystal shutter is increased, the background of the suspended image 3 becomes so transparent that the view through the back side window is not visible.

[0210] Furthermore, the LCD shutter can control intermediate color levels, so it can also be set to a state with a transmittance of 50%. For example, the transmittance of the electronically controlled transmittance variable device 1620 can be controlled by the control unit 1110 according to the operation input unit 1107 in FIG3. With this structure, when viewing the scenery through the back side window as the background of the spatial levitation image 3, but the background, i.e., the scenery through the back side window, is too bright, which reduces the recognizability of the spatial levitation image 3, the recognizability of the spatial levitation image 3 can be adjusted by adjusting the transmittance of the electronically controlled transmittance variable device 1620.

[0211] Alternatively, an illuminance sensor can be installed on the back side of the spatial levitation image display device 1000 (the side opposite to the user 230) – for example, near the back side window – to measure the brightness of the space outside the back side window. In this case, the transmittance of the electronically controlled transmittance variable device 1620 can be controlled by the control unit 1110 of FIG3 based on the detection result of the illuminance sensor. In this way, even if the user 230 does not perform operation input through the operation input unit 1107 of FIG3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted accordingly based on the brightness of the space outside the back side window, thus better maintaining the recognizability of the spatial levitation image 3.

[0212] Furthermore, the example described above uses an electrically controlled variable transmittance device 1620, which is an example of a liquid crystal shutter. However, electronic paper can also be used as another example of an electrically controlled variable transmittance device 1620. Even when using electronic paper, the same effect as described above can be achieved. Moreover, the power consumption of electronic paper in maintaining intermediate color levels is very low. Therefore, compared with the case of using a liquid crystal shutter, a low-power spatial levitation image display device can be realized.

[0213] Figure 4K is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 in Figure 4K differs from that in Figure 4G in that it has a transmissive self-emissive image display device 1650 instead of the transparent component 100. The other structures are the same as those in Figure 4G, so repeated descriptions are omitted.

[0214] In the spatial levitation image display device 1000 of Figure 4K, after the image beam passes through the display surface of the transmissive self-emissive image display device 1650, a spatial levitation image 3 is formed outside the spatial levitation image display device 1000. That is, when an image is displayed using the transmissive self-emissive image display device 1650 as a two-dimensional flat panel display, the spatial levitation image 3 can be displayed as a floating image in front of the user's image on the transmissive self-emissive image display device 1650. At this time, the user 230 can simultaneously see two images with different depth positions. The transmissive self-emissive image display device 1650 can be constructed using existing technology such as the transmissive organic EL panel disclosed in Japanese Patent Application Publication No. 2014-216761. In addition, the transmissive self-emissive image display device 1650 is not shown in Figure 3, but it can be connected to other processing units such as the control unit 1110 as a component of the spatial levitation image display device 1000 of Figure 3.

[0215] This allows for the display of both background and character objects on the transmissive self-emissive image display device 1650, followed by the movement of only the character objects to the suspended image 3 in front of the user. This provides the user 230 with a more effective "surprise effect" image experience.

[0216] Furthermore, if the interior of the spatial levitation image display device 1000 is in a light-blocking state, the background of the transmissive self-emissive image display device 1650 is sufficiently dark. Therefore, when the display device 1 does not display an image or the light source of the display device 1 does not emit light, and only the transmissive self-emissive image display device 1650 displays an image, for the user 230, the transmissive self-emissive image display device 1650 appears not to be a transmissive display but to be a typical two-dimensional flat panel display (in the embodiments of the present invention, the spatial levitation image 3 is displayed as a real optical image in a space where there is no screen, so if the light source of the display device 1 is not emitted, the predetermined display position of the spatial levitation image 3 is a space where nothing exists). Therefore, by using the transmissive self-emissive image display device 1650 to display images as if it were a typical two-dimensional flat panel display, and then suddenly displaying characters and objects in the air as spatial levitation images 3, a more effective "surprise effect" image experience can be provided to the user 230.

[0217] Furthermore, the darker the interior of the spatial levitation image display device 1000, the more the transmissive self-emissive image display device 1650 appears to be a two-dimensional flat panel display. Therefore, an absorptive polarizer (not shown) can be provided on one side of the spatial levitation image display device 1000 (the incident surface where the image light reflected from the polarization separation member 101B is incident on the transmissive self-emissive image display device 1650, i.e., the side of the transmissive self-emissive image display device 1650 opposite to the spatial levitation image 3) to transmit the polarization of the image light reflected from the polarization separation member 101B and absorb the polarization that is 90° out of phase with the polarization. In this way, the impact on the image light forming the spatial levitation image 3 is not significant, but the light incident from the outside through the transmissive self-emissive image display device 1650 into the interior of the spatial levitation image display device 1000 can be greatly reduced, making the interior of the spatial levitation image display device 1000 darker, which is preferable.

[0218] Figure 4L is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 in Figure 4L is a variation of the spatial levitation image display device in Figure 4K. The arrangement direction of the structures in the spatial levitation image display device 1000 is different from that in the spatial levitation image display device in Figure 4K, and is similar to that in the spatial levitation image display device in Figure 4F. Regarding the functions and operations of each structure, since the structures are the same as those in the spatial levitation image display device in Figure 4K, repeated descriptions are omitted.

[0219] The spatial levitation image display device in Figure 4L is the same. After the beam of image light passes through the transmissive self-emissive image display device 1650, a spatial levitation image 3 is formed on the side closer to the user 230 than the transmissive self-emissive image display device 1650.

[0220] In both the spatial levitation image display device examples in Figure 4K and Figure 4L, from the user's perspective (230), the spatial levitation image 3 is displayed superimposed on the image of the transmissive self-emissive image display device 1650 (on the side in front of the user). Here, the position of the spatial levitation image 3 differs from the position of the image of the transmissive self-emissive image display device 1650 in the depth direction. Therefore, when the user moves their head (viewpoint), they can perceive the depth of the two images due to parallax. Thus, by displaying two images with different depth positions, a better naked-eye 3D image experience can be provided to the user without the need for stereoscopic glasses.

[0221] Figure 4M is a diagram showing an example of the structure of a spatial levitation image display device. In the spatial levitation image display device 1000 of Figure 4M, a second display device 1680 is provided on the deeper side as viewed from the user, relative to the polarization separation component 101B of the spatial levitation image display device of Figure 4G. The other structures are the same as those of the spatial levitation image display device of Figure 4G, so repeated descriptions are omitted.

[0222] In the structural example shown in Figure 4M, the second display device 1680 is disposed at a depth of the display position of the spatial levitation image 3, with its image display surface facing the spatial levitation image 3. With this structure, from the user 230's perspective, the image of the second display device 1680 and the spatial levitation image 3, both displayed at different depth positions, can be seen overlappingly. That is, it can be considered that the second display device 1680 is configured to display the image in a direction facing the user 230 viewing the spatial levitation image 3. Furthermore, although the second display device 1680 is not shown in Figure 3, it can be considered a component of the spatial levitation image display device 1000 of Figure 3, and connected to other processing units such as the control unit 1110.

[0223] Furthermore, the image light from the second display device 1680 of the spatial levitation image display device 1000 in FIG4M is seen by the user 230 after passing through the polarization separation member 101B. Therefore, in order to better transmit the image light from the second display device 1680 through the polarization separation member 101B, it is preferable that the image light output from the second display device 1680 is light with a polarization direction more suitable for transmission through the polarization separation member 101B. That is, it is preferable that the polarization direction is the same as that of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized light, it is preferable that the image light output from the second display device 1680 is also S-polarized light. Similarly, if the image light output from the display device 1 is P-polarized light, it is preferable that the image light output from the second display device 1680 is also P-polarized light.

[0224] The example of the spatial levitation image display device in Figure 4M also displays a second image at the depth of the spatial levitation image 3, achieving the same effect as the examples of the spatial levitation image display devices in Figures 4K and 4L. However, unlike the examples of the spatial levitation image display devices in Figures 4K and 4L, in the example of the spatial levitation image display device in Figure 4M, the beam of image light used to form the spatial levitation image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-emissive image display device; it can be a liquid crystal display (LCD) as a two-dimensional flat panel display. The second display device 1680 can also be an organic EL display. Therefore, compared to the examples of the spatial levitation image display devices in Figures 4K and 4L, the example of the spatial levitation image display device in Figure 4M allows for the realization of the spatial levitation image display device 1000 at a lower cost.

[0225] Here, depending on the polarization distribution of the image light output from display device 1 and the performance of polarization separation component 101B, there is a possibility that a portion of the image light output from display device 1 may be reflected on polarization separation component 101B and go to second display device 1680. There is a possibility that this light (a portion of the image light) may be reflected again on the surface of second display device 1680 as stray light and be seen by the user.

[0226] Therefore, to prevent stray light, an absorptive polarizer can be provided on the surface of the second display device 1680. In this case, the absorptive polarizer can be an absorptive polarizer that transmits polarized image light output from the second display device 1680 and absorbs polarized light with a polarization 90° out of phase with the image light output from the second display device 1680. Furthermore, when the second display device 1680 is a liquid crystal display (LCD), an absorptive polarizer is also present on the image emission side inside the LCD. However, if the emission surface of the absorptive polarizer on the image emission side inside the LCD has a glass cover (a glass cover on the image display side), it will be impossible to prevent stray light generated by reflection of light from outside the LCD from the glass cover. Therefore, it is necessary to additionally provide the aforementioned absorptive polarizer on the surface of the glass cover.

[0227] Furthermore, when displaying images using the second display device 1680, which is a two-dimensional flat panel display, the spatially suspended image 3 can be displayed as an image on the side of the image on the second display device 1680 closer to the user (the side in front of the user). At this time, the user 230 can simultaneously see two images with different depth positions. By displaying the character using the spatially suspended image 3 and displaying the background on the second display device 1680, the user 230 can be provided with an effect as if they can see the space where the character is located in three dimensions.

[0228] Another effect can be achieved by displaying both the background and the character on the second display device 1680, and then moving only the character and the other objects to the space in front of the user to float the image 3. This can provide the user 230 with a more effective "surprise effect" image experience.

[0229] Next, Figure 4N is a diagram showing an example of the structure of a spatial levitation image display device. The spatial levitation image display device 1000 of Figure 4N is a spatial levitation image display device that adopts the optical system of Figure 2D. Similar to the examples of spatial levitation image display devices that adopt the optical systems of Figures 2A to 2C, the spatial levitation image 3 is imaged in the air using image light that passes through the transparent component 100. In addition, the user's finger 9004 can be used to detect the operation of the spatial levitation image 3 by using the sensing light of the air operation detection sensor 1351, which is disposed deep inside the transparent component 100 from the user's perspective.

[0230] In both the example of the spatial levitation image display device employing the optical systems of Figures 2A-2C and the example of the spatial levitation image display device employing the optical system of Figure 2D, the spatial levitation image 3 is imaged in front of the transparent component 100 (on the side closest to the user). The user's finger operation on the spatial levitation image 3 can be detected using the sensing light of the air operation detection sensor 1351, which is located deep within the transparent component 100 from the user's perspective. Therefore, the optical system in the spatial levitation image display device employing the optical system of Figure 2D is different from that in the spatial levitation image display device where the optical systems of Figures 2A-2C are located deep within the transparent component 100 from the user's perspective.

[0231] However, in terms of ease of use from the user's perspective, the spatial levitation image display device using the optical system of Figure 2D has roughly the same ease of use as the spatial levitation image display device using the optical systems of Figures 2A to 2C.

[0232] Next, Figure 4O is a diagram showing an example of the structure of a spatial levitation image display device. Figure 4O is a diagram that visually shows the structure of the internal optical system in the spatial levitation image display device 1000 of Figure 4N. The spatial levitation image display device 1000 shown in Figure 4O is equipped with an optical system corresponding to the optical system of Figure 2D. The spatial levitation image display device 1000 shown in Figure 4O is horizontally arranged with the side forming the spatial levitation image 3 facing upwards.

[0233] That is, in Figure 40, the transparent component 100 of the spatial levitation image display device 1000 is provided on the top surface of the device. The spatial levitation image 3 is formed above the surface of the transparent component 100 of the spatial levitation image display device 1000. The light of the spatial levitation image 3 travels obliquely upward. When the air operation detection sensor 1351 is provided as shown, it is possible to detect the user 230's finger operation on the spatial levitation image 3.

[0234] Here, the structures of FIG4O and FIG4A are compared to confirm their differences. In FIG4A, the display device 1 and the spatial levitation image 3 are symmetrical about the surface of the polarization separation component 101. In contrast, in FIG4O, the display device 1 and the spatial levitation image 3 are symmetrical about the surface of the retroreflector 5. Furthermore, the retroreflector 2 and the λ / 4 waveplate 21 are present in the structure of FIG4A, but they are absent in FIG4O. Additionally, the absorption polarizer 12 is more preferably present in FIG4A, but the absorption polarizer 12 is not particularly necessary in FIG4O.

[0235] That is, to replace the optical system of Figure 2A in the structure of Figure 4A with the optical system of Figure 2D, thereby replacing it with the structure of Figure 4O, the following operation can be performed: The polarization separation component 101 in the structure of Figure 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 waveplate 21 are removed from the structure of Figure 4A. The absorptive polarizer 12 is optional. By performing this replacement based on the concept, the optical systems of Figures 2A to 2C mounted in the structures of the spatial levitation image display devices of Figures 4A to 4G can be replaced with the optical system of Figure 2D, thereby replacing it with a spatial levitation image display device equipped with the optical system of Figure 2D. At this time, in Figures 4A and 4B, the polarization separation component 101 can be replaced with the retroreflector 5, and in Figures 4C to 4G, the polarization separation component 101B can be replaced with the retroreflector 5.

[0236] In this way, it is possible to realize a spatial levitation image display device by replacing the optical system with the optical system of Figure 2D in the structure of the spatial levitation image display device of Figures 4A to 4G. In these spatial levitation image display devices with the optical system of Figure 2D, it is also possible to realize a spatial levitation image display device with approximately the same ease of use as the spatial levitation image display devices of Figures 4A to 4G.

[0237] <Display Device>

[0238] Next, the display device 1 of this embodiment will be described using the accompanying drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 constituting its light source. In FIG5, the light source device 13 is shown in an unfolded perspective view together with the liquid crystal display panel.

[0239] As shown by arrow 30 in Figure 5, the liquid crystal display panel (image display element 11) receives an illumination beam from the light source device 13, which serves as a backlight device, exhibiting characteristics similar to laser light, including narrow-angle diffusion, strong directivity (linear travel), and a uniform polarization plane in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination beam according to the input image signal. The modulated image light is reflected by the retroreflector 2 and passes through the transparent member 100 to form a spatially suspended real image (see Figure 1).

[0240] Furthermore, Figure 5 includes a liquid crystal display panel 11 constituting the display device 1, and a light direction conversion panel 54 that controls the directional characteristics of the emitted beam from the light source device 13, and may include a narrow-angle diffuser (not shown). Specifically, polarizers are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted according to the intensity of the image signal (see arrow 30 in Figure 5). Thus, the desired image is projected as highly directional (linear) specifically polarized light through the light direction conversion panel 54 onto the retroreflector 2, reflected by the retroreflector 2, and transmitted to the viewer's eye outside the shop (space), forming a spatially suspended image 3. Alternatively, a protective cover 50 may be provided on the surface of the aforementioned light direction conversion panel 54 (see Figures 6 and 7).

[0241] <Example 1 of a display device>

[0242] Figure 6 shows an example of the specific structure of the display device 1. Figure 6 shows a liquid crystal display panel 11 and a light direction conversion panel 54 arranged on the light source device 13 of Figure 5. The light source device 13 is formed of a housing, for example, plastic, as shown in Figure 5, and houses LED elements 201 and a light guide 203 inside. On the end face of the light guide 203, a lens shape is provided to convert the diverging light from each LED element 201 into a substantially parallel beam, as shown in Figure 5. This lens has a shape where the cross-sectional area gradually increases towards the surface opposite the light-receiving part, and it has the function of causing multiple total internal reflections during light propagation inside, thereby gradually reducing the divergence angle. The liquid crystal display panel 11 constituting the display device 1 is mounted on the upper surface of the display device 1. Furthermore, an LED substrate 202 is mounted on one side of the housing of the light source device 13 (the left end face in this example), on which LED (Light Emitting Diode) elements 201, serving as semiconductor light sources, and their control circuitry are mounted. On the outer side of the LED substrate 202, a heat sink can be installed to cool the heat generated in the LED components and control circuit.

[0243] Furthermore, on the frame (not shown) of the liquid crystal display panel mounted on the upper surface of the housing of the light source device 13, the liquid crystal display panel 11 mounted on the frame, and the FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11 are mounted. That is, the liquid crystal display panel 11, as an image display element, and the LED element 201, as a solid-state light source, together modulate the intensity of the transmitted light based on the control signal from the control circuit (image control unit 1160 in FIG. 3) constituting the electronic device to generate a display image. At this time, the generated image light has a narrow diffusion angle and only a specific polarization component, so it is close to the surface-emitting laser image source driven by the image signal, and a novel image display device that has not been obtained before can be obtained. In addition, under the current circumstances, it is technically and safely impossible to obtain a laser beam of the same size as the image obtained by the above-mentioned display device 1 by means of a laser device. Therefore, in this embodiment, for example, a light beam emitted by a conventional light source having LED elements is used to obtain the light that is close to the surface-emitting laser image light.

[0244] Next, the structure of the optical system housed in the housing of the light source device 13 will be described in detail with reference to Figures 6 and 7.

[0245] Since Figures 6 and 7 are cross-sectional views, only one of the multiple LED elements 201 constituting the light source is shown. These elements are transformed into approximately collimated light by the shape of the light-receiving end face 203a of the light guide 203. Therefore, the light-receiving portion of the light guide end face is mounted in a predetermined positional relationship with the LED element.

[0246] Furthermore, the light guide 203 is formed, for example, using a light-transmitting resin such as acrylic resin. The LED light-receiving surface at the end of the light guide 203 has, for example, an outer peripheral surface in the shape of a convex cone obtained by rotating a parabolic section. Its top has a recess, and a convex portion (i.e., a convex lens surface) is formed in the center of this recess. Furthermore, a convex lens surface protruding outwards (or a concave lens surface recessed inwards) is formed in the center of its flat portion (not shown). Additionally, the light-receiving portion of the light guide on which the LED element 201 is mounted has a parabolic shape forming a conical outer peripheral surface, and is set within an angle range that allows light emitted from the LED element in the peripheral direction to be totally internally reflected, or a reflective surface is formed.

[0247] On the other hand, LED elements 201 are respectively disposed at predetermined positions on the surface of its circuit board, i.e., LED substrate 202. The LED substrate 202 is disposed and fixed relative to the LED collimator (light-receiving end face 203a) such that the LED elements 201 on its surface are respectively located at the center of the aforementioned recess.

[0248] According to this structure, the shape of the light-receiving end face 203a of the light guide 203 can be used to make the light emitted from the LED element 201 become approximately parallel light and output, thereby improving the utilization efficiency of the generated light.

[0249] As described above, the light source device 13 is constructed by mounting a light source unit consisting of a plurality of LED elements 201 arranged as light sources at the light-receiving end face 203a, which is provided on the end face of the light guide 203. For the diverging light beam from the LED elements 201, the light-receiving end face 203a of the light guide end face makes it approximately parallel light, and guides the light inside the light guide 203 (in a direction parallel to the paper surface) as shown by the arrow. The beam direction conversion unit 204 directs the light beam to be emitted towards the liquid crystal display panel 11 (in a direction perpendicular to the paper surface) which is arranged approximately parallel to the light guide 203. By optimizing the distribution (density) of the beam direction conversion unit 204 with the shape of the inside or surface of the light guide, the uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled.

[0250] The aforementioned beam direction conversion unit 204, by utilizing the shape of the light guide surface or by providing portions with different refractive indices within the light guide, directs the beam propagating within the light guide towards the liquid crystal display panel 11 (in a direction perpendicular to the paper surface) arranged approximately parallel to the light guide 203. At this time, for the liquid crystal display panel 11, with the viewpoint positioned directly opposite the center of the screen and the diagonal dimension of the screen aligned with the center, the brightness of the center and periphery of the screen is compared. A relative brightness ratio of 20% or higher is sufficient for practical use, and a ratio exceeding 30% indicates even better performance.

[0251] Additionally, Figure 6 is a cross-sectional view illustrating the structure and function of the light source in this embodiment, which includes a light guide 203 and an LED element 201, for performing polarization transformation. In Figure 6, the light source device 13 includes, for example, a light guide 203 made of plastic or the like, on which a beam direction transformation unit 204 is provided on or inside; an LED element 201 serving as a light source; a reflector 205; a phase difference plate 206; a cylindrical lens; etc.; and a liquid crystal display panel 11 with polarizers on the light source incident surface and the image light exit surface is mounted on its upper surface.

[0252] Furthermore, a thin film or sheet-like reflective polarizer 49 is provided on the light incident surface (lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, so that one polarization (e.g., P-light) 212 of the natural light beam 210 emitted from the LED element 201 is selectively reflected. The reflected light is reflected again on a reflective sheet 205 provided on one surface (lower surface in the figure) of the light guide 203 and goes to the liquid crystal display panel 11. Therefore, a phase retardation plate (λ / 4 waveplate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizer 49, so that the light is reflected on the reflective sheet 205 and passes through the phase retardation plate twice, thereby changing the reflected beam from P-polarization to S-polarization, improving the utilization efficiency of the light source light as image light. The light intensity is modulated by the image signal of the liquid crystal display panel 11 into the retroreflector 2. After reflection on the retroreflector 2, a spatial levitation image of a real image can be obtained.

[0253] Figure 7 is a cross-sectional view, similar to Figure 6, illustrating the structure and function of the light source in this embodiment, which performs polarization transformation in the light source device 13 including the light guide 203 and the LED element 201. The light source device 13 also includes, for example, a light guide 203 made of plastic or the like, with a beam direction transformation unit 204 provided on its surface or internally, an LED element 201 serving as the light source, a reflector 205, a phase retardation plate 206, a cylindrical lens, etc. A liquid crystal display panel 11, having polarizers on both the light source incident surface and the image light exit surface, is mounted as an image display element on the upper surface of the light source device 13.

[0254] A thin film or sheet-like reflective polarizer 49 is provided on the light incident surface (lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, so that one polarization (e.g., S-polarization) 211 of the natural light beam 210 emitted from the LED element 201 is selectively reflected. That is, in the example of FIG7, the selective reflection characteristics of the reflective polarizer 49 are different from those in FIG7. The reflected light is reflected on a reflective sheet 205 provided on one surface (lower surface in the figure) of the light guide 203 and goes back to the liquid crystal display panel 11. A phase retardation plate (λ / 4 waveplate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizer 49, so that the light is reflected on the reflective sheet 205 and passes through the phase retardation plate twice, thereby changing the reflected beam from S-polarization to P-polarization and improving the utilization efficiency of the light source light as image light. The light intensity is modulated by the image beam (arrow 214 in FIG7) by the liquid crystal display panel 11 according to the image signal and then incident on the retroreflector 2. A spatially suspended image of a real image can be obtained after reflection on the retroreflector plate 2.

[0255] In the light source devices shown in Figures 6 and 7, besides the function of the polarizers on the light incident surface of the corresponding liquid crystal display panel 11, the theoretically achievable contrast ratio is the product of the reciprocal of the orthogonal transmittance of the reflective polarizer and the reciprocal of the orthogonal transmittance obtained from the two polarizers attached to the liquid crystal display panel, since a polarization component is reflected by the reflective polarizer. Therefore, a high contrast ratio can be obtained. In fact, experiments have confirmed that the contrast ratio of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those obtained with self-emissive organic EL displays can be obtained.

[0256] <Example 2 of a display device>

[0257] Figure 8 shows another example of the specific structure of the display device 1. The light source device 13 is constructed by housing LEDs, collimators, composite diffusers, light guides, etc., in a housing such as plastic, and a liquid crystal display panel 11 is mounted on its upper surface. In addition, an LED substrate is mounted on one side of the housing of the light source device 13, on which LED (Light Emitting Diode) elements 14a and 14b, which serve as semiconductor light sources, and their control circuits are mounted. A heat sink 103, which is used to cool the heat generated in the LED elements and the control circuits, is mounted on the outer side of the LED substrate.

[0258] Furthermore, a liquid crystal display panel 11 mounted on the upper surface of the housing, and an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11 are provided on the liquid crystal display panel frame. That is, the liquid crystal display panel 11, as a liquid crystal display element, and the LED elements 14a and 14b, as solid-state light sources, together modulate the intensity of transmitted light based on control signals from the control circuit (not shown here) constituting the electronic device to generate a display image.

[0259] <Example 3 of a display device>

[0260] Next, another example of the specific structure of the display device 1 (Example 3 of the display device) will be described using FIG. 9. The light source device of this display device 1 uses a collimator 18 to convert the diverging beam of light (a mixture of P-polarized and S-polarized light) from the LED into a substantially parallel beam, and uses the reflective surface of the reflective light guide 304 to reflect it towards the liquid crystal display panel 11. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 allows light of a specific polarization (e.g., P-polarized light) to pass through, causing the transmitted polarized light to enter the liquid crystal display panel 11. Here, other polarizations (e.g., S-polarized light) are reflected by the reflective polarizer 49 and return to the reflective light guide 304.

[0261] The reflective polarizer 49 is tilted relative to the liquid crystal display panel 11 in a manner that is not perpendicular to the principal ray of light from the reflective surface of the reflective light guide 304. The principal ray of light reflected from the reflective polarizer 49 is incident on the transmissive surface of the reflective light guide 304. Light incident on the transmissive surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304 and is reflected on the reflective plate 271 by the λ / 4 waveplate 270, which serves as a phase retardation plate. The light reflected on the reflective plate 271 passes through the λ / 4 waveplate 270 again and passes through the transmissive surface of the reflective light guide 304. The light passing through the transmissive surface of the reflective light guide 304 is then incident on the reflective polarizer 49 again.

[0262] At this point, the light that is incident on the reflective polarizer 49 again passes through the λ / 4 waveplate 270 twice, so its polarization is transformed into a polarization that can pass through the reflective polarizer 49 (e.g., P-polarization). Thus, the polarization-transformed light passes through the reflective polarizer 49 and enters the liquid crystal display panel 11. Alternatively, the polarization design in the polarization transformation can be reversed compared to the above description (switching S-polarization and P-polarization).

[0263] As a result, the light from the LEDs is uniformly polarized (e.g., P-polarized) and incident on the liquid crystal display panel 11, and the brightness is modulated accordingly according to the image signal to display an image on the panel surface. Similar to the example above, multiple LEDs constituting the light source are shown (but only one is shown in Figure 9 because it is a longitudinal cross-sectional view), which are mounted in designated positions relative to the collimator 18.

[0264] Furthermore, the collimator 18 is formed, for example, using a light-transmitting resin such as acrylic resin or glass. The collimator 18 may have an outer peripheral surface with a convex conical shape obtained by rotating a parabolic section. Additionally, at the center of the top portion of the collimator 18 (the side opposite to the LED substrate 102), a recess with a protrusion (i.e., a convex lens surface) may be formed. Furthermore, at the center of the planar portion of the collimator 18 (the side opposite to the aforementioned top), a convex lens surface protruding outwards (or it may be a concave lens surface recessed inwards) may be formed. Additionally, the parabolic surface forming the conical outer peripheral surface of the collimator 18 is set within an angle range that allows light emitted from the LED in the peripheral direction to undergo total internal reflection within it, or it forms a reflective surface.

[0265] In addition, the LEDs are respectively disposed at predetermined positions on the surface of its circuit board, i.e., the LED substrate 102. The LED substrate 102 is configured and fixed relative to the collimator 18 such that the LEDs on its surface are respectively located at the center of the top of the convex conical shape (or the recess if the top has a recess).

[0266] According to this structure, under the action of the collimator 18, the light emitted from the LED, especially the light emitted from its central portion, is converged into parallel light by the convex lens surface forming the shape of the collimator 18. Furthermore, light emitted from other portions towards the periphery is reflected by the parabolic surface of the conical outer peripheral surface of the collimator 18, and similarly converged into parallel light. In other words, by using the collimator 18, which has a convex lens in its central portion and a parabolic surface in its peripheral portion, almost all the light generated by the LED can be output as parallel light, thereby improving the utilization efficiency of the generated light.

[0267] Furthermore, the light, which has been converted into approximately parallel light by the collimator 18 as shown in Figure 9, is reflected on the reflective light guide 304. In this light, light with a specific polarization is transmitted through the reflective polarizer 49, and light with another polarization, reflected by the reflective polarizer 49, again passes through the light guide 304. This light is reflected by a reflective plate 271 located opposite the reflective light guide 304 to the liquid crystal display panel 11. At this time, the light undergoes polarization transformation twice by passing through the λ / 4 waveplate 270, which acts as a phase difference plate. The light reflected from the reflective plate 271 passes through the light guide 304 again and is incident on the reflective polarizer 49 located on the opposite side. Because the incident light has been polarized, it can pass through the reflective polarizer 49, resulting in a uniform polarization direction incident on the liquid crystal display panel 11. As a result, all the light from the light source can be utilized, thus achieving a 2-fold increase in the geometrical optics efficiency. Furthermore, the polarization degree (extinction ratio) of the reflective polarizer is multiplied by the overall extinction ratio of the system, thus significantly improving the overall contrast of the display device by using the light source device of this embodiment. Additionally, by adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflective plate 271, the reflection diffusion angle of light on each reflective surface can be adjusted. To improve the uniformity of light incident on the liquid crystal display panel 11, the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflective plate 271 can be adjusted according to each design.

[0268] Furthermore, for polarized light incident perpendicular to the λ / 4 waveplate 270 in Figure 9, the phase difference does not need to be λ / 4. In the structure of Figure 9, any phase retardation plate that allows polarized light to pass through twice, thus changing the phase by 90° (λ / 2), is sufficient. The thickness of the phase retardation plate can be adjusted accordingly based on the incident angle distribution of the polarized light.

[0269] <Example 4 of a display device>

[0270] Furthermore, another example of the structure of an optical system such as a light source device in a display device (Example 4 of the display device) will be explained using FIG. 10. This is a structural example in the light source device of Example 3 of the display device, in which a diffuser is used instead of a reflective light guide 304. Specifically, on the light emitting side of the collimator 18, two optical sheets (optical sheet 207A and optical sheet 207B) are used to transform the diffusion characteristics in the vertical and horizontal directions (front and back directions in the figure, not shown), so that light from the collimator 18 is incident between the two optical sheets (diffusers).

[0271] Alternatively, the aforementioned optical sheet can be replaced with a single sheet instead of a two-sheet structure. In the case of a single-sheet structure, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes of the front and back sides of the single optical sheet. Alternatively, multiple diffuser sheets can be used to share the effect. Here, in the example of Figure 10, for the reflection diffusion characteristics determined by the front and back shapes of optical sheets 207A and 207B, the number of LEDs, the divergence angle of the LED substrate (optical element) 102, and the optical specifications of the collimator 18 can be used as design parameters for optimization, thereby making the surface density of the light beam emitted from the liquid crystal display panel 11 more uniform. That is, the diffusion characteristics are adjusted by using the surface shapes of multiple diffuser sheets instead of a light guide.

[0272] In the example of Figure 10, the polarization conversion is performed using the same method as in Example 3 of the display device described above. That is, in the example of Figure 10, the reflective polarizer 49 can be configured to reflect S-polarized light (transmit P-polarized light). In this case, P-polarized light emitted from the light source, i.e., the LED, is transmitted, and the transmitted light is incident on the liquid crystal display panel 11. S-polarized light emitted from the light source, i.e., the LED, is reflected, and the reflected light passes through the phase retardation plate 270 shown in Figure 10. The light passing through the phase retardation plate 270 is reflected on the reflector plate 271. The light reflected by the reflector plate 271 passes through the phase retardation plate 270 again and is converted into P-polarized light. The polarization-converted light passes through the reflective polarizer 49 and is incident on the liquid crystal display panel 11.

[0273] Furthermore, for polarized light incident perpendicular to the λ / 4 waveplate 270 in Figure 10, the phase difference does not need to be λ / 4. In the structure of Figure 10, any phase retardation plate that allows polarized light to pass twice, thus changing the phase by 90° (λ / 2), is sufficient. The thickness of the phase retardation plate can be adjusted accordingly based on the incident angle distribution of the polarized light. Similarly, in Figure 10, the polarization design in the polarization transformation can be reversed compared to the above description (switching S-polarization and P-polarization).

[0274] Regarding the emitted light from the liquid crystal display panel 11, in conventional TV devices, it exhibits the same diffusion characteristics in both the horizontal direction (represented by the X-axis in FIG12(a)) and the vertical direction (represented by the Y-axis in FIG12(b)). In contrast, the diffusion characteristics of the emitted light beam from the liquid crystal display panel of this embodiment are shown in Example 1 of FIG12. The viewing angle at 50% of the brightness when viewed from the front (angle 0 degrees) is 13 degrees, approximately 1 / 5 of the 62 degrees of a conventional TV device. Similarly, for the vertical viewing angle, the reflection angle of the reflective light guide and the area of ​​the reflective surface are optimized to make it uneven vertically, suppressing the upper viewing angle to approximately 1 / 3 of the lower viewing angle. As a result, compared to existing LCD TVs, the amount of image light heading towards the viewing direction is significantly increased, with brightness more than 50 times greater.

[0275] Furthermore, if the viewing angle characteristics shown in Example 2 of Figure 12 are adopted, the viewing angle when the brightness is 50% of that for frontal viewing (0 degrees) is 5 degrees, which is 1 / 12 of the 62 degrees of a typical TV device. Similarly, for the vertical viewing angle, the reflection angle of the reflective light guide and the area of ​​the reflective surface are optimized to make it uniform vertically and to suppress the viewing angle to about 1 / 12 of that of a typical TV device. As a result, compared with existing LCD TVs, the amount of image light heading towards the viewing direction is greatly increased, and the brightness is more than 100 times greater.

[0276] By narrowing the viewing angle as described above, the amount of light beam heading in the viewing direction can be concentrated, thus significantly improving light utilization efficiency. As a result, even when using a typical LCD panel for TV applications, a significant increase in brightness can be achieved with the same power consumption by controlling the light diffusion characteristics of the light source device, enabling an image display device compatible with information display systems facing bright outdoor environments.

[0277] When using a large LCD panel, the light from the periphery of the screen is directed inwards towards the viewer when the viewer is facing the center of the screen, thereby improving the overall brightness of the screen. Figure 11 shows the convergence angles of the long and short sides of the panel with the distance L between the viewer and the panel and the panel size (16:10 aspect ratio) as parameters. In portrait mode, the convergence angle can be set accordingly for the short side. For example, when using a 22″ panel in portrait mode at a viewing distance of 0.8m, setting the convergence angle to 10 degrees allows the image light from the four corners of the screen to effectively reach the viewer.

[0278] Similarly, when viewing a 15″ panel in portrait mode at a viewing distance of 0.8m, setting the convergence angle to 7 degrees allows image light from the four corners of the screen to effectively reach the viewer. As described above, by directing image light from the periphery of the screen to the viewer in the most suitable position for viewing the center of the screen, based on the size of the LCD panel and whether it is used in portrait or landscape mode, the overall brightness of the screen can be improved.

[0279] As a basic structure, as shown in Figure 9 above, a light source device is used to direct a beam of light with narrow-angle pointing characteristics onto the liquid crystal display panel 11. The brightness is modulated accordingly according to the image signal, so that the image information displayed on the screen of the liquid crystal display panel 11 is reflected on the retroreflector to obtain a spatial levitation image, which is then displayed outdoors or indoors via the transparent component 100.

[0280] Using the display device and light source device of one embodiment of the present invention described above, a spatial levitation image display device with higher light utilization efficiency can be realized.

[0281] <Examples of image display processing in spatial levitation image display devices>

[0282] Next, an example of the technical problem to be solved by the image processing in this embodiment will be described using FIG13A. In the spatial levitation image display device 1000, from the user's perspective, the deep side of the spatial levitation image 3 is located inside the housing of the spatial levitation image display device 1000, and in sufficiently dark conditions, the user sees the background of the spatial levitation image 3 as black.

[0283] Here, FIG13A is used to illustrate an example of displaying the character "Panda" 1525 in the spatial levitation image 3. First, the image control unit 1160 of FIG3 distinguishes between the pixel area depicting the character "Panda" 1525 and the transparent information area 1520 serving as the background image for the image shown in FIG13A (1) – which includes the pixel area depicting the image of the character "Panda" 1525 and the transparent information area 1520 serving as the background image.

[0284] Regarding the method for distinguishing and recognizing character images from background images, for example, the image processing of the image control unit 1160 can be configured to process the background image layer and the character image layer located in front of the background image layer as different layers, and to distinguish and recognize the character image and the background image based on the overlap relationship when these layers are composited.

[0285] Here, the image control unit 1160 identifies the black pixels of the pixels depicting the character image (object) as different from the transparent information pixels. However, the pixel brightness of both the black pixels depicting the object and the transparent information pixels is 0. In this case, when displaying the spatial levitation image 3, there is no brightness difference between the black pixels in the image depicting the character "panda" 1525 and the pixels of the transparent information area 1520, which serves as the background image. Therefore, in the spatial levitation image 3, as shown in (2) of FIG13A, neither the black pixels in the image depicting the character "panda" 1525 nor the pixels of the transparent information area 1520 have brightness, and are perceived by the user as optically identical black spaces. That is, the black parts in the image depicting the object, i.e., the character "panda" 1525, blend into the background, and only the non-black parts of the character "panda" 1525 can be recognized as images floating in the display area of ​​the spatial levitation image 3.

[0286] An example of image processing in this embodiment will be described using FIG13B. FIG13B is a diagram illustrating an example of image processing that can better solve the technical problem of the black image area of ​​the object described in FIG13A blending into the background. In (1) and (2) of FIG13B, the display state of the spatial levitation image 3 is shown on the upper side, and the input and output characteristics of the image processing of the object image are shown on the lower side. In addition, the image of the object (character "panda" 1525) and / or its corresponding data can be read from the storage unit 1170 or the memory 1109 of FIG3. Alternatively, it can be input from the image signal input unit 1131. Alternatively, it can be obtained via the communication unit 1132.

[0287] Here, in the state shown in Figure 13B (1), the input / output characteristics of the image processing of the object's image are linear and not specially adjusted. In this case, it is the same display state as in Figure 13A (2), where the black image area of ​​the object blends into the background. In contrast, in Figure 13B (2), the image control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the object (character "Panda" 1525) to the input / output characteristics shown below.

[0288] That is, the image control unit 1160 performs image processing on the image of the object (character "Panda" 1525) with the following input-output characteristics, wherein for the pixels of the input image, the brightness value of the pixels in the low-brightness area is increased to produce an output pixel. After the image processing with these input-output characteristics is performed on the image of the object (character "Panda" 1525), the image including the image of the object (character "Panda" 1525) is input to the display device 1 for display. Thus, as shown in the upper part of (2) of FIG13B, the display state of the spatial floating image 3 increases the brightness of the black pixel area in the image depicting the character "Panda" 1525. As a result, the black areas in the area depicting the character "Panda" 1525 are not blended into the black background, allowing the user to distinguish them and display the object better.

[0289] That is, by using the image processing of (2) in FIG13B, the area displaying the image of the object, namely the character "panda" 1525, can be distinguished from the black background inside the housing of the spatial levitation image display device 1000 seen through the window, thus improving the recognizability of the object. As a result, even if the object is an object whose pixels contain pixels with a brightness value of 0 before the above image processing (i.e., when reading the image of the object and / or its corresponding data from the storage unit 1170 or memory 1109 of FIG3, or when inputting the image of the object from the image signal input unit 1131, or when obtaining the data of the object via the communication unit 1132, etc.), the image processing of the input-output characteristics performed by the image control unit 1160 can transform it into an object whose brightness value of pixels in the low brightness area is increased, and then display it on the display device 1, and transform it into a spatial levitation image 3 by the optical system of the spatial levitation image display device 1000.

[0290] That is, the pixels of the object after image processing that constitutes the input and output characteristics do not contain pixels with a brightness value of 0. After being transformed into such a state, it is displayed on the display device 1 and transformed into a spatial levitation image 3 by the optical system of the spatial levitation image display device 1000.

[0291] In addition, in the image processing of FIG13B (2), as a method of performing image processing of the input-output characteristics of FIG13B (2) only on the area of ​​the image of the object (character "panda" 1525), for example, the image processing of the image control unit 1160 can process the background image layer and the character image layer located in front of the background image layer as different layers, perform image processing of the input-output characteristics of FIG13B (2) on the character image layer, and not perform the image processing on the background image layer.

[0292] These layers are then composited, and thus, as shown in (2) of Figure 13B, only the character image is subjected to image processing that brightens the low-brightness areas of the input image. Alternatively, as another method, the image processing of the input-output characteristics of Figure 13B (2) can be performed only on the area of ​​the character image after compositing the character image layer with the background image layer.

[0293] Furthermore, in image processing that brightens low-brightness areas in the input-output characteristics of an input image, the input-output image characteristics used are not limited to the example of (2) in Figure 13B. Any image processing that can brighten low-brightness areas is acceptable, such as so-called brightness adjustment. Alternatively, image processing as disclosed in International Publication No. 2014 / 162533 can be performed, which improves discriminability by controlling the gain used to change the weights of the Retinex processing.

[0294] According to the image processing of Figure 13B (2) described above, for the black areas in the area of ​​the image depicting characters and objects, it is possible to prevent them from blending into the black background, so that the user can recognize them and achieve better display.

[0295] Furthermore, in the examples of Figures 13A and 13B, taking a spatial levitation image display device with a black background (such as the spatial levitation image display device 1000 in Figures 4A to 4G, and the spatial levitation image display device 1000 in Figures 4I and 4J with the rear side window blocked from light) as an example, the technical problems and better image processing are explained. However, this image processing is also effective in devices other than these spatial levitation image display devices.

[0296] Specifically, in the spatial levitation image display device 1000 of Figure 4H and the spatial levitation image display device 1000 in Figures 4I and 4J with the rear side window not blocked from light, the background of the spatial levitation image 3 is not black, but rather the scenery behind the spatial levitation image display device 1000 through the window. In this case, the technical problem illustrated in Figures 13A and 13B also exists.

[0297] That is, the black parts in the image depicting the object, namely the character "Panda" 1525, blend into the scenery behind the spatial levitation image display device 1000 through the window. Similarly, by using the image processing of (2) in FIG13B, the black parts in the image depicting the object, namely the character "Panda" 1525, can be distinguishably identified from the scenery behind the spatial levitation image display device 1000 through the window, thus improving the recognizability of the object.

[0298] That is, by using the image processing of (2) in Figure 13B, the area showing the image of the object, i.e. the character "panda" 1525, can be distinguished from the scenery behind the space-suspended image display device 1000 through the window, and the object, i.e. the character "panda" 1525, can be better identified as being in front of the aforementioned scenery, thus improving the recognizability of the object.

[0299] Furthermore, in the spatial levitation image display device 1000 of Figures 4K, 4L, and 4M, as described above, when other images (such as images from the transmissive self-emissive image display device 1650 or the second display device 1680) are displayed at a depth position different from that of the spatial levitation image 3, the background of the spatial levitation image 3 is not black but rather the other image. In this case, the technical problem illustrated in Figures 13A and 13B also exists.

[0300] That is, the black parts in the image depicting the object, i.e., the character "panda" 1525, are integrated with the other images shown at a depth position different from the spatial levitation image 3. Similarly, in this case, by using the image processing of (2) in FIG13B, the black parts in the image depicting the object, i.e., the character "panda" 1525, can be distinguished from the other images, and the recognizability of the object is improved.

[0301] That is, by using the image processing of (2) in Figure 13B, the area of ​​the image showing the object, i.e. the character "panda" 1525, can be distinguished from the other images mentioned above, and the object, i.e. the character "panda" 1525, can be better identified as being in front of the other images mentioned above, thus improving the recognizability of the object.

[0302] An example of the image display processing in this embodiment will be described using FIG13C. FIG13C is an example of image display in this embodiment, showing a simultaneous display of the spatially suspended image 3 and other images, namely the second image 2050. The second image 2050 may correspond to the display image of the transmissive self-emissive image display device 1650 of FIG4K or FIG4L. Alternatively, the second image 2050 may also correspond to the display image of the second display device 1680 of FIG4M.

[0303] That is, the image shown in Figure 13C is a specific example of the image display example of the spatial levitation image display device 1000 of Figures 4K, 4L, and 4M. In this example, a bear character is shown in the spatial levitation image 3. The area other than the bear character in the spatial levitation image 3 is black, as the spatial levitation image is transparent. In addition, the second image 2050 is a background image depicting a plain, mountains, and the sun.

[0304] Here, in Figure 13C, the spatial levitation image 3 and the second image 2050 are displayed at different depth positions. The user 230 views the two images, spatial levitation image 3 and the second image 2050, in the direction of the line of sight of arrow 2040, thus allowing the user 230 to view the images in an overlapping state. Specifically, the bear character in spatial levitation image 3 can be seen overlapping in front of the background of plains, mountains, and sun depicted in the second image 2050.

[0305] Here, the spatially suspended image 3 is projected as a real image in the air, so when the user 230 slightly moves their viewpoint, they can discern the distance between the spatially suspended image 3 and the second image 2050 due to parallax. Therefore, the user 230 can view the two images in an overlapping state, while simultaneously experiencing a stronger sense of spatial suspension from the spatially suspended image 3.

[0306] An example of the image display processing in this embodiment will be described using FIG13D. FIG13D (1) is an example of the image display in this embodiment of FIG13C, showing the spatial levitation image 3 viewed from the line of sight of user 230. Here, the bear character is displayed in the spatial levitation image 3. The area outside the bear character in the spatial levitation image 3 is black, as the spatial levitation image is transparent.

[0307] Figure 13D (2) is an example of the image display of this embodiment of Figure 13C, showing the second image 2050 viewed from the line of sight of user 230. In this example, the second image 2050 is a background image depicting a plain, mountains, and the sun.

[0308] Figure 13D (3) is a diagram showing an example of the image display of this embodiment of Figure 13C, in which the second image 2050 and the spatial levitation image 3 are seen overlapping in the line of sight of the user 230. Specifically, the bear character of the spatial levitation image 3 can be seen overlapping in front of the background of the plains, mountains and sun depicted in the second image 2050.

[0309] Here, when both the spatial levitation image 3 and the second image 2050 are displayed simultaneously, it is preferable to maintain a balance in the brightness of both images to better ensure the recognizability of the spatial levitation image 3. If the second image 2050 is too bright compared to the brightness of the spatial levitation image 3, the displayed image of the spatial levitation image 3 will become transparent, allowing the background, i.e., the second image 2050, to be seen clearly and transparently.

[0310] Therefore, the output of the light source of the spatial levitation image 3 and the display image brightness of the display device 1, the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device can be set such that the brightness per unit area of ​​the spatial levitation image 3 at the display position of the spatial levitation image 3 is greater than the brightness per unit area of ​​the image light from the second image 2050 to the display position of the spatial levitation image 3.

[0311] Furthermore, as long as this condition is met when both the spatial floating image 3 and the second image 2050 are displayed simultaneously, the brightness of the second image 2050 can be reduced by decreasing the output of the light source of the display device displaying the second image 2050 and / or the brightness of the displayed image on the display device. This control can be achieved by controlling the display device 1 and the display device displaying the second image 2050 (the transmissive self-emissive image display device 1650 of FIG. 4K or FIG. 4L, or the second display device 1680 of FIG. 4M) through the control unit 1110 of FIG. 3.

[0312] Furthermore, during the switching from the first display mode to the second display mode, if it is necessary to reduce the brightness of the second image 2050, the brightness can be reduced uniformly across the entire screen of the second image 2050. Alternatively, instead of uniformly reducing the brightness across the entire screen of the second image 2050, the portion of the spatial levitation image 3 where the object to be displayed is made to have the highest brightness reduction effect, with the brightness reduction effect gradually reduced around its perimeter. This is because reducing the brightness of the second image 2050 only for the overlapping portion of the spatial levitation image 3 within the second image 2050 is sufficient to ensure the recognizability of the spatial levitation image 3.

[0313] Here, the spatial levitation image 3 and the second image 2050 are displayed at different depth positions, so when the user 230 slightly changes their viewpoint, the overlapping position of the spatial levitation image 3 on the second image 2050 will change due to parallax. Therefore, in the switching from the first display mode to the second display mode, when the brightness of the entire screen of the second image 2050 is reduced unevenly, it is not preferable to sharply reduce the brightness based on the outline of the object displayed in the spatial levitation image 3. Instead, it is preferable to gradually change the brightness reduction effect according to the position as described above, and perform a gradual processing of the brightness reduction effect.

[0314] Furthermore, if the position of the object displayed in the spatial levitation image 3 is approximately the center of the spatial levitation image 3, then in the spatial levitation image display device 1000, the position with the highest brightness reduction effect in the gradient processing of the brightness reduction effect can be set to the center of the spatial levitation image 3.

[0315] Based on the image display processing of this embodiment described above, user 230 is able to better view the spatial levitation image 3 and the second image 2050.

[0316] Alternatively, it can be controlled so that the second image 2050 is not displayed when the spatial levitation image 3 is displayed. When the second image 2050 is not displayed, the spatial levitation image 3 is more recognizable, so it is suitable for applications such as the spatial levitation image display device 1000 where the user must reliably see the spatial levitation image 3 when it is displayed.

[0317] <Example 2>

[0318] As Embodiment 2 of the present invention, another structural example of a spatial levitation image display device will be described. Furthermore, the optical system incorporated in the spatial levitation image display device described in Embodiment 1 is changed to the optical system shown in FIG. 14(1) or FIG. 14(2). The differences from Embodiment 1 will be described in this embodiment; repeated descriptions of structures identical to those in Embodiment 1 will be omitted. In the following description of this embodiment, it will be specified that polarization (polarized light) and another polarization (polarized light) are polarizations (polarized light) with a phase difference of 90°.

[0319] Figure 14(1) is an example of the optical system and optical path of this embodiment. The optical system shown in Figure 14(1) is closer to the polarization separation component 101B in the optical system of Figure 2C, making the overall optical system more compact. For structures in Figure 14(1) labeled with the same reference numerals as those in Figure 2C, repeated detailed descriptions are omitted.

[0320] In Figure 14(1), similar to Figure 2C, image light of a predetermined polarization (P-polarized in the figure) emitted from the display device 1 travels vertically from the image display surface of the display device 1. Here, the polarization separation component 101B, similar to Figure 2C, selectively transmits the predetermined polarized light (P-polarized in the figure) emitted from the display device 1 and reflects the other polarized light (S-polarized in the figure).

[0321] Thus, the image light of a predetermined polarization (P-polarized in the figure) traveling vertically from the image display surface of the display device 1 passes through the polarization separating member 101B and reaches the retroreflector 2 to which the λ / 4 waveplate 21 is attached. The image light, retroreflected on the retroreflector 2 and traveling again towards the polarization separating member 101B, passes through the λ / 4 waveplate 21 twice, thereby changing from the predetermined polarization (P-polarized in the figure) emitted from the display device 1 to another polarization (S-polarized in the figure). The image light traveling again towards the polarization separating member 101B is now of another polarization (S-polarized in the figure), and is therefore reflected on the polarization separating member 101B towards the position where the user should be located. The direction of travel of the image reflected on the polarization separating member 101B is determined based on the configuration angle of the polarization separating member 101B.

[0322] In the example of (1) in Figure 14, the image light traveling toward the polarization separation component 101B is reflected at a right angle onto the polarization separation component 101B and travels as shown in the figure. The image light reflected onto the polarization separation component 101B forms a spatial levitation image 3A. The spatial levitation image 3A can be viewed well by the user from the direction of arrow A.

[0323] Here, regarding the retroreflective characteristics of the retroreflector 2, there exists a relationship where the optical path length of the image light emitted from the display device 1 to the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to the formation position of the spatially suspended image 3A. The formation position of the spatially suspended image 3A in the direction of travel of the image light reflected from the polarization separation component 101B is determined by this relationship.

[0324] In the example of Figure 14(1), the display device 1, the polarization separation component 101B, and the retroreflector 2 are arranged closer together than in the example of Figure 2C. This allows for a more compact overall optical system configuration. However, the amount by which the spatial levitation image 3A is lifted from the optical system of Figure 14(1) is not significant. For example, as an indicator of the amount by which the spatial levitation image 3A is lifted from the optical system, the distance shown in the figure is L1 (in the example of Figure 14(1)) from the position where the light from the center of the image light is reflected on the polarization separation component 101B to the position where the image light forms the spatial levitation image 3A.

[0325] Furthermore, regarding the polarization design in the optical system of Figure 14 (1), the characteristics of P-polarization and S-polarization can be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 can be made S-polarized, and the characteristics of P-polarization and S-polarization can be interchanged regarding the reflection characteristics of the polarization separation component 101B. In this case, the P-polarization and S-polarization shown in the figure become opposite, but the optical design of the optical path, etc., can be implemented exactly the same.

[0326] Next, another example of the optical system and optical path of this embodiment is shown in Figure 14(2). The optical system of Figure 14(2) is modified to achieve the same compactness as the optical system of Figure 14(1) while increasing the amount of space-suspended image floating from the optical system. For the structures in Figure 14(2) labeled with the same reference numerals as those in Figure 14(1), repeated detailed descriptions are omitted.

[0327] In Figure 14(2), similar to Figure 14(1), image light of a predetermined polarization (P-polarized in the figure) emitted from the display device 1 travels vertically from the image display surface of the display device 1. Here, the polarization characteristics of the polarization separation member 101B differ by 90 degrees from those in Figure 14(1). The image light of the predetermined polarization (P-polarized in the figure) traveling vertically from the image display surface of the display device 1 passes through the polarization separation member 101B.

[0328] Here, unlike (1) in Figure 14, instead of a retroreflector 2 with a λ / 4 waveplate 21 attached, a specular reflector 4 with a λ / 4 waveplate 21B attached is arranged in front of the image light passing through the polarization separation component 101B. Here, the reflection on the specular reflector 4 is specular reflection (also known as orthographic reflection), not retroreflection.

[0329] Therefore, the image light passing through the polarization separation component 101B undergoes specular reflection on the mirror reflector 4 to which the λ / 4 waveplate 21B is attached. The image light, after being specularly reflected on the mirror reflector 4 and traveling again towards the polarization separation component 101B, passes through the λ / 4 waveplate 21 twice, thus changing from the predetermined polarized light (P-polarized light in the figure) emitted from the display device 1 to another polarized light (S-polarized light in the figure). The image light traveling again towards the polarization separation component 101B is now another polarized light (S-polarized light in the figure), and therefore reflects off the polarization separation component 101B.

[0330] Here, the polarization separation component 101B in Figure 14(2) is arranged in a different direction than in Figure 14(1), so the image light reflected from the polarization separation component 101B travels in the opposite direction to the position where the user should be located. A retroreflector 2 with a λ / 4 waveplate 21C attached is positioned in front of the image light reflected from the polarization separation component 101B. The image light is retroreflected by the retroreflector 2. The image light, retroreflected on the retroreflector 2 and traveling again towards the polarization separation component 101B, passes through the λ / 4 waveplate 21C twice, thereby changing from another polarized light (S-polarized light in the figure) back to the specified polarized light (P-polarized light in the figure).

[0331] The image light that travels again towards the polarization separation component 101B is predefined polarized light (P-polarized light in the figure), so it travels through the polarization separation component 101B to the position where the user should be. The image light passing through the polarization separation component 101B forms a spatial levitation image 3B. The spatial levitation image 3B can be viewed well by the user from the direction of arrow A.

[0332] Here, in Figure 14(2), similarly to Figure 14(1), there exists a relationship regarding the retroreflective characteristics of the retroreflector 2, where the optical path length of the image light emitted from the display device 1 to the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to the formation position of the spatially suspended image 3B. The formation position of the spatially suspended image 3B in the direction of travel of the image light passing through the polarization separation component 101B is determined by this relationship.

[0333] The optical path length from the image light emitted from the display device 1 to the retroreflector 2 in Figure 14(2) is longer than that in Figure 14(1). This is because the optical system in Figure 14(2) adds a reciprocating optical path between the polarization separation component 101B and the specular reflector 4 to the optical path length from the image light emitted from the display device 1 to the retroreflector 2, which is not present in the optical system in Figure 14(1).

[0334] Therefore, in the optical system of Figure 14(2), the distance from the position where the light ray from the center of the image light passes through the polarization separation member 101B to the position where the image light forms the spatially suspended image 3B (L2 in the example of Figure 14(2)) is much longer than the distance from the position where the light ray from the center of the image light is reflected on the polarization separation member 101B to the position where the image light forms the spatially suspended image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0335] Furthermore, regarding the polarization design in the optical system of Figure 14 (2), the characteristics of P-polarization and S-polarization can be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 can be made S-polarized, and the characteristics of P-polarization and S-polarization can be interchanged regarding the reflection characteristics of the polarization separation component 101B. In this case, the P-polarization and S-polarization shown in the figure become opposite, but the optical design of the optical path, etc., can be implemented exactly the same.

[0336] Based on the optical systems of FIG14(1) and FIG14(2) in Embodiment 2 of the present invention described above, a more compact optical system can be achieved. In particular, based on the optical system of FIG14(2), a greater amount of spatial levitation image can be raised from the optical system with a more compact optical system.

[0337] Furthermore, if the optical system of Figure 14(1) or Figure 14(2) is to be installed in the space levitation image display device, it can be achieved simply by replacing the optical system in the space levitation image display device described in Embodiment 1 with the optical system of Figure 14(1) or Figure 14(2). Specifically, the optical system of Figure 14(1) can be replaced with the optical system of the space levitation image display device of Figures 4E, 4F, 4G, 4H, 4I, 4J, 4K, 4L, or 4M. In this case, since the optical system becomes more compact, the housing of the space levitation image display device in each figure can be made smaller.

[0338] Specifically, the optical system of (2) in Figure 14 can be replaced with the optical system of the space levitation image display device in Figures 4E, 4F, 4G, 4K, or 4L. In this case, the amount by which the space levitation image floats off the optical system can be increased. In addition, since the optical system becomes more compact, the housing of the space levitation image display device in each figure can be made smaller.

[0339] <Example 3>

[0340] As Embodiment 3 of the present invention, a system including the spatial levitation image display device 1000 will be described. The spatial levitation image display device 1000 of Embodiment 3 can be any of the spatial levitation image display devices 1000 shown in the figures of Embodiment 1 or Embodiment 2. In this embodiment, the differences from Embodiment 1 or Embodiment 2 will be described, and repeated descriptions of structures identical to those embodiments will be omitted.

[0341] In Example 3, a case is shown where images captured by a camera at a live concert are distributed and displayed to a display device such as a spatial levitation image display device in real time or non-real time. This example illustrates the distribution of live concert images. However, the present invention is not limited to this example and can be applied to various image distribution, broadcasting, and image distribution of encapsulated media. Furthermore, while this image distribution may include sound distribution, it is not limited to this and can also be applied to distributing only images. Additionally, the image distribution in this example is not limited to capturing images of real people and can also be applied to capturing images of virtual characters. In the latter case, the image distribution can include the distribution of 3D models and motion information (in other words, motion data), as described later.

[0342] Furthermore, the image distribution in this embodiment is not limited to real-time live streaming distribution during events such as live concerts, but can also be applied to on-demand delivery methods. It can also be applied to VOD (Video on Demand) methods, where footage from a live concert is first uploaded and registered to a server, and then streamed and received from the server to a display device at the user's request. Additionally, it can be applied to download-and-play methods where image data is downloaded to the display device beforehand and played (also known as "replay") at the user's request. In other words, the image distribution in this embodiment also includes situations where images from a live concert are distributed later after they have been recorded.

[0343] Furthermore, the image data format in this embodiment can be various, such as MPEG1, MPEG2, MPEG4, H.264, HEVC, VVC, FLV, etc. The audio data format in this embodiment can be various, such as MP3, MPEG4-AAC, MPEG4-ALS, MP4, AAC, MPEG-H 3D Audio, AC-4, Enhanced AC-3, etc. Additionally, when transmitting motion information, formats such as C3D, CSV, FBX, BVH, BVA, and TRC can be used. Image and audio can be transmitted using system formats or container formats such as MOV, MP4, FLV, HLS, MPEG-DASH, CMAF, ISO BMFF, MMT, etc. When transmitting motion information, if these system formats or container formats do not have a dedicated area for motion information, it can be stored in the user data area, extended information area, or additional information area, etc.

[0344] In addition, this embodiment mainly describes the processing of image data captured by the camera, but it is not limited to this. It can also process combined data of image data and other types of data such as accompanying sound data and text data.

[0345] [Display System]

[0346] Figure 15 illustrates a structural example of the system, or display system, of Embodiment 3, including the spatial levitation image display device 1000. In other words, the display system in Figure 15 is a live camera system, a camera image transceiver system, etc. The display system in Figure 15 is a system formed by connecting a system including a camera 1501 (concert venue 1500) and a system including the spatial levitation image display device 1000 used by user 230 via a communication network 1509 and a server 1503. The communication network 1509 includes WAN or LAN networks such as the Internet and mobile networks. The server 1503 is a computer system such as an operator's server that controls the distribution of camera images.

[0347] The display system in Figure 15 has the function of using camera 1501 to capture the live concert (hereinafter sometimes referred to as the live performance) of a group (e.g., a band) 1510 performing in concert venue 1500, and displaying the obtained image as an aerial image (spatial levitation image) 3 on the spatial levitation image display device 1000 of user 230 via communication and server 1503. Server 1503 has the function of distributing the image to spatial levitation image display device 1000, i.e., live camera distribution function 1530. Spatial levitation image display device 1000 has the function of displaying the image distributed by server 1503 as aerial image 3L, i.e., live camera function 1511. User 230 can view the aerial image 3L of the live performance on spatial levitation image display device 1000.

[0348] Furthermore, in Embodiment 3, there are images obtained from a camera 1501 that captures the live performance and the group 1510 as a whole (camera image 1600 of the overall camera C0 in FIG. 16), and additionally, there are images from various perspectives obtained from individual cameras 1501 that capture individual members 1502, etc. (camera images 1601 of individual cameras C1 in FIG. 16, etc.). The user 230 can select and view the camera image from the desired perspective. In this display system, which includes the spatial levitation image display device 1000 and the server 1503, there is a function that changes the camera image selected from the multiple camera images based on the user 230's operation input to the specified user interface (UI) and displays it, namely, the live camera switching function 1512.

[0349] At concert venue 1500, group 1510 performs live on a designated date and time. Group 1510, in other words, is a musician, artist, performer, etc. Live performance, in other words, is a concert, performance, etc. Live performance includes expressions such as music, video, and dance. These live performances, concerts, and performances can be referred to as events. Alternatively, it can be a live performance such as a talk show centered on conversation. Group 1510 has one or more members 1502. Member 1502 is a singer or dancer, etc. It can also be a narrator performing a talk show. In the example of Figure 15, member 1502 is a real person, but in other examples described later, it may be an artificial character / virtual image, etc. In the example of Figure 15, as shown in Figure 16 described later, group 1510 has multiple members 1502, and the subject of the filming is these multiple members 1502. It is not limited to this; the subject can also be only one person. Multiple cameras 1501 can be used to film a single subject from various directions.

[0350] [Live performance and cameras at the concert venue]

[0351] Figure 16 illustrates a structural example of multiple members 1502 of group 1510 and multiple cameras 1501 in concert venue 1500. In this example, group 1510 performing live has three members 1502: M1, M2, and M3. The diagram uses circles to represent the view from above. Cameras 1501 film the live performance of group 1510. In this example, as multiple cameras 1501, concert venue 1500 is equipped with a general camera C0 and individual cameras C1 to C3. The general camera C0 is a dedicated camera device for filming the entire live concert, including group 1510. Individual cameras are dedicated camera devices for filming individual members. Individual camera C1 primarily films member M1, individual camera C2 primarily films member M2, and individual camera C3 primarily films member M3.

[0352] Furthermore, during a live performance, each camera 1501 can be moved, and its position, orientation, and field of view can be changed. Each camera 1501 can be controlled by the filming personnel or automatically. Each camera 1501 can be equipped with a microphone or similar device to record live sound, and audio data is obtained along with the video data 1504. In a modified example, when the subject is only one person, multiple cameras 1501 are used to film that person from various positions and directions.

[0353] In Figure 15, each camera 1501 has communication functions and transmits the image data 1504 of the captured images to the server 1503 via the communication network 1509. Alternatively, other communication devices connected to the camera 1501 can also obtain the image data 1504 from the camera 1501 and transmit it to the server 1503.

[0354] This embodiment (Figure 15) uses a live broadcast distribution method, employing camera 1501 to capture a live performance taking place on a specific date and time, and then distributing and displaying it in real-time to the spatial levitation image display device 1000 via server 1503. Because processing and communication by server 1503 are required, a certain degree of delay occurs, but from the user 230's perspective, the live concert video can be viewed in roughly real-time. Not limited to this, other embodiments (Figure 21 described later) use an on-demand distribution method, first capturing a live performance taking place on a specific date and time and registering it with server 1503, and then distributing it from server 1503 to the spatial levitation image display device 1000 on other dates and times according to user 230's request.

[0355] Server 1503 acquires image data 1504 from camera 1501 at concert venue 1500 via communication and stores it in a database (DB) or similar memory. Server 1503 performs the necessary processing on the image data 1504 for distribution and stores the processed image 1505 in the DB or similar memory. Server 1503 then sends the image data 1505 to the spatial floating image display device 1000 of user 230 via communication.

[0356] The spatial levitation image display device 1000 for user 230 is, for example, a spatial levitation image display device 1000 owned by user 230, but is not limited to this; it can also be a device installed in a shop or public space, etc. The spatial levitation image display device 1000 receives image data 1505 from server 1503, accumulates it in memory, and forms an aerial image 3L of the live concert image as aerial image 3 based on the image data 1505. Thus, a mini live stage composed of aerial image 3L can be realized in the spatial levitation image display device 1000, and user 230 can watch and appreciate the aerial image 3L of the live concert image in this mini live stage.

[0357] The spatial levitation image display device 1000 of this embodiment includes a field camera function 1511 and a field camera switching function 1512. The field camera switching function 1512 is a function that can switch the camera images displayed in the aerial image 3 based on the input operation and selection of the user 230. The user 230 can select a camera image corresponding to the viewpoint of the requested member 1502, etc., via a predefined UI, and switch to the aerial image 3L of the selected camera image.

[0358] [Space-based levitation image display device]

[0359] Figure 17 shows a structural example of the spatial levitation image display device 1000 of the display system of Figure 15. This structural example is based on the structure of the optical system of Figures 2D and 4O, and the structure is approximately the same if it is rotated 45 degrees in the figure. In the side view (YZ plan view) shown on the right, image light is emitted upward in the Z direction from the display device 10 (e.g., the same as in Figure 2D) located in the lower part of the housing 1190. The image light is reflected forward in the Y direction by the tilted retroreflector 5. The image light forms a real image in the air (spatial levitation image) 3 at a predetermined position where it floats forward in the Y direction. In this example, the air image 3 is formed on a stage 1190S formed by a part of the housing 1190. The stage 1190S is a part of the stage model that extends forward in the Y direction from the lower part of the housing 1190, constituting a simulated stage for the air image 3, a mini stage that imitates a live concert stage.

[0360] In the front view (XZ plan view) shown on the left, a display area 3R for aerial image 3 is formed on stage 1190S. Within this display area 3R, an aerial image 3L is formed that includes (i.e., captures) members of the live performance, such as member 1502. User 230 can view this aerial image 3L well in the Y direction.

[0361] The housing 1190 is configured to include a light-shielding material, internally fixing components such as the display device 10 and the optical system in a predetermined positional relationship. A transparent component 100, etc., may be provided on the front surface of the retroreflector 5, or on the side of the front surface in the Y direction (closer to the user) (e.g., between it and the stage 1190S shown in the figure). An eaves / roof formed by a portion of the housing 1190 may be present on the upper side relative to the stage 1190S.

[0362] Furthermore, in the structural example shown in Figure 17, a control communication device 1701, corresponding to the control unit 1110 and communication unit 1132 (Figure 3), is provided in a part of the housing 1190 and connected to the display device 10, etc. The control communication device 1701 can be connected to the communication network 1509 (Figure 15). The control communication device 1701 can be implemented as a control device and a communication device. The control communication device 1701 can be located inside or outside the housing 1190.

[0363] Furthermore, the aerial operation detection sensor 1351 and camera unit 1180 are not illustrated in the structural example of Figure 17 (Figure 3), but they can be set up in the same way as in the above embodiment. Additionally, as a method for user 230 to input operations onto the spatial levitation image display device 1000, operation input unit 1107 and microphone 1139 can also be used (Figure 3). As for operation input unit 1107, in addition to a remote control, dedicated buttons or other hardware can also be provided. Furthermore, as an output method for user 230, in addition to displaying the aerial image 3, various displays, sound output unit 1140, etc., can be used in the same way as in the above embodiment (Figure 3).

[0364] In the structural example of Figure 17, a speaker 1702 and an indicator light 1703 are provided near the stage 1190S of the housing 1190. The speaker 1702 corresponds to the sound output unit 1140 in Figure 3. The spatial levitation image display device 1000 outputs sound from the speaker 1702 in sync with the display of the aerial image 3, based on the sound data provided in the image data 1505 from the server 1503. In addition, the sound output of the speaker 1702 also includes guide voice, alarms, and other sound outputs as part of the UI. The indicator light 1703 uses LEDs or other display devices to indicate the status of the spatial levitation image display device 1000, and can be operated by buttons or other operating devices, such as power on / off, mode switching, volume, cursor movement, execution / cancellation, etc.

[0365] [Hardware: First Structure Example]

[0366] Figure 18A shows a first structural example of the hardware structure of the spatial levitation image display device 1000 in Embodiment 3. Similar to Figure 17, the first structural example is based on Figures 2D and 4O. Image light from the display device 10 is emitted along the Y direction via the retroreflection area 5R of the retroreflector 5, forming an aerial image 3 (display area 3R) on the stage 1190S. In the first structural example, as shown in the top and side views, a light-shielding plate 1801 is vertically erected in the housing 1190 at a position deeper than the retroreflector 5 in the Y direction, having an XZ plane. The light-shielding plate 1801 can be a plate with a black film. Therefore, as shown in the front view, when viewing the aerial image 3 from the user 230's viewpoint in direction A (Y direction), the background of the aerial image 3 is dark, thus allowing for a clear view of the aerial images 3L of the performers on stage.

[0367] Additionally, Figure 18A shows an example of the installation of the airborne operation detection sensor 1351 and the camera unit 1180 (i.e., camera 1180). To sense the formation position of the aerial image 3, i.e., the display area 3R and its vicinity, the airborne operation detection sensor 1351 is installed, for example, on the stage 1190S facing upwards. Alternatively, the airborne operation detection sensor 1351 can be installed on the upper part of the housing 1190 facing downwards. Similar to Figure 4M, the airborne operation detection sensor 1351 can also be installed on the left and right sides of the housing 1190. Furthermore, in Figure 18A, the camera 1180 is installed on the upper part of the housing 1190 in a direction capable of capturing the user 230 located in front of the aerial image 3. The camera 1180 can also be installed in other locations in a direction capable of capturing the aerial image 3.

[0368] When the air operation detection sensor 1351 is present, the touch operation performed by the user 230's finger on the air image 3L can be detected as an air operation, and the display of the air image 3L can be changed accordingly based on the air operation. If a GUI is displayed in the air image 3L, the GUI can be operated. When the camera unit 1180 is present, the presence of the user 230 can be detected, and the display of the air image 3 can be changed accordingly based on the detection.

[0369] [Hardware: Example of Second Structure]

[0370] Figure 18B shows a second structural example of the hardware. In this second structural example, a second display panel 1902 is set in the background relative to the aerial image 3. The second display panel 1902 corresponds to the second display device 1680 in Figure 3, and is the second display panel when the liquid crystal display panel 11 included in the display device 10 is used as the first display panel 1901. The second structural example is based on the structure of Figure 4M. The display device 10, including the first display panel 1901, is disposed at the lower part of the housing 1190 and emits image light upward in the Z direction. This image light passes through the tilted polarization beam splitter 101B, is reflected back onto the retroreflector 2 with a λ / 4 waveplate 21 disposed on the upper panel 1903 at the upper part of the housing 1190, and is polarized. The polarized image light is reflected at the polarization beam splitter 101B, forming the aerial image 3 at a predetermined position in front in the Y direction.

[0371] The second structural example is similar to that in Figure 4M, where the air operation detection sensor 1351 can be positioned to the left or right relative to the air image 3. In the example of Figure 18B, the air operation detection sensor 1351 is positioned downwards on the top plate 1903, which can detect air operations when the user 230's finger intrudes into the deep side of the air image 3L.

[0372] [Second display panel and background image]

[0373] The second structural example of the spatial levitation image display device 1000 in Figure 18B, as shown in the front view, uses a second display panel 1902 to display a background image 1904 relative to the aerial image 3L. The spatial levitation image display device 1000 displays the aerial image 3L, including members of the live performance, in the optical image, i.e., the aerial image 3, and displays the background image 1904 of the aerial image 3L on the screen of the second display panel 1902. The image light corresponding to the background image 1904 displayed on the screen of the second display panel 1902 travels forward in the Y direction and passes through the polarization beam splitter 101B. When the user 230 views the aerial image 3L in direction A (Y direction), they can see an image of the aerial image 3L of the live performance superimposed on the front of the background image 1904.

[0374] Background image 1904 can use background images or videos from a live concert, or other images or videos. Background image 1904 can initially be generated based on images of the actual concert venue 1500 captured by camera 1501, including the venue's background. Server 1503 can generate image data (background image data) for background image 1904 based on the image data 1504 from camera 1501. Server 1503 can also generate an image (simulated background) that mimics the live concert background based on the image data 1504.

[0375] Furthermore, the background image 1904 is not limited to an image of the actual scene background or an image generated by imitating it; any image / video can be used. The content of any image / video can include, for example, natural scenery, virtual space, etc. In this case, it is possible to provide an image obtained by superimposing an aerial image 3L containing members 1502, etc., onto a background image 1904 with content different from the actual scene background. The server 1503 or the spatial levitation image display device 1000 can use pre-generated / set default background image data as the background image data for constructing the background image 1904. The user 230 can also select or specify the background image to be used from multiple default background images via the UI. The default background image data can be pre-set and stored in the spatial levitation image display device 1000. The server 1503 can also specify the background image data to be used via control information. Alternatively, the default background image data held by the server 1503 can be sent to the spatial levitation image display device 1000.

[0376] Additionally, an appropriate background image 1904 can be overlaid based on the content of the camera image in the aerial image 3L (e.g., the selected member 1502). For example, a background image 1904 with appropriate color and brightness can be selected based on the color and brightness in the aerial image 3L.

[0377] Additionally, the server 1503 or the spatial levitation image display device 1000 can perform variable control of the background image 1904. The server 1503 can generate control information for controlling the background image 1904 based on the image data 1504 of the camera 1501 and send it to the spatial levitation image display device 1000. The background image 1904 can be displayed fixedly on the timeline or displayed periodically.

[0378] For example, server 1503 can control the selection and display switching of background image 1904 in accordance with the selection and display switching of camera images corresponding to member 1502, etc. by user 230.

[0379] Alternatively, server 1503 can generate image data or control information for background image 1904 in such a way that the display content of background image 1904 can be changed in conjunction with the display content of aerial image 3L of the live performance formed based on image data 1505.

[0380] The spatial levitation image display device 1000 displays an aerial image 3L according to image data 1505 from a server 1503, and displays a background image 1904 according to image data or control information from a background image from the server 1503. Thus, a background image 1904 linked with the aerial image 3L can be achieved. A stereoscopic image representation of the aerial image 3L superimposed on the front of the background image 1904 can be provided. In other structural examples, the spatial levitation image display device 1000 can also generate image data or control information for the background image 1904 based on the image data 1505 from the server 1503, and control the display of the background image 1904.

[0381] For example, server 1503 can extract an image portion of object member 1502 from image data 1504 of camera 1501 to generate image data 1505 for aerial image 3L. Then, server 1503 can generate background image data or control information for overlaying background image 1904 required for the extracted aerial image 3L of member 1502.

[0382] Additionally, the spatial levitation image display device 1000 can also display images / images sent from the server 1503 or camera 1501 during a live performance as background image 1904 on the second display panel 1902. When a camera 1501 is present for capturing the live background, its image can be used as background image 1904. The spatial levitation image display device 1000 can also display images / images specified by control information sent from the server 1503 to the spatial levitation image display device 1000 during a live performance as background image 1904 on the second display panel 1902 at a designated time.

[0383] [Lighting Effects]

[0384] Furthermore, the second structural example of FIG18B includes a lighting effect 1905 formed by the lighting effect device 1906. The lighting effect 1905 illuminates the aerial image 3 or the vicinity of the aerial image 3 or the user 230. In the example of FIG18B, the lighting effect device 1906 is provided on the upper part of the housing 1190, near the upper side of the formation position of the aerial image 3 in front of the user on the top plate 1903. The direction of the lighting effect 1905 formed by the lighting effect device 1906 is in the direction from the aerial image 3 to the stage 1190S between the aerial image 3 and the user 230. The range of the lighting effect 1905 covers the aerial image 3 and the upper surface of the stage 1190S. The direction and range of the lighting effect 1905 are preferably set to illuminate the upper surface of the stage 1190S and the front side of the aerial image 3L.

[0385] Lighting effect 1905 is a different element from background image 1904. Lighting effect 1905 adds effects to the aerial image 3L and background image 1904 in the live performance. More importantly, lighting effect 1905 allows for changes not only to the on / off timing of the light emission, but also to the color, intensity, and direction of the emitted light. Using lighting effect 1905 can enhance the sense of presence.

[0386] As a variation, the background image 1904 may be displayed using only the second display panel 1902 without the lighting effect 1905. Alternatively, as a variation, the background image 1904 may be displayed using only the lighting effect 1905 instead of the second display panel 1902.

[0387] As variable control for the lighting effect 1905, the server 1503 or the spatial levitation image display device 1000 can control the timing and color of the light emission. Specifically, the server 1503 can generate control information for the lighting effect 1905 based on the image data 1504 from the camera 1501 and send it to the spatial levitation image display device 1000. This control information is used to variably control the timing and color of the lighting effect 1905 to coordinate with the display content of the aerial image 3L formed based on the image data 1505. The spatial levitation image display device 1000 displays the aerial image 3L according to the image data 1505 from the server 1503 and drives the lighting effect device 1906 according to the control information from the server 1503. Thus, a lighting effect 1905 coordinated with the display content of the aerial image 3L is achieved. A three-dimensional image representation of the aerial image 3L overlaid with the lighting effect 1905 is provided. In other structural examples, the spatial levitation image display device 1000 can also generate control information for the lighting effect 1905 based on the image data 1505 from the server 1503, and drive the lighting effect device 1906.

[0388] [Hardware: Example of a Third Architecture]

[0389] Figure 18C shows a third structural example of the hardware. The difference between the third and second structural examples is that the display device 10, including the first display panel 1901, is disposed on the upper panel 1903 of the housing 1190, and a retroreflector 2 is disposed at the lower part. Image light from the display device 10 is emitted downwards, passes through a polarizing beam splitter 101B, and is retroreflected on the retroreflector 2 with a λ / 4 waveplate. The polarized image light is reflected on the polarizing beam splitter 101B, forming an aerial image 3 at a position on the stage 1190S.

[0390] The third structural example is similar to the second structural example, capable of displaying the aerial image 3L of the live performance in the aerial image 3, and displaying the background image 1904 on the second display panel 1902. The third structural example can also similarly include an aerial operation detection sensor 1351 and a camera unit 1180. In the example of Figure 18C, the aerial operation detection sensor 1351 is positioned forward of the formation location of the aerial image 3, facing downwards. When the user 230's finger approaches the aerial image 3L, it can be detected as an aerial operation.

[0391] Furthermore, the third structural example, like the second structural example, can also include a lighting effect 1905. Based on the control information of the lighting effect 1905, the spatial levitation image display device 1000, in conjunction with the aerial image 3L, illuminates the stage 1190S from the lighting effect device 1906 by changing the timing and color of the illumination. In the example of Figure 18C, the lighting effect 1905 illuminates the aerial image 3L downwards and slightly deeper from the lighting effect device 1906.

[0392] [Hardware: Example of the Fourth Architecture]

[0393] Figure 18D shows a fourth structural example of the hardware. The fourth structural example differs from the second structural example in that the display device 10, including the first display panel 1901, is disposed on the side portion of the housing 1190 in the left-right direction, i.e., the X direction, i.e., the side panel 1907 in Figure 18D. As shown in the top view, for example, the display device 10 including the first display panel 1901 is disposed on the right side panel 1907 viewed from the user 230 in the X direction, and a retroreflector 2 is disposed on the left side panel 1907. Image light from the display device 10 exits to the left, passes through the polarizing beam splitter 101B, and is retroreflected on the retroreflector 2 with a λ / 4 waveplate. The polarized image light is reflected on the polarizing beam splitter 101B, forming an aerial image 3 at a position on the stage 1190S.

[0394] The fourth structural example is similar to the second structural example, capable of displaying the aerial image 3L of the live performance in the aerial image 3, and displaying the background image 1904 on the second display panel 1902. The fourth structural example can also similarly include an aerial operation detection sensor 1351 and a camera unit 1180. With the aerial operation detection sensor 1351 positioned to the left or right of the display range 3R of the aerial image 3, the side panel 1907 is extended to the left or right of the stage 1190S. Furthermore, the fourth structural example can also similarly include a lighting effect 1905.

[0395] [Example of Combined Control]

[0396] As described above, the structural elements present in the example structure of FIG18B, etc., of Embodiment 3 include: an aerial image 3L formed based on image data 1504 and image data 1505 from camera 1501, a background image 1904 from the second display panel 1902, and a lighting effect 1905 formed by the lighting effect device 1906. Various controls and image representations combining these structural elements are possible. By controlling these structural elements, the visual effects and image representations of the aerial image 3L, etc., seen by user 230 can be realized. When the background image 1904 and lighting effect 1905 are applied to the aerial image 3L in combination, the sense of depth and immersion can be further enhanced.

[0397] Figure 19 shows a table summarizing various control examples. The table shows the relationship between the aerial image 3L, the background image 1904, and the lighting effect 1905 corresponding to the camera image, with control examples shown in each row. In Embodiment 3, as shown in the control examples, the various components can be controlled in a combined and linked manner. The specific control to be applied can be selected according to the design and implementation of this display system. Furthermore, users can select and set various controls via the UI.

[0398] (Example 1) An aerial image 3L that only displays footage from a live concert camera. Background image 1904 and lighting effects 1905 are set to "None (OFF)". The content of aerial image 3L includes images of people or characters (described later) from the live concert group 1510. Alternatively, the "None (OFF)" setting for background image 1904 and lighting effects 1905 can also be the default output setting.

[0399] (Example 2) Display a combination of aerial image 3L and background image 1904. The content of background image 1904 can be selected from background images corresponding to the background of the live concert, background images independent of the background of the live concert, default images / videos, etc. Server 1503 or spatial floating image display device 1000 can control the display of background image 1904 through control information or background image data.

[0400] (Example 3) Displaying a combination of aerial image 3L and lighting effect 1905. The timing, color, amount, and direction of illumination of lighting effect 1905 can be variably controlled. Server 1503 or spatial levitation image display device 1000 can control the variable control of lighting effect 1905 via control information.

[0401] Furthermore, as an example of controlling the lighting effect 1905, the content of the lighting effect 1905 can be used in conjunction with the display of the aerial image 3L of the selected camera / member's camera image, depending on the camera / member. For example, the lighting effect 1905 can be red in the case of member M1's image, and blue in the case of member M2's image, etc.

[0402] (Example 4) Display a combination of aerial image 3L, background image 1904, and lighting effect 1905. The display content of background image 1904 and the variable control content of lighting effect 1905 are controlled in association. Server 1503 or spatial levitation image display device 1000 controls both background image 1904 and lighting effect 1905 through control information, etc.

[0403] (Example 5) Displaying a combination of aerial image 3L and lighting effect 1905. Furthermore, the content displayed in aerial image 3L can be supplemented with image representations and display effects that are linked to the variable control content of lighting effect 1905. For example, display control could be implemented such that the brightness and color of the surface of the person / character in aerial image 3L are changed according to the timing and color of the illumination of lighting effect 1905. In other words, this is a simulated lighting effect that makes it seem as if lighting effect 1905 is actually illuminating the person / character in aerial image 3L.

[0404] [An example of variable control for lighting effects 1905]

[0405] The following is an example of variable control of the lighting effect 1905 corresponding to Examples 3 to 5 above. When lighting effects are present during a live performance at the concert venue 1500, these effects are captured by the camera 1501 and reflected in the image data 1504. The server 1503 and the spatial floating image display device 1000 can link the lighting effect 1905 with the lighting effects contained in the image data 1504. For example, suppose that at a certain moment during the live performance, a member 1502 is illuminated with red light. The server 1503 performs image processing or rendering based on the image data 1504 to generate image data 1505 from the perspective of the member 1502, and generates control information for the lighting effect 1905. At this time, the server 1503 generates control information for the lighting effect 1905 by illuminating the aerial image 3L with red light 1905 in sync with the actions of the member 1502.

[0406] Furthermore, server 1503 can variably control the content of image data 1505 of background image 1904 or aerial image 3L in a manner synchronized with the red lighting effect 1905. For example, server 1503 can generate control information or background image data for background image 1904 in a manner that synchronizes the red light effect 1905 with the display content of background image 1904.

[0407] Additionally, for example, server 1503 can process the image data 1504 to calculate the lighting, thereby generating image data 1505 of the aerial image 3L in a manner that can present brightness and other representations corresponding to the state of the object member 1502 being illuminated by red light. Thus, by superimposing the lighting effect 1905 with the aerial image 3L, their synergistic effect can be used to achieve a visual effect as if the light is shining on the members (people / characters) of the aerial image 3L.

[0408] [On-site camera switching function]

[0409] The on-site camera switching function 1512 in Example 3 (Fig. 15) can display an aerial image 3 (aerial image 3L) selected from the overall view of the live concert (overall view camera image 1600 in Fig. 16) and the individual member images (individual view camera images 1601-1603 in Fig. 16). "View" is in other words, the field of view or viewpoint.

[0410] Figure 20 is an explanatory diagram of the live camera switching function 1512, specifically illustrating an example of a GUI (Graphical User Interface) that allows user 230 to select a desired camera / viewpoint. The spatial levitation image display device 1000 displays the GUI image 1550 shown in Figure 20 in the aerial image 3. The GUI image 1550 is a GUI that allows selection of a desired camera image from both the overall camera image and individual camera images, i.e., the camera images of each member. In other words, the GUI image 1550 is a GUI that allows selection of a desired viewpoint from both the overall viewpoint and individual viewpoints, i.e., the viewpoints of each member. The GUI image 1550 includes, for example, an overall viewpoint icon 1551 and individual viewpoint icons 1552, 1553, and 1554. The overall viewpoint icon 1551 is a GUI for selecting the overall viewpoint camera image 1600, which is captured by the overall camera C0 in Figure 16 and is an image from the perspective of the entire live performance and all group members. Individual viewpoint icons 1552 are GUI elements used to select individual viewpoint camera images 1601, which are captured by individual camera C1 in Figure 16 and are images from the perspective of member M1. Each icon may include an identification number or ID, a thumbnail of the camera image, etc.

[0411] User 230 inputs an action on the icons of GUI image 1550, thereby selecting the desired camera / viewpoint image. Regarding the input, for example, touch operations can be detected using an air operation detection sensor 1351. When a camera / viewpoint is selected, the spatial levitation image display device 1000 displays the image of that camera / viewpoint as aerial image 3L in aerial image 3. The GUI image 1550 is not limited to being displayed in aerial image 3; if the spatial levitation image display device 1000 is equipped with other displays (e.g., a second display panel 1902), the GUI image 1550 can also be displayed on that display.

[0412] With such a GUI, users only need to select one of the required cameras / members from multiple pre-prepared options, making it easier for them to select images.

[0413] [Aerial Operation Detection Sensor]

[0414] Figure 29 shows an example of the structure of the airborne operation detection sensor 1351, which serves as a supplementary explanation of the user operation detection mechanism. Figure 29 shows an implementation example of the airborne operation detection sensor 1351 that can be applied to Embodiment 3, etc. The airborne operation detection unit 1350, control unit 1110, or image control unit 1160, etc. (in other words, image processing unit) in Figure 3 use the detection signal of the airborne operation detection sensor 1351 to determine / detect the airborne operation performed by the user on the display range 3R of the spatial levitation image 3.

[0415] Figure 29 shows an xy-plane view corresponding to the display range 3R of the space-suspended image 3. The airborne operation detection sensor 1351 has multiple optical elements 1351c arranged in the x-direction. Each optical element 1351c is a pair of light-emitting elements 1351a and light-receiving elements 1351b. The light-emitting element 1351a is, for example, composed of an infrared element. The light-emitting element 1351a emits light a1 (e.g., infrared light) in the y-direction. This light a1 passes through the display range 3R without being obstructed by an object. If the light a1 is obstructed by an object, it is reflected back as reflected light a2. The reflected light a2 is received by the light-receiving element 1351b.

[0416] For example, if a user 230's finger touches a point a3 within the xy plane of the display area 3R, light a1 is reflected at this point a3 as reflected light a2 and returns. A light-receiving element 1351b at a certain x-direction position detects this reflected light a2. Therefore, the air-operation detection unit 1350 can determine that the touch point a3 is located at that x-direction position. Furthermore, the air-operation detection unit 1350 can calculate the distance using a TOF method based on the time it takes for the emitted light a1 to return as reflected light a2. For example, it can calculate the distance a4 to the touch point a3. Thus, the position coordinates of the touch point a3 on the xy plane of the display area 3R can be determined.

[0417] Not limited to this example, the air operation detection sensor 1351 can be configured on the upper side, left or right side, or offset in the forward / backward direction (i.e., the z-direction) relative to the xy-plane of the display range 3R. Multiple air operation detection sensors 1351 can also be configured at multiple positions in the forward / backward direction.

[0418] [Display System: Real-world Space: On-demand Distribution]

[0419] As an example of the display system structure in Embodiment 3, Figure 21 shows the case of a live concert in a real space. Figure 21 also shows the case of on-demand distribution via server 1503. The members 1502 of the group 1510 being filmed are real people. In this display system, for multiple cameras 1501 filming the live performance of group 1510 in concert venue 1500, image data 1504 of the images captured by each camera 1501 is uploaded to server 1503. Server 1503 registers and stores the distributed image data 2130, generated based on the image data 1504 through image processing, etc., in a database. In response to a request (request information 2101) from display device 2000 of user 230, server 1503 distributes image data 1505 of the camera images selected from the live camera image data 2130. User 230's display device 2000 displays the camera images based on the image data 1505 received from server 1503. The images in this display system can be distributed via the Internet or via radio waves.

[0420] In the example of Figure 21, group 1510 has two members, M1 and M2, as members 1502 and cameras 1501. There is a general camera C0 and two individual cameras C1 and C2. For example, member M1's ID / name is Koto, and member M2's ID / name is Airia. For example, camera C1 films member M1, and camera C2 films member M2. Each camera 1501 sends and uploads the image data 1504 of the captured images to server 1503 via communication. Server 1503 accumulates the image data 1504 from each camera 1501 in its memory, performs prescribed image processing, generates image data for distribution, and registers it in the database as on-site camera image data 2130. On-site camera image data 2130 includes images from various perspectives.

[0421] User 230 uses either the spatial levitation image display device 1000 or a television (television display device) or other display device 3000 as display device 2000. In this description, display device 2000 refers to both the spatial levitation image display device 1000 and / or display device 3000. Display device 2000 is, in other words, the user's display device. For example, user 230 has either the spatial levitation image display device 1000 or a television (or other display device 3000) in their home. The example in Figure 21 shows a case where user 230 uses either the spatial levitation image display device 1000 or the television (or display device 3000) as display device 2000, but there are also cases where only one of these devices is used, and there can be three or more. In this example, either display device 2000 can use the live camera function 1511, etc. The spatial levitation image display device 1000 and the display device 3000 each have specified functions, namely, the same on-site camera function 1511 and on-site camera switching function 1512 as described above. In the case of the spatial levitation image display device 1000, the camera image is displayed as an aerial image 3, as described above. In the case of the display device 3000, such as a television set, the camera image is displayed on a screen with a fixed pixel surface. The display device 3000, in other words, is a fixed pixel surface display device, which is a device that displays images and videos on a two-dimensional surface formed by fixed pixels, i.e., a display screen, including various display devices such as television display devices and monitor devices.

[0422] Figure 21 illustrates a scenario where user 230 simultaneously uses two display devices 2000. User 230 selects to view the overall viewpoint image on the television (display device 3000), which has a relatively larger screen, and selects to view individual viewpoint images on the spatial levitation image display device 1000, which has a relatively smaller screen. User 230 can view camera footage in parallel using both display devices 2000. Furthermore, the size relationship of the screens on the multiple display devices 2000 is not limited; it is also possible for one side of display device 3000 to have a relatively smaller screen, and the other side of the spatial levitation image display device 1000 to have a relatively larger screen.

[0423] Server 1503 can perform pre-defined processing such as image processing on image data 1504 from camera 1501 to generate image data for distribution. Examples of this processing include cropping or extracting object members 1502 to generate a two-dimensional image, as well as correction and noise reduction processing to obtain a clear image / video, brightness adjustment, and zooming / reduction. Alternatively, image processing may not be performed on server 1503.

[0424] The display device 2000 can select the desired camera 1501's image from multiple cameras 1501 based on user input via the UI during the on-site camera switching function 1512. Alternatively, it can select a desired member 1502 from multiple members 1502. The UI for selecting a camera 1501 or member 1502 can, for example, use the GUI of FIG20. It is not limited to this; the selection UI can also use the ID or name of the member 1502 that the user 230 is interested in. Alternatively, the selection UI can also select the camera or member associated with a number by specifying a number using a remote control or button. Furthermore, it can also select a camera or member by inputting a name and number via voice input or voice recognition. Additionally, it is possible to pre-select and set which camera / member's image to display for each display device 2000 via the UI.

[0425] The display device 2000 generates and determines selection information representing the selection made by the user 230 through the UI, such as camera / member selection, and sends request information 2101 containing the selection information to the server 1503. The selection information may be camera selection information or member selection information, etc. Camera selection information is information about selecting camera 1501 or its associated camera image. Member selection information is information about selecting member 1502. In other words, this selection information is information about selecting a viewpoint or display object. Request information 2101 contains identification information of the requesting display device 2000 and the requested server 1503, etc. Based on the selection information in request information 2101, the server 1503 selects the camera image to be distributed from the on-site camera image data 2130 in the database and sends the image data 1505 of that camera image to the requesting display device 2000. For example, if a member 1502 is selected, the image to be distributed is the image of the camera 1501 that captured that member 1502.

[0426] The display device 2000 can provide not only the aforementioned selection UI, but also a UI that allows for user settings. This UI, for example, can perform user account registration and live concert search.

[0427] Furthermore, in the structural example of Figure 21, the display device 2000 of user 230 can cooperate with a portable terminal 4000 (in other words, a portable information terminal device) such as a smartphone held by user 230. When the selected UI is a GUI, it is not limited to the GUI on the screen of the display device 2000; a GUI on the display screen of the portable terminal 4000, which is wirelessly connected to the display device 2000, can also be used. In this case, user 230 inputs an operation on the GUI of the portable terminal 4000, and the portable terminal 4000 sends selection information based on the operation input to the display device 2000. The display device 2000 then sends a request message 2101 containing the selection information to the server 1503. Alternatively, the request message 2101 containing the selection information can be sent directly from the portable terminal 4000 to the server 1503.

[0428] [Camera Image Selection Method]

[0429] The following is a supplementary explanation of the selection methods for the camera images to be displayed in the on-site camera switching function 1512. In the structural example in Figure 21, the selection methods are 1. Camera selection method and 2. Member selection method.

[0430] 1. Camera Selection Method: In the camera selection method, based on the UI specified in the example shown in Figure 20, the user 230 selects and specifies the camera 1501 as the display object, and the display device 2000 generates camera selection information. The server 1503 determines the camera image to be distributed based on the camera selection information. In this case, the user 230's operation is only to select one of the required cameras from multiple options prepared in advance by the UI, so the user 230 can more easily set the image to be displayed.

[0431] 2. Member Selection Method: In the member selection method, user 230 selects and specifies the entire group 1510 or a member 1502 (a person in Figure 21) within group 1510 based on the prescribed UI. The display device 2000 generates member selection information. Server 1503 determines the camera footage to be distributed based on the member selection information. In this case, user 230's operation is simply to select one member (the one they are interested in) from multiple options prepared in advance by the UI. This allows them to select the camera footage from the perspective corresponding to that member, making it easier for user 230 to view the footage of the requested member 1502.

[0432] In the structural example of Figure 21, because there are two display devices 2000, the user 230 can display the selected camera images on each display device 2000 and watch them simultaneously. In one display device 2000, such as a television or display device 3000, the camera image corresponding to the camera / member selected by the user 230 can be displayed on the screen. In the other display device 2000, such as a spatial levitation image display device 1000, other camera images corresponding to other cameras / members selected by the user 230 can be displayed as aerial images 3L. The user 230 can watch images from different perspectives in parallel. In the example of Figure 21, the user 230 can use the large screen of the television or display device 3000 to watch the overall perspective image of the entire live performance as the main camera image, and in parallel, watch individual perspective images focusing on a specific member 1502 on the side of the spatial levitation image display device 1000, which has a relatively small display area 3R.

[0433] Additionally, the on-site camera switching function 1512 can switch to other camera images, or in other words, other perspectives, during camera image playback. When user 230 selects a new camera / member via UI input during camera image playback, the display device 2000 and server 1503, based on this operation, perform the same processing as described above to control the display to switch to the camera image corresponding to the newly selected camera / member.

[0434] [Select Information Form]

[0435] Figure 22A shows a structural example of a selection information table corresponding to the structural example in Figure 21. This table relates to the selection of cameras / members in the UI and is maintained by the server 1503 or display device 2000 for control purposes. The server 1503 determines the camera image to be distributed based on this selection information table in response to the selection information in request information 2101 from the display device 2000. The selection information sent from the display device 2000 to the server 1503, as described above, can be either camera selection information or member selection information. Regardless of the type of selection information, the image to be distributed can be determined according to this table.

[0436] The table in Figure 22A includes items such as image (corresponding to image data), shooting camera, subject (corresponding to viewpoint), camera selection information, and member selection information, and sets their corresponding relationships. For example, row #1 specifies that image A is the overall viewpoint camera image, the shooting camera is camera A (the overall camera C0 in Figure 21), and the subject is group 1510, in other words, all members 1502 {M1, M2}. Furthermore, row #1 specifies that image A is selected when the camera selection information is "camera A (number 1)" or the member selection information is "group (all members)". Additionally, if cameras / members are pre-assigned numbers, they can also be specified using those numbers (Figure 20). If member 1502 has a name, it can also be specified using that name.

[0437] The selection information table in Figure 22B also shows the selection information table for other structural examples (Figure 26) described later.

[0438] [Display device (television)]

[0439] Figure 23 illustrates a structural example of the display device 3000. The display device 3000 can be, for example, a large-screen display device such as a television. Figure 23 shows a functional module structure example of a television, i.e., the display device 3000. The display device 3000 includes a display panel 30011, a control unit (including a processor) 30012, an external power input interface 30013, a power supply 30014, a secondary battery 30015, a storage unit 30016, an image control unit 30017, a communication unit 30020, an audio output unit (including a speaker) 30021, an audio input unit (including a microphone) 30022, an image signal input unit 30023, an audio signal input unit 30024, and a camera unit 30025, etc.

[0440] Display panel 30011 can be configured to display images or videos on a two-dimensional display screen (in other words, a fixed pixel surface) using devices such as liquid crystal panels or organic EL panels. Display panel 30011 may also be referred to as a display unit.

[0441] The communication unit 30020 is configured using devices that employ communication interfaces including Wi-Fi (registered trademark), Bluetooth (registered trademark), or mobile communication interfaces such as 4G or 5G. The communication unit 30020 can use these communication methods to communicate with the communication units of external devices (e.g., the spatial levitation image display device 1000). Furthermore, the communication unit 30020 can communicate with external devices (e.g., server 1503) connected to a communication network 1509 such as the Internet using any of these communication methods.

[0442] The control unit 30012 includes a processor such as a CPU and internal memory such as ROM and RAM, and controls various units such as the image control unit 30017 and the communication unit 30020, as well as the display device 3000 as a whole. The control unit 30012 can execute processing according to programs read from internal memory or memory 30026 using the processor. The control unit 30012 can cooperate with programs stored in memory 30026 to perform calculations based on information obtained from various units within the display device 3000.

[0443] The memory 30026 stores data processed by the control unit 30012 and the image control unit 30017, etc. For example, the memory 30026 stores image data for images to be displayed on the display panel 30011, and control data for the display device 3000. The control unit 30012 can read various software programs from the storage unit 30016 and load them into the memory 30026.

[0444] The non-volatile memory 30027 stores various data and information used in the display device 3000. The data stored in the non-volatile memory 30027 includes, for example, various operational data to be displayed on the display panel 30011, display icons, data of user-operated objects, and layout information.

[0445] Power supply 30014 converts AC current input from the outside via external power input interface 30013 into DC current, supplying the required DC current to various parts of display device 3000. Secondary battery 30015 stores the power supplied by power supply 30014. Furthermore, when no external power is received via external power input interface 30013, secondary battery 30015 supplies power to the various parts requiring electricity.

[0446] The video signal input unit 30023 connects to an external video output device to input video data. The video signal input unit 30023 can utilize various digital video input interfaces. For example, it can use video input interfaces of the HDMI (High-Definition Multimedia Interface) standard, the DVI (Digital Visual Interface) standard, or the DisplayPort standard. Alternatively, analog video input interfaces such as analog RGB and component video can also be provided. The video signal input unit 30023 can also utilize various USB interfaces.

[0447] The audio signal input unit 30024 connects to an external audio output device to input audio data. The audio signal input unit 30024 can use HDMI standard audio input interfaces, optical digital terminal interfaces, or coaxial digital terminal interfaces, etc. The audio signal input unit 30024 can also use various USB interfaces, etc. When using an HDMI standard interface, the video signal input unit 30023 and the audio signal input unit 30024 can be configured as an interface integrating terminals and cables.

[0448] The audio output unit 30021 includes a speaker and is capable of outputting sound based on audio data input to the audio signal input unit 30024. The audio output unit 30021 can also output sound based on audio data stored in the storage unit 30016. The audio output unit 30021 can also output built-in operation tones and error warning tones. Alternatively, the audio output unit 30021 can also employ a structure that outputs digital signals to external devices, similar to the Audio Return Channel function specified in the HDMI standard.

[0449] The sound input unit 30022 includes a microphone, which collects the sound around the display device 3000 and converts it into a signal to generate a sound signal. The sound input unit 30022 can be configured to use the microphone to collect the voice of a person such as the user 230 (speech), and the control unit 30012 and the like perform sound recognition processing on the generated sound signal to obtain text information based on the sound signal.

[0450] The camera unit 30025 can be equipped with a camera having an image sensor. The camera can be mounted on the front side of the display panel 30011 of the display device 3000, or on the back side. Both a front-facing camera and a rear-facing camera can also be used.

[0451] Storage unit 30016 is a storage device for recording various data and information such as image data, audio data, text data, and programs. Storage unit 30016 can be configured, for example, by a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor device memory such as a solid-state drive (SSD). In storage unit 30016, various data and information such as image data can be pre-recorded, for example, at the time of product shipment. Storage unit 30016 can also record various data and information such as image data, audio data, etc., obtained from external devices such as servers via communication unit 30020. The image data recorded in storage unit 30016 is output to display panel 30011. The image data recorded in storage unit 30016 can also be output to external devices via communication unit 30020. Alternatively, various data can be stored in a storage device connected externally to display device 3000.

[0452] The image control unit 30017 performs various controls on the image signals input to the display panel 30011. The image control unit 30017 can be constructed from hardware such as an ASIC, FPGA, or image processor (GPU). Alternatively, the image control unit 30017 can be referred to as an image processing circuit, image processing unit, picture processing unit, display processing unit, etc. The image control unit 30017 can be included as a part of the display panel 30011.

[0453] The image control unit 30017 performs control such as image switching, switching which image signal to input to the display panel 30011 from among multiple image signals, including image signals (and corresponding image data) stored in the memory 30026 and image signals (and corresponding image data) input to the image signal input unit 30023. Additionally, the image control unit 30017 can also control image processing of the image signals input from the image signal input unit 30023 and the image signals stored in the memory 30026. Image processing includes, for example, scaling (enlarging, reducing, and distorting the image), brightness adjustment (changing brightness), contrast adjustment (changing the image's contrast curve), and Retinex processing (decomposing the image into light components and changing the weights of each component).

[0454] In the above description, the display device 3000 is a fixed-pixel surface display device, which includes a display panel 30011 that displays images / videos on a two-dimensional display screen using fixed pixels, but is not limited to this. The concept of this fixed-pixel surface display device also includes projection-type image display devices, in other words, projectors. In a projector, a projection optical system is provided after the display panel having a fixed pixel surface, and the optical image of the image displayed on the fixed-pixel surface display panel is projected onto a predetermined screen or other surface for display.

[0455] [server]

[0456] Figure 24 shows an example of the structure of the computer system of server 1503. Server 1503 includes a control unit 15031 with a processor, memory 15032, a communication unit 15033, and an input / output interface 15034, which are interconnected by a bus or similar architecture. Input / output interface 15034 externally connects to input device 15035 and output device 15036. Communication unit 15033 implements a communication interface corresponding to communication network 1509, etc. Users such as administrators operate server 1503 through input device 15035 and output device 15036. Users can also operate server 1503 using a client terminal device that is communicatively connected to server 1503.

[0457] The memory 15032 stores the control program 2401, setting information 2402, database (DB) 2403, and image data 2404. The control program 2401 is a program used to implement the specified functions, such as the field camera distribution function 1530. The processor of the control unit 15031 executes processing according to the control program 2401 to implement the specified functions, such as the field camera distribution function 1530. The setting information 2402 contains system setting information and user setting information. The DB 2403 registers image data used for distribution. The image data 2404 includes, for example, data that constitutes a web page or other screen. The server 1503 can manage and provide information about the field camera distribution service and web pages with a GUI.

[0458] [Transmission and reception of image data and control information]

[0459] Figure 25 illustrates a structural example of the transmission and reception of image data and control information between server 1503 and display device 2000 in the display system of Figure 21. Server 1503 receives image data 1504 from camera 1501 in concert venue 1500. The image processing unit 2501 of server 1503 processes the image data 1504 to generate image data 1505 for distribution. The image data 1505 includes audio data, etc. The image processing unit 2501 is implemented by the control unit 15031 (Figure 24). In addition, the image processing unit 2501 generates control information 2502 regarding the image data 1505. The control information 2502 is used to control the background image 1904, lighting effects 1905 (Figure 18B, etc.), but is not limited to this. The control information 2502 is managed in association with the image data 1505. The control information 2502 can also be described as part of the attribute information / metadata of the image data 1505. Server 1503 sends image data 1505 and control information 2502 to display device 2000.

[0460] The display device 2000 receives image data 1505 and control information 2502 from the server 1503. The image processing unit 20001 of the display device 2000 generates display data, or in other words, image signals, for displaying images on the display screen 20005 of the display unit 20002 based on the image data 1505. Furthermore, if the display device 2000 is a spatial levitation image display device 1000, the display unit 20002 is equivalent to the display device 10 in FIG. 17, and the rear stage of the display unit 20002 includes a retroreflector 5 as shown in FIG. 17 as an optical system 20003. The display screen (i.e., the display image) 20005 corresponds to the display range 3R of the aerial image 3. If the display device 2000 is a display device 3000 such as a television set, the optical system 20003 can be omitted, and the display screen 20005 of the display unit 20002 corresponds to the screen of the display panel 30011 (FIG. 23). In addition, if the display device 2000 is a projector, the optical system 20003 is equivalent to a projection optical system, and the display screen 20005 is equivalent to a screen (the surface to be projected).

[0461] The display device 2000 uses the user operation detection mechanism 20004 to detect the operation of the user 230. The user 230 performs user operations on the display screen 20005 or the user operation detection mechanism 20004. In the case where the display device 2000 is a spatial levitation image display device 1000, the user operation detection mechanism 20004 is, for example, the aforementioned air operation detection sensor 1351 (Figs. 3 and 29). In other examples, the user operation detection mechanism 20004 is a remote control, touch panel, operation button, etc.

[0462] As an example of control information 2502, as shown in Example 1, when the background image 1904 in Figure 18B is displayed and control of the background image 1904 in Figure 19 exists, a file for sending background image control information is sent as control information for controlling the background image 1904. If the display device 2000 already retains this file, only this file needs to be specified. Furthermore, as an example 2, when the lighting effect 1905 in Figure 18B exists and control of the lighting effect 1905 in Figure 19 exists, a file for sending lighting effect control information is sent as control information for controlling the lighting effect 1905. If the display device 2000 already retains this file, only this file needs to be specified.

[0463] For example, when the spatial levitation image display device 1000 is equipped with the aforementioned lighting effect device 1906, the server 1503 sends image data 1505 of the selected camera image to the spatial levitation image display device 1000, and simultaneously sends control information for the lighting effect 1905. Then, in the spatial levitation image display device 1000, in conjunction with displaying the aerial image 3L based on the received image data 1505, the lighting effect device 1906 is driven based on the received control information to change the color of the lighting effect 1905, etc.

[0464] [Display System: Virtual Space: First Structure Example]

[0465] As another structural example of the display system in Embodiment 3, Figure 26 shows a first structural example regarding virtual space. In this first structural example, the concert venue 2600 is not an actual venue, but a virtual space constructed in the server 2603. The live concert and group 2610 are constructed within this virtual space. Members 2602 generate images by rendering based on 3D model objects and virtual cameras 2601 set within the virtual space, calculated by roles / virtual images constructed within the virtual space. The server 2603 calculates the concert venue 2600 constituting the virtual space, conducts the live performance of group 2610, and generates images for distribution by rendering. In this example, members 2602 include members M1 and M2. The structural example of Figure 26 can be applied to any of the live distribution methods shown in Figure 15 and the on-demand distribution method shown in Figure 21.

[0466] Server 2603 constructs and maintains 3D models of objects (such as the stage and background objects) configured in the virtual space concert venue 2600, as well as 3D models of members 2602, i.e., characters. Server 2603 controls the actions of the 3D model characters during the live performance. Server 2603 uses the motion information (in other words, motion data) of the 3D model characters to generate and execute the character's actions and animations.

[0467] In addition, server 2603 controls multiple virtual cameras 2601 configured in the virtual space. Server 2603 controls the position, orientation, field of view, and movement of each virtual camera 2601 within the virtual space. Server 2603 maintains virtual camera information used to control the virtual cameras 2601. Based on the 3D model object and the virtual cameras 2601, server 2603 uses each virtual camera 2601 to capture images of the set shooting targets (objects), namely the overall performance and individual members 2602, etc., and generates images from the perspective of each virtual camera 2601 through rendering processing. Server 2603 registers and stores the image data corresponding to the generated images as image data 2630 for distribution, i.e., on-site camera image data, in the database (DB) of memory. Server 2603 maintains 3D models of the concert venue 2600 and characters, character motion information, virtual camera information, and image data 2630 in the DB.

[0468] In the example of Figure 26, virtual cameras 2601 include virtual cameras C0, C1, C2, and Cf. Virtual camera C0 captures an overall view of the live concert and all members 2602 of the group 2610. Virtual camera C1 captures individual view of member M1. Virtual camera C2 captures individual view of member M2. Virtual camera Cf captures an image from a viewpoint that can be freely set by the user. In the example of Figure 26, virtual cameras C0, C1, and C2 are virtual cameras 2601 pre-set by server 2603, and virtual camera Cf is virtual camera 2601 corresponding to a virtual viewpoint that the user can freely set. The user 230 can freely specify and set the position, direction, etc. of this virtual viewpoint through the UI.

[0469] In the user 230 environment, the display device 2000, like in Figure 21, includes a spatial floating image display device 1000 and a display device 3000. The display device 2000, like in Figures 15 and 21, has a live camera function 1511 and a live camera switching function 1512. In addition to selecting the desired live concert, the user 230 can also select the desired virtual camera 2601 or member 2602, etc., through a predefined UI. The display device 2000 sends a request message 2606 containing selection information to the server 2603.

[0470] In the case of live broadcast distribution, server 2603 distributes image data 2605 to display device 2000 on the date and time of the live concert. Alternatively, in the case of on-demand distribution, server 2603 distributes image data 2605 to display device 2000 based on request information 2606 from display device 2000. Server 2603 distributes image data 2605 selected from image data 2630 to display device 2000 based on selection information in request information 2606 from display device 2000. Display device 2000 displays the image on the display screen of display unit based on the image data 2605 received from server 2603.

[0471] Regarding the live camera switching function 1512, in the spatial floating image display device 1000 and the display device 3000, based on the operation input made by the user 230 via the UI, one virtual camera 2601 or multiple members 2602 (i.e., characters) in the live concert is selected, and virtual camera selection information or character selection information (i.e., selection information) is generated and determined. This selection information is the same as the structural example in Figure 21, which is equivalent to the information used to select the image from the desired viewing angle. The server 2603 pre-sets the positions, directions, etc. of multiple virtual cameras 2601 (e.g., C0, C1, C2) corresponding to the viewpoints of each character, and prepares the image data generated by rendering accordingly, and registers it in the DB.

[0472] Additionally, user 230 can set a virtual camera Cf corresponding to a free virtual viewpoint. For such a virtual camera, no virtual camera 2601 or image data is pre-registered. In this case, server 2603 sets the virtual camera Cf according to the position, direction, etc., specified by user 230 through the UI, and generates image data of the virtual camera Cf's perspective through rendering.

[0473] When selecting virtual camera 2601, you can select the number of virtual camera 2601 that has been pre-set and registered in the virtual space. Virtual camera 2601 corresponds to viewpoint in the virtual space, i.e., virtual viewpoint. When specifying / setting virtual camera Cf corresponding to a free virtual viewpoint, this specification refers to specifying at least one of the following parameters: position, direction (angle corresponding to the direction), or field of view of the virtual viewpoint in the virtual space.

[0474] When selecting a pre-defined virtual camera 2601, or when selecting a group 2610 or a member 2602 (i.e., a role), the UI can be the same as the GUI shown in Figure 20 above. Additionally, as described above, hardware buttons, voice recognition, and a portable terminal 4000 can also be used.

[0475] [Virtual camera in virtual space]

[0476] Figure 27 is an explanatory diagram of a virtual camera 2601 corresponding to a virtual viewpoint within the concert venue 2600 in the virtual space. In Figure 27, the coordinate system of the virtual space is set to (x, y, z), schematically showing the xy-plane view of the characters on the stage 2700 from above. In this example, the user 230 selects and focuses on member M2, i.e., character 2702. An individual virtual camera C2 films member M2, i.e., character 2702. The position and movement of the virtual camera C2 are preset / determined by the server 2603. Alternatively, a virtual camera Cf, which can be freely specified by the user 230, can be used to film character 2702. The user 230 can specify the position and movement of the virtual camera Cf through a defined UI.

[0477] On stage 2700, character 2702 moves in a predetermined sequence within virtual space. An example of character 2702's movement trajectory is shown using arrows. The following describes an example of setting up virtual camera C2 or virtual camera Cf corresponding to character 2702's movements. For example, at the first time point t1, virtual camera C2 / Cf is located at position p1, facing forward in the +y direction; this is the field of view for filming character 2702's front. At the second time point t2, virtual camera C2 / Cf is located at position p2, facing diagonally to the left; this is the field of view for filming character 2702's right diagonal front. At the third time point t3, virtual camera C2 / Cf is located at position p3, facing left in the -x direction; this is the field of view for filming character 2702's right side. The virtual viewpoint of virtual camera C2 / Cf moves along the arc trajectory shown in the diagram. For each virtual camera 2601, the zoom amount, or focus amount, can be set at each time point.

[0478] [GUI used to specify the virtual viewpoint]

[0479] The display system provides a UI that allows the user 230 to specify a free virtual viewpoint. As an example of the UI in the live camera switching function 1512, Figure 28 shows an example of a GUI that allows for precise selection and specification of the position and orientation of the virtual viewpoint corresponding to the virtual camera Cf. Figure 28 shows an example of a GUI 2800 displayed on the screen of the display device 3000. In this GUI 2800, the position (x, y, z) and orientation (angle α, β) of the virtual viewpoint corresponding to the virtual camera Cf can be specified via the GUI. For example, angle α is the azimuth angle, and angle β is the pitch angle. Additionally, the preview frame 2801 can display a preview of the captured image character corresponding to the temporarily specified position and orientation. The example in Figure 28 shows the character in the preview captured image. The user 230 can later change or adjust the temporarily specified position and orientation in the UI. The position and orientation of the virtual viewpoint can be specified precisely or coarsely using numerical values. For example, a pre-defined position and orientation can be selected, such as being in front, diagonally to the right, to the right, or behind the group 2610.

[0480] The position (x, y, z) and direction (angle α, β) of the virtual viewpoint corresponding to the virtual camera Cf, which can be specified using the GUI 2800, can be absolute coordinates and absolute angles in virtual space. Alternatively, the position (x, y, z) and direction (angle α, β) of the virtual viewpoint corresponding to the virtual camera Cf, which can be specified using the GUI 2800, can also be relative coordinates and relative angles relative to the reference coordinates and direction of a specific character. For example, the reference coordinates and direction of a specific character might be the position and direction of a specific bone in the skeleton data of the character's 3D data (described later). When using relative coordinates and relative angles relative to the reference coordinates and direction of a specific character, even if the specific character moves or changes direction, the virtual camera Cf can track the movement and direction of the specific character and film that specific character using the relative coordinates and relative angles specified by the user through the GUI.

[0481] According to the selection information for the virtual camera / character / virtual viewpoint, if the corresponding image data already registered in the database exists, the server 2603 reads the data and distributes it to the display device 2000. If the corresponding image data does not exist, the server 2603 determines the position and orientation of the virtual camera 2601 (Cf) corresponding to the specified virtual viewpoint in the virtual space according to the selection information. Then, the server 2603 performs rendering processing on the 3D model object in the virtual space from the virtual viewpoint of the specified virtual camera 2601 (Cf), including a series of actions and animations for each character, thereby generating a two-dimensional image from the corresponding perspective. The rendered image data is stored in the memory. The above rendering process can be implemented by the processor of the control unit 15031 through program-based processing based on the structure shown in FIG24.

[0482] [Select Information]

[0483] The above selection information is further explained.

[0484] 1. When the selection information is virtual camera selection information, the server 2603 selects one virtual camera 2601 from a pre-set list of multiple virtual camera 2601 options (e.g., virtual cameras C0, C1, C2) based on this selection information. The server 2603 obtains rendered image data based on the position, orientation, and / or field of view of the virtual viewpoint corresponding to that virtual camera 2601, and sends it to the display device 2000. In this case, only a selection from the pre-set list of multiple virtual camera 2601 options is required, thus allowing the user 230 to more easily set the desired viewpoint and view the corresponding image.

[0485] 2. When the selection information is role selection information, the server 2603 selects a role from the members 2602 of the group 2610 based on this selection information. Based on the virtual viewpoint settings of the virtual camera 2601 that captures the role, the server obtains rendered image data and sends it to the display device 2000. In this case, the user 230 only needs to specify the desired role to select the image focusing on that role's viewpoint, thus making it easier for the user 230 to select and view the desired role's image.

[0486] 3. When the selection information is virtual viewpoint selection information, the server 2603, based on this selection information and the position, orientation, and / or field of view of the virtual camera Cf corresponding to the specified virtual viewpoint, obtains the rendered image and sends it to the display device 2000. In this case, the user 230 can directly specify the virtual viewpoint, and an image corresponding to the specified virtual viewpoint can be generated, thus allowing the user 230 to view the image at the desired perspective.

[0487] When user 230 uses both the spatial levitation image display device 1000 and the display device 3000 as the display device 2000, similar to the structural example in Figure 21, it is possible to view images from two perspectives in parallel. Furthermore, during the playback of live performance images, the user 230 can switch to different perspectives by changing the virtual camera, character, or virtual viewpoint through the live camera switching function 1512 of the display device 2000, based on user input via the UI.

[0488] Similar to Figure 18B, the structural example in Figure 26 can also combine the background image 1904 and the lighting effect 1905. In this case, when the server 2603 generates images of live performances in a virtual space, it can generate control information for controlling the background image 1904 and lighting effect 1905 corresponding to the image, and register it in the database in association with the image data. Then, when the server 2603 distributes the image selected from the display device 2000, it distributes the control information along with the image data. The display device 2000 changes and outputs the background image 1904 and lighting effect 1905 in conjunction with the display of the live performance image according to the received image data and control information.

[0489] [Select Information Form]

[0490] Figure 22B shows an example of a selection information table maintained in server 2603 corresponding to the structure example in Figure 26. In the selection information table of Figure 22B, rows #1 to #3 are the same as those in Figure 22A, the difference being the selection format for the virtual camera, character, and virtual viewpoint. For the pre-set virtual cameras C0, C1, and C2, the position, direction, and field of view of their respective virtual viewpoints are pre-set. Although not shown, the temporal movement of the virtual camera 2601 can also be set (Figure 27). Row #4 is a setting example for the virtual camera Cf corresponding to the free virtual viewpoint, setting its position, direction, and field of view. Furthermore, in the setting of virtual camera Cf, the character of interest can also be set simultaneously; in this case, the image mainly captures the character of interest.

[0491] The selection information table in Figure 22B can be stored in server 2603 and sent from server 2603 to display device 2000, and also stored in display device 2000. This selection information table can be included as part of a dataset such as a 3D model on server 2603, or it can be set as independent control information.

[0492] [Rendering Processing]

[0493] As a supplementary explanation, the rendering process performed on the server (e.g., server 2603 in Figure 26) is as follows. The server renders 3D model objects in a computationally virtual three-dimensional space; in other words, it performs rendering-based processing, thereby generating image data for distribution, such as image data for displaying aerial image 3L in the spatial levitation image display device 1000. The server performs image processing on the image generated by rendering the virtual three-dimensional space configured with 3D model objects. The rendering of the virtual three-dimensional space is performed by a virtual three-dimensional space camera (virtual camera 2601 in Figure 26) set within the virtual three-dimensional space using perspective projection. The image generated by rendering is an image obtained by setting the field of view of the virtual three-dimensional space camera to meet specified conditions, based on the lens focal length of the virtual three-dimensional space camera, the distance between the user's 230-degree view and the display range of the spatial levitation image 3, and the diagonal length of the screen, and by capturing and rendering the virtual three-dimensional space configured with objects.

[0494] [Display System: Virtual Space: Second Structure Example]

[0495] As another structural example of the display system in Embodiment 3, FIG30 shows a second structural example regarding virtual space. Similar to the first structural example in FIG26, in the second structural example, the live performance in the concert venue 2600 is constructed in a virtual space calculated by the server 3003. The difference is that in the second structural example of FIG30, the virtual live performance image is generated by the display device 2000 on the user 230 side based on the dataset through rendering processing, not by the server 3003. The second structural example of FIG30 can be applied to any method of live broadcast distribution or on-demand distribution. The display device 2000, as described above, has a live camera function 1511 and a live camera switching function 1512, which are omitted from the illustration.

[0496] Server 3003 stores 3D models of the concert venue 2600 and characters configured in the virtual space, motion information of the 3D model objects, and virtual camera information in the database. This data is a series of raw data required for rendering virtual live performance images on the display device 2000 side, sometimes referred to as a dataset.

[0497] In the display device 2000 used by user 230, namely the spatial levitation image display device 1000 and display device 3000, based on the operation input made by user 230 via the UI, a request message 3006 is sent to server 3003 regarding the virtual live performance that the user wants to watch. The request message 3006 is a dataset sending request message or a data stream sending request message. Unlike the example of FIG26, the example of FIG30 performs rendering processing in the display device 2000. Therefore, the display device 2000 of FIG30 does not need to include the above-mentioned selection information in the request message 3006.

[0498] In response to the request information 3006 received from the display device 2000, the server 3003 sends a dataset 3005 corresponding to the requested virtual live performance to the display device 2000. The dataset 3005 sent may also be a data stream described later. The dataset 3005 sent here is a dataset capable of generating images corresponding to various selectable viewpoints in the display device 2000. Additionally, metadata (containing options for generating the selection information table of Figure 22B—such as preset virtual camera information) may be included in the dataset 3005 and sent to the display device 2000.

[0499] Display device 2000 stores the dataset 3005 received from server 3003 in its memory. User 230 can select and specify the virtual live performance of group 2610, virtual camera 2601, or member 2602 they wish to watch through the UI of display device 2000. This UI can be generated based on the selection information table mentioned above. In the UI of display device 2000, the user selects / specifies the virtual live performance of group 2610, virtual camera 2601, or member 2602 they wish to watch, and the corresponding selection information is used for rendering processing in display device 2000. Based on the selection information corresponding to the selection / specification in the UI of display device 2000, the image processing unit 3020 of display device 2000 uses the dataset 3005 to generate image data 3030 for displaying the virtual live performance image from the selected viewpoint on the display unit through rendering processing. As image data 3030, image data corresponding to the virtual camera / character / virtual viewpoint selected / specified by user 230 is generated. The display device 2000 displays a virtual live performance image on the display screen of the display unit based on the image data 3030. In the case of the spatial levitation image display device 1000, the image is displayed as an aerial image 3L.

[0500] Furthermore, regarding the live camera switching function 1512, during the playback of virtual live performance images, the display device 2000 can select / specify a virtual camera, character, or virtual viewpoint requested by the user 230 based on the operation input made by the user 230 via the UI, generating and determining new selection information. For the new selection / specification made by the user 230 via the UI of the live camera switching function 1512, the image processing unit 3020 of the display device 2000 performs rendering processing based on the corresponding selection information, generating image data 3030 for displaying the virtual live performance image of the newly selected viewpoint on the display unit. As image data 3030, image data corresponding to the newly selected / specified virtual camera / character / virtual viewpoint requested by the user 230 is generated. The display device 2000 displays the virtual live performance image on the display screen of the display unit based on this image data 3030. In the case of the spatial levitation image display device 1000, this image is displayed as an aerial image 3L.

[0501] Here, for the new selection / specification made by user 230 via the UI of the on-site camera switching function 1512, rendering processing is performed based on the corresponding selection information. If the dataset 3005 received by display device 2000 from server 3003 lacks the data required for this rendering processing, display device 2000 sends a request message 3006 to server 3003 requesting the missing data. Server 3003 sends the dataset 3005 required to generate the image of the selected viewpoint to display device 2000 based on the request message 3006. Based on the received dataset 3005, display device 2000 generates image data 3030 for the image of the selected viewpoint through rendering, and switches to display the image of the selected viewpoint according to the image data 3030.

[0502] In other structural examples, selection information can also be included in the request information 3006 sent to the server 3003. The display device 2000 first selects a virtual camera, character, or virtual viewpoint requested by the user 230 based on the operation input made by the user 230 via the UI, generates and determines the selection information, and sends the request information 3006 containing the selection information to the server 3003. Based on the request information 3006, the server 3003 sends a dataset 3005 to the display device 2000, containing data capable of generating an image of at least the selected viewpoint. The image processing unit 3020 of the display device 2000 generates image data 3030 based on the received dataset 3005 and the initial selection information, for an image corresponding to the viewpoint requested by the user 230. The display device 2000 displays the image of that viewpoint based on the image data 3030.

[0503] Similarly, regarding the live camera switching function 1512, during the playback of virtual live performance images, the display device 2000 generates and determines corresponding new selection information based on the selection and specification of virtual cameras, etc., requested by the user 230 in the UI. Based on the dataset 3005 and this new selection information, the display device 2000 generates image data 3030 for the image of the newly selected viewpoint, and switches to display the image of the selected viewpoint according to the image data 3030. Furthermore, if the dataset 3005 lacks the data required to generate the image of the newly selected viewpoint, the display device 2000 can, in the same manner as described above, send a request message 3006 regarding the missing data to the server 3003 to obtain the dataset 3005 containing the missing data from the server 3003.

[0504] In the second structural example of FIG30, the display device 2000 may, for example, maintain the selection information table shown in FIG22B and perform the selection processing corresponding to the selection information as described above. Regarding the selection and rendering processing of the virtual camera / character / virtual viewpoint, the display device 2000 in FIG30 can perform the same processing as the server 2603 in the first structural example of FIG26.

[0505] In the second structural example of Figure 30, in the user 230's environment, the display device 2000 with display function processes the image data 3030 of the virtual live performance generated by rendering based on the dataset 3005. In other structural examples, the device performing this rendering and other processing may not be the display device 2000, but other devices without display function (referred to as receiving devices or communication devices). For example, in the user 230's environment, a receiving device may be included as another device connected to the display device 2000. The receiving device sends request information 3006 to the server 3003 and receives the dataset 3005 from the server 3003. Based on the dataset 3005 and selection information, the receiving device generates image data 3030 of the image from the selected viewpoint through rendering processing and outputs it to the display device 2000. The display device 2000 displays the image on the display unit according to the image data 3030.

[0506] [3D Model and Motion Information]

[0507] As a supplementary explanation, Figure 31 is an explanatory diagram of a character's 3D model and motion information. Figure 31 schematically illustrates the skeleton data in a 3D model of a character (e.g., member M1). Skeleton data is, for example, data called bones or rigs. Body parts such as the head, hands, and feet, composed of vertices, edges, and faces (not shown), are associated with the skeleton data. When the position and orientation of each part of the skeleton data changes, the corresponding vertices, edges, and faces of the body parts deform, which can appear as if the 3D model of the character is moving. Motion information includes the position, orientation, etc., of each part of the skeleton data at each time point. The motion information describes the temporal changes of these skeleton data. Motion information can also be generated based on images captured by a motion capture system of actual human movements. A two-dimensional image of the character can be generated by rendering based on the skeleton data and the character's 3D model data at each time point. In addition, the skeleton data needs to be in a data format that can be well understood by both the server 3003 side and the display device 2000 side. For example, the Humanoid format is used in the example in Figure 31. The motion information also uses a format corresponding to that of the skeleton data. Therefore, the display device 2000 can readily understand the motion information transmitted from the server 3003.

[0508] In the second structural example of Figure 30, in the case of on-demand distribution, a dataset 3005 containing 3D models and motion information of characters can be distributed from server 3003 to display device 2000 at a time specified by user 230, and then a virtual live performance video can be generated and viewed from the requested perspective at a time specified by user 230.

[0509] Not limited to this, in the case of live distribution, for example, the date and time of the virtual live performance event are predetermined on the server 3003 side. Before the event time, a pre-set dataset 3005 for the virtual live performance is first distributed from the server 3003 to the display device 2000. The pre-set dataset 3005 contains 3D models of the concert venue 2600 and characters configured in the virtual space for the virtual live performance, as well as objects of these 3D models. On the display device 2000 side, the rendering data contained in the dataset 3005 is pre-loaded into memory before the event time. Therefore, after the event time (which is later than the distribution of the dataset 3005), a data stream for the virtual live performance can be distributed from the server 3003 to the display device 2000 in real time. The action information of the characters can be included in the distributed data stream and sent, which is received by the display device 2000. The display device 2000 continuously reflects the real-time received action information in the rendering process. This enables the generation of images where 3D objects of characters contained in a pre-received dataset 3005 move in real time based on that motion information. Additionally, sound data from the virtual live performance can also be included in the data stream and transmitted and received in real time, along with motion information.

[0510] Display device 2000 can generate a two-dimensional image of the selected character's animation through rendering processing based on the 3D model and motion information received from the data stream of server 3003, and can play the image and sound synchronously. Therefore, display device 2000 can display virtual live performance images in approximately real-time during the event, and user 230 can watch the images. Furthermore, during the playback of the virtual live performance, regarding the live camera switching function 1512, display device 2000 can perform the same selection processing based on the user 230's operation input via the UI, thereby switching the display perspective.

[0511] Furthermore, the background image 1904 and lighting effect 1905, as described in Figure 18B, can be applied in the same way. In this case, the control information for the background image 1904 and lighting effect 1905 can be generated on the server 3003 side and distributed in the dataset 3005, or it can be generated on the display device 2000 side.

[0512] [Display system of the variant]

[0513] Figure 32 illustrates the structure of a display system in a modified example of Embodiment 3. For example, the systems in Figures 21 and 26 are configured such that the spatial levitation image display device 1000 and display device 3000, as display devices 2000, jointly receive image data rendered on the server side and play the image. On the other hand, the system in Figure 30 is configured such that the spatial levitation image display device 1000 and display device 3000, as display devices 2000, jointly receive unrendered data (i.e., a dataset) from the server side, render it on the display device 2000 side, and play the image. That is, in these structures, multiple display devices 2000 have the same function. In contrast, in the modified example of Figure 32, the spatial levitation image display device 1000 and the television (i.e., display device 3000), which are multiple display devices 2000, have different functions, corresponding to a combination of a structure rendering on the server side and a structure rendering on the display device 2000 side. In the structural example of Figure 32, the television set, i.e., the display device 3000, has a function corresponding to the structure shown in Figures 21 and 26 where rendering is performed on the server side, and the spatial levitation image display device 1000 has a function corresponding to the structure shown in Figure 30 where rendering is performed on the display device 2000 side. The server 3203 in Figure 32 has a structure corresponding to both the function of distributing rendered image data and distributing unrendered datasets.

[0514] The television display device 3000, for example, sends a request message 3201 to the server 3203, and receives image data 3202 rendered based on dataset 3210 from the server 3203 in response to the request message 3201. Based on the image data 3202, it displays the live camera image (the aforementioned virtual live performance image) on the display screen of the display unit. On the other hand, the spatial levitation image display device 1000, for example, sends a request message 3211 to the server 3203, and receives an unrendered dataset 3212 or data stream obtained based on dataset 3210 from the server 3203 in response to the request message 3211. Based on the dataset 3212 or data stream, it generates image data 3213 through rendering, and displays the live camera image as aerial image 3L on the display unit based on the image data 3213.

[0515] In other structural examples, the functions of the spatial levitation image display device 1000 and the display device 3000 in Figure 32 can be interchanged. The television, i.e., the display device 3000, generates and displays camera images by rendering based on the received dataset. The spatial levitation image display device 1000 displays camera images based on the received rendered image data.

[0516] [Display System: Virtual Space: Third Structure Example]

[0517] As another structural example of the display system in Embodiment 3, Figure 33 shows a third structural example regarding the virtual space. In the third structural example, similar to the structural examples in Figures 21 and 26 above, a live performance of the concert venue 2600 is constructed in the virtual space of server 3303. The structural example of the virtual space of server 3303 is the same as that in Figure 26. The difference in the third structural example is that the display device 2000 in the environment of user 230 is a display of camera images that is collaboratively realized by the spatial floating image display device 1000 and the television, i.e., display device 3000, through communication. In the third structural example, the fixed pixel surface display device, i.e., display device 3000, is referred to as the first screen device, and the spatial floating image display device 1000 is referred to as the second screen device. Each display device 2000 in Figure 33 has the above-mentioned live camera function 1511 and live camera switching function 1512, and at least one of them has a UI for selection, but these are omitted from the illustration.

[0518] Based on the operation of user 230, the first screen device, i.e., display device 3000, sends request information 3301 to server 3303. Request information 3301 is a dataset / data stream transmission request. In response 3302, server 3303 sends the requested dataset or data stream to display device 3000. The response 3302 from server 3303 to display device 3000 includes both data for the first screen device (referred to as first data) and data for the second screen device (referred to as second data).

[0519] Based on the user's operation 230, the second screen device, namely the spatial levitation image display device 1000, sends a request message 3311 to the first screen device, namely the display device 3000. The request message 3311 is a request for a partial dataset / partial data stream. In response 3312, the display device 3000 sends the requested partial dataset or partial data stream to the spatial levitation image display device 1000. The data from the partial dataset or partial data stream in response 3312 sent by the display device 3000 to the spatial levitation image display device 1000 is data extracted from the dataset / data stream from the aforementioned server 3303 to obtain second data for the second screen device.

[0520] In the example of Figure 33, the image processing unit of the display device 3000 generates image data for displaying virtual live performance images (e.g., an overall viewpoint) on the display device 3000 based on the first data in the dataset / data stream received from the server 3303, and displays the virtual live performance images on the display screen according to the image data. On the other hand, the image processing unit of the spatial levitation image display device 1000 generates image data for displaying virtual live performance images (e.g., the viewpoint of a selected character) on the spatial levitation image display device 1000 based on the second data in the partial dataset / partial data stream received from the display device 3000, and displays the aerial image 3L of the virtual live performance images according to the image data.

[0521] In the example of Figure 33, rendering can be performed on the server 3303 side as shown in Figure 26. In this case, as a dataset / data stream, the display device 3000 receives first image data and second image data, which consist of rendered image data, from the server 3303. The display device 3000 displays the virtual live performance image locally (display device 3000) based on the first image data. As a partial dataset / partial data stream, the second screen device, namely the spatial levitation image display device 1000, receives the second image data from the display device 3000 and displays the aerial image 3L of the virtual live performance image locally (spatial levitation image display device 1000) based on the second image data.

[0522] In the example of Figure 33, rendering can also be performed on the display device 2000 side instead of the server 3303, as shown in Figure 30. In this case, as a dataset / data stream, the display device 3000 receives first data and second data consisting of unrendered data from the server 3303. Based on the first data in the dataset received from the server 3303, the image processing unit of the display device 3000 generates image data for displaying virtual live performance images (e.g., an overall view) on the display device 3000 through rendering, and displays the virtual live performance images on the display screen according to the image data. On the other hand, as a partial dataset / partial data stream received from the display device 3000, the image processing unit of the spatial levitation image display device 1000 receives second data consisting of unrendered data. Based on the second data received from the display device 3000, image data for displaying virtual live performance images (e.g., the viewpoint of a selected character) on the spatial levitation image display device 1000 is generated through rendering, and the aerial image 3L of the virtual live performance images is displayed according to the image data.

[0523] In this way, when there are multiple display devices 2000 on the user 230 side, it is possible to communicate with the server 3303 through only one display device 2000, and to perform display through cooperation between the one display device 2000 and other display devices 2000.

[0524] As a variation, the opposite of the above can be adopted, in which the spatial levitation image display device 1000 is used as the first screen device and the display device 3000 is used as the second screen device, and the spatial levitation image display device 1000 receives / obtains data from the server 3303.

[0525] In the structural example of Figure 33, the UI for selecting the required camera, etc., related to the on-site camera switching function 1512, can be at least one of the UI in the display device 3000 and the UI in the spatial levitation image display device 1000. Each display device 2000 may have its own UI, or the UI may be centralized in one display device 2000. For example, a GUI serving as a centralized UI can be provided in the television, i.e., the display device 3000, where not only the viewing angle for display on the display device 3000 can be selected, but also the viewing angle for displaying the aerial image 3L on the spatial levitation image display device 1000. The GUI can similarly apply to the example of Figure 20, etc.

[0526] [Dataset / Data Flow]

[0527] Figure 34A shows a table summarizing specific structural examples of the datasets / data streams that can be applied in the collaborative display structure of the first screen device (display device 3000) and the second screen device (spatial levitation image display device 1000) in Figure 33.

[0528] (Example 1) For datasets / data streams sent from server 3303 to the first screen device, MMT (MPEG Media Transport) is applied to transmit video, sound, data, and / or motion information. For datasets / data streams sent from the first screen device to the second screen device, Partial MMT is applied to transmit video, sound, data, and / or motion information. As a note, NTP (Network Time Protocol) format (UTC base) is used in synchronization control.

[0529] (Example 2) For datasets / data streams sent from server 3303 to the first screen device, CMAF (Common Media Application Format) is applied to transmit image, sound, data, and / or motion information. For datasets / data streams sent from the first screen device to the second screen device, a partial CMAF is applied to transmit image, sound, data, and / or motion information. As a note, NTP (Network Time Protocol) format (UTC base) is used in synchronization control.

[0530] (Example 3) For the dataset / data stream sent from server 3303 to the first screen device, MMT, CMAF, etc., are used to transmit the IP data stream or data file (containing motion information and sound). In the dataset / data stream sent from the first screen device to the second screen device, motion information, data, and / or sound are transmitted; this transmission does not include video. Note that the video is rendered and played on both the first and second screen devices separately. Furthermore, the video on the first screen device and the video on the second screen device are played synchronously.

[0531] (Example 4) For datasets / data streams sent from server 3303 to the first screen device, MMT, CMAF, etc., are used to transmit IP data streams or data files (containing images, sound, and motion information). In the datasets / data streams sent from the first screen device to the second screen device, motion information, data, and / or sound are transmitted; this transmission does not include images. Note that the first screen device receives and plays a stream of rendered images. Unrendered data containing motion information is sent from the first screen device to the second screen device, and images are played in the second screen device based on the unrendered data through rendering. Furthermore, the images from the first screen device and the images from the second screen device are played synchronously.

[0532] Examples 1 and 2 above can be adopted using one of the following 1 to 4.

[0533] 1. Both the spatial levitation image display device 1000 and the display device 3000 receive the image data of the rendered image and play the image respectively.

[0534] 2. Both the spatial levitation image display device 1000 and the display device 3000 receive unrendered data, render it, and play the image respectively.

[0535] 3. The display device 3000 receives the rendered image and plays the image, while the spatial floating image display device 1000 receives the unrendered data, renders it, and plays the image.

[0536] 4. The display device 3000 receives the unrendered data, renders it, and plays the image. The spatial floating image display device 1000 receives the rendered image and plays the image.

[0537] Regarding Example 3 above, as shown in 2. above, both the spatial levitation image display device 1000 and the display device 3000 receive unrendered data, render it, and play the image respectively. Regarding Example 4 above, as shown in 3. above, the display device 3000 receives the rendered image and plays it, while the spatial levitation image display device 1000 receives unrendered data, renders it, and plays the image.

[0538] (Example 4): Display System

[0539] Figure 34B illustrates a summary of data transmission and reception in the display system using Example 4 described above. This is an example of displaying an overall view image in the display device 3000 and individual view images in the spatial levitation image display device 1000. The dataset / data stream 3302 sent from the server 3303 to the display device 3000 contains first data, which is rendered image data, and second data, which is an unrendered dataset. The dataset / data stream 3312 sent from the display device 3000 to the spatial levitation image display device 1000 contains second data, which is an unrendered dataset. The image processing unit of the display device 3000 plays a virtual live performance image based on the first data. The spatial levitation image display device 1000 renders and plays the virtual live performance image based on the second data. The display device 3000 and the spatial levitation image display device 1000 synchronously play the image based on the first data and the image based on the second data.

[0540] (Example 4): Sending and receiving data

[0541] Figures 35A and 35B illustrate an example of the sending-side processing and receiving-side processing for implementing the display collaboration and synchronized playback in Example 4 above. Figure 35A shows an example of the processing by the server 3303 as an example of the sending-side system's processing. The sending-side system 3510 (specifically, the server 3303) performs the processing as described below. In Example 4, a data stream 3504 containing image and sound information 3501 and motion information 3502 is sent from the server 3303 to the display device 3000. This data stream 3504 includes the first data and the second data shown in Figure 34B. For example, the image and sound information 3501 is transmitted as part of the first data, and the motion information 3502 is transmitted as part of the second data. The character's motion information 3502 is transmitted from the display device 3000 to the spatial levitation image display device 1000 as the second data, and there is a possibility of transmission delay during this transmission. Furthermore, in order to reflect the character's motion information 3502 in the image generation that is linked to the video's image and sound information 3501, rendering processing is required in the receiving system (in Example 4, the spatial levitation image display device 1000). Therefore, there is a possibility that a playback delay greater than that of the image and sound may occur.

[0542] Therefore, in this embodiment, for the data 3503 (represented by squares A, V, M in Figure 35A) such as fragments or packets of the video and audio information 3501 and motion information 3502 that should be played (reproduced) synchronously, when transmitting them in the data stream 3504, the motion information 3502 is configured to be transmitted at a position more than a predetermined time (e.g., more than 1 second) earlier than the video and audio information 3501 at the transmission time, regarding the storage position of the data 3503 in the data stream 3504. In each fragment or packet of data 3503, time information that can be used in the synchronization control on the playback side, such as NTP format (UTC reference), can be attached or stored. In this way, even if the data positions at the time of transmission are separated in time, the playback side can still perform synchronization control. This transmitted data, i.e., the data stream 3504, is transmitted from the server 3303 to the receiving system via the communication network 1509 (Figure 35B).

[0543] As an example of the processing of the receiving-side system, Figure 35B shows a processing example of the display device 2000. The receiving-side system 3520 (specifically, the first screen device, i.e., the display device 3000, and the second screen device, i.e., the spatial levitation image display device 1000) performs the processing as described below. In the data stream 3504 transmitted by the transmitting-side system 3510 in Figure 35A, data 3503 of motion information 3502 that was transmitted more than a predetermined time (e.g., 1 second) before the aforementioned segments or groups of image information and / or segments or groups of sound information is stored. In the receiving-side display device 2000, the display device 3000 of Example 4 extracts the data 3503 containing motion information 3502 from the data stream 3504 and transmits it to the spatial levitation image display device 1000. The spatial levitation image display device 1000 renders the motion information 3502 reproduced from the data 3503, thereby generating and displaying an image. In sync with the display of the image generated by rendering based on motion information 3502 in the spatial levitation image display device 1000, the display device 3000 displays or outputs the image information and / or sound information stored in the segments or groups of image information and / or sound information received after receiving motion information 3502.

[0544] In the example of Figure 35B, as also shown in Figure 34B, the first screen device of the display device 2000, namely the display device 3000, generates and displays the virtual live performance image 3521 according to the rendered image and sound information 3501. The second screen device, namely the spatial levitation image display device 1000, uses the motion information 3502 received from the first screen device, namely the display device 3000, to render and generate the virtual live performance image 3522, which is displayed as an aerial image 3L. The spatial levitation image display device 1000 plays the image 3522 synchronously with the image 3521 on one side of the display device 3000. In addition, the 3D data required for rendering based on the motion information 3502 (e.g., 3D models of concert venues and characters configured in the virtual space of the virtual live performance, objects of the 3D models, etc.) can be received in advance as a dataset by the spatial levitation image display device 1000 before transmitting the data stream 3504 of the motion information 3502 in real time. The spatial levitation image display device 1000 can r...

Claims

1. A display system including a display device, characterized in that: The display device receives image data of images captured by a camera, i.e., camera images, and displays the camera images on a display screen based on the image data. Based on user input to the display device, it selects a camera, viewpoint, or subject from multiple cameras, multiple viewpoints, or camera subjects as the target camera, receives image data of the camera images corresponding to the target camera, viewpoint, or subject according to the selection, and displays the images on the display screen based on the received image data.

2. The display system as described in claim 1, characterized in that: The display device is an aerial levitation image display device, comprising: an image processing unit that performs image processing based on image data; a display unit that displays the image processed by the image processing unit; and an optical system that generates an aerial levitation image based on the image displayed by the display unit.

3. The display system as described in claim 1, characterized in that: The images captured by the camera are images of a live concert or a sporting event.

4. The display system as described in claim 1, characterized in that: The images captured by the camera are images of real people or characters created in the virtual space of the server.

5. The display system as described in claim 4, characterized in that: The subjects are multiple members of a group consisting of real people or characters constructed in the virtual space of a server. The camera has multiple cameras, including a camera that captures the entire group or the activities performed by the group as a whole, and a camera that captures individual members by camera.

6. The display system as described in claim 1, characterized in that: The system includes a server that receives image data from the camera, performs image processing on the image data to generate image data for distribution, and distributes the image data for distribution to the display device.

7. The display system as described in claim 2, characterized in that: The display device has a display panel that displays a background image on the back side of the aerial suspended image, and is controlled to display the background image corresponding to the aerial suspended image.

8. The display system as described in claim 2, characterized in that: The display device has a lighting effect device for generating lighting effects for the suspended aerial image, and controls the device to generate lighting effects corresponding to the suspended aerial image.

9. The display system as described in claim 1, characterized in that: The display device is capable of displaying a graphical user interface (GUI) for selecting the camera, viewpoint, or subject from the plurality of cameras, multiple viewpoints, or multiple subjects based on the user's input.

10. The display system as claimed in claim 1, characterized in that: The system includes a server that constructs the subject and a virtual camera that captures the subject in a virtual space, maintains a dataset including a 3D model of the subject, motion information of the subject, and virtual camera information of the virtual camera, and generates image data for distribution of camera images of the subject through rendering processing based on the dataset.

11. The display system as claimed in claim 1, characterized in that: The system includes a server that constructs the subject and a virtual camera that captures the subject in a virtual space, maintains a dataset including a 3D model of the subject, motion information of the subject, and virtual camera information of the virtual camera, and distributes the dataset to the display device. The display device generates image data of the camera image of the subject based on the dataset through rendering processing.

12. The display system as claimed in claim 1, characterized in that: The display device has a first screen and a second screen. The display system includes a server. The server sends first data and second data to the first screen. The first screen receives the first data and the second data and displays an image on the first screen, which serves as a display screen, based on the image data of the camera image included in the first data. The second screen receives the second data from the first screen and displays an image different from the image displayed on the first screen on the second screen, which serves as a display screen, based on the second data, thereby synchronizing the display of the image on the first screen with the display of the image on the second screen.

13. The display system as described in claim 12, characterized in that: The system includes a server that sends image data of the camera image to the first screen device as the first data, and sends motion information as the second data at least a predetermined time in advance of the image data. The first screen device and the second screen device use the image data and the motion information sent at least a predetermined time in advance of the image data to synchronize the display of the image on the first screen with the display of the image on the second screen.

14. The display system as claimed in claim 1, characterized in that: The display device switches between displaying images from the camera corresponding to the camera, viewpoint, or subject based on the user's input.

15. The display system as claimed in claim 1, characterized in that: The system includes AI, which analyzes images captured by the camera and selects camera images to be distributed to the display device based on the time period of the images.

16. The display system as claimed in claim 1, characterized in that: The system includes AI, which selects camera images to be distributed to the display device from images captured by the camera based on input information about the user's interests, including the display device.

17. The display system as claimed in claim 4, characterized in that: The AI ​​includes multiple members of a group consisting of real people or characters constructed in a virtual space on a server. The AI ​​selects content information associated with camera footage distributed to the display device based on input information of the user's interests, including the display device. Candidates for the content information include information about the group, information about the members, and explanatory information about the activities performed by the group.

18. The display system as claimed in claim 1, characterized in that: The display device displays an image based on the camera corresponding to the camera, viewpoint, or subject, and displays an evaluation input image for inputting the user's evaluation of the displayed content of the image.

19. The display system as claimed in claim 18, characterized in that: The system includes a server that collects and accumulates user evaluation information for the evaluation input image input from multiple display devices of multiple users, which are the display devices, and analyzes the user evaluation information.

20. The display system as claimed in claim 1, characterized in that: The display device records the image data in a recording medium, plays and displays the recorded image in the recording medium, and for the played and displayed recorded image, selects the camera, viewpoint or subject as the object from the plurality of cameras, multiple viewpoints or multiple subjects based on the operation input of the user, receives image data of the camera image corresponding to the camera, viewpoint or subject as the object according to the selection, and displays the image based on the camera on the display screen based on the received image data.

21. A display device, characterized in that: The device receives image data, i.e., camera images, captured by a camera; displays the camera images on a display screen based on the image data; selects a camera, viewpoint, or subject from multiple cameras, multiple viewpoints, or camera subjects based on user input to the display device; receives image data of the camera images corresponding to the selected camera, viewpoint, or subject; and displays the images on the display screen based on the received image data.

22. A display system, characterized in that, include: A first display device receives image data via communication and displays an image based on the image data; A second display device receives motion information from the first display device and plays and displays images based on the motion information.

23. The display system as described in claim 22, characterized in that: The motion information sent from the first display device to the second display device is received by the first display device via the communication, along with the image data.

24. The display system as described in claim 22, characterized in that: Simultaneously play the images from the first display device and the second display device.

25. The display system as described in claim 24, characterized in that: The motion information used to synchronize playback with the image displayed on the first display device is received by the first display device via the communication at least a predetermined time before the image data of the image.

26. The display system as described in claim 22, characterized in that: The second display device uses the motion information received from the first display device to perform rendering processing, and plays and displays the image.

27. The display system as claimed in claim 22, characterized in that: The motion information is sent from the first display device to the second display device, but the image data is not sent.

Images