Display system and display device

Abstract

This invention provides a more suitable display device. The present invention contributes to, among the Sustainable Development Goals (SDGs), "3. Good health and well-being for all", "9. Creating the basis for industry and technology innovation", and "11. Creating sustainable cities and communities". In this display system, a display device (for example, a spatial floating image display device): receives video data of a camera video captured by a camera; displays the camera video on a display screen on the basis of the video data; selects a target camera, viewpoint, or subject from a plurality of cameras, a plurality of viewpoints, or camera subjects on the basis of an operation input by a user; receives video data of the camera video corresponding to the target camera, viewpoint, or subject in accordance with the selection; and displays the video on the display screen on the basis of the received video data.

Description

Display system and display device

[0001] The present invention relates to a floating-in-the-air image display device.

[0002] The floating information display technology is disclosed in, for example, Patent Document 1.

[0003] Japanese Patent Application Laid-Open No. 2019-128722

[0004] However, the disclosure of Patent Document 1 does not sufficiently consider configurations for obtaining practical brightness and quality for the floating image, or configurations for allowing users to view the floating image more enjoyably.

[0005] An object of the present invention is to provide a more suitable display device such as a floating-in-the-air image display device.

[0006] In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problems, and one example thereof may be configured as follows: A display system including a display device, wherein the display device receives video data of camera video, which is video captured by a camera, displays the camera video on a display screen based on the video data, selects a target camera, viewpoint, or subject from multiple cameras, multiple viewpoints, or camera subjects based on an operation input by a user of the display device, receives video data of camera video corresponding to the target camera, viewpoint, or subject in accordance with the selection, and displays the video on the display screen based on the received video data.

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments.

[0008] FIG. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a projection diagram of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a top view of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a perspective view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 9 is a top view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 10 is a side view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of a configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention.1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 3 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 4 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 5 is a layout diagram showing main parts of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 7 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 8 is an explanatory diagram for explaining light source diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 9 is an explanatory diagram for explaining diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram for an example of a problem solved by image processing according to an embodiment of the present invention. FIG. 11 is an explanatory diagram for an example of image processing according to an embodiment of the present invention. FIG. 12 is an explanatory diagram for an example of image display processing according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of the configuration of a main part and a retro-reflection part of a space-floating image display device according to an embodiment of the present invention. FIG. 14 is an explanatory diagram for a display system including a space-floating image display device according to an embodiment of the present invention. FIG. 15 is an explanatory diagram for an example of camera arrangement according to an embodiment of the present invention. FIG. 16 is an explanatory diagram for an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 1 is an explanatory diagram of an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram of an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is an explanatory diagram of an example of a control according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of an example of a GUI according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of an example of the configuration of a display system according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of an example of a selection information table according to an embodiment of the present invention. FIG. 8 is an explanatory diagram of an example of a selection information table according to an embodiment of the present invention. FIG. 9 is an explanatory diagram of an example of the configuration of a display device that is a television according to an embodiment of the present invention. FIG. 10 is an explanatory diagram of an example of the configuration of a server according to an embodiment of the present invention. FIG. 11 is an explanatory diagram regarding transmission and reception of data between a server and a display device according to an embodiment of the present invention. FIG. 12 is an explanatory diagram of an example of the configuration of a display system according to an embodiment of the present invention. FIG. 13 is an explanatory diagram of a virtual camera in a virtual space according to an embodiment of the present invention.FIG. 1 is an explanatory diagram of an example GUI according to an embodiment of the present invention. FIG. 2 is an explanatory diagram of an example configuration of an aerial operation detection sensor according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of an example configuration of a display system according to an embodiment of the present invention. FIG. 4 is an explanatory diagram of motion information according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of an example configuration of a display system according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of an example configuration of a display system according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of an example configuration of a display system according to an embodiment of the present invention. FIG. 8 is an explanatory diagram of a specific example of data distribution according to an embodiment of the present invention. FIG. 9 is an explanatory diagram of data distribution in a display system according to an embodiment of the present invention. FIG. 10 is an explanatory diagram of data transmission by a sending system according to an embodiment of the present invention. FIG. 11 is an explanatory diagram of data reception by a receiving system according to an embodiment of the present invention. FIG. 12 is an explanatory diagram of a display system including a space-floating image display device according to an embodiment of the present invention. FIG. 13 is an explanatory diagram of an example usage of a display system according to an embodiment of the present invention. FIG. 14 is an explanatory diagram of an example usage of a display system according to an embodiment of the present invention. FIG. 15 is an explanatory diagram of an example of camera image switching according to an embodiment of the present invention. FIG. 16 is an explanatory diagram of an example of camera image display ON / OFF according to an embodiment of the present invention. FIG. 17 is an explanatory diagram of an example configuration of a display system using AI according to an embodiment of the present invention. FIG. 18 is an explanatory diagram of an example of distribution image selection that uses AI according to an embodiment of the present invention. FIG. 19 is an explanatory diagram of an example of distribution image selection that varies for each user using AI according to an embodiment of the present invention. FIG. 1 is an explanatory diagram of an example of video-related content distribution according to an embodiment of the present invention. FIG. 2 is an explanatory diagram of an example of a GUI when AI is used according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of an example of an evaluation input image when AI is used according to an embodiment of the present invention. FIG. 4 is an explanatory diagram of an example of an evaluation input image when AI is not used according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of an example of an evaluation input image when AI selection is evaluated according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of an example of using evaluation information according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of an example of distribution video selection using AI in the case of on-demand distribution according to an embodiment of the present invention. FIG. 8 is an explanatory diagram of an example of using video recorded on a recording medium according to an embodiment of the present invention. FIG. 9 is an explanatory diagram of an example of using AI and video recorded on a recording medium according to an embodiment of the present invention.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0010] The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member. In the following explanation of the examples, the image floating in space is expressed using the term "floating image in space." Instead of this term, it is also acceptable to express it as "aerial image," "spatial image," "floating image in space," "floating optical image of displayed image," "floating optical image of displayed image," etc. The term "floating image in space," which is mainly used in the explanation of the examples, is used as a representative example of these terms.

[0011] According to the following embodiments, a suitable image display device can be realized for, for example, bank ATMs, train station ticket machines, digital signage, and the like. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels, but by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflector, resulting in high light utilization efficiency and suppressing the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflection systems, thereby achieving a clear floating image. Furthermore, a device including the light source of this embodiment can provide a novel, highly usable floating image display device (floating image display system) that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle. Example 1

[0012] <Example of Usage of the Space-Floating Image Display Device> Figure 1 is a diagram showing an example of usage of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Figure 2 and other figures. Light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, reflected by the optical system within the space-floating image display device, and then incident on the retroreflector 2, retroreflected and transmitted through a transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface. In the following examples, the retroreflector 2 (retroreflector) will be used as an example of the retroreflector. However, the retroreflector 2 of the present invention is not limited to a planar plate, and is used as an example of a concept including a sheet-like retroreflector attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflector is attached to a planar or non-planar member. Furthermore, since the light rays reflected by the retroreflector 2 have the optical property of forming an image, the retroreflector 2 may also be expressed as an imaging optical member or an imaging optical plate.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a transparent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves in the window glass 105, it is possible to form an aerial image at a desired position inside the store by reflecting the specific polarized waves.

[0015] <Configuration Example of Optical System of Space-Floating Image Display Device> Figure 2A is a diagram showing an example of the configuration of an optical system of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using Figure 2A. As shown in Figure 2A (1), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

[0016] Image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100) provided on a transparent member 100 and has a film that selectively reflects image light of a specific polarization, and then enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when entering the retroreflector 2 and once when exiting. Here, the polarization separation member 101, which selectively reflects image light of a specific polarization, has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a spatially floating image 3, which is a real image, outside the transparent member 100. 2A shows an example in which the chief ray of the image light incident on the retroreflector 2 is incident at an angle of 90° to the retroreflector 2. However, the incident angle of the chief ray of the image light on the retroreflector 2 is not limited to 90°, and an angle of, for example, 90°±15° can also be used.

[0017] Here, a first example of polarization design for the optical system of FIG. 2A will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separation member 101, which may have the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized light to P-polarized light. The P-polarized image light then travels back toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting S-polarized light and transmitting P-polarized light, so the P-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0018] Next, a second example of polarization design for the optical system of FIG. 2A will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and proceeds toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the image light from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light, so the S-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0019] The light forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2A, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in systems that display images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may become irregular. The reflection angle may also become irregular. Such irregular light may not maintain the polarization state and propagation angle assumed in the design. For example, light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 directly from the position of the retroreflector 2 without passing through the polarization separation member. Such light with an unintended polarization state and propagation angle may be reflected by components within the space-floating image display device and then re-enter the image display surface of the liquid crystal display panel 11. Such light re-entering the image display surface of the liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, potentially generating ghost images and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizer 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizer 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images. Specifically, if the display device 1 is configured to emit S-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs P-polarized light. Furthermore, if the display device 1 is configured to emit P-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs S-polarized light.

[0021] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0022] 2A(2) shows an example of the surface shape of a typical retroreflector 2. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real floating image is displayed on the display device 1 based on the image displayed.

[0023] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflector 2 shown in Figure 2A (2). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. As a result, the effective resolution of the floating image in space will be reduced to about one-third.

[0024] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps any of the sides of one pixel of the liquid crystal display panel.

[0025] The surface shape of the retroreflector according to this embodiment is not limited to the above example. Various surface shapes that achieve retroreflection may be used. Specifically, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or a combination thereof are periodically arranged. Alternatively, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which these prisms are periodically arranged to form cube corners. These may also be referred to as corner reflector arrays or polyhedral reflector arrays. Alternatively, the surface of the retroreflector according to this embodiment may be provided with capsule lens-type retroreflection elements in which glass beads are periodically arranged. The detailed configuration of these retroreflection elements can be achieved using existing technology, so a detailed description will be omitted. Specifically, the techniques disclosed in Japanese Patent Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, and 2009-229942 may be used.

[0026] <Another configuration example 1 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2B. In Fig. 2B, components with the same reference numerals as Fig. 2A have the same functions and configurations as Fig. 2A. For such components, repeated explanations will be omitted to simplify the explanation.

[0027] In the optical system of FIG. 2B , similar to FIG. 2A , image light of a specific polarization is output from the display device 1. The image light of a specific polarization output from the display device 1 is input to a polarization separation member 101B. The polarization separation member 101B is a member that selectively transmits image light of a specific polarization. Unlike the polarization separation member 101 of FIG. 2A , the polarization separation member 101B is not integrated with the transparent member 100 but has an independent plate-like shape. Therefore, the polarization separation member 101B may also be referred to as a polarization separation plate. The polarization separation member 101B may be configured as a reflective polarizing plate configured by attaching a polarization separation sheet to a transparent member. Alternatively, the transparent member may be formed of a metal multilayer film that selectively transmits specific polarization and reflects polarization of other specific polarizations. In FIG. 2B , the polarization separation member 101B is configured to transmit image light of a specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarized and converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it enters the retroreflector and once when it leaves. Here, the polarization separation member 101B has the property of reflecting the polarized light of the other polarization that has been polarized and converted by the λ / 4 plate 21, so the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of polarization design for the optical system of FIG. 2B will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, and the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again to the polarization separation member 101B. Here, the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light, so the S-polarized image light is reflected by the polarization separation member 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0030] Next, a second example of polarization design for the optical system of FIG. 2B will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separator 101B, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching the polarization separator 101B from the display device 1 passes through the polarization separator 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized image light to P-polarized light. The P-polarized image light then proceeds again to the polarization separator 101B. Here, the polarization separator 101B has the property of reflecting P-polarized light and transmitting S-polarized light, so the P-polarized image light is reflected by the polarization separator 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0031] In FIG. 2B , the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separation member 101B is arranged at an angle α (e.g., 30°) with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. Then, upon reflection by the polarization separation member 101B, the direction of travel of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) differs by an angle β (e.g., 60°) from the direction of travel of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light). With this configuration, in the optical system of FIG. 2B , the image light is output at a predetermined angle shown toward the outside of the transparent member 100, forming a real image, the floating-in-space image 3. In the configuration of FIG. 2B , when viewed by a user from the direction of arrow A, the floating-in-space image 3 is perceived as a bright image. However, when viewed by another person from the direction of arrow B, the floating-in-space image 3 cannot be perceived as an image at all. This characteristic is extremely suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0032] As described above, the optical system of FIG. 2B has a different configuration from the optical system of FIG. 2A, but can form a suitable floating image in space, similar to the optical system of FIG. 2A.

[0033] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2B due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0034] <Another configuration example 2 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2C. Note that in Fig. 2C, components with the same reference numerals as Fig. 2B have the same functions and configurations as Fig. 2B. For the sake of simplicity, such components will not be described repeatedly.

[0035] The only difference between the optical system of Fig. 2B and the optical system of Fig. 2C is the arrangement angle of the polarization separation member 101B with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. All other configurations are the same as those of the optical system of Fig. 2B, so repeated explanations will be omitted. The polarization design of the optical system of Fig. 2C is also the same as that of the optical system of Fig. 2B, so repeated explanations will be omitted.

[0036] In the optical system of FIG. 2C , the polarization separation member 101B is tilted at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C , the angle α is 45°. With this configuration, when the polarization separation member 101B reflects, the angle β between the direction of propagation of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light) and the direction of propagation of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, thereby simplifying the angular relationship between the surfaces that make up the optical system. If the surface of the transparent member 100 is positioned perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, the angular relationship between the surfaces that make up the optical system can be further simplified. In the configuration of Figure 2C, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0037] As described above, the optical system of Fig. 2C has a different configuration from the optical systems of Fig. 2A and Fig. 2B, but can form a suitable floating image in space similar to the optical systems of Fig. 2A and Fig. 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2C due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0039] <Another Configuration Example 3 of the Optical System of the Space-Floating Image Display Device> Another configuration example of the optical system of the space-floating image display device will be described using FIG. 2D. The optical system of FIG. 2D is an optical system that uses a retroreflector 5 that is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, Another Configuration Example 3 of the optical system will be described in more detail using FIGS. 2D to 2I. In FIG. 2D, components that are assigned the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. Such components will not be described repeatedly in order to simplify the explanation.

[0040] 2D is a diagram showing an example of the main components and retroreflection components of a space-floating image display device according to an embodiment of the present invention. A display device 10 that emits image light is provided obliquely on a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] A chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and may also be, for example, 45°±15°.

[0042] The retroreflector 5 is an optical element having the optical property of retroreflecting light rays in at least some directions. Furthermore, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may also be referred to as an imaging optical element or an imaging optical plate.

[0043] 2E, 2F, etc., the principal ray 9020 travels in the z direction and is retroreflected in the x and y directions by the retroreflector 5. As a result, the reflected light ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the principal ray 9020 with the retroreflector 5 as the reference, passes through the transparent member 100, and forms the floating image 3 in space as a real image on the imaging plane.

[0044] The light beam forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 5 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike a diffuse image formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be perceived as an image at all. This characteristic is suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described using Figures 2E and 2F. The retroreflector 5 has a configuration in which a plurality of corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be called a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Light rays 9111, 9112, 9113, and 9114 emitted from a light source 9110 are reflected twice by two mirror surfaces 9041 and 9042 of the corner reflector 9040, becoming reflected light rays 9121, 9122, 9123, and 9124. This double reflection is retroreflection in the x and y directions, where the light is reflected back in the same direction as the incident direction (traveling in a direction rotated 180 degrees), and in the z direction, where the angle of incidence and the angle of reflection match due to total reflection.

[0046] That is, the light rays 9111 to 9114 generate reflected light rays 9121 to 9124 on straight lines symmetrical in the z direction with respect to the corner reflector 9040, forming an aerial real image 9120. Note that the light rays 9111 to 9114 emitted from the light source 9110 are four light rays representing the diffused light from the light source 9110, and although the light rays incident on the retroreflector 5 are not limited to these, depending on the diffusion characteristics of the light source 9110, all incident light rays cause similar reflections and form an aerial real image 9120. Note that to make the drawing easier to see, the position of the light source 9110 and the position of the aerial real image 9120 are shown shifted in the x direction, but in reality, the position of the light source 9110 and the position of the aerial real image 9120 in the x direction are the same, and are overlapping when viewed from the z direction.

[0047] 2G, 2H, and 2I, the configuration and effects of the corner reflector 9040 that constitutes the retroreflector 5 will be described. The corner reflector 9040 is a rectangular parallelepiped with only two specific surfaces being mirror surfaces 9041 and 9042, and the other four surfaces being made of transparent materials. The retroreflector 5 has a configuration in which these corner reflectors 9040 are arrayed so that the corresponding mirror surfaces face in the same direction.

[0048] When viewed from the top (+z direction), a light ray 9111 emitted from the light source 9110 enters the mirror surface 9041 (or the mirror surface 9042) at a specific angle of incidence, is totally reflected at the reflection point 9130, and then is totally reflected again at the reflection point 9132 on the mirror surface 9042 (or the mirror surface 9041).

[0049] If the angle of incidence of light ray 9111 with respect to mirror surface 9041 (or mirror surface 9042) is θ, then the angle of incidence of first reflected light ray 9131 reflected by mirror surface 9041 (or mirror surface 9042) with respect to mirror surface 9042 (or mirror surface 9041) can be expressed as 90°-θ. Therefore, with respect to light ray 9111, second reflected light ray 9121 undergoes a rotation of 2θ after the first reflection and 2×(90°-θ) after the second reflection, resulting in a total reversal optical path of 180°. On the other hand, when viewed from the side (the direction halfway between -x and -y), total reflection in the z direction occurs only once. Therefore, if the angle of incidence with respect to mirror surface 9041 or mirror surface 9042 is φ, then reflected light ray 9121 undergoes a rotation of 2×φ after one reflection with respect to light ray 9111.

[0050] As described above, the light rays incident on the corner reflector 9040 undergo retroreflection, which results in an inverted optical path in the x and y directions, and specular reflection due to total reflection in the z direction. Considering the retroreflector 5, similar reflection occurs in each optical path, so that an image is formed at a point symmetrical with respect to the z axis direction by an inverted optical path that is convergent in the x and y directions.

[0051] 2A to 2C, the retroreflector 2 has retroreflection properties in three axes. As a result, when a diffusive incident light beam is incident on the retroreflector 2, a convergent reflected light beam travels toward the side of the retroreflector 2 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air, forming a floating image 3. The traveling direction of the chief ray of the convergent reflected light beam reflected from the retroreflector 2 is opposite to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 2.

[0052] 2D, the retroreflector 5 has retroreflection properties in two axial directions and specular reflection in the other axial direction. As a result, when a diffusive incident light beam is incident on the retroreflector 5, the convergent reflected light beam is reflected by the corner reflector array and travels in the direction opposite to the side of the retroreflector 5 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air and forms the space floating image 3.

[0053] The traveling direction of the chief ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflector 5 is not the opposite direction to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5. The normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5 and the normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray after being reflected by the retroreflector 5 and becoming a convergent reflected light beam, travel in a straight line, unchanged before and after reflection by the corner reflector array.

[0054] That is, the diffusive incident light beam is converted into a convergent reflected light beam by reflection on the retroreflector 5, but in the normal direction to the plate-shaped surface of the retroreflector 5, the light beam travels as if passing through the retroreflector 5. Here, the diffusive incident light beam incident on the retroreflector 5 and the convergent reflected light beam emerging from the retroreflector 5 are in a geometrically symmetrical relationship with respect to the plate-shaped surface of the retroreflector 5.

[0055] The resolution of the space-floating image formed by the light beams from the video output unit 10 depends not only on the resolution of the liquid crystal display panel 11, but also on the diameter D and pitch P (not shown) of the retroreflective portion of the retroreflector 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image will be equivalent to 300 μm. As a result, the effective resolution of the space-floating image will be reduced to about one-third.

[0056] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0057] The shape of the retroreflector (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that achieve retroreflection. Specifically, it may be a variety of cubic corner bodies, corner reflector arrays, slit mirror arrays, dihedral corner reflector arrays, polyhedral reflector arrays, or shapes in which a combination of these reflective surfaces is periodically arranged. Alternatively, capsule lens-type retroreflecting elements in which glass beads are periodically arranged may be provided on the surface of the retroreflector according to this embodiment. Since existing technology can be used for the detailed configuration of these retroreflecting elements, a detailed description will be omitted. Specifically, the technology disclosed in JP 2017-33005 A, JP 2019-133110 A, JP 2017-67933 A, WO 2009 / 131128 A, etc. may be used.

[0058] 2D, the image light emitted from the display device 10 may be in any polarization state, either S-polarized or P-polarized.

[0059] As described above, the optical system of FIG. 2D is an optical system that uses a retroreflector different from the optical systems of FIGS. 2A to 2C, but it can form a more suitable floating image in space, similar to the optical systems of FIGS. 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, it is possible to provide brighter and higher quality floating images in space.

[0061] <<Block diagram of the internal configuration of the space floating image display device>>

[0062] Next, a description will be given of a block diagram of the internal configuration of the space-floating image display device 1000. Fig. 3 is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0063] The space-floating image display device 1000 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power supply input interface 1111, an operation input unit 1107, a nonvolatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. In addition, the space-floating image display device 1000 may also include a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112.

[0064] Each component of the space floating image display device 1000 is disposed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0065] The retroreflecting portion 1101 in Fig. 3 corresponds to the retroreflector 2 in Fig. 2A, Fig. 2B, and Fig. 2C. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0066] The image display unit 1102 in Fig. 3 corresponds to the liquid crystal display panel 11 in Fig. 2A, 2B, and 2C. The light source 1105 in Fig. 3 corresponds to the light source device 13 in Fig. 2A, 2B, and 2C. The image display unit 1102, the light guide 1104, and the light source 1105 in Fig. 3 correspond to the display device 1 in Fig. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0068] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts AC current input from the outside via the external power input interface 1111 into DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each component within the space-floating image display device 1000. The secondary battery 1112 stores the power supplied from the power source 1106. The secondary battery 1112 also supplies power to the light source 1105 and other components requiring power via the external power input interface 1111 when power is not supplied from the outside. In other words, when the space-floating image display device 1000 is equipped with the secondary battery 1112, the user can use the space-floating image display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the video display unit 1102. The light guide 1104 may be configured mainly using glass. The light guide 1104 may be configured mainly using plastic. The light guide 1104 may be configured using a mirror. Various methods are possible for combining the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The aerial operation detection sensor 1351 is a sensor that detects an operation of the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0071] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible light laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) sensor or an image sensor.

[0072] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0073] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and determines whether or not the finger of the user 230 has made contact with an object in the floating-in-space image 3 based on the sensing signal, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0074] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as separate entities from the space-floating image display device 1000. When provided as separate entities from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Furthermore, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000, which does not have an aerial operation detection function, is used as the main body, and only the aerial operation detection function can be added as an option. Furthermore, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases where it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0076] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in detecting the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path, etc.

[0077] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object that has not intruded into the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the images captured by the multiple image capturing units 1180 and object depth information from the depth sensor. These pieces of information and various other information such as the distance between the object and the intrusion detection plane are then used for various display controls for the floating in space image 3.

[0079] Furthermore, without using the aerial operation detection sensor 1351 , the aerial operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180 .

[0080] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of the user 230 operating the space-floating image 3 and a range including the area surrounding the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs signals for operations different from the aerial operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0082] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces are possible for the video signal input unit 1131. For example, the video signal input unit 1131 may be configured as a video input interface conforming to the HDMI (registered trademark) standard (High-Definition Multimedia Interface), a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured as an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. In the case of an HDMI-standard interface, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an interface in which a terminal and a cable are integrated. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be configured as a speaker.

[0084] The audio output unit 1140 may also output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output a digital signal to an external device, like the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that collects sounds around the space-floating image display device 1000, converts them into signals, and generates audio signals. The microphone may record a person's voice, such as a user's voice, and the control unit 1110, described later, may perform voice recognition processing on the generated audio signal to obtain text information from the audio signal.

[0085] The non-volatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0086] The control unit 1110 controls the operation of each connected unit. The control unit 1110 may also work in cooperation with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. If the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, by an Ethernet-standard LAN interface. If the communication unit 1132 has a wireless communication interface, the interface may be configured, for example, by a Wi-Fi communication interface, a Bluetooth communication interface, or a mobile communication interface such as 4G or 5G. Various types of data, such as video data, image data, and audio data, are transmitted and received through communication via the communication unit 1132.

[0088] The removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the floating image 3 via the video display unit 1102 and the retroreflection unit 1101.

[0089] The storage unit 1170 is a storage device that records various types of information, such as video data, image data, and audio data. The storage unit 1170 may be configured with a magnetic recording medium recording device, such as a hard disk drive (HDD), or a semiconductor element memory, such as a solid state drive (SSD). For example, various types of information, such as video data, image data, and audio data, may be recorded in advance in the storage unit 1170 at the time of product shipment. The storage unit 1170 may also record various types of information, such as video data, image data, and audio data, acquired from an external device, an external server, or the like via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of display icons, objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0091] Layout information of display icons, objects, etc. displayed as the spatial floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0092] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 may be referred to as a video processing circuit, and may be configured with hardware such as an ASIC, FPGA, or video processor. The video control unit 1160 may also be referred to as a video processing unit or an image processing unit. The video control unit 1160 controls video switching, such as determining which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0093] In addition, the image control unit 1160 may generate a superimposed image signal by superimposing the image signal to be stored in the memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby controlling the formation of a composite image as a floating image in space 3.

[0094] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing that enlarges, reduces, or deforms the image, brightness adjustment processing that changes the brightness, contrast adjustment processing that changes the contrast curve of the image, and Retinex processing that decomposes the image into light components and changes the weighting of each component.

[0095] Furthermore, the video control unit 1160 may perform special effect video processing or the like on the video signal input to the video display unit 1102 to assist the aerial operation (touch operation) of the user 230. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 or the image of the user 230 captured by the imaging unit 1180.

[0096] The attitude sensor 1113 is a sensor configured with a gravity sensor, an acceleration sensor, or a combination of these, and can detect the attitude in which the space-floating image display device 1000 is installed. Based on the attitude detection result of the attitude sensor 1113, the control unit 1110 may control the operation of each connected unit. For example, if an undesirable attitude is detected as the user's usage state, the control unit 1110 may perform control such that the image displayed on the image display unit 1102 is stopped and an error message is displayed to the user. Alternatively, if the attitude sensor 1113 detects a change in the installation attitude of the space-floating image display device 1000, the control unit 1110 may perform control such that the display orientation of the image displayed on the image display unit 1102 is rotated.

[0097] As explained above, various functions are installed in the space floating image display device 1000. However, the space floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space floating image 3.

[0098] <Configuration Example of Space-Floating Image Display Device> Next, a configuration example of the space-floating image display device will be described. The layout of the components of the space-floating image display device according to this embodiment can be various depending on the usage form. Below, the layouts of each of Figs. 4A to 4M will be described. Note that in each of the examples of Figs. 4A to 4M, the thick line surrounding the space-floating image display device 1000 indicates an example of the housing structure of the space-floating image display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system shown in FIG. 2A. The space-floating image display device 1000 shown in FIG. 4A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4A, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user's finger. Note that the x direction is the left-right direction as seen from the user, the y direction is the front-back direction (depth direction) as seen from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x direction, y direction, and z direction are the same in each drawing of FIG. 4, so repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4B, the space-floating image display device is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user 230's finger. 4B , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0101] FIG. 4C is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4C is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4C, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0102] FIG. 4D is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4D, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger. 4D , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0103] FIG. 4E is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4E is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4E, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels directly upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0104] FIG. 4F is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4F, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels toward the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger.

[0105] FIG. 4G is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system shown in FIG. 2C. In the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, in the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the image light traveled in the front-to-back and up-to-down directions as seen from the user. In contrast, in the optical system of the space-floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, in the optical system of the space-floating image display device shown in FIG. 4G, the image light travels in the left-to-right and front-to-back directions as seen from the user. In the space-floating image display device 1000 shown in FIG. 4G, the surface on which the space-floating image 3 is formed is installed so that it faces the front of the device (toward the user 230). 4G, the space-floating image display device 1000 has the transparent member 100 installed on the front side of the device (toward the user 230). The space-floating image 3 is formed on the user side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels towards the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0106] 4H is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4H differs from the space-floating image display device of FIG. 4G in that it has a window with a transparent plate 100B made of glass or plastic on the back of the device (opposite the position where the user 230 views the space-floating image 3, i.e., opposite the traveling direction of the image light of the space-floating image 3 toward the user 230). The rest of the configuration is the same as the space-floating image display device of FIG. 4G , so repeated explanations will be omitted. The space-floating image display device 1000 of FIG. 4H has a window with a transparent plate 100B on the opposite side of the traveling direction of the image light of the space-floating image 3 from the space-floating image 3. Therefore, when the user 230 views the space-floating image 3, they can recognize the scenery behind the space-floating image display device 1000 as the background of the space-floating image 3. Therefore, the user 230 can perceive the space floating image 3 as floating in the air in front of the scenery behind the space floating image display device 1000. This can further emphasize the floating feeling of the space floating image 3.

[0107] Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, in order to prevent this stray light, the transparent plate 100B may not be provided in the window on the back of the space-floating image display device 1000.

[0108] Fig. 4I is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4I differs from the space-floating image display device of Fig. 4H in that a light-blocking door 1410 is provided in the window of the transparent plate 100B located on the back of the device (opposite the position where the user 230 views the space-floating image 3). The rest of the configuration is the same as that of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted.

[0109] The opening and closing door 1410 of the space-floating image display device 1000 in FIG. 4I has, for example, a light blocking plate and is equipped with a mechanism for moving (sliding), rotating, or attaching / detaching the light blocking plate, thereby switching between an open state and a light blocking state for the window (rear window) of the transparent plate 100B located at the back of the space-floating image display device 1000. The movement (sliding) and rotation of the light blocking plate by the opening and closing door 1410 may be electrically driven by a motor (not shown). The motor may be controlled by the control unit 1110 in FIG. 3. Note that the example in FIG. 4I discloses an example in which the opening and closing door 1410 has two light blocking plates. In contrast, the opening and closing door 1410 may have only one light blocking plate.

[0110] For example, if the view seen through the window of the transparent plate 100B of the space-floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the space-floating image 3 may become too bright, reducing the user 230's visibility of the space-floating image 3. In such a case, by moving (sliding), rotating, or attaching the light-shielding plate of the opening / closing door 1410 to block the light from the rear window, the background of the space-floating image 3 becomes dark, thereby relatively increasing the visibility of the space-floating image 3. Such a blocking operation by the light-shielding plate of the opening / closing door 1410 may be performed directly by the force of the user 230's hand. In response to an operation input via the operation input unit 1107 of FIG. 3 , the control unit 1110 may control a motor (not shown) to perform the blocking operation by the light-shielding plate of the opening / closing door 1410.

[0111] An illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 may control a motor (not shown) to perform the opening and closing operation of the light blocking plate of the opening and closing door 1410 according to the detection result of the illuminance sensor. By controlling the opening and closing operation of the light blocking plate of the opening and closing door 1410 in this way, it is possible to more suitably maintain the visibility of the space-floating image 3, even if the user 230 does not manually open or close the light blocking plate of the opening and closing door 1410.

[0112] Furthermore, the light blocking plate provided by the opening and closing door 1410 may be manually detachable. Depending on the intended use and installation environment of the space floating image display device 1000, the user can select whether to leave the rear window open or in a light blocking state. If it is planned to use the rear window in a light blocking state for a long period of time, the detachable light blocking plate can be fixed in the light blocking state. Furthermore, if it is planned to use the rear window in an open state for a long period of time, the detachable light blocking plate can be left detached. The light blocking plate may be attached and detached using screws, a hook structure, or a fitting structure.

[0113] Even in the example of the space-floating image display device 1000 shown in FIG. 4I, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, to prevent this stray light, the window on the back of the space-floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening / closing door 1410 may be provided in a window that does not have the transparent plate 100B. To prevent this stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 have a coating or material with low light reflectance.

[0114] Fig. 4J is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4J differs from the space-floating image display device of Fig. 4H in that instead of arranging a transparent plate 100B made of glass or plastic on the rear side window, an electronically controlled transmittance variable device 1620 is arranged. The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted. An example of the electronically controlled transmittance variable device 1620 is a liquid crystal shutter.

[0115] That is, the liquid crystal shutter can control the amount of light transmitted by controlling the voltage of the liquid crystal element sandwiched between two polarizing plates. Therefore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be seen through the background of the space-floating image 3. Furthermore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be hidden as the background of the space-floating image 3. Furthermore, since the liquid crystal shutter can control the intermediate length, it can also be set to a transmittance of 50% or more. For example, the control unit 1110 can control the transmittance of the electronically controlled transmittance variable device 1620 in response to an operation input via the operation input unit 1107 of FIG. 3 . With this configuration, in cases where a view through the rear window is desired as the background of the space-floating image 3, but the view through the background rear window is too bright and reduces the visibility of the space-floating image 3, the visibility of the space-floating image 3 can be adjusted by adjusting the transmittance of the electronically controlled transmittance variable device 1620.

[0116] In addition, an illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 controls the transmittance of the electronically controlled transmittance variable device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 of Fig. 3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted according to the brightness of the space beyond the rear window, making it possible to more suitably maintain the visibility of the space-floating image 3.

[0117] In the above example, a liquid crystal shutter has been described as an example of the electronically controlled transmittance varying device 1620. However, electronic paper may be used as another example of the electronically controlled transmittance varying device 1620. The same effect as described above can be obtained when electronic paper is used. Furthermore, electronic paper consumes very little power to maintain a halftone state. Therefore, a space floating image display device with lower power consumption can be realized compared to when a liquid crystal shutter is used.

[0118] Fig. 4K is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4K differs from the space-floating image display device of Fig. 4G in that it has a transmissive self-luminous image display device 1650 instead of the transparent member 100. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0119] In the space-floating image display device 1000 of FIG. 4K, an image luminous flux passes through the display surface of the transmissive self-luminous image display device 1650, and then a space-floating image 3 is formed outside the space-floating image display device 1000. That is, when an image is displayed on the transmissive self-luminous image display device 1650, which is a two-dimensional flat display, the space-floating image 3 can be displayed as a pop-up image further in front of the image displayed on the transmissive self-luminous image display device 1650. In this case, the user 230 can simultaneously view two images at different depth positions. The transmissive self-luminous image display device 1650 may be configured using existing technology, such as a transmissive organic EL panel, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although the transmissive self-luminous image display device 1650 is not shown in FIG. 3, it may be configured to be connected to other processing units, such as the control unit 1110, as a component of the space-floating image display device 1000 of FIG. 3.

[0120] Here, if the transmissive self-luminous image display device 1650 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise image experience.

[0121] Furthermore, if the interior of the space-floating image display device 1000 is kept in a light-blocking state, the background of the transmissive self-luminous image display device 1650 will be sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, the transmissive self-luminous image display device 1650 appears to the user 230 as a normal two-dimensional flat display rather than a transmissive display (since the space-floating image 3 in the embodiment of the present invention is displayed as a real optical image in a space without a screen, if the light source of the display device 1 is turned off, the intended display position of the space-floating image 3 will become empty space). Therefore, when the transmissive self-luminous image display device 1650 is used to display an image as if it were a normal two-dimensional flat display, characters, objects, etc. can suddenly be displayed in the air as the space-floating image 3, thereby providing the user 230 with a more effective surprise video experience.

[0122] Note that the darker the interior of the space-floating image display device 1000, the more the transmissive self-luminous image display device 1650 appears like a two-dimensional flat display. Therefore, an absorptive polarizer (not shown) that transmits the polarized light of the image light reflected by the polarization separation member 101B and absorbs polarized light that is 90° out of phase with the polarized light may be provided on the interior surface of the transmissive self-luminous image display device 1650 (the surface on which the image light reflected by the polarization separation member 101B enters the transmissive self-luminous image display device 1650, i.e., the surface of the transmissive self-luminous image display device 1650 opposite the space-floating image 3). While this does not have a significant effect on the image light that forms the space-floating image 3, it can significantly reduce the light that enters the space-floating image display device 1000 from the outside through the transmissive self-luminous image display device 1650, making the interior of the space-floating image display device 1000 darker, which is preferable.

[0123] 4L is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4L is a modified example of the space-floating image display device of FIG. 4K. The orientation of the components in the space-floating image display device 1000 is different from that of the space-floating image display device of FIG. 4K, and is closer to the arrangement of the space-floating image display device of FIG. 4F. The functions and operations of each component are the same as those of the space-floating image display device of FIG. 4K, so repeated explanations will be omitted.

[0124] In the space-floating image display device of FIG. 4L, after the luminous flux of image light passes through the transmissive self-luminous image display device 1650, a space-floating image 3 is formed on the user 230 side of the transmissive self-luminous image display device 1650.

[0125] In both the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, the space-floating image 3 is displayed superimposed on the image of the transmissive self-luminous image display device 1650 from the user 230. Here, the position of the space-floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves their head (the position of the viewpoint), they can recognize the depth of the two images due to parallax. Therefore, by displaying two images at different depth positions, it is possible to provide the user with a more suitable three-dimensional image experience with the naked eye without the need for stereoscopic glasses or the like.

[0126] Fig. 4M is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4M is provided with a second display device 1680 on the rear side as seen from the user relative to the polarization separation member 101B of the space-floating image display device of Fig. 4G. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0127] In the configuration example shown in Fig. 4M, the second display device 1680 is provided behind the display position of the space-floating image 3, and its image display surface faces the space-floating image 3. With this configuration, from the user 230's perspective, the image of the second display device 1680 and the space-floating image 3, which are displayed at two different depth positions, can be viewed superimposed on each other. In other words, the second display device 1680 is positioned so as to display an image in the direction of the user 230 viewing the space-floating image 3. Although the second display device 1680 is not shown in Fig. 3, it may be configured to be connected to other processing units such as the control unit 1110 as one component of the space-floating image display device 1000 of Fig. 3.

[0128] Note that the image light of the second display device 1680 of the space-floating image display device 1000 of FIG. 4M is viewed by the user 230 after passing through the polarization separation member 101B. Therefore, in order for the image light of the second display device 1680 to more suitably pass through the polarization separation member 101B, it is desirable that the image light output from the second display device 1680 be polarized with a polarization in a vibration direction that the polarization separation member 101B more suitably transmits. In other words, it is desirable that the image light be polarized with a polarization in the same vibration direction as the polarization of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized, it is desirable that the image light output from the second display device 1680 is also S-polarized. Furthermore, if the image light output from the display device 1 is P-polarized, it is desirable that the image light output from the second display device 1680 is also P-polarized.

[0129] The example of the space-floating image display device of FIG. 4M also has the same effect as the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L in that a second image is displayed behind the space-floating image 3. However, unlike the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, in the example of the space-floating image display device of FIG. 4M, the luminous flux of image light for forming the space-floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, but may be a liquid crystal display, which is a two-dimensional flat display. The second display device 1680 may also be an organic EL display. Therefore, in the example of the space-floating image display device of FIG. 4M, it is possible to realize the space-floating image display device 1000 at a lower cost than the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, there is a possibility that part of the image light output from the display device 1 will be reflected by the polarization separation member 101B and head toward the second display device 1680. This light (part of the image light) may be reflected again by the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, to prevent this stray light, an absorptive polarizer may be provided on the surface of the second display device 1680. In this case, the absorptive polarizer may be an absorptive polarizer that transmits the polarized waves of the image light output from the second display device 1680 and absorbs polarized waves that are 90° out of phase with the polarized waves of the image light output from the second display device 1680. If the second display device 1680 is a liquid crystal display, an absorptive polarizer is also present on the image output side inside the liquid crystal display. However, if a cover glass (cover glass on the image display surface side) is further provided on the output surface of the absorptive polarizer on the image output side inside the liquid crystal display, it is not possible to prevent stray light caused by reflection of the cover glass by light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-mentioned absorptive polarizer on the surface of the cover glass.

[0132] When an image is displayed on the second display device 1680, which is a two-dimensional flat display, the floating-in-space image 3 can be displayed as an image further in front of the user in relation to the image on the second display device 1680. In this case, the user 230 can simultaneously view two images at different depth positions. By displaying a character on the floating-in-space image 3 and a background on the second display device 1680, it is possible to provide the effect that the user 230 is viewing the space in which the character exists in three dimensions.

[0133] Furthermore, if the second display device 1680 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise visual experience.

[0134] Next, Fig. 4N is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4N is a space-floating image display device that employs the optical system of Fig. 2D. As with the examples of space-floating image display devices that employ the optical systems of Figs. 2A to 2C, an image is formed in the air as a space-floating image 3 by image light that has passed through a transparent member 100. Furthermore, using sensing light from an aerial operation detection sensor 1351 that is located on the back side of the transparent member 100 as seen from the user, it is possible to detect operation of the space-floating image 3 by the user's finger 9004.

[0135] 2A to 2C, and in the example of the space-floating image display device employing the optical system of Fig. 2D, the space-floating image 3 is formed in front of the transparent member 100, and the operation of the space-floating image 3 by the user's finger can be detected using the sensing light of the mid-air operation detection sensor 1351 arranged on the back side of the transparent member 100 as seen from the user. Therefore, the space-floating image display device employing the optical system of Fig. 2D has a different optical system from the space-floating image display device in which the optical system of Fig. 2A to 2C is arranged on the back side of the transparent member 100 as seen from the user.

[0136] However, from the user's perspective, the usability of the space-floating image display device employing the optical system of FIG. 2D is almost the same as that of the space-floating image display device employing the optical system of FIGS. 2A to 2C.

[0137] Next, Fig. 4O is a diagram showing an example of the configuration of a space-floating image display device. Fig. 4O is a diagram showing the configuration of the internal optical system of the space-floating image display device 1000 of Fig. 4N. The space-floating image display device 1000 shown in Fig. 4O is equipped with an optical system corresponding to the optical system of Fig. 2D. The space-floating image display device 1000 shown in Fig. 4O is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward.

[0138] 4O, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0139] Here, the configuration of Fig. 4O will be compared with the configuration of Fig. 4A to confirm the differences. In Fig. 4A, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the polarization separation member 101. In contrast, in Fig. 4O, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the retroreflector 5. Furthermore, the configuration of Fig. 4A includes the retroreflector 2 and the λ / 4 plate 21, but these are not present in Fig. 4O. Furthermore, while the presence of the absorbing polarizer 12 is more preferable in Fig. 4A, the absorbing polarizer 12 is not particularly necessary in Fig. 4O.

[0140] That is, to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and to replace it with the configuration of FIG. 4O, the following can be done. That is, the polarization separation member 101 in the configuration of FIG. 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 are removed from the configuration of FIG. 4A. The absorptive polarizer 12 is optional. By performing a replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the space-floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the space-floating image display device can be replaced with the optical system of FIG. 2D. In this case, the polarization separation member 101 in FIGS. 4A and 4B is replaced with the retroreflector 5, and the polarization separation member 101B in FIGS. 4C to 4G is replaced with the retroreflector 5.

[0141] In this way, it is possible to realize a space-floating image display device in which the optical system in the configuration of the space-floating image display device of Figures 4A to 4G is replaced with the optical system of Figure 2D. Even in these space-floating image display devices in which the optical system of Figure 2D is replaced, it is possible to realize a space-floating image display device that is almost as easy to use as the space-floating image display device of Figures 4A to 4G.

[0142] <Display Device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes the light source of the image display element 11. Fig. 5 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0143] As shown by arrow 30 in Figure 5, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has narrow-angle diffusion characteristics, i.e., has strong directivity (linearity) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam in accordance with an input video signal. The modulated image light is reflected by retroreflector 2 and passes through transparent member 100 to form a real image, a floating image in space (see Figure 1).

[0144] 5 also shows a configuration including a liquid crystal display panel 11 constituting the display device 1, a light redirection panel 54 for controlling the directional characteristics of the light beam emitted from the light source device 13, and, if necessary, a narrow-angle diffuser (not shown). Specifically, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated by a video signal (see arrow 30 in FIG. 5). This allows the desired image to be projected as highly directional (linear) light of a specific polarization via the light redirection panel 54 toward the retroreflector 2, where it is reflected by the retroreflector 2 and transmitted to the eyes of a monitor outside the store (space), forming a floating image 3. A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the light redirection panel 54.

[0145] <Display Device Example 1> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, a liquid crystal display panel 11 and a light direction conversion panel 54 are disposed on the light source device 13 shown in FIG. 5. The light source device 13 is configured on a case shown in FIG. 5, formed of, for example, plastic, and contains LED elements 201 and a light guide 203 therein. As shown in FIG. 5 and other figures, the end surface of the light guide 203 has a lens-like shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens-like shape gradually reduces the divergence angle by multiple total reflections during propagation within the light guide 203. The liquid crystal display panel 11 constituting the display device 1 is attached to the top surface of the display device 1. Furthermore, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202 on which their control circuits are mounted are attached to one side of the case of the light source device 13 (the left end surface in this example). A heat sink, which is a member for cooling the heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202 .

[0146] Furthermore, a frame (not shown) for the liquid crystal display panel is attached to the top surface of the case of the light source device 13. The frame includes the liquid crystal display panel 11, which is attached to the frame, and a flexible printed circuit (FPC) (not shown) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is an image display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (image control unit 1160 in FIG. 3 ) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, for example, an LED element.

[0147] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 as well as FIG.

[0148] 6 and 7 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown, and this light is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion on the end surface of the light guide and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Each light guide 203 is formed of a translucent resin such as acrylic. The LED light-receiving surface at the end of the light guide 203 has a cone-shaped outer periphery obtained by rotating a parabolic cross section, and the top of the light guide 203 has a concave portion with a convex portion (i.e., a convex lens surface) formed in the center, and the center of the flat portion has a convex lens surface (or a concave lens surface) that protrudes outward (not shown). The outer shape of the light-receiving portion of the light guide to which the LED element 201 is attached is a parabolic shape that forms a cone-shaped outer periphery, and is set within an angle range that allows total reflection of the light emitted from the LED element toward the periphery within the parabolic shape, or a reflective surface is formed.

[0150] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203 a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0151] With this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the utilization efficiency of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit having a plurality of LED elements 201 arranged as light sources to the light-receiving end surface 203a, which is a light-receiving section provided on the end surface of the light guide 203, and the divergent light beams from the LED elements 201 are converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, which is then guided inside the light guide 203 as shown by the arrow (in a direction parallel to the drawing), and emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is disposed approximately parallel to the light guide 203 (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape inside or on the surface of the light guide.

[0153] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 (in a direction perpendicular to the front of the drawing) that is disposed substantially parallel to the light guide 203, by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0154] 6 is a cross-sectional layout diagram illustrating the configuration and operation of the light source of this embodiment that performs polarization conversion in light source device 13 including the above-mentioned light guide 203 and LED element 201. In Fig. 6, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0155] In addition, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice to convert the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by a video signal in the liquid crystal display panel 11 (arrow 213 in Figure 6), enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0156] 7 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in a light source device 13 including a light guide 203 and an LED element 201, similar to Fig. 6. The light source device 13 is similarly configured with a light guide 203 formed of, for example, plastic, on the surface of which or inside which a light beam direction conversion means 204 is provided, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, etc. A liquid crystal display panel 11 is attached to the top surface of the light source device 13 as an image display element, and the panel has polarizing plates on the light source light entrance surface and the image light exit surface.

[0157] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED element 201. That is, in the example of FIG. 7 , the selective reflection characteristics of the reflective polarizing plate 49 differ from those in FIG. 7 . The reflected light is reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and returns to the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from S-polarized light to P-polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7 ) enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0158] In the light source devices shown in Figures 6 and 7, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side. Therefore, the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.

[0159] <Display Device Example 2> Figure 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc., in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board.

[0160] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto the liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.

[0161] <Display Device Example 3> Next, another example of the specific configuration of the display device 1 (Display Device Example 3) will be described with reference to Fig. 9 . The light source device of this display device 1 converts a divergent beam of light (a mixture of P-polarized and S-polarized light) from an LED into a substantially parallel beam by a collimator 18, and reflects the parallel beam toward the liquid crystal display panel 11 by the reflective surface of a reflective light guide 304. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 transmits light of a specific polarization (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, light polarized other than the specific polarization (e.g., S-polarized light) is reflected by the reflective polarizer 49 and directed again toward the reflective light guide 304.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so that the reflective polarizing plate 49 is not perpendicular to the chief ray of light from the reflective surface of the reflective light guide 304. The chief ray of light reflected by the reflective polarizing plate 49 is incident on the transmission surface of the reflective light guide 304. The light that has entered the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that has passed through the transmission surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, the light that re-enters the reflective polarizing plate 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarization (for example, P-polarized light) that is transmitted through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11. Note that, with regard to the polarization design related to the polarization conversion, the polarization may be configured in reverse from the above explanation (S-polarized light and P-polarized light may be reversed).

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, because this is a vertical cross section, only one is shown in Figure 9), and these are attached at predetermined positions relative to the collimator 18.

[0165] Each collimator 18 is formed of a translucent resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer circumferential surface obtained by rotating a parabolic cross section. The collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center of the apex (the side facing the LED substrate 102). The collimator 18 may have a convex lens surface (or a concave lens surface) protruding outward in the center of the flat surface (the side opposite the apex). The parabolic surface forming the cone-shaped outer circumferential surface of the collimator 18 is set within an angle range that allows total reflection of the light emitted from the LED toward the periphery, or a reflective surface is formed therein.

[0166] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED board 102. The LED board 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are positioned at the center of the apex of the convex cone shape (or in the recess if the apex has a recess).

[0167] With this configuration, the collimator 18 focuses light emitted from the LED, particularly light emitted from the central portion, into parallel light by the convex lens surface that forms the outer shape of the collimator 18. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the collimator 18, and is similarly focused into parallel light. In other words, the collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0168] Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Of this light, light of a specific polarization passes through the reflective polarizer 49 due to the action of the reflective polarizer 49, while light of the other polarization reflected by the action of the reflective polarizer 49 passes through the light guide 304 again. The light is reflected by the reflector 271 located opposite the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted by passing twice through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizer 49 provided on the opposite surface. Since the incident light has been polarized and converted, it passes through the reflective polarizer 49 and enters the liquid crystal display panel 11 with its polarization direction aligned. As a result, all of the light from the light source can be used, doubling the geometrical optical utilization efficiency of light. Furthermore, since the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 allows adjustment of the angle of light reflection and diffusion at each reflective surface. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 for each design can improve the uniformity of the light incident on the liquid crystal display panel 11.

[0169] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 9, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 9, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarized light.

[0170] <Display Device Example 4> Another example (Display Device Example 4) of the configuration of an optical system such as a light source device of a display device will be described with reference to Fig. 10. This is a configuration example in which a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Display Device Example 3. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert diffusion characteristics in the vertical and horizontal directions of the drawing (front-to-back directions not shown in the drawing) are used on the light emission side of collimator 18, and light from collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] The optical sheet may be a single sheet instead of a two-sheet configuration. In a single-sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. Alternatively, multiple diffusion sheets may be used to share the functions. In the example of FIG. 10 , the reflection and diffusion characteristics due to the front and back shapes of optical sheets 207A and 207B can be optimally designed using the number of LEDs, the divergence angle from LED substrate (optical element) 102, and the optical specifications of collimator 18 as design parameters so that the surface density of the light beam emitted from the liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of light guides.

[0172] In the example of FIG. 10 , polarization conversion is performed in the same manner as in the display device example 3 described above. That is, in the example of FIG. 10 , the reflective polarizing plate 49 may be configured to have the property of reflecting S-polarized light (transmitting P-polarized light). In this case, the reflective polarizing plate 49 transmits P-polarized light from the light source LED, and the transmitted light enters the liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light from the light source LED, and the reflected light passes through the retardation plate 270 shown in FIG. 10 . The light that passes through the retardation plate 270 is reflected by the reflector 271. The light reflected by the reflector 271 passes through the retardation plate 270 again and is converted to P-polarized light. The polarization-converted light passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11.

[0173] It should be noted that the λ / 4 plate 270, which is the retarder in FIG. 10, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of FIG. 10, any retarder that changes the phase by 90° (λ / 2) when polarized light passes through it twice may be used. The thickness of the retarder may be adjusted according to the distribution of incident angles of the polarized light. It should be noted that, in FIG. 10 as well, the polarization design for polarization conversion may be configured in reverse (reversing the S-polarized light and P-polarized light) from the above explanation.

[0174] In a typical TV device, the light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in FIG. 12(a)) and the vertical direction of the screen (shown on the Y-axis in FIG. 12(b)). In contrast, the diffusion characteristics of the light beam emitted from the liquid crystal display panel of this embodiment are, for example, as shown in Example 1 of FIG. 12, when the viewing angle at which the luminance is 50% of that at a front view (angle of 0 degrees) is set to 13 degrees, this is 1 / 5 of the 62 degrees of a typical TV device. Similarly, the vertical viewing angle is set asymmetrically between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 50 times higher.

[0175] Furthermore, assuming the viewing angle characteristics shown in Example 2 of Figure 12, if the viewing angle at which brightness is 50% of that at a front view (angle of 0 degrees) is set to 5 degrees, this is 1 / 12 of the 62 degrees of devices used for general TV applications. Similarly, the vertical viewing angle is set equal at the top and bottom, and the reflection angle and area of ​​the reflective surface of the reflective light guide are optimized to keep the viewing angle to about 1 / 12 of that of devices used for general TV applications. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, and brightness is more than 100 times higher.

[0176] As described above, by setting the viewing angle to a narrow angle, the amount of luminous flux directed in the monitoring direction can be concentrated, significantly improving the efficiency of light utilization. As a result, even when using a liquid crystal display panel for general TV applications, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant improvement in brightness with similar power consumption, making it possible to create a video display device that is compatible with information display systems aimed at bright outdoor areas.

[0177] When using a large LCD panel, the brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 11 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation and a monitoring distance of 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As described above, depending on the size of the liquid crystal display panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the centre of the screen.

[0179] As shown in Figure 9, the basic configuration involves a light source device directing a light beam with a narrow angle of directionality to a liquid crystal display panel 11, which is then luminance-modulated according to a video signal. The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflector, and the resulting spatially floating image is displayed indoors or outdoors via a transparent member 100.

[0180] By using the display device and light source device according to the embodiment of the present invention described above, it is possible to realize a space floating image display device with higher light utilization efficiency.

[0181] <Example of Image Display Processing in Space-Floating Image Display Device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to Fig. 13A. In the space-floating image display device 1000, when the far side of the space-floating image 3 as seen from the user is inside the housing of the space-floating image display device 1000 and is sufficiently dark, the user will visually recognize that the background of the space-floating image 3 is black.

[0182] Here, an example of displaying a character "panda" 1525 in a floating-in-space image 3 will be described with reference to Fig. 13A. First, for an image including a pixel region for drawing the image of the character "panda" 1525 and a transparent information region 1520 that is a background image, as shown in Fig. 13A(1), the image control unit 1160 in Fig. 3 distinguishes and recognizes the pixel region for drawing the image of the character "panda" 1525 from the transparent information region 1520 that is a background image.

[0183] A method for distinguishing and recognizing the character image from the background image is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the character image layer in front of the background image layer can be processed as separate layers, and the character image and background image can be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the image control unit 1160 recognizes black pixels that depict objects such as character images and transparent information pixels as different information. However, it is assumed that both the black pixels that depict objects and the transparent information pixels have a luminance of 0. In this case, when the Space Floating Image 3 is displayed, there is no difference in luminance between the pixels that depict black in the image of the character "panda" 1525 and the pixels of the transparent information region 1520, which is the background image. Therefore, in the Space Floating Image 3, as shown in FIG. 13A (2), neither the pixels that depict black in the image of the character "panda" 1525 nor the pixels of the transparent information region 1520 have luminance, and they are visually perceived by the user as the same optically black space. In other words, the black portions of the image of the character "panda" 1525, which is an object, blend into the background, and only the non-black portions of the character "panda" 1525 are perceived as floating in the display region of the Space Floating Image 3.

[0185] An example of image processing according to this embodiment will be described with reference to FIG. 13B. FIG. 13B is a diagram illustrating an example of image processing that more effectively resolves the issue of the black image region of the object blending into the background, as described in FIG. 13A. In FIGS. 13B(1) and 13B(2), the upper side shows the display state of the floating-in-space image 3, and the lower side shows the input / output characteristics of the image processing of the image of the object. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or memory 1109 in FIG. 3, or may be input from the video signal input unit 1131, or may be acquired via the communication unit 1132.

[0186] In the state shown in Figure 13B(1), the input / output characteristics of the image processing of the object image are in a linear state with no particular adjustment. In this case, the display state is the same as in Figure 13A(2), and the black image area of ​​the object blends into the background. In contrast, in Figure 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the image of the object (character "panda" 1525) to the input / output characteristics shown in the lower row.

[0187] That is, the video control unit 1160 performs image processing with input / output characteristics on the image of the object (character "panda" 1525), which has a characteristic of converting pixels in low-brightness areas of the input image into output pixels with increased brightness values. After the image of the object (character "panda" 1525) has been subjected to the image processing with input / output characteristics, a video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B (2), the display state of the floating in space image 3 is such that the brightness of pixel areas depicting black in the image of the character "panda" 1525 increases. This allows the user to distinguish the areas depicting black among the areas depicting the image of the character "panda" 1525 without them blending into the black background, making it possible to display the object more appropriately.

[0188] 13B(2), the area displaying the image of the object character "panda" 1525 can be distinguished from the black background inside the housing of the space-floating image display device 1000 through the window, improving the visibility of the object. Therefore, for example, even if the object includes pixels with a brightness value of 0 among the pixels constituting the object before the image processing (i.e., at the time when the image of the object and the corresponding data are read from the storage unit 1170 or memory 1109 in FIG. 3, or at the time when the image of the object is input from the video signal input unit 1131, or at the time when the data of the object is acquired via the communication unit 1132, etc.), the image processing of the input / output characteristics by the video control unit 1160 converts the object into an object with increased brightness values ​​of pixels in low-brightness areas, and then the object is displayed on the display device 1 and converted into a space-floating image 3 by the optical system of the space-floating image display device 1000.

[0189] That is, the pixels that make up the object after image processing of the input / output characteristics are converted to a state in which they do not include pixels with a brightness value of 0, and then they are displayed on the display device 1 and converted into a floating image 3 by the optical system of the floating image display device 1000.

[0190] In the image processing of Figure 13B(2), a method for applying image processing with the input / output characteristics of Figure 13B(2) only to the image area of ​​the object (character "panda" 1525) is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the image processing with the input / output characteristics of Figure 13B(2) is applied to the character image layer while not applying this image processing to the background image layer.

[0191] Then, by combining these layers, image processing with a characteristic of raising the low-brightness areas of the input image is performed only on the character image, as shown in Fig. 13B (2). Alternatively, after the character image layer and background image layer are combined, image processing with the input / output characteristics of Fig. 13B (2) may be performed only on the character image area.

[0192] Furthermore, the input / output image characteristics used in the image processing for enhancing low-luminance regions of the input / output characteristics for the input image are not limited to the example shown in FIG. 13B(2). Any image processing for enhancing low luminance may be used, including so-called brightness adjustment. Alternatively, image processing for improving visibility by controlling the gain that changes the weighting of Retinex processing, as disclosed in International Publication WO 2014 / 162533, may be performed.

[0193] According to the image processing of FIG. 13B(2) described above, it is possible to make the user aware of areas where black is drawn among areas where images of characters, objects, etc. are drawn without blending into the black background, thereby realizing a more suitable display.

[0194] 13A and 13B, the problems and more suitable image processing were explained using a space-floating image display device in which the background appears black (for example, the space-floating image display device 1000 in FIGS. 4A to 4G, or the space-floating image display device 1000 in a state where the rear side window is shaded in FIGS. 4I and 4J). However, this image processing is also effective for devices other than these space-floating image display devices.

[0195] Specifically, in the space-floating image display device 1000 of Fig. 4H and the space-floating image display device 1000 in Fig. 4I and Fig. 4J where the rear window is not shaded, the background of the space-floating image 3 is not black, but the scenery behind the space-floating image display device 1000 through the window. In this case, the problems explained in Fig. 13A and Fig. 13B also exist.

[0196] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the scenery behind the space-floating image display device 1000 through the window. In this case, by using the image processing of Fig. 13B (2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognized as being distinct from the scenery behind the space-floating image display device 1000 through the window, improving the visibility of the object.

[0197] That is, by using the image processing of FIG. 13B(2), the area displaying the image of the character "panda" 1525, which is an object, can be recognized as distinct from the scenery behind the space floating image display device 1000 through the window, and it can be more easily recognized that the character "panda" 1525, which is the object, is in front of the scenery, improving the visibility of the object.

[0198] 4K, 4L, and 4M, as described above, when another image (such as an image from the transmissive self-luminous image display device 1650 or an image from the second display device 1680) is displayed at a position different in depth from the space-floating image 3, the background of the space-floating image 3 is not black, but the other image. In this case, the problems described in FIGS. 13A and 13B also exist.

[0199] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the other image that is displayed at a different depth from the floating image in space 3. In this case as well, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognizable as being distinct from the other image, improving the visibility of the object.

[0200] In other words, by using the image processing of Figure 13B (2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the other image, and it can be more easily recognized that the object character "panda" 1525 is in front of the other image, improving the visibility of the object.

[0201] An example of the image display process of this embodiment will be described with reference to Fig. 13C. Fig. 13C shows an example of the image display process of this embodiment, in which the space floating image 3 and a second image 2050, which is another image, are simultaneously displayed. The second image 2050 may correspond to the image displayed by the transmissive self-luminous image display device 1650 of Fig. 4K or 4L. The second image 2050 may also correspond to the image displayed by the second display device 1680 of Fig. 4M.

[0202] That is, the example of the image display in Fig. 13C shows a specific example of the image display examples of the space-floating image display device 1000 in Figs. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in space-floating image 3. Areas other than the bear character in space-floating image 3 are displayed in black, and become transparent as a space-floating image. In addition, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0203] 13C , the floating-in-space image 3 and the second image 2050 are displayed at different depth positions. When the user 230 views the two images, the floating-in-space image 3 and the second image 2050, in the line of sight of the arrow 2040, the user 230 can view the two images superimposed on each other. Specifically, the bear character of the floating-in-space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0204] Here, since the space floating image 3 is formed as a real image in the air, when the user 230 moves their viewpoint slightly, they can recognize the depth of the space floating image 3 and the second image 2050 due to parallax. Therefore, the user 230 can get a stronger sense of floating in space from the space floating image 3 while viewing the two images in an overlapping state.

[0205] An example of the image display process of this embodiment will be described with reference to Fig. 13D. Fig. 13D(1) is a diagram of the floating in space image 3, from the example of image display of this embodiment in Fig. 13C, as seen from the line of sight of the user 230. Here, a bear character is displayed in the floating in space image 3. The area other than the bear character in the floating in space image 3 is displayed in black, and the floating in space image is transparent.

[0206] 13D(2) is a diagram showing the second image 2050 in the example of the video display of this embodiment shown in FIG. 13C as viewed from the line of sight of the user 230. In the example shown in this figure, the second image 2050 is a background image in which a plain, a mountain, and a sun are depicted.

[0207] 13D(3) is a diagram showing the state in which the second image 2050 and the floating in space image 3 appear superimposed in the line of sight of the user 230, among the example of image display of this embodiment in Fig. 13C. Specifically, the bear character of the floating in space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0208] Here, when the space-floating image 3 and the second image 2050 are displayed simultaneously, it is desirable to pay attention to the balance of brightness between the two images in order to more favorably ensure the visibility of the space-floating image 3. If the second image 2050 is too bright compared to the brightness of the space-floating image 3, the displayed image of the space-floating image 3 will be transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, the output of the light source of the spatially floating image 3 and the display image brightness of the display device 1, and the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device should be set so that at least the brightness per unit area of ​​the spatially floating image 3 at the display position of the spatially floating image 3 is greater than the brightness per unit area of ​​the image light that reaches the display position of the spatially floating image 3 from the second image 2050.

[0210] Note that, since this condition only needs to be satisfied when the space-floating image 3 and the second image 2050 are displayed simultaneously, when switching from the first display mode in which the space-floating image 3 is not displayed and only the second image 2050 is displayed to the second display mode in which the space-floating image 3 and the second image 2050 are displayed simultaneously, control may be performed to reduce the brightness of the second image 2050 by lowering the output of the light source of the display device that displays the second image 2050 and / or the display image brightness of the display device. These controls may be realized by the control unit 1110 in Fig. 3 controlling the display device 1 and the display device that displays the second image 2050 (the transmissive self-luminous image display device 1650 in Fig. 4K or 4L or the second display device 1680 in Fig. 4M).

[0211] Note that when switching from the above-described first display mode to the above-described second display mode, if control is performed to reduce the brightness of second image 2050, the brightness may be reduced uniformly across the entire screen of second image 2050. Alternatively, instead of reducing the brightness uniformly across the entire screen of second image 2050, the brightness reduction effect may be greatest in the portion where an object is displayed in space-floating image 3, and the brightness reduction effect may be gradually weakened around that portion. In other words, the visibility of space-floating image 3 can be sufficiently ensured by reducing the brightness of second image 2050 only in the portion where space-floating image 3 is visually recognized as being superimposed on second image 2050.

[0212] Here, since the space floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes their viewpoint, the parallax causes a change in the superimposed position of the space floating image 3 relative to the second image 2050. Therefore, when switching from the above-mentioned first display mode to the above-mentioned second display mode, if the brightness is to be reduced unevenly across the entire screen of the second image 2050, it is not desirable to reduce the brightness sharply based on the outline of the object displayed in the space floating image 3, and it is desirable to perform gradation processing of the brightness reduction effect, which changes the brightness reduction effect in stages depending on the position as described above.

[0213] In addition, in the space floating image display device 1000 where the position of the object displayed in the space floating image 3 is approximately at the center of the space floating image 3, the position where the brightness reduction effect of the gradation processing of the brightness reduction effect is greatest can be set to the center position of the space floating image 3.

[0214] According to the image display process of this embodiment described above, the user 230 can view the space floating image 3 and the second image 2050 more favorably.

[0215] Note that when displaying the space-floating image 3, control may be performed so as not to display the second image 2050. Since not displaying the second image 2050 increases the visibility of the space-floating image 3, this is suitable for applications such as the space-floating image display device 1000 in which the user must be able to reliably view the space-floating image 3 when it is displayed.

[0216] <Example 2> As Example 2 of the present invention, an example of another configuration example of a space-floating image display device will be described. Note that the space-floating image display device according to this example is obtained by changing the optical system stored in the space-floating image display device described in Example 1 to the optical system shown in Figure 14 (1) or Figure 14 (2). In this example, differences from Example 1 will be described, and repeated explanations of the same configuration as Example 1 will be omitted. Note that in the following description of this example, the predetermined polarized light and the other polarized light are polarized waves whose phases differ by 90° from each other.

[0217] Fig. 14(1) shows an example of an optical system and optical path according to this embodiment. The optical system shown in Fig. 14(1) is configured such that the display device 1 is closer to the polarization separation member 101B in the optical system of Fig. 2C, making the entire optical system more compact. In Fig. 14(1), components that are assigned the same reference numerals as those in Fig. 2C will not be described in detail again.

[0218] 14(1), similar to Fig. 2C, image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, similar to Fig. 2C, polarization separation member 101B selectively transmits the predetermined polarized light (P-polarized light in the figure) emitted from display device 1 and reflects the other polarized light (S-polarized light in the figure).

[0219] Therefore, image light of a predetermined polarization (P-polarized light in the figure) traveling in the vertical direction from the image display surface of the display device 1 passes through the polarization separation member 101B and reaches the retroreflector 2 to which the λ / 4 plate 21 is attached. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has passed through the λ / 4 plate 21 twice, and is converted from the predetermined polarization (P-polarized light in the figure) at the time of emission from the display device 1 to the other polarization (S-polarized light in the figure). The image light that travels again toward the polarization separation member 101B is the other polarization (S-polarized light in the figure), and is therefore reflected by the polarization separation member 101B toward the position where the user should be. The traveling direction of the image reflected by the polarization separation member 101B is determined based on the angle at which the polarization separation member 101B is disposed.

[0220] In the example of Figure 14 (1), the image light traveling toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The image light reflected by the polarization separation member 101B forms a floating image 3A in space. The floating image 3A can be viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the position where the space floating image 3A is formed. This relationship determines the position where the space floating image 3A is formed in the traveling direction of the image light reflected by the polarization separating member 101B.

[0222] In the example of Figure 14(1), the display device 1, the polarization separating member 101B, and the retroreflector 2 are arranged closer than in the example of Figure 2C. This allows the entire optical system to be configured more compactly. However, the amount by which the space-floating image 3A protrudes from the optical system of Figure 14(1) is not very large. For example, as an indicator of the amount by which the space-floating image 3A protrudes from the optical system, the figure shows the distance from the position where the central light beam of the image light is reflected by the polarization separating member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)).

[0223] 14(1), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0224] Next, Fig. 14(2) shows another example of an optical system and an optical path according to this embodiment. The optical system of Fig. 14(2) is a modified version of the optical system of Fig. 14(1) in order to increase the amount of the floating image projecting from the optical system while realizing the same compactness as the optical system of Fig. 14(1). In Fig. 14(2), the components with the same reference numerals as Fig. 14(1) will not be described in detail again.

[0225] 14(2), as in FIG. 14(1), image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, the polarization characteristics of polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). Image light of a predetermined polarized light (P-polarized light in the figure) traveling in a direction perpendicular to the image display surface of display device 1 passes through polarization separation member 101B.

[0226] 14(1), the image light passing through the polarization separation member 101B is not faced with the retroreflector 2 having the λ / 4 plate 21 attached thereto, but with the specular reflector 4 having the λ / 4 plate 21B attached thereto. Here, the reflection at the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the image light that has passed through polarization separation member 101B is specularly reflected by specular reflector 4 to which λ / 4 plate 21B is attached. The image light that has been specularly reflected by specular reflector 4 and travels again toward polarization separation member 101B has been converted from the predetermined polarization (P-polarized in the figure) at the time of emission from display device 1 to the other polarization (S-polarized in the figure) by having passed through λ / 4 plate 21 twice. The image light that has traveled again toward polarization separation member 101B is the other polarization (S-polarized in the figure), and is therefore reflected by polarization separation member 101B.

[0228] Here, because the orientation of the polarization separation member 101B in Figure 14(2) is different from that in Figure 14(1), the image light reflected by the polarization separation member 101B travels in the opposite direction from the position where the user should be. A retroreflector 2 with a λ / 4 plate 21C attached is disposed at the destination of the image light reflected by the polarization separation member 101B. The image light is retroreflected by the retroreflector 2. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has been converted from the other polarized light (S-polarized light in the figure) to the specified polarized light (P-polarized light in the figure) by passing through the λ / 4 plate 21C twice.

[0229] The image light that travels back toward the polarization separation member 101B is of a predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and continues toward the location where the user should be. The image light that has passed through the polarization separation member 101B forms a floating image 3B in space. The floating image 3B can be easily viewed by the user from the direction of arrow A.

[0230] 14(2), as in the case of Fig. 14(1), due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the formation position of the space floating image 3B. This relationship determines the formation position of the space floating image 3B in the traveling direction of the image light transmitted through the polarization separation member 101B.

[0231] 14(2), the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is longer than the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 in FIG. 14(1). This is because in the optical system of FIG. 14(2), an optical path going back and forth between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of FIG. 14(1), is added to the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2.

[0232] As a result, in the optical system of Figure 14(2), the distance from the position where the central ray of the image light passes through the polarization separation member 101B to the position where the image light forms the space-floating image 3B (L2 in the example of Figure 14(2)) is significantly longer than the distance from the position where the central ray of the image light is reflected by the polarization separation member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0233] 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0234] As described above, according to the optical systems of Fig. 14(1) and Fig. 14(2) in the second embodiment of the present invention, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to increase the amount of the floating image projecting from the optical system, despite the more compact optical system.

[0235] In addition, when the optical system of Fig. 14(1) or Fig. 14(2) is incorporated into a space-floating image display device, it can be realized by replacing the optical system in the space-floating image display device described in Example 1 with the optical system of Fig. 14(1) or Fig. 14(2). Specifically, the optical system of Fig. 14(1) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4H, Fig. 4I, Fig. 4J, Fig. 4K, Fig. 4L, or Fig. 4M. In this case, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0236] Specifically, the optical system of Fig. 14(2) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4K, or Fig. 4L. In this case, it is possible to increase the amount of the space-floating image projecting from the optical system. Also, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0237] <Embodiment 3> As embodiment 3 of the present invention, a system including a space-floating image display device 1000 will be described. As the space-floating image display device 1000 of embodiment 3, any of the space-floating image display devices 1000 shown in the drawings of embodiment 1 or embodiment 2 may be used. In this embodiment, differences from embodiment 1 or embodiment 2 will be described, and repeated explanations of the same configurations as those embodiments will be omitted.

[0238] In Example 3, a case is shown in which video captured by a camera of a live concert or the like is distributed and displayed in real time or non-real time on a display device such as a floating video display device. In this example, a case of live concert video distribution is described as an example. However, the present invention is not limited to this example and can also be applied to various types of video distribution, broadcasting, video distribution on packaged media, etc. Furthermore, while this video distribution is accompanied by audio distribution, it is not limited to this and may also be applied to distribution of video only. Furthermore, the video distribution in this example is not limited to video of real people, but can also be applied to video of virtual characters, etc. In the latter case, the video distribution may also be accompanied by distribution of 3D models and motion information (in other words, motion data) described below.

[0239] Furthermore, the video distribution in this embodiment is not limited to a real-time live distribution method coinciding with the time of a live concert or the like, but can also be applied to an on-demand distribution method. It can also be applied to a so-called VOD (video on demand) method, in which filmed video of a live concert or the like is uploaded and registered to a server, and then the video is received as streaming from the server to a display device in response to a user's request at a time desired by the user. It can also be applied to a download playback method, in which video data or the like is downloaded to a display device in advance and played back at a time desired by the user. In other words, the video distribution in this embodiment also includes the case where video is filmed at a live concert or the like and then distributed at a later date.

[0240] In addition, various formats, such as MPEG1, MPEG2, MPEG4, H.264, HEVC, VVC, and FLV, can be applied to the video data format of this embodiment. Various formats, such as MP3, MPEG4-AAC, MPEG4-ALS, MP4, AAC, MPEG-H 3D Audio, AC-4, and Enhanced AC-3, can be applied to the audio data format of this embodiment. Furthermore, formats such as C3D, CSV, FBX, BVH, BVA, and TRC can be applied to transmit motion information. Video and audio may be transmitted in a system format or container format such as MOV, MP4, FLV, HLS, MPEG-DASH, CMAF, ISO BMFF, and MMT. When transmitting motion information, if these system formats or container formats do not have a dedicated area for motion information, the motion information may be stored in a user data area, extended information area, or additional information area.

[0241] In addition, in this embodiment, the processing of video data of camera-captured images will be mainly described, but it is not limited to this, and it is also possible to handle data that is a combination of video data and other types of data such as accompanying audio data and text data.

[0242] [Display System] Fig. 15 shows an example of the configuration of a display system including the space-floating image display device 1000 of Example 3. The display system of Fig. 15 is, in other words, a live camera system, a camera image transmission / reception system, etc. The display system of Fig. 15 is a system in which a system including a camera 1501 at a concert venue 1500 and a system including the space-floating image display device 1000 used by a user 230 are connected via a communication network 1509 and a server 1503. The communication network 1509 includes a WAN or LAN such as the Internet or a mobile network. The server 1503 is a computer system such as a business's server that controls the camera image distribution.

[0243] The display system in Fig. 15 is a system having a function of displaying a video of a live concert (hereinafter sometimes simply referred to as a live performance) of a group 1510 performed as a performance at a concert venue 1500, taken by a camera 1501, as an aerial image (space-floating image) 3 on the space-floating image display device 1000 of a user 230 via communication and a server 1503. The server 1503 has a live camera distribution function 1530 as a function of distributing the video to the space-floating image display device 1000. The space-floating image display device 1000 has a live camera function 1511 as a function of displaying the video distributed through the server 1503 as aerial image 3L. The user 230 can watch the live aerial image 3L on the space-floating image display device 1000.

[0244] In addition, in the third embodiment, there is provided an image taken by a camera 1501 that captures the entire target live performance and group 1510 (camera image 1600 of overall camera C0 in FIG. 16), and separately, images of each view taken by individual cameras 1501 that capture individual members 1502, etc. (camera image 1601 of individual camera C1 in FIG. 16, etc.), from which the user 230 can select and view the camera image of the desired view. This display system, which includes the space-floating image display device 1000 and the server 1503, has a live camera change function 1512 that changes to and displays the selected camera image from the multiple camera images based on the user 230's operation input to a predetermined user interface (UI).

[0245] At a concert venue 1500, a live performance by a group 1510 is held on a specified date and time. The group 1510 may be, in other words, a musician, artist, performer, etc. A live performance may also be a concert, performance, etc. A live performance includes expressions such as music, video, and dance. These live performances, concerts, and performances may also be referred to as events. The live performance may also be a talk performance, etc., in which a narrator mainly talks. The group 1510 has one or more members 1502. The members 1502 may be singers, dancers, etc., or may be speakers who perform talk performances. In the example of FIG. 15 , the members 1502 are real people, but in other examples described below, they may be artificial characters / avatars, etc. In the example of FIG. 15 , as in FIG. 16 described below, there are multiple people as members 1502 of the group 1510, and the subjects of the photograph are these multiple members 1502. However, this is not limited to this, and the subject of the photograph may be only one person. A single subject may be photographed from different directions using multiple cameras 1501.

[0246] [Live Performance and Cameras at a Concert Venue] FIG. 16 shows an example of the configuration of multiple members 1502 of a group 1510 and multiple cameras 1501 at a concert venue 1500. In this example, the group 1510 performing a live performance has three members 1502, members M1, M2, and M3, and the overhead view is illustrated as a circle. Camera 1501 captures the live performance by group 1510. In this example, multiple cameras 1501 are installed at the concert venue 1500, including an overall camera C0 and individual cameras C1 to C3. Overall camera C0 is a dedicated camera device for capturing the entire concert live performance, including group 1510. The individual cameras are dedicated camera devices for capturing individual members. Individual camera C1 mainly captures member M1, individual camera C2 mainly captures member M2, and individual camera C3 mainly captures member M3.

[0247] During the live broadcast, each camera 1501 may move, and the position, orientation, and angle of view of the camera 1501 may change. Each camera 1501 may be controlled by a filming staff member or automatically. The camera 1501 may also be equipped with a microphone or the like for recording live audio, and audio data may be obtained along with the video data 1504. In a modified example, when only one person is to be filmed, multiple cameras 1501 are provided to film that person at different positions and orientations.

[0248] 15 , each camera 1501 is equipped with a communication function and transmits video data 1504 of the captured video to a server 1503 via a communication network 1509. Note that another communication device connected to the camera 1501 may acquire the video data 1504 from the camera 1501 and transmit it to the server 1503.

[0249] In this embodiment (FIG. 15), as a method of live distribution, a live performance performed on a certain date and time is filmed by a camera 1501, and is distributed in real time to the space-floating image display device 1000 via a server 1503 and displayed. Although there is a certain degree of delay due to the processing and communication of the server 1503, the user 230 can watch the live concert video in roughly real time. Not limited to this, in another embodiment (FIG. 21 described later), as a method of on-demand distribution, a live performance performed on a certain date and time is filmed once and registered in the server 1503, and later distributed from the server 1503 to the space-floating image display device 1000 in response to a request from the user 230 on a different date and time.

[0250] The server 1503 acquires the video data 1504 of the camera image from the camera 1501 in the concert venue 1500 via communication, and accumulates and stores it in a memory database (DB) or the like. The server 1503 performs the necessary processing for distribution on the video data 1504, and accumulates and stores the video data 1505 obtained after the processing in a memory DB or the like. The server 1503 transmits the video data 1505 to the space floating image display device 1000 of the user 230 via communication.

[0251] The space-floating image display device 1000 of the user 230 is, for example, a space-floating image display device 1000 owned by the user 230, but is not limited to this and may be a device installed in a store, public space, etc. The space-floating image display device 1000 receives image data 1505 from the server 1503, stores it in memory, and forms aerial image 3L of a live concert video as the aerial image 3 based on the image data 1505. In this way, the space-floating image display device 1000 realizes a mini live stage using the aerial image 3L, and the user 230 can enjoy viewing the aerial image 3L of the live concert video on the mini live stage.

[0252] The space floating image display device 1000 of this embodiment is equipped with a live camera function 1511 and a live camera change function 1512. This live camera change function 1512 is a function that can switch the camera image displayed in the aerial image 3 based on an input operation and selection by the user 230. The user 230 can select a camera image of a view corresponding to a desired member 1502, etc., via a predetermined UI, and can switch to the aerial image 3L of the selected camera image.

[0253] [Space-Floating Image Display Device] Figure 17 shows an example configuration of the space-floating image display device 1000 of the display system of Figure 15. This example configuration is based on the optical system configurations of Figures 2D and 4O, and if they are rotated 45 degrees in the drawing, they will have a roughly similar configuration. In the side view (Y-Z plan view) shown on the right, image light is emitted upward in the Z direction from a display device 10 (similar to Figure 2D, for example) installed at the bottom of the housing 1190, and the image light is retroreflected forward in the Y direction by a retroreflector 5 arranged at an angle. The image light forms a real aerial image (space-floating image) 3 at a predetermined position projecting forward in the Y direction. In this example, the aerial image 3 is formed on a stage 1190S, which is a part of the housing 1190. The stage 1190S is a model stage part at the bottom of the housing 1190 that extends forward in the Y direction, and is configured as a pseudo-stage for the aerial image 3, a miniature stage simulating a live concert stage.

[0254] In the front view (X-Z plane view) shown on the left, a display area 3R for aerial image 3 is formed on stage 1190S, and aerial video 3L showing live performance members 1502 and the like is formed within display area 3R. User 230 can easily view aerial video 3L in the Y direction.

[0255] The housing 1190 is configured to include a light-blocking material, and has components such as the display device 10 and optical system fixed therein in a predetermined positional relationship. A transparent member 100 or the like may be provided in front of the retroreflector 5 or in a position in front of the retroreflector 5 in the Y direction, for example, in a position between the retroreflector 5 and the stage 1190S shown in the figure. A part of the housing 1190 may be used as a canopy / roof above the stage 1190S.

[0256] In the configuration example of FIG. 17 , a control communication device 1701 having implementations corresponding to the control unit 1110 and the communication unit 1132 (FIG. 3) is provided in a part of the housing 1190 and is connected to the display device 10, etc. The control communication device 1701 can be connected to the communication network 1509 (FIG. 15). The control communication device 1701 may be implemented as a separate control device and communication device. The control communication device 1701 may be located inside or outside the housing 1190.

[0257] In the configuration example of Fig. 17, the aerial operation detection sensor 1351, the imaging unit 1180, etc. (Fig. 3) are not shown, but they can be provided as in the above-mentioned embodiment. Also, as means for the user 230 to input operations to the space floating image display device 1000, the operation input unit 1107, the microphone 1139, etc. (Fig. 3) can also be used. As the operation input unit 1107, in addition to a remote control, hardware such as a dedicated button can be provided. Also, as means for outputting to the user 230, in addition to the display of the aerial image 3, various displays, an audio output unit 1140, etc. (Fig. 3) can be used as in the above-mentioned embodiment.

[0258] In the configuration example of FIG. 17 , a speaker 1702 and an indicator 1703 are also installed near the stage 1190S of the housing 1190. The speaker 1702 corresponds to the audio output unit 1140 in FIG. 3 . The space-floating image display device 1000 outputs audio from the speaker 1702 in synchronization with the display of the aerial image 3, based on audio data provided in association with the video data 1505 from the server 1503. The audio output from the speaker 1702 also includes audio output such as guides and alarms as part of the UI. The indicator 1703 displays the status of the space-floating image display device 1000 using a display device such as an LED, and allows operation input to the space-floating image display device 1000, such as power on / off, mode switching, volume, cursor movement, and execute / cancel, using an operation device such as a button.

[0259] [Hardware: First Configuration Example] FIG. 18A shows a first configuration example of the hardware of the space-floating image display device 1000 in Example 3. Like FIG. 17 , the first configuration example is based on FIGS. 2D and 4O. Image light from the display device 10 is emitted in the Y direction via the retroreflector region 5R of the retroreflector 5, forming an aerial image 3 (display range 3R) on the stage 1190S. In the first configuration example, as shown in the top view and side view, a light-shielding plate 1801 is provided vertically on the X-Z plane at the rear of the housing 1190 in the Y direction, further back than the retroreflector 5. The light-shielding plate 1801 may be a plate provided with a black film. As a result, when the aerial image 3 is viewed in direction A (Y direction) from the viewpoint of the user 230, the background of the aerial image 3 appears dark, allowing for optimal viewing of the aerial image 3L of the live performance members, etc.

[0260] 18A also shows, as an example, an example of the installation of the aerial operation detection sensor 1351 and the camera 1180 serving as the imaging unit 1180. The aerial operation detection sensor 1351 is installed facing upward, for example, on the stage 1190S, so as to sense the display range 3R, which is the formation position of the aerial image 3, or its vicinity. The aerial operation detection sensor 1351 may also be installed facing downward on the top of the housing 1190. As in FIG. 4M, the aerial operation detection sensors 1351 may also be installed on the left and right sides of the housing 1190. Also, in FIG. 18A , the camera 1180 is installed on the top of the housing 1190 in an orientation that allows it to capture an image of the user 230 who is in front of the aerial image 3. The camera 1180 may also be installed in another position in an orientation that allows it to capture the aerial image 3.

[0261] If aerial operation detection sensor 1351 is provided, it is possible to detect an operation of user 230 touching aerial image 3L with their fingers as an aerial operation, and it is possible to control, for example, changing the display of aerial image 3L in accordance with the aerial operation. If a GUI is displayed on aerial image 3, it is possible to operate the GUI. If imaging unit 1180 is provided, it is possible to detect the presence of user 230, and it is possible to control, for example, changing the display of aerial image 3 in accordance with the detection.

[0262] [Hardware: Second Configuration Example] FIG. 18B shows a second configuration example of the hardware. In the second configuration example, a second display panel 1902 is provided in the background of the aerial image 3. The second display panel 1902 corresponds to the second display device 1680 in FIG. 3 and is the second display panel when the liquid crystal display panel 11 included in the display device 10 is used as the first display panel 1901. The second configuration example is based on the configuration of FIG. 4M. The display device 10 including the first display panel 1901 is disposed at the bottom of the housing 1190 and emits image light upward in the Z direction. The image light passes through a polarizing beam splitter 101B disposed at an angle and is retroreflected and polarization-converted by a retroreflector 2 with a λ / 4 plate 21 disposed on a top panel 1903 at the top of the housing 1190. The polarization-converted image light is reflected by the polarizing beam splitter 101B and forms an aerial image 3 at a predetermined position forward in the Y direction.

[0263] In the second configuration example, similarly to Fig. 4M, aerial operation detection sensors 1351 may be provided on the left or right of aerial image 3L. In the example of Fig. 18B, aerial operation detection sensor 1351 is provided facing downward on top panel 1903, and can detect an aerial operation when the fingers of user 230 penetrate deep into aerial image 3L.

[0264] [Second Display Panel and Background Image] As shown in the front view, space-floating image display device 1000 of the second configuration example in FIG. 18B uses second display panel 1902 to display background image 1904 against aerial image 3L. Space-floating image display device 1000 displays aerial image 3L, which includes images of live performance members, etc., in aerial image 3, which is an optical image, and displays background image 1904 of aerial image 3L on the screen of second display panel 1902. Image light corresponding to background image 1904 displayed on the screen of second display panel 1902 passes through polarized beam splitter 101B to the front in the Y direction. When viewing aerial image 3L in direction A (Y direction) from the viewpoint of user 230, the image can be viewed with aerial image 3L of the live performance superimposed on the foreground of background image 1904.

[0265] The background image 1904 can be a background image or background video of a live concert, or other image or video. The background image 1904 can be a video based on a video of a live background or the like captured by a camera 1501 at an actual concert venue 1500. The server 1503 may create video data (background video data) for the background image 1904 based on the video data 1504 of the camera 1501. The server 1503 may also create a video (pseudo background) that imitates the live background based on the video data 1504.

[0266] Furthermore, the background image 1904 is not limited to an image of an actual live performance background or an image created to imitate it, but can also be any image / video. Examples of the content of the arbitrary image / video include natural scenery and virtual space. In this case, a video can be provided in which a background image 1904 with content different from the actual live performance background is superimposed on an aerial video 3L showing members 1502, etc. The server 1503 or the space-floating image display device 1000 may use default background video data created and set in advance as the background video data that forms the basis of the background image 1904. The user 230 may be able to select and specify one of multiple default background videos to use via the UI. The default background video data may be pre-set and stored in the space-floating image display device 1000. The server 1503 may specify the background video data to be used via control information. Alternatively, the default background video data stored by the server 1503 may be transmitted to the space-floating image display device 1000.

[0267] It is also possible to overlay a suitable background image 1904 in accordance with the content of the camera image in aerial image 3L (e.g., selected member 1502). For example, it is possible to select background image 1904 of a suitable color and brightness depending on the color and brightness of aerial image 3L.

[0268] Furthermore, the server 1503 or the space-floating image display device 1000 may perform variable control of the background image 1904. The server 1503 may create control information for controlling the background image 1904 based on the image data 1504 of the camera 1501, and send it to the space-floating image display device 1000. The display of the background image 1904 may be fixed on the time axis, or may be periodically switched.

[0269] For example, the server 1503 may perform control to select a background image 1904 and switch the display in accordance with the user 230's selection of a camera image corresponding to the member 1502 or the like and the switching of the display.

[0270] Also, for example, server 1503 may create video data or control information for background image 1904 so that the display content of background image 1904 can be changed in conjunction with the display content of live aerial image 3L based on video data 1505.

[0271] Space-floating image display device 1000 displays aerial image 3L according to image data 1505 from server 1503, and also displays background image 1904 according to image data or control information of the background image from server 1503. This realizes background image 1904 linked with aerial image 3L. A three-dimensional image representation can be provided in which aerial image 3L is superimposed on the foreground of background image 1904. In another configuration example, space-floating image display device 1000 may create image data or control information for background image 1904 based on image data 1505 from server 1503, and control the display of background image 1904.

[0272] For example, server 1503 may extract only the image portion of target member 1502 from video data 1504 from camera 1501, and create video data 1505 for aerial video 3L. Server 1503 may then create background video data or control information for overlaying a desired background image 1904 on aerial video 3L of extracted member 1502.

[0273] Furthermore, the space-floating image display device 1000 may display video / images transmitted from the server 1503 or the camera 1501 during a live broadcast as a background image 1904 on the second display panel 1902. If there is a camera 1501 that captures the live background, the camera image may be used as the background image 1904. The space-floating image display device 1000 may display video / images specified by control information transmitted from the server 1503 to the space-floating image display device 1000 during a live broadcast as a background image 1904 on the second display panel 1902 at a specified timing.

[0274] [Stage Lighting] Furthermore, the second configuration example of FIG. 18B includes stage lighting 1905 formed by stage lighting device 1906. Stage lighting 1905 illuminates aerial image 3, the vicinity of aerial image 3, or user 230. In the example of FIG. 18B , stage lighting device 1906 is installed on the top of housing 1190, on the front side of top panel 1903, near the upper side of the position where aerial image 3 is formed. The direction of stage lighting 1905 by stage lighting device 1906 is between aerial image 3 and user 230, facing toward stage 1190S. The range of this stage lighting 1905 covers aerial image 3 and the upper surface of stage 1190S. It is preferable that the direction and range of stage lighting 1905 be such that it illuminates the upper surface of stage 1190S and the front side of aerial image 3L.

[0275] Stage lighting 1905 is a separate element from background image 1904. Stage lighting 1905 is lighting that adds a stage effect to live aerial image 3L and background image 1904. Stage lighting 1905 is more preferably lighting that can vary not only the on / off timing of light emission, but also the color, amount, and direction of light emission. Use of stage lighting 1905 can enhance the sense of realism, etc.

[0276] As a modified example, the effect lighting 1905 may be eliminated, and the background image 1904 may be displayed by the second display panel 1902. Also, as a modified example, the display of the background image 1904 by the second display panel 1902 may be eliminated, and the effect lighting 1905 may be displayed.

[0277] Server 1503 or space-floating image display device 1000 may control the timing and color of light emission as variable control of effect lighting 1905. In particular, server 1503 may create control information for effect lighting 1905 based on video data 1504 from camera 1501 and send it to space-floating image display device 1000. This control information is information for variably controlling the timing and color of light emission of effect lighting 1905 so as to be linked with the display content of aerial image 3L corresponding to video data 1505. Space-floating image display device 1000 displays aerial image 3L according to video data 1505 from server 1503, and drives effect lighting device 1906 according to the control information from server 1503. This realizes effect lighting 1905 linked with the display content of aerial image 3L. A three-dimensional image expression can be provided in which effect lighting 1905 is superimposed on aerial image 3L. In another configuration example, the space floating image display device 1000 may create control information for effect lighting 1905 based on the image data 1505 from the server 1503 and drive the effect lighting device 1906.

[0278] 18C shows a third example of the hardware configuration. The third example of the configuration differs from the second example of the configuration in that a display device 10 including a first display panel 1901 is disposed on an upper surface plate 1903 at the top of a housing 1190, and a retroreflector 2 is disposed below. Image light from the display device 10 is emitted downward, passes through a polarizing beam splitter 101B, and is retroreflected by the retroreflector 2 with a λ / 4 plate. The polarization-converted image light is reflected by the polarizing beam splitter 101B, and forms an aerial image 3 at a position on a stage 1190S.

[0279] In the third configuration example, similar to the second configuration example, aerial image 3 displays live aerial video 3L, and second display panel 1902 can display background image 1904. Similarly, in the third configuration example, aerial operation detection sensor 1351 and imaging unit 1180 may be provided. In the example of Fig. 18C, aerial operation detection sensor 1351 is provided facing downward in a position on the near side of the position where aerial image 3 is formed. When the fingers of user 230 come in front of aerial image 3L, this can be detected as an aerial operation.

[0280] Furthermore, similar to the second configuration example, the third configuration example may include effect lighting 1905. Based on control information from effect lighting 1905, space-floating image display device 1000 varies the timing and color of light emission from effect lighting device 1906 in conjunction with aerial image 3L, and illuminates stage 1190S. In the example of Fig. 18C , effect lighting 1905 is directed downward from effect lighting device 1906 toward aerial image 3L, slightly toward the back.

[0281] [Hardware: Fourth Configuration Example] Figure 18D shows a fourth configuration example of hardware. The difference between the fourth configuration example and the second configuration example is that the display device 10 including the first display panel 1901 is disposed on a side portion of the housing 1190 in the X direction, which is the left-right direction, i.e., on a side panel 1907 in Figure 18D. As shown in the top view, for example, the display device 10 including the first display panel 1901 is disposed on the side panel 1907 on the right side as viewed from the user 230 in the X direction, and the retroreflector 2 is disposed on the left side panel 1907. Image light from the display device 10 is emitted leftward, passes through the polarizing beam splitter 101B, and is retroreflected by the retroreflector 2 with a λ / 4 plate. The polarization-converted image light is reflected by the polarizing beam splitter 101B and forms an aerial image 3 at a position on the stage 1190S.

[0282] In the fourth configuration example, similar to the second configuration example, a live aerial video 3L is displayed on the aerial image 3, and a background image 1904 can be displayed on the second display panel 1902. Similarly, in the fourth configuration example, an aerial operation detection sensor 1351 and an imaging unit 1180 may be provided. When the aerial operation detection sensors 1351 are provided on the left and right sides of the display range 3R of the aerial image 3, side panels 1907 are provided so as to extend to the left and right sides of the stage 1190S. Similarly, in the fourth configuration example, performance lighting 1905 may be provided.

[0283] [Example of Control of Combinations] As described above, the configuration example of Example 3 in FIG. 18B and other figures includes components including aerial image 3L based on video data 1504 and 1505 from camera 1501, background image 1904 on second display panel 1902, and effect lighting 1905 from effect lighting device 1906. Various types of control and image representation are possible by combining these components. By controlling these components, visual effects and image representations such as aerial image 3L visible to user 230 are realized. When background image 1904 and effect lighting 1905 are combined with aerial image 3L, the sense of three-dimensionality and realism can be further enhanced.

[0284] FIG. 19 shows a table summarizing various control examples. The table shows the relationship between aerial image 3L corresponding to the camera image, background image 1904, and stage lighting 1905, with each row showing a control example. In Example 3, as in each control example, it is also possible to combine and link the various components for control. The type of control to apply can be selected depending on the design and implementation of the display system. Alternatively, the user may be able to select from multiple controls and set them via a UI.

[0285] (Example 1) Only aerial image 3L is displayed as live concert camera footage. Background image 1904 and stage lighting 1905 are turned off. The content of aerial image 3L is an image including people or characters (described below) of group 1510 performing the live concert. Note that the absence (off) of background image 1904 and stage lighting 1905 may be output as a predetermined default setting.

[0286] (Example 2) Display of a combination of aerial image 3L and background image 1904. The content of background image 1904 can be selected from a background image corresponding to the background of the live concert, a background image independent of the background of the live concert, a default image / video, etc. The display of background image 1904 can be controlled by server 1503 or space-floating image display device 1000 using control information or background image data.

[0287] (Example 3) Display of aerial image 3L combined with effect lighting 1905. Effect lighting 1905 may be variably controlled in terms of timing, color, amount, direction, etc. of light emission. Server 1503 or space floating image display device 1000 can control the variable control of effect lighting 1905 using control information.

[0288] As an example of controlling lighting effect 1905, lighting effect 1905 may be controlled in conjunction with the display of aerial image 3L of the camera image of a selected camera / member, so that lighting effect 1905 has different content depending on the camera / member. For example, lighting effect 1905 may be set to red for the image of member M1, and blue for the image of member M2.

[0289] (Example 4) Display of aerial image 3L in combination with background image 1904 and effect lighting 1905. The display content of background image 1904 and the variable control content of effect lighting 1905 are controlled in association with each other. Server 1503 or space-floating image display device 1000 controls both background image 1904 and effect lighting 1905 using control information or the like.

[0290] (Example 5) Display of aerial image 3L combined with stage lighting 1905. Furthermore, video expressions, display effects, etc. linked to the variable control content of stage lighting 1905 are added to the display content of aerial image 3L. For example, display control may be performed to change the brightness or color of the surface of the person / character in aerial image 3L in accordance with the timing and color of the light emission of stage lighting 1905. In other words, this is the expression of a pseudo-lighting effect, as if stage lighting 1905 were shining on the person / character in aerial image 3L.

[0291] [Example of Variable Control of Stage Lighting 1905] An example of variable control of stage lighting 1905 corresponding to Examples 3 to 5 above is as follows. If stage lighting is used at a live performance at concert venue 1500, the stage lighting is captured by camera 1501 and reflected in video data 1504. Server 1503 and space-floating image display device 1000 may coordinate stage lighting 1905 with the stage lighting included in video data 1504. For example, suppose that red light is shining on a certain member 1502 at a certain time during a live performance. Server 1503 performs image processing or rendering processing based on video data 1504 to create video data 1505 of a view targeted at member 1502, and also creates control information for stage lighting 1905. In doing so, server 1503 creates control information for stage lighting 1905 so that red stage lighting 1905 is shining on aerial image 3L in synchronization with the movement of member 1502.

[0292] Furthermore, server 1503 may variably control the content of background image 1904 or video data 1505 of aerial image 3L so as to synchronize with red effect lighting 1905. For example, server 1503 may create control information or background video data for background image 1904 so that the display content of background image 1904 also reflects the display of red, synchronized with the timing at which effect lighting 1905 is turned red.

[0293] Furthermore, for example, server 1503 may create video data 1505 of aerial image 3L so that the brightness and other aspects correspond to a state in which target member 1502 is illuminated with red light by calculating lighting through processing based on video data 1504. In this way, when staged lighting 1905 and aerial image 3L are overlaid, a synergistic effect can be achieved, creating a visual effect as if the member (person / character) in aerial image 3L were illuminated with light.

[0294] [Live Camera Change Function] The live camera change function 1512 ( FIG. 15 ) in Example 3 is a function that can display a desired view image selected from the overall view image of the concert live performance (overall view camera image 1600 in FIG. 16 ) and the images of each individual member (individual view camera images 1601 to 1603 in FIG. 16 ) as an aerial image 3 (aerial image 3L). The view can be expressed in other words as the angle of view or the viewpoint.

[0295] FIG. 20 is an explanatory diagram of the live camera change function 1512, particularly showing an example of a GUI (graphical user interface) that allows the user 230 to select a desired camera / view. The space-floating image display device 1000 displays a GUI image 1550 as shown in FIG. 20 in the aerial image 3. The GUI image 1550 is a GUI that allows a user to select a desired camera image from the overall camera image and individual camera images of each member. In other words, the GUI image 1550 is a GUI that allows a user to select a desired view from the overall view and individual member views. The GUI image 1550 includes, for example, an overall view icon 1551 and individual view icons 1552, 1553, and 1554. The overall view icon 1551 is a GUI for selecting an overall view camera image 1600, which is an image of the overall live view / view of all members of the group, captured by the overall camera C0 in FIG. 16. The individual view icon 1552 is a GUI for selecting an individual view camera image 1601, which is an image of a view of member M1 taken by the individual camera C1 in Fig. 16. Each icon may include an identification number or ID, a thumbnail image of the camera image, etc.

[0296] User 230 selects the image of the desired camera / view by performing an operation input on the icon of GUI image 1550. The operation input may be, for example, detection of a touch operation using aerial operation detection sensor 1351. When a camera / view is selected, space-floating image display device 1000 displays the image of that camera / view as aerial image 3L on aerial image 3. Display of GUI image 1550 is not limited to aerial image 3, and if space-floating image display device 1000 is equipped with another display (e.g., second display panel 1902), GUI image 1550 may be displayed on that display.

[0297] In the case of such a GUI, the user 230 can more easily select a desired video image from a plurality of cameras / members etc. corresponding to a plurality of options prepared in advance.

[0298] [Aerial Operation Detection Sensor] Fig. 29 shows, as a supplement, a configuration example of the aerial operation detection sensor 1351, which is an example of a user operation detection mechanism. Fig. 29 shows an implementation example of the aerial operation detection sensor 1351 applicable to Example 3 etc. The aerial operation detection unit 1350, control unit 1110, video control unit 1160 etc. in Fig. 3, in other words the video processing unit, uses the detection signal of the aerial operation detection sensor 1351 to determine and detect aerial operations with respect to the display range 3R of the floating-in-space image 3.

[0299] FIG. 29 shows an xy plane view aligned with the display range 3R of the floating-in-space image 3. The aerial operation detection sensor 1351 has multiple optical elements 1351c arranged in the x direction. The optical elements 1351c are pairs of a light-emitting element 1351a and a light-receiving element 1351b. The light-emitting element 1351a is composed of, for example, an infrared element. The light-emitting element 1351a emits light a1, for example, infrared light, in the y direction. If the light a1 is not blocked by an object, it passes through the display range 3R. If the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2. The reflected light a2 is received by the light-receiving element 1351b.

[0300] For example, if a contact point a3 made by the user 230's finger is within the x-y plane of the display range 3R, light a1 is reflected from the contact point a3 and returns as reflected light a2. A light receiving element 1351b at a certain x-direction position detects the reflected light a2. This allows the aerial operation detection unit 1350 to determine that the contact point a3 is located at that x-direction position. Furthermore, the aerial operation detection unit 1350 can calculate the distance using the TOF method from the time it takes for the emitted light a1 to return as reflected light a2. For example, it can calculate the distance a4 to the contact point a3. This also allows the position coordinates of the contact point a3 on the x-y plane of the display range 3R to be determined.

[0301] Not limited to this example, the aerial operation detection sensor 1351 may be placed above, to the left or right of, or at a position shifted in the z direction, which is the front-to-back direction, with respect to the xy plane of the display range 3R. Multiple aerial operation detection sensors 1351 may be placed at multiple positions in the front-to-back direction.

[0302] [Display System: Real Space: On-Demand Distribution] FIG. 21 illustrates an example configuration of a display system in Example 3, targeting a live concert in real space. FIG. 21 also illustrates the case of on-demand distribution via a server 1503. Members 1502 of a group 1510 being filmed are real people. This display system uploads video data 1504 of video captured by each camera 1501 of multiple cameras 1501 filming the live performance of the group 1510 at a concert venue 1500 to a server 1503. The server 1503 registers and stores live camera video data 2130, which is video data for distribution created by video processing or the like based on the video data 1504, in a database. In response to a request (request information 2101) from the display device 2000 of the user 230, the server 1503 distributes video data 1505 of a camera video selected from the live camera video data 2130. The display device 2000 of the user 230 displays the camera image based on the video data 1505 received from the server 1503. Video distribution in this display system may be via the Internet or via broadcast waves.

[0303] In the example of FIG. 21 , group 1510 includes two members M1 and M2 as members 1502, and two cameras 1501, a general camera C0 and two individual cameras C1 and C2. For example, member M1's ID / name is Koto, and member M2's ID / name is Airia. For example, camera C1 captures member M1, and camera C2 captures member M2. Each camera 1501 transmits and uploads video data 1504 of the captured video to server 1503 via communication. Server 1503 stores the video data 1504 from each camera 1501 in memory, performs predetermined processing such as video editing, creates video data for distribution, and registers it in a database as live camera video data 2130. The live camera video data 2130 includes video from each view.

[0304] The user 230 uses a display device 3000 such as the space-floating image display device 1000 or a television (television display device) as the display device 2000. For the sake of explanation, the display device 2000 refers to the space-floating image display device 1000 and / or the display device 3000. The display device 2000 is, in other words, a user display device. For example, the user 230 owns a space-floating image display device 1000 and a display device 3000 such as a television in their home. The example in FIG. 21 shows a case in which the user 230 uses two display devices 2000: the space-floating image display device 1000 and the television display device 3000. However, the user 230 may have only one display device, or three or more. In this example, the live camera function 1511 and the like can be used on any of the display devices 2000. The space-floating image display device 1000 and the display device 3000 each have predetermined functions such as the live camera function 1511 and the live camera change function 1512 described above. In the case of the space floating image display device 1000, the camera image is displayed as the aerial image 3, as described above. In the case of the display device 3000 such as a television, the camera image is displayed on a display screen with a fixed pixel surface. In other words, the display device 3000 is a fixed pixel surface display device, a device that displays images and videos on a display screen that is a two-dimensional surface made of fixed pixels, and includes various display devices such as television display devices and monitor devices.

[0305] 21 shows a state in which a user 230 simultaneously uses two display devices 2000, selecting and viewing an overall view image on the display device 3000, which is a television with a relatively large screen, and selecting and viewing an individual view image on the space-floating image display device 1000, which has a relatively small screen compared to the television. The user 230 can view camera images in parallel on the two display devices 2000. Note that the size relationship of the display screens of the multiple display devices 2000 is not limited, and the display device 3000 side may have a relatively small screen, and the space-floating image display device 1000 side may have a relatively large screen.

[0306] The server 1503 may perform predetermined processing such as image processing on the video data 1504 from the camera 1501 to create video data for distribution. Examples of such processing include trimming or extracting the target member 1502 to generate a two-dimensional video, as well as correction processing and noise removal processing to create a clear image / video, brightness adjustment processing, enlargement / reduction processing, etc. Note that the server 1503 may not necessarily perform image processing.

[0307] In the display device 2000, the live camera change function 1512 allows the user 230 to select the video of a desired camera 1501 from multiple cameras 1501 based on operational input via the UI. Alternatively, the user 230 can select a desired member 1502 from multiple members 1502. The UI for selecting the camera 1501 or member 1502 may be, for example, the GUI shown in FIG. 20 . Alternatively, the selection UI may be a method in which the user 230 specifies the ID or name of the member 1502 of interest. The selection UI may also be a method in which the user 230 specifies a number using a remote control or button to select a camera or member associated with that number. Alternatively, the camera or member may be selected by inputting a name, number, or the like using voice input or voice recognition. The camera / member video to be displayed may also be selected and set in advance for each display device 2000 using the UI.

[0308] In response to the user 230's selection of a camera, member, or the like on the UI, the display device 2000 generates and determines selection information representing the selection and transmits request information 2101 including the selection information to the server 1503. The selection information may be camera selection information or member selection information. The camera selection information is information for selecting the camera 1501 or a camera image associated with the camera. The member selection information is information for selecting the member 1502. In other words, this selection information is information for selecting a view or a display target. The request information 2101 includes identification information for the requesting display device 2000 and the requesting server 1503. Based on the selection information in the request information 2101, the server 1503 selects camera image data to be distributed from the live camera image data 2130 in the DB and transmits the video data 1505 of the camera image to the requesting display device 2000. For example, if a certain member 1502 is selected, the video to be distributed is video captured by the camera 1501 that captured the member 1502.

[0309] The display device 2000 may provide not only the above-mentioned selection UI but also a UI that allows user settings, etc. The UI allows, for example, user account registration and search for live concerts.

[0310] In the configuration example of FIG. 21 , the display device 2000 of the user 230 may be linked to a mobile terminal 4000 (in other words, a mobile information terminal device) such as a smartphone owned by the user 230. When the UI for the selection is a GUI, the GUI need not be limited to the GUI on the screen of the display device 2000, but may be a GUI on the display screen of the mobile terminal 4000 wirelessly connected to the display device 2000. In this case, the user 230 performs an operation input to the GUI of the mobile terminal 4000, and the mobile terminal 4000 transmits selection information based on the operation input to the display device 2000. The display device 2000 transmits request information 2101 including the selection information and the like to the server 1503. Alternatively, the request information 2101 including the selection information and the like may be transmitted directly from the mobile terminal 4000 to the server 1503.

[0311] [Camera Video Selection Method] A supplementary explanation will be given regarding the method for selecting camera video to be displayed in the live camera change function 1512. In the configuration example of Fig. 21, there are two selection methods: 1. camera selection method, and 2. member selection method.

[0312] 1. Camera Selection Method: In the camera selection method, the user 230 selects and specifies the camera 1501 to be displayed based on a predetermined UI such as the example in Fig. 20, and the display device 2000 generates camera selection information. The server 1503 determines the camera video to be distributed based on the camera selection information. In this case, the user 230 only needs to select a desired one from multiple cameras as multiple options prepared in advance on the UI, making it easier for the user 230 to set the video to be displayed.

[0313] 2. Member Selection Method: In the member selection method, the user 230 selects and designates the entire group 1510 or a member 1502 (a person in FIG. 21 ) of interest within the group 1510 based on a predetermined UI, and the display device 2000 generates member selection information. The server 1503 determines the camera video to be distributed based on the member selection information. In this case, the user 230 simply selects a desired member (a person of interest) from multiple members as multiple options prepared in advance on the UI, and the camera video of the view corresponding to that member can be selected, making it easier for the user 230 to view the video of the desired member 1502.

[0314] In the configuration example of FIG. 21 , there are two display devices 2000, and the user 230 can display selected camera images on each display device 2000 and view them simultaneously. One display device 2000, for example, a television display device 3000, displays camera images corresponding to the camera / member selected by the user 230 on its screen, while the other display device 2000, for example, the space-floating image display device 1000, displays a different camera image corresponding to a different camera / member selected by the user 230 as aerial image 3L. The user 230 can view different view images simultaneously. In the example of FIG. 21 , the user 230 can view an overall view image of the entire live event as a main camera image on the large screen of the television display device 3000, and simultaneously view an individual view image focusing on a specific member 1502 on the relatively small screen of the space-floating image display device 1000, which has a display range 3R.

[0315] The live camera change function 1512 can switch to another camera image, in other words, to another view, even while camera images are being played back. If a new camera / member is selected as an operation input by the user 230 via the UI while camera images are being played back, the display device 2000 and the server 1503 perform control based on that operation to switch the display to the camera image corresponding to the newly selected camera / member, using the same processing as described above.

[0316] [Selection Information Table] Fig. 22A shows an example of the configuration of a selection information table as a data table held by the server 1503 or the display device 2000 for control purposes in relation to camera / member selection on the UI, corresponding to the configuration example of Fig. 21 . The server 1503 determines the camera video to be distributed based on this selection information table in response to the selection information in the request information 2101 from the display device 2000. The selection information transmitted from the display device 2000 to the server 1503 may be either camera selection information or member selection information, as described above. Regardless of the selection information, the video to be distributed can be determined from this table.

[0317] The table in FIG. 22A has the following items: video (corresponding to video data), shooting camera, subject (corresponding to view), camera selection information, and member selection information, and their correspondence is set. For example, in row #1, it is set that video A is an overview camera video, the shooting camera is camera A (overall camera C0 in FIG. 21), and the subject is group 1510, in other words, all members 1502 {M1, M2}. Also, in row #1, it is set that video A will be selected when the camera selection information is "camera A (number 1)" or the member selection information is "group (all members)." Note that if numbers are assigned to cameras / members in advance, they can also be specified by number ( FIG. 20 ). If members 1502 have names, they can also be specified by name.

[0318] The selection information table in FIG. 22B similarly shows a selection information table relating to another configuration example (FIG. 26) described later.

[0319] [Display Device (Television)] An example configuration of a display device 3000 will be described using FIG. 23 . The display device 3000 may be, for example, a display device with a large screen, such as a television. FIG. 23 shows an example functional block configuration of the display device 3000, which is a television. The display device 3000 includes a display panel 30011, a control unit (including a processor) 30012, an external power input interface 30013, a power supply 30014, a secondary battery 30015, a storage unit 30016, a video control unit 30017, a communication unit 30020, an audio output unit (including a speaker) 30021, an audio input unit (including a microphone) 30022, a video signal input unit 30023, an audio signal input unit 30024, an imaging unit 30025, and the like.

[0320] The display panel 30011 may be a two-dimensional display screen using fixed pixels, such as a liquid crystal panel or an organic EL panel, or a device configured to display images or videos on a fixed pixel surface. The display panel 30011 may also be referred to as a display unit.

[0321] The communication unit 30020 is configured by a device that implements, for example, a Wi-Fi (registered trademark) communication interface, a Bluetooth (registered trademark) communication interface, a mobile communication interface such as 4G or 5G, etc. Using these communication methods, the communication unit 30020 can communicate with a communication unit of an external device, for example, the space floating image display device 1000. Furthermore, the communication unit 30020 can communicate with an external device, for example, the server 1503, that is connected to a communication network 1509 such as the Internet, using any of the communication methods.

[0322] The control unit 30012 includes a processor such as a CPU and internal memories such as a ROM and a RAM, and controls each unit such as the video control unit 30017 and the communication unit 30020, as well as the entire display device 3000. The control unit 30012 may execute processing in accordance with a program read by the processor into the internal memory or the memory 30026. The control unit 30012 may cooperate with the program stored in the memory 30026 to perform arithmetic processing based on information acquired from each unit in the display device 3000.

[0323] The memory 30026 stores data processed by the control unit 30012, the video control unit 30017, etc. The memory 30026 stores, for example, video data of the video to be displayed on the display panel 30011, control data for the display device 3000, etc. The control unit 30012 may read various software programs from the storage unit 30016, and expand and store them in the memory 30026.

[0324] The nonvolatile memory 30027 holds various data and information used by the display device 3000. The data stored in the nonvolatile memory 30027 includes, for example, various operation data to be displayed on the display panel 30011, display icons, object data for user operation, layout information, and the like.

[0325] The power supply 30014 converts AC current input from the outside via the external power supply input interface 30013 into DC current and supplies the DC current required for each unit of the display device 3000. The secondary battery 30015 stores the power supplied from the power supply 30014. Furthermore, when power is not supplied from the outside via the external power supply input interface 30013, the secondary battery 30015 supplies power to each unit requiring power.

[0326] The video signal input unit 30023 connects to an external video output device and inputs video data. The video signal input unit 30023 can use various digital video input interfaces. For example, a video input interface conforming to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard can be used. Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The video signal input unit 30023 may use various USB interfaces, etc.

[0327] The audio signal input unit 30024 connects to an external audio output device and inputs audio data. The audio signal input unit 30024 can be an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. The audio signal input unit 30024 may also be any of various USB interfaces. In the case of an HDMI-standard interface, the video signal input unit 30023 and the audio signal input unit 30024 may be configured as an interface in which a terminal and a cable are integrated.

[0328] The audio output unit 30021 includes a speaker and can output audio based on audio data input to the audio signal input unit 30024. The audio output unit 30021 can also output audio based on audio data stored in the storage unit 30016. The audio output unit 30021 may output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 30021 may be configured to output audio as a digital signal to an external device, such as the Audio Return Channel function defined in the HDMI standard.

[0329] The audio input unit 30022 includes a microphone, collects sounds around the display device 3000, converts the collected sounds into signals, and generates audio signals. The audio input unit 30022 may be configured to record a person's voice, such as the voice of the user 230, with the microphone, and have the control unit 30012 or the like perform voice recognition processing on the generated audio signal to acquire text information from the audio signal.

[0330] A camera having an image sensor can be applied to the imaging unit 30025. A camera may be provided on the front side of the display panel 30011 side of the display device 3000, or on the back side. Both a front camera and a back camera may be provided.

[0331] The storage unit 30016 is a storage device that records various data and information such as video data, image data, audio data, character data, and programs. The storage unit 30016 may be configured, for example, with a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor element memory such as a solid-state drive (SSD). For example, various data and information such as video data may be recorded in the storage unit 30016 before product shipment. The storage unit 30016 may record various data and information such as video data, image data, and audio data acquired from an external device, such as a server, via the communication unit 30020. The video data, etc. recorded in the storage unit 30016 is output to the display panel 30011. The video data, etc. recorded in the storage unit 30016 may also be output to an external device via the communication unit 30020. Various data may also be stored in a storage device externally connected to the display device 3000.

[0332] The video control unit 30017 performs various controls related to the video signals input to the display panel 30011. The video control unit 30017 may be configured with hardware such as an ASIC, FPGA, or video processor (so-called GPU). The video control unit 30017 may also be referred to as a video processing circuit, video processing unit, image processing unit, display processing unit, etc. The video control unit 30017 may be included as part of the display panel 30011.

[0333] The video control unit 30017 controls video switching, such as which video signal among a plurality of video signals including a video signal (corresponding video data) to be stored in the memory 30026 and a video signal (corresponding video data) input to the video signal input unit 30023 is to be input to the display panel 30011. The video control unit 30017 may also control image processing of the video signal input from the video signal input unit 30023 and the video signal to be stored in the memory 30026. Examples of image processing include scaling processing that enlarges, reduces, or deforms an image, brightness adjustment processing that changes the brightness, contrast adjustment processing that changes the contrast curve of the image, and Retinex processing that decomposes an image into light components and changes the weighting of each component.

[0334] In the above description, the display device 3000 is described as a fixed pixel surface display device including a display panel 30011 that displays images and videos on a two-dimensional display screen using fixed pixels, but is not limited to this. The concept of this fixed pixel surface display device also includes a projection-type video display device, in other words, a projector. This projector has a projection optical system provided after the display panel with a fixed pixel surface, and projects an optical image of the video displayed on the display panel with the fixed pixel surface onto a predetermined surface such as a screen via the projection optical system.

[0335] [Server] Fig. 24 shows an example of the configuration of the server 1503 as a computer system. The server 1503 includes a control unit 15031 including a processor, a memory 15032, a communication unit 15033, an input / output interface 15034, and the like, which are interconnected by an architecture such as a bus. An input device 15035 and an output device 15036 are externally connected to the input / output interface 15034. The communication unit 15033 is equipped with a communication interface compatible with the communication network 1509, etc. A user such as an administrator operates the server 1503 via the input device 15035 and the output device 15036. The user may operate the server 1503 using a client terminal device that is communicatively connected to the server 1503.

[0336] The memory 15032 stores a control program 2401, setting information 2402, a database (DB) 2403, screen data 2404, etc. The control program 2401 is a program for realizing predetermined functions such as the live camera distribution function 1530. The processor of the control unit 15031 executes processing in accordance with the control program 2401 to realize predetermined functions such as the live camera distribution function 1530. The setting information 2402 is system setting information and user setting information. Video data for distribution, etc. is registered in the DB 2403. The screen data 2404 is data that constitutes a screen such as a web page. The server 1503 may manage and provide the live camera distribution service and web pages related to the GUI, etc.

[0337] [Transmission and Reception of Video Data and Control Information] FIG. 25 shows an example of a configuration for transmission and reception of video data and control information between the server 1503 and the display device 2000 in the display system of FIG. 21 . The server 1503 receives video data 1504 from the camera 1501 at the concert venue 1500. The video processing unit 2501 of the server 1503 performs processing based on the video data 1504 to create video data 1505 for distribution. The video data 1505 also includes audio data and the like. The video processing unit 2501 is realized by processing by the control unit 15031 ( FIG. 24 ) described above. The video processing unit 2501 also creates control information 2502 for the video data 1505. The control information 2502 may include, but is not limited to, information for controlling the background image 1904 and the stage lighting 1905 ( FIG. 18B , etc.) described above. The control information 2502 is managed in association with the video data 1505. The control information 2502 may be described as part of the attribute information / metadata of the video data 1505. The server 1503 transmits the video data 1505 and the control information 2502 to the display device 2000.

[0338] The display device 2000 receives the video data 1505 and the control information 2502 from the server 1503. The video processing unit 20001 of the display device 2000 generates display data, in other words, a video signal, for displaying an image on the display screen 20005 of the display unit 20002, based on the video data 1505. If the display device 2000 is the space-floating image display device 1000, the display unit 20002 corresponds to the display device 10 in FIG. 17, and the display unit 20002 is followed by the retroreflector 5 in FIG. 17 as an optical system 20003. The display screen 20005 corresponds to the display range 3R of the aerial image 3. If the display device 2000 is a display device 3000 such as a television, the optical system 20003 can be omitted, and the display screen 20005 of the display unit 20002 corresponds to the screen of the display panel 30011 (FIG. 23). Furthermore, if the display device 2000 is a projector, the optical system 20003 corresponds to a projection optical system, and the display screen 20005 corresponds to a screen (projection surface).

[0339] The display device 2000 detects an operation by the user 230 by the user operation detection mechanism 20004. The user 230 performs a user operation on the display screen 20005 or the user operation detection mechanism 20004. When the display device 2000 is the space-floating image display device 1000, an example of the user operation detection mechanism 20004 is the above-mentioned mid-air operation detection sensor 1351 (FIGS. 3 and 29). In other examples, the user operation detection mechanism 20004 is a remote control, a touch panel, an operation button, or the like.

[0340] As an example of control information 2502, as shown in Example 1, when the background image 1904 in Fig. 18B is displayed and the background image 1904 in Fig. 19 is under control, a file of background image control information is transmitted as control information to be used for controlling the background image 1904. If the file in question is already stored in the display device 2000, it is sufficient to simply specify that file. Also, as Example 2, when the effect lighting 1905 in Fig. 18B is displayed and the effect lighting 1905 in Fig. 19 is under control, a file of effect lighting control information is transmitted as control information to be used for controlling the effect lighting 1905. If the file in question is already stored in the display device 2000, it is sufficient to simply specify that file.

[0341] For example, when the space-floating image display device 1000 is equipped with the aforementioned effect lighting device 1906, the server 1503 transmits the image data 1505 of the selected camera image to the space-floating image display device 1000, and also transmits control information for the effect lighting 1905. Then, the space-floating image display device 1000 drives the effect lighting device 1906 so as to change the color of the effect lighting 1905 based on the received control information, in conjunction with the display of the aerial image 3L based on the received image data 1505.

[0342] [Display System: Virtual Space: First Configuration Example] FIG. 26 shows a first configuration example of a virtual space as another configuration example of the display system in Example 3. In this first configuration example, the concert venue 2600 is not a real venue but a virtual space configured on a server 2603, and a concert live performance and a group 2610 are configured within the virtual space. Members 2602 are characters / avatars, etc., configured within the virtual space by calculation, and video is generated by rendering based on 3D model objects and a virtual camera 2601 set within the virtual space. The server 2603 configures the virtual concert venue 2600 by calculation, performs a live performance by the group 2610, and generates video for distribution by rendering. In this example, the members 2602 include members M1 and M2. The configuration example of FIG. 26 can be applied to both live distribution as shown in FIG. 15 and on-demand distribution as shown in FIG. 21.

[0343] The server 2603 constructs and stores 3D models of objects of the concert venue 2600 arranged in the virtual space, such as 3D models of the live stage and background objects, and 3D models of characters who are the members 2602. The server 2603 controls the motion of the 3D model characters during the live performance. The server 2603 uses motion information about the 3D model characters, in other words, motion data, to generate and execute the motion and animation of the characters.

[0344] The server 2603 also controls multiple virtual cameras 2601 placed in the virtual space. The server 2603 controls the position, orientation, angle of view, movement, etc., of each of the multiple virtual cameras 2601 within the virtual space. The server 2603 stores virtual camera information for controlling the virtual cameras 2601. Based on the 3D model object and the virtual camera 2601, the server 2603 uses each virtual camera 2601 to capture images of the entire live performance or individual members 2602, which are the respective set targets, and generates video images of the views of each virtual camera 2601 through rendering processing. The server 2603 registers and stores video data corresponding to the generated images in a database in memory as live camera video data, which is video data 2630 for distribution. The server 2603 stores, in the database, the concert venue 2600 and 3D models of characters, character motion information, virtual camera information, video data 2630, etc.

[0345] In the example of FIG. 26 , virtual cameras C0, C1, C2, and Cf are provided as virtual cameras 2601. Virtual camera C0 captures an overall view video of the entire concert live performance, targeting all members 2602 of group 2610. Virtual camera C1 captures an individual view video of member M1. Virtual camera C2 captures an individual view video of member M2. Virtual camera Cf captures a view video from a virtual viewpoint freely set based on user settings. In the example of FIG. 26 , virtual cameras C0, C1, and C2 are virtual cameras 2601 set in advance by server 2603, and virtual camera Cf is virtual camera 2601 associated with a virtual viewpoint whose position, orientation, etc. can be freely specified and set by user 230 via the UI.

[0346] The environment of the user 230 includes the space floating image display device 1000 and the display device 3000 as the display device 2000, similar to Fig. 21 . The display device 2000 has a live camera function 1511 and a live camera change function 1512, similar to Fig. 15 and Fig. 21 . The user 230 can select a desired concert live and can also select a desired virtual camera 2601 or member 2602, etc., on a predetermined UI, and the display device 2000 transmits request information 2606 including the selection information to the server 2603.

[0347] In the case of a live distribution method, the server 2603 distributes video data 2605 to the display device 2000 on the date and time of the live concert. Alternatively, in the case of an on-demand distribution method, the server 2603 distributes video data 2605 to the display device 2000 in response to request information 2606 from the display device 2000. The server 2603 distributes video data 2605 selected from video data 2630 to the display device 2000 in response to selection information in the request information 2606 from the display device 2000. The display device 2000 displays video on the display screen of the display unit based on the video data 2605 received from the server 2603.

[0348] With regard to the live camera change function 1512, the space floating image display device 1000 or the display device 3000 selects a desired one from among multiple virtual cameras 2601 or characters that are multiple members 2602 in a live concert based on operation input from the user 230 via the UI, and generates and determines selection information that is virtual camera selection information or character selection information. Similar to the configuration example of FIG. 21 , this selection information corresponds to information for selecting the image of the view you want to display. The server 2603 sets the positions and orientations of multiple virtual cameras 2601, for example, C0, C1, and C2, in advance, corresponding to the viewpoints from which each character is viewed, prepares corresponding image data generated by rendering, and registers it in the DB.

[0349] Furthermore, a virtual camera Cf corresponding to a free virtual viewpoint that is not registered as a pre-registered virtual camera 2601 or video data may be set by the user 230. In this case, the server 2603 sets the virtual camera Cf at a position, orientation, etc. specified by the user 230 on the UI, and generates video data of the view of the virtual camera Cf by rendering based on the virtual camera Cf.

[0350] The selection of the virtual camera 2601 may be made by selecting the number of the virtual camera 2601 that has been set and registered in advance in the virtual space. The virtual camera 2601 corresponds to a virtual viewpoint, which is a viewpoint in the virtual space. When specifying and setting a virtual camera Cf that corresponds to a free virtual viewpoint, the specification is made by specifying at least one of parameters such as the position, orientation (angle corresponding to the orientation), or angle of view of the virtual viewpoint in the virtual space.

[0351] When selecting a predefined virtual camera 2601 or when selecting a character that is a group 2610 or a member 2602, the GUI shown in Fig. 20 can be similarly applied as a UI. Also, similar to the above, hardware buttons, voice recognition, the mobile terminal 4000, and the like can be applied.

[0352] [Virtual Camera in Virtual Space] FIG. 27 is an explanatory diagram of a virtual camera 2601 associated with a virtual viewpoint within a concert venue 2600 in a virtual space. FIG. 27 shows a schematic x-y plan view of a character on a live stage 2700 viewed from above, with the virtual space coordinate system set as (x, y, z). In this example, the target selected and focused on by the user 230 is character 2702, who is member M2. An individual virtual camera C2 captures images of character 2702, who is member M2. The position and movement of virtual camera C2 are set and determined in advance by the server 2603. Alternatively, the character 2702 may be captured using a virtual camera Cf that can be freely specified by the user 230. The position and movement of virtual camera Cf can be specified by the user 230 using a predetermined UI.

[0353] On the live stage 2700, a character 2702 moves in a predetermined motion over a timeline in virtual space. An example of the movement trajectory of the character 2702 is indicated by an arrow. An example of setting the virtual camera C2 or virtual camera Cf to match the motion of the character 2702 is described below. For example, at a first time point t1, the virtual camera C2 / Cf is at position p1, facing forward in the +y direction, which is an angle of view capturing the front of the character 2702. At a second time point t2, the virtual camera C2 / Cf is at position p2, facing diagonally left, which is an angle of view capturing the diagonally right front of the character 2702. At a third time point t3, the virtual camera C2 / Cf is at position p3, facing left in the -x direction, which is an angle of view capturing the right side of the character 2702. The virtual viewpoint of the virtual camera C2 / Cf moves along the arc trajectory shown in the figure. Each virtual camera 2601 can also set the zoom amount, in other words, the focus amount, at each time.

[0354] [GUI for Specifying a Virtual Viewpoint] The display system also provides a UI that allows the user 230 to freely specify a virtual viewpoint. FIG. 28 shows an example of a UI for the live camera change function 1512, in which the user 230 can precisely select and specify the position and orientation of the virtual viewpoint corresponding to the virtual camera Cf. FIG. 28 shows an example of a GUI 2800 displayed on the display screen of the display device 3000. This GUI 2800 allows the user to specify the position (x, y, z) and orientation (angles α, β) of the virtual viewpoint corresponding to the virtual camera Cf. For example, the angle α is the azimuth angle, and the angle β is the elevation / depression angle. Furthermore, the preview field 2801 can display a preview of a character in a captured video corresponding to the provisionally specified position and orientation. In the example of FIG. 28 , a character is displayed in the preview captured video. The position and orientation once specified by the user 230 on the UI can be later changed or adjusted. The position and orientation of the virtual viewpoint may be specified precisely using numerical values, or may be specified roughly. For example, the selection may be made from predefined positions and orientations such as in front of the group 2610, diagonally right, right, and behind.

[0355] The position (x, y, z) and orientation (angles α, β) of the virtual viewpoint corresponding to the virtual camera Cf, which can be specified using the GUI 2800, may be absolute coordinates or absolute angles in the virtual space. Furthermore, the position (x, y, z) and orientation (angles α, β) of the virtual viewpoint corresponding to the virtual camera Cf, which can be specified using the GUI 2800, may be relative coordinates or relative angles with respect to the reference coordinates and reference orientation of a specific character. For example, the reference coordinates and reference orientation of a specific character are the position and orientation of a specific bone in the skeletal data of the character's 3D data, which will be described later. When relative coordinates and relative angles with respect to the reference coordinates and reference orientation of a specific character are used, even if the specific character moves or changes its orientation, the virtual camera Cf can track the movement and orientation of the specific character, and can capture the specific character at the desired relative coordinates and relative angle specified by the user using the GUI.

[0356] If there is corresponding video data registered in the DB according to the virtual camera / character / virtual viewpoint selection information, the server 2603 reads it and distributes it to the display device 2000. If there is no corresponding video data, the server 2603 determines settings such as the position and orientation of the virtual camera 2601 (Cf) corresponding to the specified virtual viewpoint in the virtual space according to the selection information. The server 2603 then generates a two-dimensional video of the corresponding view by performing rendering processing on the 3D model object in the virtual space, including a series of motions and animations for each character, from the virtual viewpoint of the specified virtual camera 2601 (Cf). The video data resulting from the rendering is stored in memory. The rendering processing can be realized by the processor of the control unit 15031 executing processing based on a program based on the configuration shown in FIG. 24.

[0357] [Selection Information] A supplementary note regarding the above selection information: 1. When the selection information is virtual camera selection information, the server 2603 selects one virtual camera 2601 from a plurality of preset options of virtual cameras 2601, for example, virtual cameras C0, C1, and C2, based on the selection information. The server 2603 obtains rendered video data based on the settings of the position, orientation, and / or angle of view of the virtual viewpoint corresponding to the selected virtual camera 2601, and transmits the rendered video data to the display device 2000. In this case, since the user 230 only needs to select from a plurality of preset options of virtual cameras 2601, it becomes easier for the user 230 to set a desired viewpoint and view the corresponding video.

[0358] 2. When the selection information is character selection information, the server 2603 selects a character from the members 2602 of the group 2610 based on the selection information, obtains rendered video data based on the setting of the virtual viewpoint of the virtual camera 2601 capturing the character, and transmits the rendered video data to the display device 2000. In this case, the user 230 can select a video from a viewpoint that focuses on the desired character simply by specifying the desired character, making it easier for the user 230 to select and view the video of the desired character.

[0359] 3. When the selection information is virtual viewpoint selection information, the server 2603 obtains a rendered image based on the position, orientation, and / or angle of view of the virtual camera Cf corresponding to the virtual viewpoint specified by the selection information, and transmits the rendered image to the display device 2000. In this case, the user 230 can directly specify a virtual viewpoint and generate an image corresponding to the specified virtual viewpoint, so that the user 230 can view an image from a desired view.

[0360] When the user 230 uses both the space floating image display device 1000 and the display device 3000 as the display device 2000, it is possible to view images of two different views in parallel, similar to the configuration example of Fig. 21. Note that even while playing back live images, the live camera change function 1512 of the display device 2000 can change the virtual camera, character, or virtual viewpoint based on the operation input of the user 230 via the UI, and switch to images of different views.

[0361] In the configuration example of Fig. 26 , similarly to Fig. 18B and the like, it is possible to apply a combination of background image 1904 and effect lighting 1905. In this case, when generating video of a live performance or the like in a virtual space, server 2603 may generate control information for controlling background image 1904 and effect lighting 1905 that matches the video, and may register the control information in association with the video data in a DB. Then, when distributing the video selected from display device 2000, server 2603 distributes the control information together with the video data. Display device 2000 varies and outputs background image 1904 and effect lighting 1905 in conjunction with the display of the live video according to the received video data and control information.

[0362] [Selection Information Table] FIG. 22B shows an example of a selection information table stored in the server 2603 corresponding to the configuration example of FIG. 26 . The selection information table of FIG. 22B is similar to the selection information table of FIG. 22A in rows #1 to #3, with the difference being that the selection information table includes selections related to the virtual camera, character, and virtual viewpoint. For the preset virtual cameras C0, C1, and C2, the position, orientation, angle of view, etc. of the corresponding virtual viewpoint are preset. Although not shown, the movement of the virtual camera 2601 over time ( FIG. 27 ) can also be set. Row #4 is an example of setting a virtual camera Cf corresponding to a free virtual viewpoint, where the position, orientation, angle of view, etc. are set. Furthermore, when setting the virtual camera Cf, a character of interest may also be set, in which case the character of interest is primarily captured in the video.

[0363] 22B may be held in the server 2603, and may also be transmitted from the server 2603 to the display device 2000 and held in the display device 2000. This selection information table may be included as part of a data set of a 3D model or the like in the server 2603, or may be provided as independent control information.

[0364] [Rendering Process] As a supplementary note, the rendering process performed by a server, for example, server 2603 in FIG. 26 , is as follows. The server performs rendering processing on a 3D model object in a calculated virtual three-dimensional space, in other words, rendering-based processing, to create video data for distribution, for example, video data for displaying the aerial image 3L on the floating-in-space image display device 1000. The server performs video processing on the image generated by rendering the virtual three-dimensional space in which the 3D model object is placed. Rendering of the virtual three-dimensional space is performed by capturing an image using a perspective projection method with a virtual three-dimensional space camera (virtual camera 2601 in FIG. 26 ) set within the virtual three-dimensional space. The image generated by rendering is an image rendered by capturing an image of the virtual three-dimensional space in which the object is placed, with the angle of view of the virtual three-dimensional space camera set to satisfy predetermined conditions based on the focal length of the lens of the virtual three-dimensional space camera, the distance at which the user 230 views the screen corresponding to the display range of the floating-in-space image 3, the diagonal length of the screen, etc.

[0365] [Display System: Virtual Space: Second Configuration Example] FIG. 30 shows a second configuration example of a virtual space as another configuration example of the display system in Example 3. In the second configuration example, similar to the first configuration example of FIG. 26 , a live performance at a concert venue 2600 is configured in a virtual space calculated by a server 3003. The difference is that in the second configuration example of FIG. 30 , the display device 2000 on the user 230 side, rather than the server 3003, generates a virtual live performance video by rendering processing based on a data set. The second configuration example of FIG. 30 can be applied to both live distribution and on-demand distribution. Although not shown, the display device 2000 has a live camera function 1511 and a live camera change function 1512, similar to the above.

[0366] The server 3003 stores in a database 3D models of the concert venue 2600 and characters that are placed in the virtual space, motion information for the objects in the 3D models, virtual camera information, etc. This data is a series of original data necessary for generating a virtual live video by rendering on the display device 2000 side, and may be referred to as a data set.

[0367] The display device 2000 used by the user 230, such as the space floating image display device 1000 or the display device 3000, transmits request information 3006 regarding the virtual live show that the user 230 wants to view to the server 3003 based on the operation input of the user 230 via the UI. The request information 3006 is data set transmission request information or data stream transmission request information. In the example of Fig. 30, unlike the example of Fig. 26, the rendering process is performed in the display device 2000. Therefore, in the display device 2000 of Fig. 30, the request information 3006 does not need to include the above-mentioned selection information.

[0368] In response to request information 3006 received from the display device 2000, the server 3003 transmits a data set 3005 corresponding to the requested virtual live to the display device 2000. The transmitted data set 3005 may instead be a data stream, as described below. The transmitted data set 3005 here is a data set that enables generation of images corresponding to various views that can be selected on the display device 2000. Note that the data set 3005 may be transmitted to the display device 2000 after including metadata including preset virtual camera information and the like that will be used as options for generating the selection information table of FIG. 22B .

[0369] The display device 2000 stores the data set 3005 received from the server 3003 in its memory. The user 230 can select and specify the virtual live video of the group 2610 they want to view, the virtual camera 2601, or the member 2602 of interest, etc., on the UI of the display device 2000. The UI may be generated based on the selection information table described above. Selection information corresponding to the selection and specification of the virtual live video of the group 2610 they want to view, the virtual camera 2601, or the member 2602 of interest, etc., on the UI of the display device 2000 is used for rendering processing in the display device 2000. The video processing unit 3020 of the display device 2000 uses the data set 3005 based on the selection information corresponding to the selection and specification on the UI of the display device 2000 to generate video data 3030 for displaying the virtual live video from the selected view on the display unit through rendering processing. The video data 3030 is generated as video data corresponding to the desired virtual camera / character / virtual viewpoint selected and specified by the user 230. Display device 2000 displays a virtual live video on the display screen of the display unit based on that video data 3030. In the case of space-floating image display device 1000, that video is displayed as aerial image 3L.

[0370] Furthermore, with regard to the live camera change function 1512, the display device 2000 selects and specifies the user 230's desired virtual camera, character, or virtual viewpoint based on the user 230's operation input via the UI while the virtual live video is being played, and generates and determines new selection information. The video processing unit 3020 of the display device 2000 generates video data 3030 for displaying the virtual live video from the newly selected view on the display unit by rendering processing based on the selection information corresponding to the user 230's new selection and specification via the UI using the live camera change function 1512. The video data 3030 is generated as video data corresponding to the desired virtual camera / character / virtual viewpoint newly selected and specified by the user 230. The display device 2000 displays the virtual live video on the display screen of the display unit based on the video data 3030. In the case of the space-floating image display device 1000, the image is displayed as aerial image 3L.

[0371] Here, if the data set 3005 that the display device 2000 has already received from the server 3003 is lacking data necessary for rendering processing based on selection information corresponding to a new selection or specification by the user 230 via the UI using the live camera change function 1512, the display device 2000 transmits request information 3006 requesting transmission of the missing data to the server 3003. In response to the request information 3006, the server 3003 transmits the data set 3005 necessary for generating an image of the selected view to the display device 2000. The display device 2000 generates video data 3030 for the image of the selected view by rendering based on the received data set 3005, and displays the image by switching to the image of the selected view in accordance with the video data 3030.

[0372] In another configuration example, selection information may be included in the request information 3006 to the server 3003. The display device 2000 first selects a virtual camera, character, or virtual viewpoint desired by the user 230 based on an operation input from the user 230 via the UI, generates and determines selection information, and transmits request information 3006 including the selection information to the server 3003. In response to the request information 3006, the server 3003 transmits to the display device 2000 a dataset 3005 capable of generating an image of the view, including data capable of generating at least an image of the selected view. The image processing unit 3020 of the display device 2000 generates image data 3030 for an image of a view corresponding to the virtual camera / character / virtual viewpoint desired by the user 230, based on the received dataset 3005 and the initial selection information. The display device 2000 displays an image of the view based on the image data 3030.

[0373] Similarly, with regard to the live camera change function 1512, the display device 2000 generates and determines new selection information in response to the selection and specification of a desired virtual camera, etc., by the user 230 on the UI while playing back the virtual live video. The display device 2000 generates video data 3030 for the video of the newly selected view based on the data set 3005 and the new selection information, and displays the video so as to switch to the video of the selected view in accordance with the video data 3030. Note that if the data set 3005 lacks data necessary to generate the video of the newly selected view, the display device 2000 may transmit request information 3006 regarding the missing data to the server 3003 and obtain the data set 3005 regarding the missing data from the server 3003, as described above.

[0374] In the second configuration example of Fig. 30, the display device 2000 may hold a selection information table such as that shown in Fig. 22B and perform selection processing according to the selection information described above. The selection of the virtual camera / character / virtual viewpoint and the rendering processing performed by the server 2603 in the first configuration example of Fig. 26 may be similarly performed by the display device 2000 in Fig. 30.

[0375] In the second configuration example of FIG. 30 , in the user 230's environment, the display device 2000 with a display function performs a process of generating virtual live video data 3030 by rendering based on a data set 3005. In another configuration example, the device performing this rendering process, etc., may be another device (referred to as a receiving device or a communication device) without a display function, rather than the display device 2000. For example, in the user 230's environment, a receiving device is included as another device connected to the display device 2000. The receiving device transmits request information 3006 to a server 3003 and receives a data set 3005 from the server 3003. The receiving device generates video data 3030 of the video of the selected view by rendering based on the data set 3005 and selection information, and outputs the video data 3030 to the display device 2000. The display device 2000 displays the video on a display unit in accordance with the video data 3030.

[0376] [3D Model and Motion Information] FIG. 31 is a supplementary explanatory diagram of a character's 3D model and motion information. FIG. 31 shows a schematic diagram of skeletal data for a 3D model of a character, e.g., member M1. Skeletal data is data called, for example, bones or rigs. Body parts such as the head, limbs, and so on, composed of vertices, edges, faces, and so on (not shown), are associated with the skeletal data. When the position or orientation of each part of the skeletal data changes, the vertices, edges, and faces of the corresponding body parts are deformed, making the character's 3D model appear to be moving. Motion information includes the position and orientation of each part of the skeletal data for each point in time. The motion information describes the time-series fluctuations of this skeletal data. Motion information can also be generated based on captured video of an actual person's movements using a motion capture system. A two-dimensional image of the character can be generated by rendering based on the skeletal data and the character's 3D model data at each point in time. The skeletal data must be in a data format that can be easily understood by both the server 3003 and the display device 2000. For example, in the example of Fig. 31, a Humanoid data format is used. The motion information also uses a format corresponding to the data format of the skeletal data. This allows the motion information transmitted from the server 3003 to be appropriately understood on the display device 2000 side.

[0377] In the second configuration example of Figure 30, in the case of an on-demand distribution method, a data set 3005 including a 3D model of a character and motion information is distributed from a server 3003 to a display device 2000 at a timing specified by a user 230, and then a virtual live video can be generated and viewed at the timing specified by the user 230 and in the desired selected view.

[0378] Without being limited to this, in the case of a live distribution method, for example, the server 3003 predetermines the date and time, etc., of the virtual live event to be held. Prior to the event, a preliminary dataset 3005 for the virtual live is first distributed from the server 3003 to the display device 2000. The preliminary dataset 3005 includes 3D models of the concert venue 2600 and characters to be placed in the virtual space where the virtual live event will be held, as well as objects of the 3D models. On the display device 2000 side, rendering data included in the dataset 3005 is loaded into memory in advance of the event. Therefore, after the event is held, which is a timing after the transmission of the dataset 3005, a data stream for the virtual live event may be distributed in real time from the server 3003 to the display device 2000. Motion information for the characters may be included in the distributed data stream and transmitted to the display device 2000, which receives the data. The display device 2000 continuously reflects the motion information received in real time in the rendering process. This makes it possible to generate a video in which a 3D character object included in the previously received data set 3005 moves in real time based on the motion information. Also, audio data of the virtual live performance may be included in the data stream together with the motion information and the like, and transmitted and received in real time.

[0379] The display device 2000 can generate a two-dimensional animated image of the selected character through rendering processing based on the 3D model and motion information in the data stream received from the server 3003, and can play back the image and audio in synchronization. This allows the display device 2000 to display virtual live video in roughly real time at the timing of the event, which the user 230 can view. Furthermore, even during playback of the virtual live event, the display device 2000 can switch the displayed view by performing the same selection processing as described above using the live camera change function 1512 based on operation input by the user 230 via the UI.

[0380] 18B and the like can be similarly combined and applied. In this case, the control information for the background image 1904 and the effect lighting 1905 may be generated on the server 3003 side and distributed as part of the data set 3005, or may be generated on the display device 2000 side.

[0381] [Display System of Modified Example] Fig. 32 shows the configuration of a display system of a modified example of Example 3. For example, in the systems of Figs. 21 and 26 described above, the space-floating image display device 1000 and the display device 3000 as the display device 2000 are configured to receive rendered image data on the server side and play back images. On the other hand, in the system of Fig. 30 described above, the space-floating image display device 1000 and the display device 3000 as the display device 2000 are configured to receive a data set, which is unrendered data, from the server side and perform rendering on the display device 2000 side to play back images. That is, in these configurations, the multiple display devices 2000 have similar functions. In contrast, in the modified example of Fig. 32, the multiple display devices 2000, for example, the space-floating image display device 1000 and the television display device 3000, have different functions and correspond to a combination of a configuration in which rendering is performed on the server side and a configuration in which rendering is performed on the display device 2000 side. In the configuration example of Fig. 32, the display device 3000 which is a television has a function corresponding to a configuration in which rendering is performed on the server side as in Fig. 21 or 26, and the space floating image display device 1000 has a function corresponding to a configuration in which rendering is performed on the display device 2000 side as in Fig. 30. The server 3203 in Fig. 32 has a configuration corresponding to both the functions of distributing rendered image data and distributing unrendered data sets.

[0382] Display device 3000, which is a television, for example, sends request information 3201 to server 3203, receives rendered video data 3202 based on dataset 3210 from server 3203 in response to request information 3201, and displays live camera video (the aforementioned virtual live video) on the display screen of the display unit based on that video data 3202. On the other hand, space-floating image display device 1000, for example, sends request information 3211 to server 3203, receives unrendered dataset 3212 or a data stream based on dataset 3210 from server 3203 in response to request information 3211, generates video data 3213 by rendering based on dataset 3212 or the data stream, and displays the live camera video as aerial image 3L on the display unit based on that video data 3213.

[0383] In another configuration example, the functions of the space-floating image display device 1000 and the display device 3000 in Fig. 32 may be interchanged. The display device 3000, which is a television, generates and displays camera images by rendering based on the received data set. The space-floating image display device 1000 displays camera images based on the received rendered image data.

[0384] [Display System: Virtual Space: Third Configuration Example] FIG. 33 shows a third configuration example of a virtual space as another configuration example of the display system in Example 3. In the third configuration example, similar to the configuration examples of FIGS. 21 and 26 described above, a live performance at a concert venue 2600 is generated in a virtual space by a server 3303. The configuration example of the virtual space by the server 3303 is the same as that of FIG. 26. The difference with the third configuration example is that, as the display device 2000 in the user 230's environment, the space-floating image display device 1000 and the display device 3000, which is a television, communicate with each other to display camera images. In the third configuration example, the display device 3000, which is a fixed pixel surface display device, serves as the first screen device, and the space-floating image display device 1000 serves as the second screen device. Although not shown in FIG. 33, each display device 2000 has the aforementioned live camera function 1511 and live camera change function 1512, and at least one of them has a UI for selection.

[0385] The display device 3000, which is the first screen device, transmits request information 3301 to the server 3303 based on an operation by the user 230. The request information 3301 is data set / data stream transmission request information. The server 3303 transmits the requested data set or data stream to the display device 3000 as a response 3302. The response 3302 of the data set or data stream transmitted from the server 3303 to the display device 3000 includes both data for the first screen device (referred to as first data) and data for the second screen device (referred to as second data).

[0386] The space-floating image display device 1000, which is the second screen device, transmits request information 3311 to the display device 3000, which is the first screen device, based on the operation of the user 230. The request information 3311 is partial dataset / partial data stream transmission request information. The display device 3000 transmits the requested partial dataset or partial data stream to the space-floating image display device 1000 as a response 3312. The partial dataset or partial data stream data of the response 3312 transmitted from the display device 3000 to the space-floating image display device 1000 is data obtained by extracting the second data for the second screen device from the dataset / data stream from the server 3303.

[0387] 33, the video processing unit of the display device 3000 generates video data for displaying a virtual live video (e.g., an overall view) on the display device 3000 based on first data of the data set / data stream received from the server 3303, and displays the virtual live video on the display screen according to the video data. On the other hand, the video processing unit of the space-floating image display device 1000 generates video data for displaying a virtual live video (e.g., a view of a selected character) on the space-floating image display device 1000 based on second data of the partial data set / partial data stream received from the display device 3000, and displays an aerial image 3L of the virtual live video according to the video data.

[0388] In the example of Fig. 33, rendering may be performed on the server 3303 side, as in Fig. 26. In this case, the display device 3000 receives the first video data and the second video data, which are rendered video data, from the server 3303 as a data set / data stream. The display device 3000 displays a virtual live video on its own device (display device 3000) based on the first video data. The space-floating image display device 1000, which is the second screen device, receives the second video data from the display device 3000 as a partial data set / partial data stream, and displays an aerial image 3L of the virtual live video on its own device (space-floating image display device 1000) based on the second video data.

[0389] In the example of Fig. 33, rendering may be performed on the display device 2000 side instead of the server 3303, as in Fig. 30. In this case, the display device 3000 receives first data and second data, which are unrendered data, from the server 3303 as a data set / data stream. The video processing unit of the display device 3000 generates video data for displaying a virtual live video (e.g., an overall view) on the display device 3000 by rendering based on the first data of the data set received from the server 3303, and displays the virtual live video on the display screen according to the video data. On the other hand, the video processing unit of the space-floating image display device 1000 receives second data, which is unrendered data, as a partial data set / partial data stream received from the display device 3000. Based on the second data received from the display device 3000, generates video data for displaying a virtual live video (e.g., a view of a selected character) on the space-floating image display device 1000 by rendering, and displays the aerial image 3L of the virtual live video according to the video data.

[0390] In this way, when there are multiple display devices 2000 on the user 230 side, communication with the server 3303 side can be performed by only one display device 2000, and the display can be performed in cooperation between that one display device 2000 and the other display devices 2000.

[0391] As a variant, the opposite of the above may be adopted, with the space-floating image display device 1000 as the first screen device and the display device 3000 as the second screen device, and the space-floating image display device 1000 may receive and acquire data from the server 3303.

[0392] In the configuration example of Fig. 33, as a UI for selecting a desired camera etc. for the live camera change function 1512, at least one of the UI of the display device 3000 and the UI of the space-floating image display device 1000 can be used. Each display device 2000 may have a UI, or the UI may be integrated into one display device 2000. For example, the display device 3000, which is a television, may have a GUI as an integrated UI, and the GUI may enable not only the selection of a view for display on the display device 3000, but also the selection of a view for displaying the aerial image 3L on the space-floating image display device 1000. As a GUI, the example of Fig. 20 etc. can be similarly applied.

[0393] [Data Set / Data Stream] Fig. 34A shows a table summarizing specific configuration examples of data sets / data streams that can be applied to the configuration of linked display between the first screen device (display device 3000) and the second screen device (space-floating image display device 1000) in Fig. 33 .

[0394] (Example 1) MMT (MPEG Media Transport) is applied to the data set / data stream from the server 3303 to the first screen device, and video, audio, data, and / or motion information is transmitted. Partial MMT is applied to the data set / data stream from the first screen device to the second screen device, and video, audio, data, and / or motion information is transmitted. Note that the NTP (Network Time Protocol) format (UTC standard) is used for synchronization control.

[0395] (Example 2) The data set / data stream from the server 3303 to the first screen device uses the Common Media Application Format (CMAF) to transmit video, audio, data, and / or motion information. The data set / data stream from the first screen device to the second screen device uses the partial CMAF to transmit video, audio, data, and / or motion information. Note that the NTP (Network Time Protocol) format (UTC standard) is used for synchronization control.

[0396] (Example 3) The data set / data stream from the server 3303 to the first screen device applies MMT, CMAF, or other standards, and transmits an IP data stream or data file (including motion information and audio). The data set / data stream from the first screen device to the second screen device transmits motion information, data, and / or audio, and does not include video. Note that the first screen device and the second screen device each perform rendering to generate and play video. In addition, the video on the first screen device and the video on the second screen device are played back in sync.

[0397] (Example 4) The data set / data stream from the server 3303 to the first screen device applies MMT, CMAF, or other standards, and transmits an IP data stream or data file (including video, audio, and motion information). The data set / data stream from the first screen device to the second screen device transmits motion information, data, and / or audio, and does not include video. As a note, the first screen device receives a rendered video stream and plays the video. The first screen device transmits unrendered data including motion information to the second screen device, and the second screen device plays the video by rendering based on the unrendered data. In addition, the video on the first screen device and the video on the second screen device are played back in sync.

[0398] For the above examples 1 and 2, any of the following 1 to 4 may be adopted. 1. The space-floating image display device 1000 and the display device 3000 both receive image data of a rendered image, and each play that image. 2. The space-floating image display device 1000 and the display device 3000 both receive unrendered data, and each perform rendering to play the image. 3. The display device 3000 receives the rendered image and plays that image, and the space-floating image display device 1000 receives the unrendered data and performs rendering to play the image. 4. The display device 3000 receives the unrendered data and performs rendering to play the image, and the space-floating image display device 1000 receives the rendered image and plays that image.

[0399] In the above example 3, as in the above 2., both the space-floating image display device 1000 and the display device 3000 receive unrendered data and perform rendering to play images. In the above example 4, as in the above 3., the display device 3000 receives rendered images and plays the images, and the space-floating image display device 1000 receives unrendered data and performs rendering to play images.

[0400] [(Example 4): Display System] Fig. 34B shows an overview of data transmission and reception in a display system when Example 4 above is applied. This is an example in which an overall view image is displayed on the display device 3000 and individual view images are displayed on the space-floating image display device 1000. The first data included in the dataset / data stream 3302 transmitted from the server 3303 to the display device 3000 is rendered image data, and the second data is an unrendered dataset. The second data included in the dataset / data stream 3312 transmitted from the display device 3000 to the space-floating image display device 1000 is an unrendered dataset. The video processing unit of the display device 3000 plays virtual live video based on the first data. The space-floating image display device 1000 performs rendering based on the second data to play virtual live video. The display device 3000 and the space-floating image display device 1000 synchronously play video based on the first data and video based on the second data.

[0401] [Example 4: Transmission and Reception Processing] An example of transmission-side processing and reception-side processing for realizing display cooperation and synchronized playback in Example 4 above will be described with reference to FIGS. 35A and 35B . FIG. 35A shows an example of processing by the server 3303 as an example of processing by the transmission-side system. The transmission-side system 3510, specifically the server 3303, performs the following processing. In Example 4, the server 3303 transmits a data stream 3504 including video and audio information 3501 and motion information 3502 to the display device 3000. This data stream 3504 includes the first data and second data in FIG. 34B . For example, the video and audio information 3501 is transmitted as part of the first data, and the motion information 3502 is transmitted as part of the second data. The character's motion information 3502 is transmitted from the display device 3000 to the space-floating image display device 1000 as the second data, and a transmission delay may occur during this transmission. Furthermore, in order to reflect the character motion information 3502 in generating an image linked with the video audio / video information 3501, a rendering process is required in the receiving system (the space floating image display device 1000 in Example 4), which may cause a playback delay that is larger than that of the video or audio.

[0402] Therefore, in this embodiment, when transmitting fragments, packets, and other data 3503 (shown as boxes A, V, and M in FIG. 35A ) for video / audio information 3501 and motion information 3502 to be played back synchronously, included in a data stream 3504, the data 3503 is stored in a position in the data stream 3504 such that, at the transmission time, the motion information 3502 is transmitted at least a predetermined time, e.g., at least one second, earlier than the video / audio information 3501. Time information usable for synchronization control on the playback side, such as NTP format (UTC standard), may be added to or stored in each fragment or packet data 3503. In this way, synchronization control is possible on the playback side even if the data positions at the time of transmission are separated in time. This transmission data, or data stream 3504, is transmitted from server 3303 via communication network 1509 to the receiving system ( FIG. 35B ).

[0403] FIG. 35B shows an example of processing by the display device 2000 as an example of processing by the receiving system. The receiving system 3520, specifically the display device 3000 as the first screen device and the space-floating-image display device 1000 as the second screen device, performs the following processing. The data stream 3504 transmitted by the transmitting system 3510 in FIG. 35A contains data 3503 of fragments and packets of motion information 3502 that were transmitted a predetermined time (e.g., one second) or more earlier than the fragments and packets of video information and / or audio information. In the receiving display device 2000, the display device 3000 of Example 4 extracts data 3503 containing motion information 3502 from the data stream 3504 and transmits it to the space-floating-image display device 1000. The space-floating-image display device 1000 generates and displays an image by rendering based on the motion information 3502 reproduced from the data 3503. The display device 3000 displays or outputs the video information and / or audio information stored in the fragments or packets of video information and / or the fragments or packets of audio information received after the reception of the motion information 3502 in synchronization with the display of the video generated by rendering based on the motion information 3502 by the space floating video display device 1000.

[0404] In the example of FIG. 35B , as also shown in FIG. 34B , the display device 3000, which is the first screen device serving as the display device 2000, generates and displays a virtual live performance image 3521 according to rendered video and audio information 3501. The space-floating image display device 1000, which is the second screen device, performs rendering using motion information 3502 received from the display device 3000, which is the first screen device, to generate a virtual live performance image 3522 and display it as aerial image 3L. The space-floating image display device 1000 plays back image 3522 in synchronization with image 3521 on the display device 3000 side. Note that 3D data required for rendering processing based on motion information 3502, such as 3D models of the concert venue and characters to be placed in the virtual space where the virtual live performance will take place, and objects of the 3D models, can be received in advance by the space-floating image display device 1000 as a data set before the data stream 3504 in which the motion information 3502 is transmitted in real time is transmitted. The data set may be received by the space-floating image display device 1000 from the server 3303 by communication via the display device 3000. Also, the data set may be received by the space-floating image display device 1000 from the server 3303 by communication not via the display device 3000.

[0405] For synchronized playback, time information such as the NTP format (UTC standard) added to or stored in the data 3503 of each fragment or packet can be used to perform synchronized playback using a clock based on UTC time or a clock based on corrected time obtained by adding or subtracting the same predetermined time from UTC time on the first screen device and the second screen device.

[0406] The display device 2000, for example, the space-floating image display device 1000, can acquire fragment or packet data 3503 of the motion information 3502, for example, by one second or more earlier than the fragment or packet data 3503 of the video and audio information 3501. Therefore, even if there is a rendering playback delay or a delay due to data transmission between the first screen device and the second screen device due to this time difference, synchronous playback can be achieved. The technology of transmitting motion information first, as in this embodiment, is more suitable for synchronous playback than conventional technology. This transmission and synchronous playback process is also suitable even when the audio of the video and audio information 3501 is not taken into consideration. That is, the video is generated on the second screen device using the motion information 3502, and the video synchronously played on the first screen device is based on the video information of the video and audio information 3501. In this case, the audio information of the video and audio information 3501 may be played as audio on the first screen device, or may be transmitted from the first screen device to the second screen device and played as audio on the second screen device. In this case, audio information does not necessarily need to be transmitted a predet...

Claims

1. A display system comprising a display device, the display device receiving video data of camera footage which is footage captured by a camera, displaying the camera footage on a display screen based on the video data, selecting a target camera, viewpoint or subject from a plurality of cameras, a plurality of viewpoints or camera subjects based on operational input by a user of the display device, receiving video data of camera footage corresponding to the target camera, viewpoint or subject in accordance with the selection, and displaying an image on a display screen based on the received video data.

2. A display system according to claim 1, wherein the display device is a floating-in-the-air image display device, and comprises: an image processing unit that performs image processing based on image data; a display unit that displays the image that has been processed by the image processing unit; and an optical system that generates a floating-in-the-air image based on the image displayed by the display unit.

3. The display system according to claim 1, wherein the image captured by the camera is an image of a live concert or an image of a sporting event.

4. A display system according to claim 1, wherein the image captured by the camera is an image of a real person or a character configured in a virtual space of a server.

5. A display system as described in claim 4, wherein the subject is a real person or a plurality of members of a group consisting of characters configured in a virtual space of a server, and the plurality of cameras include a camera that photographs the entire group or an entire event held by the group, and a camera that photographs each individual member.

6. A display system according to claim 1, further comprising a server, said server receiving video data of the camera image from said camera, performing video processing on said video data of the camera image to create video data for distribution, and distributing said video data for distribution to said display device.

7. A display system according to claim 2, wherein the display device has a display panel for displaying a background image on the rear side of the floating-in-the-air image, and is controlled so as to display the background image in accordance with the floating-in-the-air image.

8. A display system according to claim 2, wherein the display device has a lighting device for producing lighting effects for the floating-in-the-air image, and controls the lighting effects to be produced in accordance with the floating-in-the-air image.

9. A display system according to claim 1, wherein the display device displays a graphical user interface (GUI) for selecting the target camera, viewpoint, or subject from the multiple cameras, multiple viewpoints, or multiple subjects based on operational input by the user.

10. A display system according to claim 1, further comprising a server, wherein the server configures the subject and a virtual camera that photographs the subject in a virtual space, holds a dataset including a 3D model of the subject, motion information of the subject, and virtual camera information of the virtual camera, and generates video data for distributing camera footage of the subject by rendering processing based on the dataset.

11. A display system according to claim 1, comprising a server, the server configuring the subject and a virtual camera that photographs the subject in a virtual space, holding a dataset including a 3D model of the subject, motion information of the subject, and virtual camera information of the virtual camera, and distributing the dataset to the display device, and the display device generating video data of the camera image of the subject by rendering processing based on the dataset.

12. A display system as claimed in claim 1, wherein the display device has a first screen device and a second screen device, and is provided with a server, wherein the server transmits first data and second data to the first screen device, the first screen device receives the first data and the second data, and displays an image on a first screen which is a display screen based on image data of the camera image contained in the first data, the second screen device receives the second data from the first screen device, and displays an image different from the image displayed on the first screen on a second screen which is a display screen based on the second data, and synchronizes the display of the image on the first screen and the display of the image on the second screen.

13. A display system as described in claim 12, further comprising a server, said server transmitting video data of the camera image as the first data to said first screen device, and transmitting motion information as the second data at least a predetermined time earlier than the video data, said first screen device and said second screen device synchronizing the display of the image on said first screen and the display of the image on said second screen using said video data and said motion information transmitted at least a predetermined time earlier than the video data.

14. A display system as described in claim 1, wherein the display device switches ON / OFF the display of the image based on the camera corresponding to the target camera, viewpoint, or subject based on operational input by the user.

15. A display system according to claim 1, further comprising an AI that analyzes images captured by the camera and selects camera images to be delivered to the display device for each time period of the images.

16. A display system as described in claim 1, comprising an AI, wherein the AI ​​selects camera images to be delivered to the display device from images captured by the camera in response to input information including the interests of the user of the display device.

17. A display system as described in claim 4, comprising an AI, wherein the subject is a real person or a plurality of members of a group consisting of characters configured in a virtual space of a server, and the AI ​​selects content information related to the camera image to be distributed to the display device according to input information including the interests of the user of the display device, and the content information has, as candidates, information on the group, information on the members, and commentary information on an event held by the group.

18. A display system as described in claim 1, wherein the display device displays an image based on the camera corresponding to the target camera, viewpoint, or subject, as well as an evaluation input image for inputting the user's evaluation of the display content of the image.

19. A display system according to claim 18, further comprising a server, said server collecting and storing the user's evaluation information input in response to the evaluation input image from a plurality of display devices of a plurality of users as the display devices, and analyzing the user's evaluation information.

20. A display system as claimed in claim 1, wherein the display device records the video data on a recording medium, plays and displays the recorded video from the recording medium, selects the target camera, viewpoint or subject from the multiple cameras, multiple viewpoints or multiple subjects based on operational input by the user for the played and displayed recorded video, receives video data of a camera image corresponding to the target camera, viewpoint or subject in accordance with the selection, and displays the camera-based image on a display screen based on the received video data.

21. A display device which receives video data of camera footage, which is footage captured by a camera, displays the camera footage on a display screen based on the video data, selects a target camera, viewpoint, or subject from multiple cameras, multiple viewpoints, or camera subjects based on operational input by a user of the display device, receives video data of camera footage corresponding to the target camera, viewpoint, or subject in accordance with the selection, and displays an image on a display screen based on the received video data.

22. A display system comprising: a first display device that receives video data via communication and displays video based on the video data; and a second display device that receives motion information from the first display device and plays and displays video based on the motion information.

23. A display system according to claim 22, wherein the motion information transmitted from the first display device to the second display device is received by the first display device via the communication together with the video data.

24. A display system according to claim 22, wherein the image on the first display device and the image on the second display device are reproduced in a synchronous manner.

25. A display system as claimed in claim 24, wherein the motion information for synchronous playback with the video to be displayed on the first display device is received by the first display device via the communication at least a predetermined time earlier than the video data for the video.

26. A display system according to claim 22, wherein the second display device performs a rendering process using the motion information received from the first display device to play back and display an image.

27. The display system of claim 22, wherein said motion information but not said video data is transmitted from said first display device to said second display device.

Images