Aerial image display system and display control device therefor

Abstract

In an aspect of the present invention, a retroreflection member is disposed such that a reflection surface is orthogonal to or nearly orthogonal to the viewing direction of a viewer. In addition, a first optical member is disposed between the viewer and the retroreflection member such that an action surface is parallel to or nearly parallel to the reflection surface of the retroreflection member, to thereby form a mirror image space between the first optical member and the retroreflection member and form a real space in a space where the viewer is present. Further, in the mirror image space, a second optical member is disposed in such a state that an action surface is inclined to the viewing direction at a prescribed angle, to thereby form a mirror image space movement region between the second optical member and the retroreflection member. Furthermore, the display position of a movable device is set to be movable between the mirror image space movement region and a real space movement region, and the display direction is set to be changeable between a direction orthogonal to or a direction nearly orthogonal to the viewing direction and a direction parallel to or a direction nearly parallel to the viewing direction.

Description

Aerial image display system and its display control device

[0001] One aspect of the present invention relates to an aerial image display system for displaying, for example, context information or the like in space, and a display control device used in this system.

[0002] As a method for presenting context information etc. to a user, for example, a method of presenting information using a mirror that people normally use has been proposed. For example, Non-Patent Document 1 describes a method that uses a special mirror that has the function of transmitting part of incident light and reflecting part of it, such as a one-way mirror, and displays context information from a display placed on the back side of the mirror in the front direction through the mirror, thereby simultaneously presenting a virtual image of the user himself / herself and a real image of the context information to the user.

[0003] Furthermore, Non-Patent Document 2 describes a method for presenting highly realistic information in the field of MR (Mixed Reality), which displays digital information in the real world, by simultaneously displaying an aerial image of the front side and an aerial image of the back side in the real space in front of a mirror and the mirror image space behind the mirror, respectively.

[0004] Kaori Fujinami, Kausar Fahim, Tatsuo Nakajima, "Context-aware information display device using an extended mirror," Transactions of the Information Processing Society of Japan, vol. 49 No. 6 pp. 1972-1983 (June 2008) Hiroki Yamamoto, Han-Yeoul Kim, Naoya Koizumi, Ken Naemura, "Mixed reality system presenting horizontal and vertical aerial images in front of and behind a mirror," 3D Imaging Conference, pp. 23-26, (July 2015)

[0005] However, both Non-Patent Document 1 and Non-Patent Document 2 present information in a fixed manner in either real space or mirror image space. In contrast, digital information such as context information is inherently independent of the space in which it is presented. Therefore, there is a demand for the development of a more flexible information presentation method that is not bound by physical phenomena in the real world.

[0006] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that makes it possible to present aerial images of digital information in either real space or mirror image space, thereby improving the freedom of information presentation.

[0007] In order to solve the above problems, one aspect of the aerial image display system according to the present invention includes a retroreflective member, first and second optical members that transmit and reflect a portion of incident light, and a display device that displays display information. The retroreflective member is arranged with its reflective surface perpendicular or nearly perpendicular to the viewing direction of the viewer. The first optical member is arranged between the viewer and the retroreflective member with its active surface parallel or nearly parallel to the reflective surface of the retroreflective member, forming a mirror image space between the retroreflective member and the first optical member, and forming a real space in the space where the viewer is present. The second optical member is arranged in the mirror image space with its active surface at a predetermined inclination angle with respect to the viewing direction, forming a mirror image space movement region between the second optical member and the second optical member. The display device includes a movement mechanism and a rotation mechanism. The movement mechanism is capable of moving the display position of the display information between the mirror image space movement region and a real space movement region formed in a space facing the second optical member via the mirror image space movement region. The rotation mechanism section can change the display direction of the display information between a direction perpendicular to the viewing direction or nearly so, and a direction parallel to the viewing direction or nearly so.

[0008] According to one aspect of the present invention, when the display position of the display device is set to a real space movement region and the display direction of the display device is set to a direction, for example, perpendicular to the first optical member, i.e., the direction of the second optical member, the display information displayed on the display device is reflected by the second optical member, then retroreflected by the retroreflecting member, and passes through the second optical member and the first optical member to be formed as a real aerial image in the real space where the viewer is present. Furthermore, when the display position of the display device is moved to a mirror image space movement region while the display direction of the display device is facing the second optical member, the display information displayed on the display device is reflected by the second optical member, then retroreflected by the retroreflecting member, and formed as an aerial image in the mirror image space. Furthermore, when the display position of the display device is moved to the mirror image space movement region and the display direction is turned toward the viewer, it is possible to present an aerial image that the viewer would see directly on the display screen of the display device, rather than directly through the first optical member.

[0009] In other words, by arbitrarily changing the display position and display direction of the display device, it is possible to form an aerial image of the displayed information at any position, seamlessly crossing the boundary between real space and mirror image space. In addition to changing the imaging position of the aerial image of the displayed information, it is also possible to present the aerial image of the displayed information in a state that the viewer can view it directly.

[0010] According to one aspect of the present invention, it is possible to provide a technology that makes it possible to present an aerial image of digital information in either real space or mirror image space, thereby improving the degree of freedom in presenting information.

[0011] FIG. 1 is a diagram showing a first example of an optical system in an aerial image display system according to a first embodiment of the present invention. FIG. 2 is a perspective view showing an example of the configuration of a display device provided in the aerial image display system shown in FIG. 1. FIG. 3 is a diagram showing an example of an operation when an aerial image is formed in real space in the aerial image display system shown in FIG. 1. FIG. 4 is a diagram showing an example of an operation when an aerial image is formed in mirror image space in the aerial image display system shown in FIG. 1. FIG. 5 is a diagram showing an example of an operation when a direct-view aerial image is presented in the aerial image display system shown in FIG. 1. FIG. 6 is a diagram showing a second example of an optical system in the aerial image display system according to the first embodiment of the present invention. FIG. 7 is a diagram showing a third example of an optical system in the aerial image display system according to the first embodiment of the present invention. FIG. 8 is a cross-sectional view showing an example of the structure of a beam splitter used in the aerial image display system shown in FIG. 7. FIG. 9 is a diagram showing a fourth example of an optical system in the aerial image display system according to the first embodiment of the present invention. FIG. 10 is a diagram showing a fifth example of an optical system in the aerial image display system according to the first embodiment of the present invention. FIG. 11 is a block diagram showing an example of the functional configuration of a display control device provided in an aerial image display system according to a second embodiment of the present invention. FIG. 12 is a flowchart showing an example of the processing procedure and processing content of display control processing executed by a control unit of the display control device shown in FIG. 11. FIG. 13 is a diagram for explaining an example of a method for calculating image parameters in the display control processing shown in FIG. 12. FIG. 14 is a diagram for explaining another example of a method for calculating image parameters in the display control processing shown in FIG. 12. FIG. 15 is a diagram for explaining an overview of an aerial image display system according to a third embodiment of the present invention. FIG. 16 is a diagram showing an example of the configuration of an optical system of the aerial image display system according to the third embodiment of the present invention. FIG. 17 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to the third embodiment of the present invention. FIG. 18 is a flowchart showing an example of the processing procedure and processing content of display control processing executed by a control unit of the display control device shown in FIG. 17. FIG. 19 is a flowchart showing an example of the processing procedure and processing content of rotation angle calculation processing in the display control processing shown in FIG. 18.FIG. 20 is a diagram illustrating a first example of the rotation angle calculation process shown in FIG. 19 . FIG. 21 is a diagram illustrating a second example of the rotation angle calculation process shown in FIG. 19 . FIG. 22 is a diagram illustrating an overview of an aerial image display system according to a fourth embodiment of the present invention. FIG. 23 is a diagram illustrating an overview of an aerial image display system according to the fourth embodiment of the present invention. FIG. 24 is a diagram illustrating an example of the configuration of an optical system of an aerial image display system according to the fourth embodiment of the present invention. FIG. 25 is a block diagram illustrating an example of the functional configuration of a display control device provided in the aerial image display system according to the fourth embodiment of the present invention. FIG. 26 is a flowchart illustrating an example of the processing procedure and processing content of display control processing executed by a control unit of the display control device shown in FIG. 25 . FIG. 27 is a diagram illustrating an example of processing for calculating the position of a retroreflective member with respect to a display device, part of the display control processing shown in FIG. 26 . FIG. 28 is a diagram illustrating an example of processing for calculating the angle of a retroreflective member, part of the display control processing shown in FIG. 27 . FIG. 29 is a diagram illustrating an example of processing for calculating the angle of a retroreflective member, part of the display control processing shown in FIG. 27 . Fig. 30 is a diagram showing a first example of an optical system in an aerial image display system according to a fifth embodiment of the present invention. Fig. 31 is a diagram showing an example of an aerial image and a background image displayed by the aerial image display system shown in Fig. 30. Fig. 32 is a diagram showing a second example of an optical system in the aerial image display system according to the fifth embodiment of the present invention. Fig. 33 is a diagram showing a third example of an optical system in the aerial image display system according to the fifth embodiment of the present invention.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [First Embodiment] (First Example) (Configuration Example) FIG. 1 is a diagram showing a first example of an aerial image display system according to a first embodiment of the present invention.

[0014] (1) Optical System In Fig. 1, 1A is a retroreflective member, and this retroreflective member 1A is arranged so that its reflective surface is perpendicular to the viewing direction of the viewer (hereinafter also referred to as user) US. A first beam splitter 2 is arranged between the user US and the retroreflective member 1A so that its active surface is perpendicular to the viewing direction of the user US, i.e., parallel to the reflective surface of the retroreflective member 1A. Furthermore, a second beam splitter 3 is arranged in the space between the retroreflective member 1A and the first beam splitter 2. The second beam splitter 3 is arranged so that its active surface is obliquely arranged at a predetermined angle, for example, 45°, with respect to the viewing direction of the user US.

[0015] The first and second beam splitters 2 and 3 each have the optical property of transmitting a portion of the incident light and reflecting a portion of it, thereby forming a real space RS on the user US side of the first beam splitter 2, and a mirror image space MS between the first beam splitter 2 and the retroreflective member 1A.

[0016] Furthermore, a mirror image space movement region ME is formed in the triangular region in the mirror image space MS between the retroreflective member 1A and the second beam splitter 3. A real space movement region RE is formed in the region adjacent to the mirror image space movement region ME on the extension of each surface of the retroreflective member 1A and the first beam splitter 2. The boundary between the mirror image space movement region ME and the real space movement region RE is a virtual mirror plane VM.

[0017] (2) Display Device DS The display device DS is disposed so as to be movable within the mirror space movement area ME and the real space movement area RE.

[0018] 2 is a perspective view showing an example of the configuration of a display device DS. The display device DS has a display device main body 51 equipped with, for example, a liquid crystal panel, an organic EL panel, or an LED panel, placed on a base 52. Legs 53 equipped with casters 54 are provided on the underside of the display device main body 51 to movably support the display device main body 51. A rotation mechanism 55 is also provided on the upper surface of the base 52. The rotation mechanism 55 is used to variably set the display direction of the display device main body 51 within a predetermined angle range, for example, a range of 180 degrees.

[0019] (Example of Operation) With the above configuration, the aerial image display system operates as follows.

[0020] (1) When an aerial image is formed in the real space RS: Figure 3 is a diagram for explaining the operation in this case. The display device DS is placed in the real space movement area RE, and the display direction is set to be, for example, perpendicular to the viewing direction of the user US. Note that the position and display direction of the display device DS may be set manually by an administrator or viewer, or automatically by a display control device shown in the figure.

[0021] When the display position and display direction of the display device DS are set in this manner, the display information displayed on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, and passes sequentially through the second beam splitter 3 and the first beam splitter 2, and is formed as an aerial image MI1 in the real space RS in which the user US exists.

[0022] Therefore, the user US can visually recognize, for example, figures, photographs, etc. as aerial images in the real space RS in which the user US exists.

[0023] (2) When an aerial image is formed in the mirror image space MS: Figure 4 is a diagram for explaining the operation in this case. The display device DS is moved from the real space movement area RE to the mirror image space movement area ME. The display direction is set to be perpendicular to the viewing direction of the user US. When the display position and display direction of the display device DS are set in this manner, the display information displayed on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, and passes through the second beam splitter 3 to be formed as an aerial image MI2 in the mirror image space MS.

[0024] Therefore, the user US can use the first beam splitter 2 as a mirror to perform tasks such as brushing their teeth, applying makeup, or grooming, while checking news information such as traffic information and weather forecasts on the aerial image MI2 formed in the mirror image space MS.

[0025] (3) Presenting a Direct-View Aerial Image to the User US Figure 5 is a diagram illustrating the operation in this case. The display device DS is moved from the real space movement area RE to the mirror image space movement area ME. The display direction is set to face the viewing direction of the user US. By setting the display position and display direction of the display device DS in this manner, the display information displayed on the display screen of the display device DS is transmitted sequentially through the second beam splitter 3 and the first beam splitter 2 and presented to the user US as a direct-view aerial image RI in the real space RS.

[0026] Therefore, the user US can view the displayed information in the real space RS as if he or she were looking directly at the display screen of the display device DS.

[0027] Second Example Fig. 6 is a diagram showing a second example of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as those in Fig. 1 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0028] In the second embodiment, the retroreflective member 1B is positioned on the opposite side of the virtual mirror surface VM from the second beam splitter 3 in the mirror image space MS, so that the reflective surface is oriented parallel to the viewing direction of the user US.

[0029] With this configuration, when the display device DS is placed, for example, in the real space movement area RE and the display direction is set to a direction perpendicular to the viewing direction of the user US, the display information displayed on the display device DS passes through the second beam splitter 3, is retroreflected by the retroreflective member 1B, is reflected by the second beam splitter 3, passes through the first beam splitter 2, and is formed as an aerial image MI3 in the real space RS where the user US is present.

[0030] Furthermore, when the display device DS is placed in the mirror image space movement area ME and the display direction is set to be perpendicular to the viewing direction of the user US, the display information displayed on the display screen of the display device DS passes through the second beam splitter 3 and is then retroreflected by the retroreflecting member 1B, and is formed as an aerial image MI2 in the mirror image space MS.

[0031] Furthermore, when the display device DS is placed in the mirror image space movement area ME and the display direction is set in a direction opposite to the viewing direction of the user US, the display information displayed on the display device DS passes sequentially through the second beam splitter 3 and the first beam splitter 2 and is presented to the user US in the real space RS as a direct-view aerial image RI.

[0032] Therefore, the second embodiment also provides the same effect as the first embodiment.

[0033] 7 is a diagram showing a third embodiment of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as those in FIGS. 1 and 6 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0034] In the third embodiment, retroreflective members 1A and 1B are arranged on two mutually perpendicular sides of the mirror image space MS so that their reflective surfaces are perpendicular to each other. As the second beam splitter 3, for example, as shown in Fig. 8, a reflective member 3a such as a half mirror with equal transmittance and reflectance is sandwiched between two transparent members 3b and 3c such as acrylic plates with equal thickness and refractive index is used.

[0035] With this configuration, when the display device DS is placed in the real space movement area RE and the display direction is set to a direction perpendicular to the viewing direction of the user US, the display information displayed on the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflector 1A, and sequentially transmitted through the second beam splitter 3 and the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is present. At the same time, the display information also transmits through the second beam splitter 3, then retroreflected by the retroreflector 1B, reflected by the second beam splitter 3, and then transmitted through the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is present. The same applies when an aerial image is formed in the mirror image space MS.

[0036] That is, a composite aerial image MI4 is formed in the real space RS and the mirror image space MS by combining the aerial image retroreflected by the retroreflecting member 1A and the aerial image retroreflected by the retroreflecting member 1B.

[0037] Therefore, according to the third embodiment, it is possible to present a brighter aerial image MI4 than when the retroreflective members 1A and 1B are used alone. Furthermore, since the second beam splitter 3 has a structure in which the reflective member 3a is sandwiched between two transparent members 3b and 3c having the same thickness and refractive index on both sides as shown in Figure 8, the optical path lengths corresponding to the first and second beam splitters 2 and 3 can be set to be equal, thereby making it possible to prevent double images from appearing in the aerial image MI4.

[0038] (Fourth Example) FIGS. 9 and 10 are diagrams showing a fourth example of the aerial image display system according to the first embodiment of the present invention.

[0039] In the fourth embodiment, a retardation film 4 is disposed on the reflective surface of the retroreflective member 1A. The retardation film 4 is, for example, a quarter-phase retardation film, and has the optical property of rotating the polarization direction of transmitted light by 45 degrees. Furthermore, optical elements that switch between reflection and transmission depending on the polarization direction of incident light are used as the first and second beam splitters 2 and 3. Examples of such optical elements include reflective polarizers or wire grids. The following description will discuss, as an example, the operation of the device when it is installed in a direction that reflects S-polarized light and transmits P-polarized light.

[0040] In contrast, a depolarizing film, a diffusion plate, or a retardation film is attached to the display screen of the display device DS so that the output light of the display information is unpolarized or contains S-polarized and P-polarized light. Alternatively, a polarizing plate or a retardation film is rotatably disposed facing the display screen of the display device DS, and the polarization direction of the output light is switched by the rotation.

[0041] With this configuration, when forming an aerial image in the real space RS, the display device DS is placed in the real space movement region RE with its display direction facing perpendicular to the viewing direction. If a polarizing plate is placed on the display screen of the display device DS, this polarizing plate is rotated so that the polarized light component reflected from the display device DS to the beam splitter 3, for example, S-polarized light, is output.

[0042] 9, for example, S-polarized light of the display light output from the display device DS is reflected by the second beam splitter 3 and converted into circularly polarized light by passing through the phase difference film 4 arranged in front of the retroreflective member 1A. The circularly polarized light has its direction of rotation reversed by being retroreflected by the retroreflective member 1A, and is converted into P-polarized light by passing through the phase difference film 4 again. This retroreflected P-polarized light passes through the second beam splitter 3 and further passes through the first beam splitter 2 to be formed as an aerial image MI5 in the real space RS.

[0043] Next, when forming an aerial image in the mirror image space MS, the display device DS is placed in the mirror image space movement region ME with its display direction facing perpendicular to the viewing direction, and the polarizing plate placed on the display screen of the display device DS is rotated so that S-polarized light is output from the display device DS.

[0044] 10, for example, S-polarized light out of the display light output from the display device DS1 is reflected by the second beam splitter 3 and, as described above, is retroreflected as P-polarized light by the phase difference film 4 arranged in front of the retroreflecting member 1A and the retroreflecting member 1A. The retroreflected P-polarized light then passes through the second beam splitter 3 and is formed as an aerial image MI6 in the mirror image space MS.

[0045] When the display information of the display device DS is presented as a direct-view aerial image RI, the display device DS is positioned in the mirror image space movement area ME with the display direction facing the user US. At the same time, the polarizer and retarder arranged on the display screen of the display device DS are rotated so that P-polarized light is output from the display device DS.

[0046] When set in this manner, the P-polarized light among the display light output from the display device DS passes through the second beam splitter 3 and the first beam splitter 2 sequentially, and is presented as a direct-view aerial image RI to the user US present in the real space RS.

[0047] Therefore, according to the fourth embodiment, by selectively switching the display light output from the display device DS between S-polarized and P-polarized light, it is possible to suppress light attenuation due to the beam splitter on the optical path and present a bright aerial image.

[0048] 10, for example, display devices DS1 and DS2 may be arranged in an L-shape, with each display device configured to output S-polarized light and P-polarized light, respectively. This configuration makes it possible to simultaneously present multiple aerial images with high brightness in the mirror image space MS and the real space RS.

[0049] [Second Embodiment] (Overview) In the system described in the first embodiment, an aerial image can be displayed seamlessly across the boundary between the mirror image space MS and the real space RS, thereby achieving a practically very useful effect of diversifying the display of aerial images. However, the imaged aerial image MI undergoes many reflections and transmissions by the beam splitters 2 and 3 and the retroreflective members 1A and 1B on the optical path before the image is formed, resulting in a decrease in contrast and other factors, resulting in uneven image quality compared to the direct-view aerial image RI.

[0050] Therefore, in the second embodiment, when displaying the imaged aerial image MI, the image parameters of the display image on the display device DS are adjusted in advance to achieve uniform image quality between the imaged aerial image MI and the direct-view aerial image RI.

[0051] (Configuration Example) FIG. 11 is a block diagram showing the functional configuration of a display control device CS1 provided in an aerial image display system according to the second embodiment of the present invention, together with a display device DS.

[0052] The display device DS is provided with a movement mechanism 56 for moving the display position and a rotation mechanism 55 for changing the display direction. Both of these mechanisms 56 and 55 operate in accordance with control signals output from the display control device CS1.

[0053] The display control device CS1 is, for example, a personal computer, and includes a control unit 100 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 200 and a data storage unit 300, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 400 are connected to the control unit 100 via a bus (not shown).

[0054] An input device 500 such as a keyboard and a mouse is connected to the input / output I / F unit 400. Note that a display device, a USB (Universal Serial Bus) memory, or other external storage medium may also be connected to the input / output I / F unit 400. The input / output I / F unit 400 may also be provided with a communication interface function.

[0055] The program storage unit 200 is configured by combining, for example, a nonvolatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read only memory (ROM), and stores application programs necessary for control according to the second embodiment in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

[0056] The data storage unit 300 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and in its storage area, a display position / direction control data storage unit 301, a video parameter storage unit 302, and a display information storage unit 303 are provided as the main storage units required to implement the second embodiment of the present invention.

[0057] Display position / direction control data storage unit 301 stores control data required to control the display position and display direction of display device DS in accordance with input display instructions. Image parameter storage unit 302 stores control data required to control image parameters of display information in accordance with the type of aerial image to be displayed. Display information storage unit 303 is used to store information to be displayed, such as content information.

[0058] The control unit 100 includes, as processing functions necessary for implementing the second embodiment of the present invention, a display instruction acquisition processing unit 101, a display position / direction control processing unit 102, and a video parameter control processing unit 103. These processing units 101 to 103 are all realized by causing a hardware processor of the control unit 100 to execute an application program stored in the program storage unit 200.

[0059] Note that part or all of the processing units 101 to 103 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0060] The display instruction acquisition processing unit 101 acquires display instruction information when the display instruction information for an aerial image is input via the input device 500. The display instruction information instructs the user to display either an imaged aerial image in the real space RS, an imaged aerial image in the mirror image space MS, or a direct-view aerial image in the real space RS.

[0061] The display position / orientation control processing unit 102 reads out control data from the display position / orientation control data storage unit 301 in accordance with the instructions in the display instruction information. Then, in accordance with the read out control data, the input / output I / F unit 400 outputs a position control signal and a direction control signal to the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS, respectively, to set the display position and display direction of the display device DS.

[0062] The image parameter control processing unit 103 determines whether the aerial image to be presented is a "focused aerial image" or a "direct-view aerial image" based on the setting data for the display position and display direction of the display device DS. Then, the image parameter control processing unit 103 controls the image parameters of the display image to be displayed on the display device DS based on the determination result and the image parameters stored in the image parameter storage unit 302. An example of this image parameter control processing will be described in the operation example.

[0063] (Example of Operation) Next, the operation of the display control device CS1 configured as above will be described.

[0064] FIG. 12 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 100 of the display control device CS1.

[0065] The control unit 100 of the display control device CS1 monitors input of display instruction information in step S10 under the control of the display instruction acquisition processing unit 101. When display instruction information is input via the input device 500 in this state, the control unit 100 of the display control device CS1 determines in step S11 under the control of the display position / direction control processing unit 102 whether the display instruction information is an instruction to display a "formed aerial image" or a "direct-view aerial image."

[0066] If the result of the above determination is that the display instruction is "imaging aerial image," the display position / orientation control processing unit 102 proceeds to step S12, where it generates a display position control signal based on the display position / orientation control data stored in the display position / orientation control data storage unit 301. Then, the display position / orientation control processing unit 102 outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the real space movement region RE or the mirror image space movement region ME.

[0067] The display position / orientation control processing unit 102 generates a display orientation control signal based on the display position / orientation control data, and outputs the generated display orientation control signal to the rotation mechanism unit 55 of the display device DS, thereby setting the display orientation of the display device DS to a direction perpendicular to the viewing direction.

[0068] On the other hand, if the result of the above determination is that the display instruction is "direct-view aerial image," the display position / orientation control processing unit 102 proceeds to step S14, where it generates a display position control signal based on the position / orientation control data stored in the display position / orientation control data storage unit 301. Then, the display position / orientation control processing unit 102 outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the mirror image space movement area ME.

[0069] At the same time, the display position / orientation control processing unit 102 generates a display orientation control signal based on the position / orientation control data, and outputs the generated display orientation control signal to the rotation mechanism unit 55 of the display device DS, thereby setting the display orientation of the display device DS to the direction of the user US.

[0070] In the above description, the display instruction information specifies whether to display a "focused aerial image" or a "direct-view aerial image." However, if the content included in the display information includes information specifying the type of aerial image to be displayed, i.e., whether it is a "focused aerial image" or a "direct-view aerial image," or information that can determine this, this information may be used.

[0071] When the setting control of the display position and display direction of the display device DS is completed, the control unit 100 of the display control device CS1 then executes the process of controlling the image parameters of the display information as follows under the control of the image parameter control processing unit 103.

[0072] That is, the image parameter control processing unit 103 receives the display instruction (either "imaged aerial image" or "direct-view aerial image") from the display position / direction control processing unit 102, and sets image parameters corresponding to the received display instruction.

[0073] Here, the image parameters used are control data of image parameters stored in advance in image parameter storage unit 302. The following two methods are conceivable for calculating the control data of the image parameters. Here, contrast (intensity of blur) is taken as an example of the image parameter to be controlled.

[0074] (1) First Calculation Method Fig. 13 is a diagram used to explain the first calculation method. First, a formed aerial image MI is displayed in a mirror image space MS. At this time, a chart CIM divided into two parts, white and black, is used as the displayed image. In this state, a region of the formed aerial image MI including the chart CIM is photographed with a camera, and the photographed image CMI is saved.

[0075] Next, a direct-view aerial image RI is displayed in the mirror image space MS. At this time, a chart CRI divided into two parts, black and white, is used as the displayed image, and blur processing is performed on the image to indicate the blur strength β. Then, an image of the direct-view aerial image including the chart CRI is captured by a camera, and the captured image CRI is saved. Note that blur processing refers to image processing using, for example, a Gaussian filter. β is the standard deviation.

[0076] Next, SSIM (Structural Similarity), which indicates an image quality evaluation index, is calculated for each of the captured images CMI and CRI. This SSIM calculation process is performed each time the blur strength β is changed by a fixed amount, and the β that minimizes the difference in SSIM between the captured images CMI and CRI is stored in the image parameter storage unit 302 as a parameter indicating the blur strength. Note that, in addition to SSIM, MSE (Mean Squared Error) or PSNR (Peak Signal to Noise Ratio) may also be used as the image quality evaluation index.

[0077] (2) Second Calculation Method Figure 14 is a diagram used to explain the second calculation method. First, a formed aerial image MI is displayed in a mirror image space MS. As with the first calculation method, a chart CIM divided into two parts, white and black, is used as the displayed image. In this state, an area of ​​the formed aerial image MI including the chart CIM is photographed with a camera, and the photographed image is Fourier transformed to obtain and save a modulation transfer function (MSF).

[0078] Next, a direct-view aerial image RI is displayed in the mirror image space MS. In this case, a chart CIM divided into two parts, black and white, is used as the display image. The area of ​​the direct-view aerial image including the chart CRI is then photographed with a camera, and the photographed image is Fourier transformed to obtain the MSF, which is then saved.

[0079] Next, the MTF obtained for the focused aerial image MI and the MTF obtained for the direct-view aerial image RI are normalized and then compared to calculate the attenuation rate β of the spatial frequency component, and the calculated β is stored in the image parameter memory unit 302 as a parameter indicating the strength of blur.

[0080] When actually displaying an aerial image, if the aerial image to be displayed is an "imaged aerial image," then in step S13, image parameter control processing unit 103 outputs the display information read from display information storage unit 303 directly from input / output I / F unit 400 to display device DS. That is, the display image is displayed as is without adjusting the image parameters.

[0081] In contrast, if the aerial image to be displayed is a "direct-view aerial image," the image parameter control processing unit 103 adjusts the parameters of the display information read out from the display information storage unit 303 in step S15 based on the control data for the image parameters, i.e., the control data for adjusting the blur strength β.

[0082] For example, a display image is blurred using a Gaussian filter. Alternatively, the display image is Fourier transformed to a frequency space image, and then this frequency space image is subjected to processing to attenuate the spatial frequency components. Then, the attenuated frequency space image is subjected to an inverse Fourier transform to return it to a two-dimensional display image.

[0083] Then, in step S13, the image parameter control processing unit 103 outputs the display image after the image parameter adjustment from the input / output I / F unit 400 to the display device DS, causing it to be displayed. That is, the image is displayed with reduced contrast by forcibly applying a process of adding blur β.

[0084] (Effects) Therefore, according to the second embodiment, when a direct-view aerial image is presented, the image is displayed on the display device DS with a blur β added to the display image, i.e., with contrast previously reduced. As a result, the contrast of the direct-view aerial image RI can be made equivalent to that of the formed aerial image MI, thereby homogenizing the image quality between the formed aerial image MI and the direct-view aerial image RI. Therefore, the boundary between the mirror image space MS and the real space RS can be seamlessly crossed to reduce the sense of discomfort felt by the user due to the change in image quality of the aerial image when switching the display of the aerial image between the formed aerial image MI and the direct-view aerial image RI, thereby enabling the display of an aerial image with high reproducibility.

[0085] (Variations) In the above description, the contrast (intensity of blur) is controlled as an image parameter. However, brightness or color may be controlled as an image parameter. In this case, by adjusting the RGB of the displayed image, the brightness and color of the direct-view aerial image RI can be made equivalent to those of the focused aerial image MI, thereby enabling the image quality of both aerial images to be homogenized.

[0086] [Third Embodiment] (Overview) Generally, the viewing area, i.e., the viewing angle, of the display screen of a display device DS is limited. Therefore, even when an aerial image is displayed by the display device DS, the viewing area of ​​the user US for the aerial image is naturally limited. As a result, as shown in FIG. 15 , depending on the viewing position of the user US, problems may arise, such as only being able to view an aerial image with reduced brightness, or the user US's viewing position being outside the viewing area of ​​the aerial image, making the aerial image completely unviewable. This problem is particularly pronounced when the viewing area is limited by, for example, attaching a viewing area-limiting film to the display screen of the display device DS to reduce the generation of stray light.

[0087] In order to solve the above problem, the third embodiment of the present invention detects the position of the user US in the real space RS and variably controls the display position and display direction of the display device DS according to the detected position of the user US, thereby enabling the user US to always view the aerial image from the front. Note that, although the following description will be given taking as an example a case where both the display position and display direction of the display device DS are variably controlled, it is also possible to variably control at least the display direction alone.

[0088] 16 is a diagram showing an example of the configuration of an aerial image display system according to a third embodiment of the present invention. In this figure, the same parts as those in FIG. 1 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0089] The system according to the third embodiment includes a display control device CS2 for controlling the display position and display direction of the display device DS. To enable control of the display position and display direction, the display device DS is provided with a rotation mechanism 55 and a movement mechanism 56. Furthermore, a camera CM is disposed in the optical system. The camera CM may be, for example, a depth camera, and captures an image of the area in the real space RS where the user US is present, and outputs the captured image to the display control device CS2.

[0090] FIG. 17 is a block diagram showing the functional configuration of the display control device CS2 together with the display device DS and the camera CM.

[0091] As in the second embodiment, the display control device CS2 is, for example, a personal computer, and includes a control unit 110 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 210 and a data storage unit 310, and an input / output I / F unit 410 are connected to the control unit 110 via a bus (not shown).

[0092] The display device DS and the camera CM are connected to the input / output I / F unit 410. More specifically, the display device main body 51 of the display device DS, the movement mechanism unit 56, the rotation mechanism unit 55, and the camera CM are each connected via a wireless interface such as a signal cable or a wireless LAN (Local Area Network).

[0093] The program storage unit 210 is configured by combining, for example, a nonvolatile memory such as an SSD as a storage medium that can be written to and read from at any time, with a nonvolatile memory such as a ROM, and stores application programs necessary for the control processing according to the third embodiment in addition to middleware such as an OS. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0094] The data storage unit 310 is, for example, a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from as needed with a volatile memory such as a RAM, and the storage area thereof is provided with a display information storage unit 311. The display information storage unit 311 stores content information for displaying aerial images to the user US. This content information is obtained, for example, by reading it from an external storage medium or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0095] Control unit 110 includes, as processing functions necessary for implementing the third embodiment of the present invention, a user position acquisition processing unit 111, an aerial image display position acquisition processing unit 112, a movement position calculation processing unit 113, a rotation angle calculation processing unit 114, and a display position / direction control processing unit 115. These processing units 111 to 115 are all realized by causing a hardware processor of control unit 110 to execute application programs stored in program storage unit 210.

[0096] Note that some or all of the processing units 111 to 115 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0097] The user position acquisition processing unit 111 acquires the captured image output from the camera CM via the input / output I / F unit 410, and calculates the position information of the user US in the real space RS based on the acquired captured image. Note that if the user's viewing position is fixed, the user position acquisition processing unit 111 may acquire and store the fixed viewing position as a parameter in advance. In this case, the camera CM can be omitted.

[0098] The aerial image display position acquisition processing unit 112 acquires information indicating the display position of the aerial image to be displayed from the content information stored in the display information storage unit 311 .

[0099] The movement position calculation processing unit 113 calculates the display position of the display device DS based on the position information of the user US and the display position information of the aerial image.

[0100] The rotation angle calculation processing unit 114 calculates the rotation angle θ of the display device DS based on the position information of the user US, the display position information of the aerial image, and information indicating whether the aerial image to be displayed is an imaged aerial image or a direct-view aerial image. An example of the calculation process of this rotation angle θ will be described in the operation example.

[0101] The display position / orientation control processing unit 115 drives the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS according to the calculated display position and rotation angle θ of the display device DS, respectively, to control the display position and display angle of the display device DS.

[0102] (Example of Operation) Next, the operation of the aerial image display system configured as above will be described.

[0103] FIG. 18 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 110 of the display control device CS2.

[0104] (1) Acquisition of User Position When displaying an aerial image, the control unit 110 of the display control device CS2 first acquires user position information using the user position acquisition processing unit 111 in step S20. For example, the user position acquisition processing unit 111 acquires a captured image output from the camera CM via the input / output I / F unit 410 and recognizes an image of the user US from the acquired captured image. Then, based on the position coordinates of the recognized user US in the image, it calculates the position information of the user US in the real space RS. As mentioned above, if the user's viewing position is fixed, the user position acquisition processing unit 111 may acquire and store the fixed viewing position as a parameter in advance.

[0105] (2) Acquisition of Aerial Image Display Position Subsequently, in step S21, the control unit 110 of the display control device CS2 specifies the display position of the aerial image using the aerial image display position acquisition processing unit 112. For example, when the content information stored in the display information storage unit 311 includes information indicating the display target position of the aerial image, this display target position is acquired as is as display position information of the aerial image.

[0106] (3) Calculation of the display position of the display device DS After obtaining the position information of the user US and the display position information of the aerial image, in step S22, the control unit 110 of the display control device CS2 calculates the display position of the display device DS using the movement position calculation processing unit 113. This display position of the display device DS can be determined as a position that is plane-symmetric with respect to the display position information of the aerial image.

[0107] (4) Calculation of the display angle of the display device DS Next, in step S23, the control unit 110 of the display control device CS2 calculates the rotation angle θ for controlling the display direction of the display device DS using the rotation angle calculation processing unit 114 as follows.

[0108] 19 is a flowchart showing an example of the processing procedure and processing content of the calculation process of the rotation angle θ. That is, in step S231, the rotation angle calculation processing unit 114 first calculates, from information representing the display position of the aerial image and the position of the user US, the orientation of the user US with respect to the aerial image, i.e., the angle α formed by the straight line connecting the user US and the aerial image and the perpendicular line dropped from the user US to the first beam splitter 2.

[0109] The rotation angle calculation processing unit 114 then calculates the rotation angle θ. The calculation process for this rotation angle θ differs depending on whether the aerial image is formed to the right or left of the user US.

[0110] For example, when an aerial image is formed to the right of the user US, the rotation angle calculation processing unit 114 first calculates (90-α)° from the angle α and the known angle of 90°, and then calculates (45+α)° from the (90-α)° and the known angle of 45°, as shown in Fig. 20. Then, using the (90-α)° and the known angle of 90°, 180° = 90° + 90-α° + θ is calculated, thereby calculating θ = |α|.

[0111] On the other hand, when the aerial image is to be formed on the left side of the user US, the rotation angle calculation processing unit 114 calculates the rotation angle θ using the calculated angle α and known angles 90° and 45°, as shown in Figure 21, for example.

[0112] Furthermore, in step S232, the rotation angle calculation processing unit 114 determines whether the aerial image to be displayed is a "formed aerial image" or a "direct-view aerial image." If it is a "formed aerial image," then in step S233, the final rotation angle θ is calculated with -90° as the reference. On the other hand, if it is a "direct-view aerial image," then in step S234, the final rotation angle θ is calculated with 180° as the reference.

[0113] (5) Control of Movement Position and Display Orientation of Display Device DS Once the movement position and rotation angle θ have been calculated as described above, the control unit 110 of the display control device CS2 generates, in step S24, control signals for changing the position and angle by the calculated movement position and rotation angle θ under the control of the display position / orientation control processing unit 115. The display position / orientation control processing unit 115 then outputs the generated movement position control signal and rotation angle control signal from the input / output I / F unit 410 to the movement mechanism unit 56 and rotation mechanism unit 55 of the display device DS, respectively.

[0114] Thus, the display position and display direction of the display device DS are controlled, and the imaging position and display direction of the aerial image are set so as to face the user US directly.

[0115] (Effects) As described above, according to the third embodiment, the display position and display direction of the display device DS are controlled according to the position of the user US, thereby setting the imaging position and orientation of the aerial image so that it always faces the user US. Therefore, the user US can always reliably view the aerial image with high brightness, regardless of their position, without having to adjust their own position according to the imaging position of the aerial image. This effect is particularly effective when the viewing zone is limited by, for example, attaching a viewing zone limiting film to the display screen of the display device DS to reduce stray light.

[0116] (Modification) In the above description, an example was given in which the optical system is provided with the first beam splitter 2, but the first beam splitter 2 does not necessarily have to be installed, and the same effect can be achieved even in this case.

[0117] [Fourth Embodiment] (Overview) In general, in an optical system using a retroreflective member, when a user US faces the retroreflective member 1A, a virtual image VI of the display device DS is displayed as stray light behind the formed aerial image MI due to specular reflection from the surface of the retroreflective member 1A, as shown in Fig. 22. This stray light appears brighter and clearer than the formed aerial image MI, particularly when the retroreflective component is weaker than the specular reflection component, and therefore reduces the visibility of the formed aerial image MI to the user US.

[0118] In order to solve the above problem, the fourth embodiment of the present invention detects the position of the user US, and tilts the angle of the reflective surface of the retroreflective member 1A by x / 2 according to the detected position of the user US, as shown in Figure 23, so that the retroreflective member 1A does not face the user US directly, thereby causing the virtual image VI of the display device DS to deviate from the viewing direction of the user US with respect to the aerial image MI.

[0119] (Configuration Example) FIG. 24 shows an example of the configuration of an aerial image display system according to the fourth embodiment of the present invention.

[0120] The system according to the fourth embodiment has a rotation mechanism 55 and a movement mechanism 56 in the display device DS, and also has a movement mechanism 11 and a rotation mechanism 12 (shown in Figure 25) in the retroreflective member 1A for varying its position and orientation.

[0121] Furthermore, a display control device CS3 is provided to control the display position and display direction of the display device DS and the arrangement position and reflection direction of the retroreflective member 1A.

[0122] The optical system also includes a camera CM, such as a depth camera, that captures the area in the real space RS where the user US is present and outputs the captured image to the display control device CS3. Note that if the user's viewing position is fixed, the camera CM can be omitted.

[0123] FIG. 25 is a block diagram showing the functional configuration of the display control device CS3 together with the display device DS, the retroreflective member 1A, and the camera CM.

[0124] The display control device CS3 is, for example, a personal computer, and includes a control unit 120 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 220 and a data storage unit 320, and an input / output I / F unit 420 are connected to the control unit 120 via a bus (not shown).

[0125] The display device DS, the mechanism of the retroreflective member 1A, and the camera CM are connected to the input / output I / F unit 420. More specifically, via a wireless interface such as a signal cable or a wireless LAN (Local Area Network), the display device main body 51, the movement mechanism 56, and the rotation mechanism 55 of the display device DS are connected, as well as the mechanism of the retroreflective member 1A and the camera CM.

[0126] The program storage unit 220 is configured by combining, for example, a nonvolatile memory such as an SSD as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a ROM, and stores application programs necessary for control according to the fourth embodiment in addition to middleware such as an OS. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0127] The data storage unit 320 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time and a volatile memory such as a RAM as a storage medium, and a display information storage unit 321 is provided in the storage area.

[0128] The display information storage unit 321 stores content information for displaying an aerial image to the user US. This content information is obtained, for example, by reading it from an external storage medium or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0129] The control unit 120 includes, as processing functions necessary for implementing the fourth embodiment, a user position acquisition processing unit 121, an aerial image display position acquisition processing unit 122, a display position and angle calculation processing unit 123, a retroreflective member position and angle calculation processing unit 124, a display position and direction control processing unit 125, and a retroreflective member position and direction control processing unit 126. These processing units 121 to 126 are all realized by causing a hardware processor of the control unit 120 to execute application programs stored in the program storage unit 210.

[0130] Note that some or all of the processing units 121 to 126 may be realized using hardware such as an LSI or an ASIC.

[0131] The user position acquisition processing unit 121 acquires the captured image output from the camera CM via the input / output I / F unit 420, and calculates the position information of the user US in the real space RS based on the acquired captured image. Note that if the user's viewing position is fixed, the user position acquisition processing unit 121 may acquire and store the viewing position as a parameter. In this case, the camera CM can be omitted.

[0132] The aerial image display position acquisition processing unit 122 acquires information indicating the display position of the aerial image to be displayed from the content information stored in the display information storage unit 321 .

[0133] The display position / angle calculation processor 123 calculates the display position of the display device DS based on the position information of the user US and the display position information of the aerial image. The display position / angle calculation processor 123 also calculates the rotation angle θ of the display device DS based on the position information of the user US, the display position information of the aerial image, and information indicating whether the aerial image to be displayed is an imaged aerial image or a direct-view aerial image.

[0134] The retroreflective member position / angle calculation processing unit 124 calculates the position of the retroreflective member 1A so that the optical path length between the user US is shortest, based on the position information of the user US obtained by the user position acquisition processing unit 121, and also taking into consideration the display position of the display device DS and the known position of the first beam splitter 2.

[0135] In addition, the retroreflective member position / angle calculation processing unit 124 calculates the rotation angle x / 2 of the reflective surface of the retroreflective member 1A to face the user US directly, based on the position information of the user US obtained by the user position acquisition processing unit 121, the display position of the display device DS, and the viewing zone (viewing angle) of the display device DS defined by the standard.

[0136] An example of the process for calculating the position of the retroreflective member 1A and the process for calculating the rotation angle will be described in the operation example.

[0137] The display position / orientation control processing unit 125 drives the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS according to the calculated display position and rotation angle θ of the display device DS, respectively, to control the display position and display angle of the display device DS.

[0138] The retroreflective member position / direction control processing unit 126 drives the moving mechanism unit 11 and the rotating mechanism unit 12 of the retroreflective member 1A in accordance with the position and rotation angle x / 2 of the retroreflective member 1A calculated by the retroreflective member position / angle calculation processing unit 124, thereby controlling the position and reflection direction of the retroreflective member 1A.

[0139] (Example of Operation) Next, the operation of the aerial image display system configured as above will be described.

[0140] FIG. 26 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 120 of the display control device CS3.

[0141] (1) Acquisition of User Position When displaying an aerial image, the control unit 120 of the display control device CS3 first acquires user position information using the user position acquisition processing unit 121 in step S30. For example, the user position acquisition processing unit 121 acquires a captured image output from the camera CM via the input / output I / F unit 420 and recognizes an image of the user US from the acquired captured image. Then, based on the position coordinates of the recognized user US in the image, it calculates the position information of the user US in the real space RS. As mentioned above, if the user's viewing position is fixed, the user position acquisition processing unit 121 may acquire and store the fixed viewing position as a parameter in advance.

[0142] (2) Acquisition of Aerial Image Display Position Subsequently, in step S31, the control unit 120 of the display control device CS3 specifies the display position of the aerial image using the aerial image display position acquisition processing unit 122. For example, when the content information stored in the display information storage unit 321 includes information indicating the display target position of the aerial image, this display target position is acquired as is as display position information of the aerial image.

[0143] (3) Calculation of Display Position and Display Angle of Display Device DS (3-1) Calculation of Display Position After obtaining the position information of the user US and the display position information of the aerial image, in step S32, the control unit 120 of the display control device CS3 calculates the display position of the display device DS using the display position / angle calculation processing unit 123. The display position of the display device DS can be determined as a position that is plane-symmetric with respect to the display position information of the aerial image.

[0144] (3-2) Calculation of rotation angle θ for setting the display orientation Next, in step S33, the display position / angle calculation processing unit 123 calculates the rotation angle θ for setting the display orientation of the display device DS as follows: Note that the calculation process of the rotation angle θ is the same as the process described in FIG. 19, so a description thereof will be omitted here.

[0145] (4) Calculating the position of the retroreflective member 1A and the angle x / 2 of the reflective surface Next, under the control of the retroreflective member position / angle calculation processing unit 124, the control unit 120 of the display control device CS3 calculates the position of the retroreflective member 1A and the angle x / 2 of the reflective surface in steps S34 and S35, respectively.

[0146] (4-1) Calculation of the moving position of the retroreflective member 1A When presenting an aerial image MI to a user US, in order to present a high-quality aerial image MI, it is desirable to minimize the optical path length from the light of the displayed image output from the display device DS to the point where it is perceived by the user US.

[0147] Therefore, the retroreflective member position / angle calculation processing unit 124 calculates the movement position of the retroreflective member 1A so as to satisfy the following conditions: Fig. 27 is a diagram for explaining the calculation process of the movement position.

[0148] The retroreflective member position / angle calculation processor 124 first defines the retroreflective member 1A as moving on a straight line connecting the user US and the aerial image formation position, and then sets safety areas E1 and E2 for the retroreflective member 1A and the display device DS to prevent collisions.

[0149] For example, the safety area E1 of the retroreflective member 1A is defined as a circle that includes the maximum width of the reflective surface of the retroreflective member 1A. In contrast, the safety area E2 of the display device DS is defined as a circle that includes the maximum width of the display device main body 51 or the base 52 and legs 53 that support it.

[0150] In this state, the retroreflective member position / angle calculation processor 124 calculates the optimal position of the retroreflective member 1A. For example, the retroreflective member position / angle calculation processor 124 first determines the position where the safety area E1 of the retroreflective member 1A maintains a safe distance between the safety area E2 of the display device DS and the second beam splitter 3. Then, the retroreflective member position / angle calculation processor 124 determines the position where the y-axis coordinate value of the circle representing the safety area E1 of the retroreflective member 1A is minimum, and calculates the position where the retroreflective member 1A is closest to the user US within a range smaller than the position where the y-axis coordinate value of the circle representing the safety area E2 of the display device DS is maximum. The calculated position is then set as the position to which the retroreflective member 1A should be moved.

[0151] (4-2) Calculation of the reflection angle of the retroreflective member 1A The retroreflective member position / angle calculation processing unit 124 then calculates the angle α calculated by the display position / angle calculation processing unit 12, that is, the angle α formed by the direction in which the user US faces the first beam splitter 2 and the straight line connecting the user US and the aerial image MI, plus a rotation angle x / 2 for further rotating the retroreflective member 1A.

[0152] 28 and 29 are diagrams used in the calculation process of the rotation angle x / 2. Here, the following values ​​are known parameters: a: distance from the display position of the display device DS to the retroreflective member 1A, b: distance from the display device DS to the viewpoint of the user US, and t: viewing zone of the display device DS.

[0153] The distance b from the display device DS to the viewpoint of the user US is calculated based on the position information of the user US obtained by the user position acquisition processing unit 121. Furthermore, the viewing zone t of the display device DS is expressed by the rating of a viewing zone limiting film, for example, if the viewing zone limiting film is attached to the display screen.

[0154] The retroreflective member position / angle calculation processing unit 124 first calculates the distance A from the viewpoint position of the user US to the retroreflective member 1A and the distance B as follows: Note that B is a symbol defined only in the formula as B = tan(t) for convenience.

[0155]

[0156] Next, the retroreflective member position / angle calculation processing unit 124 calculates the angle x using the calculated A and B as follows:

[0157]

[0158] In this way, the rotation angle α+x / 2 of the reflective surface of the retroreflective member 1A is calculated.

[0159] (5) Control of Movement Position and Display Orientation of Display Device DS In step S36, the control unit 120 of the display control device CS3, under the control of the display position / orientation control processing unit 115, generates control signals for adjusting the position and angle of the display device DS by the calculated movement position and rotation angle θ of the display device DS. Then, the display position / orientation control processing unit 125 outputs the generated movement position control signal and rotation angle control signal from the input / output I / F unit 420 to the movement mechanism unit 56 and rotation mechanism unit 55 of the display device DS, respectively. In this way, the display position and display orientation of the display device DS are controlled.

[0160] (6) Control of the Position and Reflection Direction of the Retroreflective Member 1A In step S37, the control unit 120 of the display control device CS3, under the control of the retroreflective member position / direction control processing unit 126, generates control signals for adjusting the position and angle of the retroreflective member 1A by the calculated movement position and rotation angle α+x / 2 of the retroreflective member 1A. The retroreflective member position / direction control processing unit 126 then outputs the generated position control signal and rotation angle control signal from the input / output I / F unit 420 to the movement mechanism 11 and rotation mechanism 12 of the retroreflective member 1A, respectively. In this way, the position and reflection direction of the retroreflective member 1A are controlled.

[0161] (Effect) As described above, according to the fourth embodiment, the display position and display angle of the display device DS are controlled according to the position of the user US and the display position of the aerial image, and the position and reflection direction of the retroreflective member 1A are controlled according to the display position of the display device DS and the position of the user US, thereby setting the retroreflective member 1A so that it does not face the user US directly.

[0162] As a result, it is possible to prevent the virtual image VI of the display device DS from entering the field of view of the user US who is viewing the aerial image MI, thereby improving the visibility of the aerial image MI for the user US.

[0163] In the above description, the position and angle of the reflective surface of the retroreflective member 1A are controlled, but it is also possible to control only the angle of the reflective surface. Furthermore, although the position and display direction of the display device DS are also controlled, it is not necessary to control the position and display direction of the display device DS.

[0164] Fifth Embodiment (Summary) Displaying a background image behind an aerial image to reinforce the sense of reality of the aerial image is extremely effective in practice. For example, displaying an aerial image of a character, object, etc. in conjunction with an image representing its shadow enables a more realistic display.

[0165] However, even if a background image is to be displayed, there may not be enough space to install a background display device in the real space where the first beam splitter on the user's side is located. Furthermore, if the display device for displaying the aerial image is moved to the mirror image space movement area, as shown in Figure 4, for example, the moving device will block the background image. As a result, areas of the background image other than those corresponding to the display device will be perceived as black, hindering the display of the aerial image.

[0166] Therefore, in the fifth embodiment of the present invention, a second display device for background display is placed in the space on the opposite side of the second beam splitter from the placement position of the first display device for displaying the aerial image, and the background image displayed on this second display device is reflected by the second beam splitter and displayed in the direction of the user US.

[0167] 30 is a diagram showing a first example of an optical system of an aerial image display system according to a fifth embodiment of the present invention. In this figure, the same parts as those in FIG. 1 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0168] 30, a display device BD for background display is placed in the space on the opposite side of the second beam splitter 3 from the position of the display device DS for displaying the aerial image. The background image displayed on this display device BD for background display is reflected by the second beam splitter 3, then passes through the first beam splitter 2 and is presented to a user US present in the real space RS.

[0169] With this configuration, the display device BD for displaying the background can be installed even when there is no installation space available in the real space RS where the user US is present. Furthermore, as shown in Figure 30, even when the display device DS for displaying the aerial image is moved into the mirror image space movement area ME to form the aerial image MI in the mirror image space MS, the display device DS does not interfere with the virtual image BI of the background image.

[0170] As a result, the user US is presented with an aerial image with a background, in which only the formed aerial image MI is superimposed on the background video BI. An example of such a display image is shown in Figure 31. Therefore, in addition to resolving the problem of the placement space for the display device BD for displaying the background, it is possible to present the user US with a clear aerial image with a background without interfering with the display of the aerial image MI displayed by the display device DS.

[0171] In order to resolve the occlusion conflict between the background image BI and the aerial image MI, the display of the background range that overlaps with the aerial image MI may be erased based on the position of the user US.

[0172] Second Example Fig. 32 is a diagram showing a second example of the optical system of the aerial image display system according to the fifth embodiment of the present invention. In this figure, the same parts as those in Fig. 30 are given the same reference numerals.

[0173] In the second embodiment, an optical element whose transmission and reflection change depending on the polarization direction, such as a reflective polarizer, is used as the second beam splitter 3. A phase difference film 4 is disposed on the reflective surface side of the retroreflective member 1A. Note that the first beam splitter 2 is a half mirror whose reflection characteristics do not change depending on the polarization direction, for example, a transparent plate with metal deposition.

[0174] With this configuration, by matching the polarization characteristics of the second beam splitter 3 with the polarization direction of the display image output from the display device DS, it is possible to suppress attenuation of the brightness level of the aerial image, thereby making it possible to present a bright and clear aerial image with a background.

[0175] (Third Example) Figure 33 is a diagram showing a third example of the optical system of the aerial image display system according to the fifth embodiment of the present invention. In this figure, the same parts as in Figure 30 are given the same reference numerals.

[0176] In the third embodiment, a display device VS for displaying a virtual image is used instead of the display device BD for displaying a background. Like the display device DS for displaying an aerial image, the display device VS for displaying a virtual image is provided with a movement mechanism and a rotation mechanism for varying the display position and display direction, thereby enabling it to move within the mirror image space MS on the opposite side of the second beam splitter 3 from the arrangement position of the display device DS.

[0177] With this configuration, for example, if the display device DS for aerial image display displays a front image of a character or an object, and the display device VS for virtual image display displays a rear image of the character or object in synchronization with this, then an aerial image equivalent to a virtual image of the formed aerial image corresponding to the front image of the character or object, and an aerial image corresponding to the rear image of the character or object are simultaneously presented to the user US present in the real space RS.

[0178] Therefore, it is possible to display an aerial image displayed in real space using a virtual image representation, as if it were reflected in a mirror.

[0179] Other Embodiments The materials, functions, and positional relationship of the retroreflective members and first and second beam splitters that constitute the optical system, the type and configuration of the display device, the type of content to display the aerial image, etc. may be modified in various ways without departing from the spirit and scope of the present invention. Furthermore, the functional configuration of the display control devices CS1, CS2, and CS3 and the procedures and processing contents of their control processes may also be modified in various ways without departing from the spirit and scope of the present invention.

[0180] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0181] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0182] US...viewer (user) DS...display device for displaying aerial image RS...real space MS...mirror image space RE...real space movement area ME...mirror image space movement area RI...direct-view aerial image MI...imaged aerial image BD...display device for background display 1A, 1B...retroreflective member 2...first beam splitter 3...second beam splitter 4...phase difference film 11, 56...movement mechanism unit 12, 55...rotation mechanism unit 51...display device main body 52...base 53...legs 54...casters 100, 110, 120...control unit 101...display instruction acquisition processing unit 102...display position / direction control processing unit 103...image parameter control processing unit 111, 121...user position acquisition processing unit 112, 122...aerial image display position acquisition processing unit 113...movement position calculation processing unit 114...rotation angle calculation processing unit DESCRIPTION OF SYMBOLS 115...Display position and direction control processing unit 123...Display position and angle calculation processing unit 124...Retroreflective member position and angle calculation processing unit 125...Display position and direction control processing unit 126...Retroreflective member position and direction control processing unit 200, 210, 220...Program storage unit 300, 310, 320...Data storage unit 301...Display position and direction control data storage unit 302...Image parameter storage unit 303...Display information storage unit 311, 321...Display information storage unit 400, 410, 420...Input / output I / F unit 500...Input device

Claims

1. A retroreflective member arranged so that a reflective surface is perpendicular or nearly perpendicular to a viewing direction from a viewer; a first optical member having a first optical characteristic of transmitting a part of incident light and reflecting a part of it, the first optical member being disposed between the viewer and the retroreflective member with an operating surface parallel or nearly parallel to the reflecting surface of the retroreflective member, forming a mirror image space between the first optical member and the retroreflective member, and forming a real space in the space in which the viewer is present; A second optical member having a second optical characteristic of transmitting a part of the incident light and reflecting a part of the incident light, the second optical member being disposed in the mirror image space with an active surface having a predetermined inclination angle with respect to the viewing direction, and forming a mirror image space movement area between the second optical member and the retroreflective member; A display device for displaying display information; Equipped with The display device includes: a movement mechanism that can move a display position between the mirror image space movement region and a real space movement region that is formed in a space facing the second optical member via the mirror image space movement region; a rotation mechanism that can change a display direction between a direction perpendicular to the viewing direction or a direction close to the viewing direction and a direction parallel to the viewing direction or a direction close to the viewing direction; An aerial image display system comprising:

2. A retroreflective member arranged such that a reflective surface is parallel or nearly parallel to a viewing direction from a viewer; A first optical member having a first optical characteristic of transmitting a part of incident light and reflecting a part of it, the reflective surface and the active surface of the retroreflective member being arranged in a state where they are perpendicular or nearly perpendicular to each other, forming a mirror image space in a rectangular area between the retroreflective member and the first optical member, and forming a real space in the space where the viewer is present; A second optical member having a second optical characteristic of transmitting a part of the incident light and reflecting a part of the incident light, the second optical member being disposed in the mirror image space with an active surface having a predetermined inclination angle with respect to the viewing direction, and forming a mirror image space movement area in a triangular space on the opposite side to the retroreflective member; A display device for displaying display information; Equipped with The display device includes: a movement mechanism that can move a display position between the mirror image space movement region and a real space movement region that is formed in a space facing the second optical member via the mirror image space movement region; a rotation mechanism that can change a display direction between a direction perpendicular to the viewing direction or a direction close to the viewing direction and a direction parallel to the viewing direction or a direction close to the viewing direction; An aerial image display system comprising:

3. A first retroreflective member arranged such that a reflective surface is perpendicular or nearly perpendicular to a viewing direction from a viewer; A second retroreflective member arranged such that a reflective surface is parallel or nearly parallel to the viewing direction; a first optical member having a first optical characteristic of transmitting a part of incident light and reflecting a part of the incident light, the first optical member being disposed such that an active surface of the first optical member has a predetermined inclination angle with respect to the viewing direction; a display device that is disposed in an area of ​​the first optical member opposite to the second retroreflective member and displays display information toward the first optical member or the viewer; Equipped with An aerial image display system in which the first optical element has a configuration in which both sides of a reflective element, which has equal transmittance and reflectance and both sides function as reflective surfaces, are sandwiched between two transparent elements having equal thicknesses and refractive indices to provide symmetry with respect to both sides.

4. A first retroreflective member arranged such that a reflective surface is perpendicular or nearly perpendicular to a viewing direction from a viewer; A second retroreflective member arranged such that a reflective surface is parallel or nearly parallel to the viewing direction; a first optical member having a first optical characteristic of transmitting a part of incident light and reflecting a part of the incident light, the first optical member being disposed such that an active surface of the first optical member has a predetermined inclination angle with respect to the viewing direction; A display device that is disposed in an area of ​​the first optical member opposite to the second retroreflective member and displays display information toward the first optical member or the viewer; a second optical member having a second optical characteristic of transmitting a part of the incident light and reflecting a part of the incident light, forming a mirror image space in a surrounding space together with the first retroreflective member and the second retroreflective member, and forming a real space in a space where the viewer is present; Equipped with The display device includes: a moving mechanism that can move a display position within the area; a rotation mechanism that can change a display direction between a direction perpendicular to the viewing direction or a direction close to the viewing direction and a direction parallel to the viewing direction or a direction close to the viewing direction; An aerial image display system comprising:

5. The retroreflective member further includes a phase difference member that imparts a phase difference to the polarization direction of incident light, the first optical member and the second optical member are configured by optical elements having optical characteristics in which transmission and reflection change depending on the polarization direction of incident light; 3. The aerial image display system according to claim 1 or 2.

6. A display control device connected to the display device, The display control device includes: a video parameter control unit that variably controls video parameters including at least one of brightness, color, and contrast of the display information in response to a moving operation of a display position of the display device and a varying operation of a display direction of the display device; The aerial image display system according to claim 1 or 2, comprising:

7. The image parameter control unit includes: When the display direction is oriented in a direction perpendicular to the viewing direction, the image parameter is set to a first value, and when the display position is set in the mirror image space movement area and the display direction is oriented in a direction parallel to the viewing direction, the image parameter is set to a second value lower than the first value.

7. The aerial image display system according to claim 6.

8. 1. A display control device used in an aerial image display system that displays an image displayed on a display device as an aerial image by selectively passing a first optical path formed by an optical system and having a first optical path length and a second optical path having a second optical path shorter than the first optical path length, A display control device comprising: an image parameter control unit that sets an image parameter of the displayed image to a first value when the first optical path is selected, and sets the image parameter to a second value lower than the first value when the second optical path is selected.

9. The display control device according to claim 8 , wherein the image parameter control unit controls at least one of brightness, color, and contrast as the image parameters.