Aerial image display system, display control device, display control method and program

Abstract

An aerial image display system according to an aspect of the present invention comprises: a display device that displays a display video; and an optical system that has a retroreflection member and an optical member having optical characteristics of partially reflecting and partially transmitting incident light, and that displays an aerial image corresponding to the display video to a viewer. The aerial image display system acquires positional information of the viewer, and causes, on the basis of the acquired positional information of the viewer, a reflection surface of the retroreflection member to be rotated by an angle that is specified using the direction of the viewer as a reference.

Description

[Technical Field]

[0001] One aspect of the present invention relates to an aerial image display system for displaying, for example, context information or the like in space, and a display control device, a display control method, and a program used in this system. [Background technology]

[0002] As a method for presenting context information etc. to a user, for example, a method of presenting information using a mirror that people normally use has been proposed. For example, Non-Patent Document 1 describes a method that uses a special mirror that has the function of transmitting part of incident light and reflecting part of it, such as a one-way mirror, and displays context information from a display placed on the back side of the mirror in the front direction through the mirror, thereby simultaneously presenting a virtual image of the user himself / herself and a real image of the context information to the user.

[0003] Furthermore, Non-Patent Document 2 describes a method for presenting highly realistic information in the field of MR (Mixed Reality), which displays digital information in the real world, by simultaneously displaying an aerial image of the front side and an aerial image of the back side in the real space in front of the mirror and the mirror image space behind the mirror, respectively. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kaori Fujinami, Kausar Fahim, Tatsuo Nakajima, "Context-aware information display device using an extended mirror," Transactions of Information Processing Society of Japan, vol. 49 No. 6 pp. 1972-1983 (June 2008) [Non-patent document 2] Hiroki Yamamoto, Han-Yeoul Kim, Naoya Koizumi, Ken Naemura, "Mixed Reality System Presenting Horizontal and Vertical Aerial Images in Front and Behind a Mirror," 3D Imaging Conference, pp.23-26, (July 2015) Summary of the Invention [Problem to be solved by the invention]

[0005] However, both Non-Patent Document 1 and Non-Patent Document 2 present information in a fixed manner in either real space or mirror image space. In contrast, digital information such as context information is inherently independent of the space in which it is presented. Therefore, there is a demand for the development of a more flexible information presentation method that is not bound by physical phenomena in the real world.

[0006] This invention was made in light of the above circumstances, and aims to provide a technology that enables aerial images of digital information to be presented in either real space or mirror image space, thereby increasing the freedom of information presentation and improving the visibility of the aerial images. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is an aerial image display system including a display device that displays a display image, and an optical system that has a retroreflective member and an optical member that has optical properties of reflecting part of incident light and transmitting part of it, and displays an aerial image corresponding to the display image toward a viewer, the system acquiring position information of the viewer and position information of the display device, and Based on the acquired position information of the viewer, the angle of the reflective surface of the retroreflective member is rotated by a specified angle with respect to the direction of the viewer, The position of the retroreflective member relative to the viewer and the angle of the reflective surface are controlled based on the acquired position information of the viewer and the position information of the display device.

[0009] According to one aspect of the present invention, at least the angle of the reflective surface of the retroreflective member is controlled in accordance with viewer position information, thereby setting the retroreflective member so that it does not face the viewer directly. This makes it possible to prevent the virtual image of the display device from entering the field of view of the viewer viewing the aerial image, thereby improving the viewer's visibility of the aerial image. [Effects of the Invention]

[0010] According to one aspect of the present invention, a technology can be provided that enables aerial images of digital information to be presented in either real space or mirror image space, thereby increasing the freedom of information presentation and improving the visibility of the aerial images. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a first example of an optical system in an aerial image display system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a perspective view showing an example of the configuration of a display device provided in the aerial image display system shown in FIG. [Figure 3] FIG. 3 is a diagram showing an example of the operation of the aerial image display system shown in FIG. 1 when forming an aerial image in real space. [Figure 4] FIG. 4 is a diagram showing an example of the operation of the aerial image display system shown in FIG. 1 when an aerial image is formed in a mirror image space. [Figure 5] FIG. 5 is a diagram showing an example of operation when a direct-view aerial image is presented in the aerial image display system shown in FIG. [Figure 6] FIG. 6 is a diagram showing a second example of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing a third example of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 8] FIG. 8 is a cross-sectional view showing an example of the structure of a beam splitter used in the aerial image display system shown in FIG. [Figure 9] FIG. 9 is a diagram showing a fourth example of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a fifth example of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to the second embodiment of the present invention. [Figure 12] FIG. 12 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit of the display control device shown in FIG. [Figure 13] FIG. 13 is a diagram for explaining an example of a method for calculating image parameters in the display control process shown in FIG. [Figure 14] FIG. 14 is a diagram for explaining another example of the method for calculating image parameters in the display control process shown in FIG. [Figure 15] FIG. 15 is a diagram illustrating an overview of an aerial image display system according to the third embodiment of the present invention. [Figure 16] FIG. 16 is a diagram showing an example of the configuration of an optical system of an aerial image display system according to the third embodiment of the present invention. [Figure 17] FIG. 17 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to the third embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit of the display control device shown in FIG. [Figure 19] FIG. 19 is a flowchart showing an example of the processing procedure and processing content of the rotation angle calculation processing in the display control processing shown in FIG. [Figure 20] FIG. 20 is a diagram illustrating a first example of the rotation angle calculation process shown in FIG. [Figure 21] FIG. 21 is a diagram illustrating a second example of the rotation angle calculation process shown in FIG. [Figure 22] FIG. 22 is a diagram illustrating an overview of an aerial image display system according to the fourth embodiment of the present invention. [Figure 23] FIG. 23 is a diagram illustrating an overview of an aerial image display system according to the fourth embodiment of the present invention. [Figure 24] FIG. 24 is a diagram showing an example of the configuration of an optical system of an aerial image display system according to a fourth embodiment of the present invention. [Figure 25]FIG. 25 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to the fourth embodiment of the present invention. [Figure 26] FIG. 26 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit of the display control device shown in FIG. [Figure 27] FIG. 27 is a diagram for explaining an example of the process of calculating the position of the retroreflective member relative to the display device, which is part of the display control process shown in FIG. [Figure 28] FIG. 28 is a diagram for explaining an example of the process of calculating the angle of the retroreflective member in the display control process shown in FIG. [Figure 29] FIG. 29 is a diagram for explaining an example of the process of calculating the angle of the retroreflective member in the display control process shown in FIG. [Figure 30] FIG. 30 is a diagram showing a first example of an optical system in an aerial image display system according to a fifth embodiment of the present invention. [Figure 31] FIG. 31 is a diagram showing an example of the aerial image and background image displayed by the aerial image display system shown in FIG. [Figure 32] FIG. 32 is a diagram showing a second example of the optical system in the aerial image display system according to the fifth embodiment of the present invention. [Figure 33] FIG. 33 is a diagram showing a third example of an optical system in an aerial image display system according to a fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [First embodiment] (First Example) (Configuration example) FIG. 1 is a diagram showing a first example of an aerial image display system according to a first embodiment of the present invention.

[0014] (1)Optical system In FIG. 1, 1A denotes a retroreflective member, and this retroreflective member 1A is disposed so that its reflective surface is perpendicular to the viewing direction of a viewer (hereinafter also referred to as a user) US. A first beam splitter 2 is disposed between the user US and the retroreflective member 1A so that its active surface is perpendicular to the viewing direction of the user US, i.e., parallel to the reflective surface of the retroreflective member 1A. A second beam splitter 3 is disposed in the space between the retroreflective member 1A and the first beam splitter 2. The second beam splitter 3 is disposed so that its active surface is obliquely disposed at a predetermined angle, for example, 45°, relative to the viewing direction of the user US.

[0015] The first and second beam splitters 2 and 3 each have the optical property of transmitting part of the incident light and reflecting part of it, thereby forming a real space RS on the user US side of the first beam splitter 2, and a mirror image space MS between the first beam splitter 2 and the retroreflective member 1A.

[0016] Furthermore, a mirror image space movement region ME is formed in the triangular region in the mirror image space MS between the retroreflective member 1A and the second beam splitter 3. A real space movement region RE is formed in the region adjacent to the mirror image space movement region ME on the extension of the surfaces of the retroreflective member 1A and the first beam splitter 2. The boundary between the mirror image space movement region ME and the real space movement region RE is a virtual mirror plane VM.

[0017] (2)Display device DS The display device DS is arranged to be movable within the mirror image space movement area ME and the real space movement area RE.

[0018] 2 is a perspective view showing an example of the configuration of a display device DS. The display device DS has a display device main body 51 equipped with, for example, a liquid crystal panel, an organic EL panel, or an LED panel, placed on a base 52, and legs 53 equipped with casters 54 are provided on the underside of the display device main body 51 to movably support the display device main body 51. In addition, a rotation mechanism 55 is provided on the upper surface of the base 52. The rotation mechanism 55 is used to variably set the display direction of the display device main body 51 within a predetermined angle range, for example, a range of 180 degrees.

[0019] (Example of operation) With the above configuration, the aerial image display system operates as follows.

[0020] (1) When forming an aerial image in real space RS FIG. 3 is a diagram for explaining the operation in this case. The display device DS is placed in the real-space movement area RE, and furthermore, the display direction is set so as to be, for example, perpendicular to the viewing direction of the user US. The setting of the position and the display direction of the display device DS may be performed manually by an administrator or a viewer, or may be performed automatically by a display control device shown in the figure.

[0021] When the display position and display direction of the display device DS are set in this manner, the display information displayed on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, and passes through the second beam splitter 3 and the first beam splitter 2 in sequence, to be formed as an aerial image MI1 in the real space RS where the user US is present.

[0022] Therefore, the user US can visually recognize, for example, figures, photographs, etc. as aerial images in the real space RS in which the user US exists.

[0023] (2) When forming an aerial image in the mirror image space MS FIG. 4 is a diagram for explaining the operation in this case. The display device DS is moved from the real space movement region RE to the mirror image space movement region ME. In addition, the display direction is set to be perpendicular to the viewing direction of the user US. When the display position and display direction of the display device DS are set in this manner, the display information displayed on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, and passes through the second beam splitter 3 to be formed as an aerial image MI2 in the mirror image space MS.

[0024] Therefore, the user US can use the first beam splitter 2 as a mirror to perform tasks such as brushing their teeth, applying makeup, or grooming, while checking news information such as traffic information and weather forecasts on the aerial image MI2 formed in the mirror image space MS.

[0025] (3) When presenting a direct-view aerial image to the user US FIG. 5 is a diagram for explaining the operation in this case. The display device DS is moved from the real space movement region RE to the mirror image space movement region ME. In addition, the display direction is set to face the viewing direction of the user US. When the display position and display direction of the display device DS are set in this manner, the display information displayed on the display screen of the display device DS is transmitted sequentially through the second beam splitter 3 and the first beam splitter 2, and is presented to the user US in the real space RS as a direct-view aerial image RI.

[0026] Therefore, the user US can view the displayed information in the real space RS as if he or she were looking directly at the display screen of the display device DS.

[0027] (Second Example) 6 is a diagram showing a second example of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as those in FIG. 1 are designated by the same reference numerals and detailed explanations thereof will be omitted.

[0028] In the second embodiment, the retroreflective member 1B is positioned on the opposite side of the virtual mirror surface VM from the second beam splitter 3 in the mirror image space MS, so that the reflective surface is oriented parallel to the viewing direction of the user US.

[0029] With this configuration, when the display device DS is placed, for example, in the real space movement area RE and the display direction is set to a direction perpendicular to the viewing direction of the user US, the display information displayed on the display device DS passes through the second beam splitter 3, is retroreflected by the retroreflective member 1B, is reflected by the second beam splitter 3, passes through the first beam splitter 2, and is formed as an aerial image MI3 in the real space RS where the user US is present.

[0030] Furthermore, when the display device DS is placed in the mirror image space movement area ME and the display direction is set to be perpendicular to the viewing direction of the user US, the display information displayed on the display screen of the display device DS passes through the second beam splitter 3 and is then retroreflected by the retroreflecting member 1B, and is formed as an aerial image MI2 in the mirror image space MS.

[0031] Furthermore, when the display device DS is placed in the mirror image space movement area ME and the display direction is set to a direction opposite to the viewing direction of the user US, the display information displayed on the display device DS passes through the second beam splitter 3 and the first beam splitter 2 sequentially and is presented to the user US in the real space RS as a direct-view aerial image RI.

[0032] Therefore, the second embodiment also provides the same effect as the first embodiment.

[0033] (Third Example) 7 is a diagram showing a third example of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as those in FIGS. 1 and 6 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0034] In the third embodiment, a retroreflective member 1A and a retroreflective member 1B are arranged on two mutually perpendicular sides of the mirror image space MS so that their reflective surfaces are perpendicular to each other. As the second beam splitter 3, for example, as shown in Fig. 8, a reflecting member 3a such as a half mirror with equal transmittance and reflectance is sandwiched between two transparent members 3b and 3c such as acrylic plates with equal thickness and refractive index.

[0035] With this configuration, when the display device DS is placed in the real-space movement area RE and its display direction is set to be perpendicular to the viewing direction of the user US, the display information displayed on the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflecting member 1A, and sequentially transmitted through the second beam splitter 3 and the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is present. At the same time, the display information transmits through the second beam splitter 3, then retroreflected by the retroreflecting member 1B, reflected by the second beam splitter 3, and then transmitted through the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is present. The same applies when forming an aerial image in the mirror image space MS.

[0036] In other words, a composite aerial image MI4 is formed in the real space RS and the mirror image space MS by combining the aerial image retroreflected by the retroreflecting member 1A and the aerial image retroreflected by the retroreflecting member 1B.

[0037] Therefore, according to the third embodiment, it is possible to present a formed aerial image MI4 with higher brightness than when retroreflective members 1A and 1B are used alone. Furthermore, since the second beam splitter 3 has a structure in which both sides of the reflective member 3a are sandwiched between two transparent members 3b and 3c with the same thickness and refractive index, as shown in Figure 8, it is possible to set the optical path lengths corresponding to the first and second beam splitters 2 and 3 to be equal, thereby making it possible to prevent double images from appearing in the formed aerial image MI4.

[0038] (Fourth Example) 9 and 10 are diagrams showing a fourth example of the aerial image display system according to the first embodiment of the present invention.

[0039] In the fourth embodiment, a retardation film 4 is disposed on the reflective surface of the retroreflective member 1A. The retardation film 4 is, for example, a 1 / 4 retardation film, and has the optical property of rotating the polarization direction of transmitted light by 45 degrees. Furthermore, optical elements that switch between reflecting and transmitting light depending on the polarization direction of the incident light are used as the first and second beam splitters 2 and 3. These optical elements may be, for example, reflective polarizing plates or wire grids. Note that the following description will discuss the operation of a device installed in a direction that reflects S-polarized light and transmits P-polarized light.

[0040] In contrast, a depolarizing film, a diffusion plate, or a retardation film is attached to the display screen of the display device DS so that the output light of the display information is unpolarized or contains S-polarized and P-polarized light. Alternatively, a polarizing plate or a retardation film is rotatably disposed facing the display screen of the display device DS, and the polarization direction of the output light can be switched by rotation.

[0041] With this configuration, when forming an aerial image in the real space RS, the display device DS is placed in the real space movement region RE with its display direction facing perpendicular to the viewing direction. If a polarizing plate is placed on the display screen of the display device DS, this polarizing plate is rotated so that the polarized light component reflected by the beam splitter 3 from the display device DS, for example, S-polarized light, is output.

[0042] With this setting, as shown in Figure 9, for example, S-polarized light out of the display light output from the display device DS is reflected by the second beam splitter 3 and converted into circularly polarized light by passing through the phase difference film 4 placed in front of the retroreflective member 1A. The circularly polarized light has its direction of rotation reversed by being retroreflected by the retroreflective member 1A, and is converted into P-polarized light by passing through the phase difference film 4 again. This retroreflected P-polarized light passes through the second beam splitter 3 and then the first beam splitter 2 to be formed as an aerial image MI5 in the real space RS.

[0043] Next, when forming an aerial image in the mirror image space MS, the display device DS is placed in the mirror image space movement region ME with its display direction facing perpendicular to the viewing direction. At the same time, the polarizing plate arranged on the display screen of the display device DS is rotated so that S-polarized light is output from the display device DS.

[0044] 10, for example, S-polarized light out of the display light output from the display device DS1 is reflected by the second beam splitter 3 and, as described above, is retroreflected as P-polarized light by the phase difference film 4 arranged in front of the retroreflecting member 1A and the retroreflecting member 1A. The retroreflected P-polarized light then passes through the second beam splitter 3 and is formed as an aerial image MI6 in the mirror image space MS.

[0045] When display information on the display device DS is presented as a direct-view aerial image RI, the display device DS is positioned in the mirror image space movement area ME and its display direction is directed toward the user US. At the same time, the polarizers and retardation plates arranged on the display screen of the display device DS are rotated so that P-polarized light is output from the display device DS.

[0046] When set up in this manner, the P-polarized light among the display light output from the display device DS passes through the second beam splitter 3 and the first beam splitter 2 in sequence, and is presented as a direct-view aerial image RI to the user US present in the real space RS.

[0047] Therefore, according to the fourth embodiment, by selectively switching the display light output from the display device DS between S-polarized and P-polarized light, it is possible to suppress light attenuation due to the beam splitter on the optical path and present a bright aerial image.

[0048] (Fifth Example) For example, as shown in FIG. 10, the display devices DS1 and DS2 may be arranged in an L-shape, and each may be set to output S-polarized light and P-polarized light. With this configuration, it becomes possible to simultaneously present multiple aerial images with high brightness in both the mirror image space MS and the real space RS.

[0049] [Second embodiment] (overview) In the system described in the first embodiment, an aerial image can be displayed seamlessly across the boundary between the mirror image space MS and the real space RS, which provides a practically very useful effect of diversifying the display of aerial images. However, the imaged aerial image MI undergoes many reflections and transmissions by the beam splitters 2 and 3 and the retroreflective members 1A and 1B on the optical path before the image is formed, which reduces the contrast and causes uneven image quality compared to the direct-view aerial image RI.

[0050] Therefore, in the second embodiment, when displaying the imaged aerial image MI, the image parameters of the display image on the display device DS are adjusted in advance to achieve uniform image quality between the image and the direct-view aerial image RI.

[0051] (Configuration example) FIG. 11 is a block diagram showing the functional configuration of a display control device CS1 provided in an aerial image display system according to the second embodiment of the present invention, together with a display device DS.

[0052] The display device DS is provided with a movement mechanism 56 for moving the display position and a rotation mechanism 55 for changing the display direction. Both of these mechanisms 56 and 55 operate in accordance with control signals output from the display control device CS1.

[0053] The display control device CS1 is, for example, a personal computer, and includes a control unit 100 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 200 and a data storage unit 300, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 400 are connected to the control unit 100 via a bus (not shown).

[0054] An input device 500 such as a keyboard and a mouse is connected to the input / output I / F unit 400. Note that a display device, an external storage medium such as a USB (Universal Serial Bus) memory, etc. may also be connected to the input / output I / F unit 400. The input / output I / F unit 400 may also be provided with a communication interface function.

[0055] The program storage unit 200 is configured by combining, for example, a nonvolatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs required for control according to the second embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0056] The data storage unit 300 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and in its storage area, a display position / direction control data storage unit 301, a video parameter storage unit 302, and a display information storage unit 303 are provided as main storage units required to implement the second embodiment of the present invention.

[0057] Display position / direction control data storage unit 301 stores control data required to control the display position and display direction of display device DS in accordance with input display instructions. Image parameter storage unit 302 stores control data required to control image parameters of display information in accordance with the type of aerial image to be displayed. Display information storage unit 303 is used to store information to be displayed, such as content information.

[0058] The control unit 100 includes, as processing functions necessary for implementing the second embodiment of the present invention, a display instruction acquisition processing unit 101, a display position / direction control processing unit 102, and a video parameter control processing unit 103. These processing units 101 to 103 are all realized by causing a hardware processor of the control unit 100 to execute an application program stored in the program storage unit 200.

[0059] Note that some or all of the processing units 101 to 103 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0060] Display instruction acquisition processing unit 101 acquires display instruction information when the display instruction information for an aerial image is input via input device 500. The display instruction information instructs whether to display an imaged aerial image in real space RS, an imaged aerial image in mirror image space MS, or a direct-view aerial image in real space RS.

[0061] The display position / orientation control processing unit 102 reads out control data from the display position / orientation control data storage unit 301 in accordance with the instructions in the display instruction information. Then, in accordance with the read out control data, the input / output I / F unit 400 outputs a position control signal and a direction control signal to the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS, respectively, to set the display position and display direction of the display device DS.

[0062] The image parameter control processing unit 103 determines whether the aerial image to be presented is a "focused aerial image" or a "direct-view aerial image" based on the setting data for the display position and display direction of the display device DS. Then, the image parameter control processing unit 103 controls the image parameters of the display image to be displayed on the display device DS based on the determination result and the image parameters stored in the image parameter storage unit 302. An example of this image parameter control processing will be described in the operation example.

[0063] (Example of operation) Next, the operation of the display control device CS1 configured as above will be described.

[0064] FIG. 12 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 100 of the display control device CS1.

[0065] The control unit 100 of the display control device CS1 monitors input of display instruction information in step S10 under the control of the display instruction acquisition processing unit 101. When display instruction information is input via the input device 500 in this state, the control unit 100 of the display control device CS1 determines in step S11 under the control of the display position / orientation control processing unit 102 whether the display instruction information is an instruction to display a "formed aerial image" or a "direct-view aerial image."

[0066] If the result of the above determination is that the display instruction is "imaging aerial image," the display position / orientation control processing unit 102 proceeds to step S12, where it generates a display position control signal based on the display position / orientation control data stored in the display position / orientation control data storage unit 301. Then, the display position / orientation control processing unit 102 outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the real space movement region RE or the mirror image space movement region ME.

[0067] Furthermore, the display position / orientation control processing unit 102 generates a display orientation control signal based on the display position / orientation control data, and outputs the generated display orientation control signal to the rotation mechanism unit 55 of the display device DS, thereby setting the display orientation of the display device DS to a direction perpendicular to the viewing direction.

[0068] On the other hand, if the result of the above determination is that the display instruction is a "direct-view aerial image," the display position and orientation control processing unit 102 proceeds to step S14, where it generates a display position control signal based on the position and orientation control data stored in the display position and orientation control data storage unit 301. Then, the display position and orientation control processing unit 102 outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the mirror image space movement area ME.

[0069] At the same time, the display position / orientation control processing unit 102 generates a display orientation control signal based on the position / orientation control data, and outputs the generated display orientation control signal to the rotation mechanism unit 55 of the display device DS, thereby setting the display orientation of the display device DS in the direction of the user US.

[0070] In the above description, the display instruction information specifies whether to display a "focused aerial image" or a "direct-view aerial image." However, if the content included in the display information includes information specifying the type of aerial image to be displayed, i.e., whether it is a "focused aerial image" or a "direct-view aerial image," or information that can determine this, this information may be used.

[0071] When the setting control of the display position and display direction of the display device DS is completed, the control unit 100 of the display control device CS1 then executes the process of controlling the image parameters of the display information under the control of the image parameter control processing unit 103 as follows.

[0072] That is, the image parameter control processing unit 103 receives the display instruction (either "image-formed aerial image" or "direct-view aerial image") from the display position / direction control processing unit 102, and sets image parameters corresponding to the received display instruction.

[0073] Here, the image parameters used are control data of image parameters stored in advance in image parameter storage unit 302. The following two methods are conceivable for calculating the control data of the image parameters. Here, contrast (intensity of blur) is taken as an example of the image parameter to be controlled.

[0074] (1) First calculation method FIG. 13 is a diagram used to explain the first calculation method. First, a formed aerial image MI is displayed in a mirror image space MS. At this time, a chart CIM divided into two parts, black and white, is used as the display image. In this state, the area of ​​the formed aerial image MI including the chart CIM is photographed with a camera and the photographed image CMI is saved.

[0075] Next, a direct-view aerial image RI is displayed in the mirror image space MS. At this time, a chart CRI divided into two parts, black and white, is used as the displayed image, and blur processing is performed on the image to indicate the blur strength β. Then, the area of ​​the direct-view aerial image including the chart CRI is captured with a camera, and the captured image CRI is saved. Note that blur processing refers to image processing using, for example, a Gaussian filter. β is the standard deviation.

[0076] Next, SSIM (Structural Similarity), which indicates an image quality evaluation index, is calculated for each of the captured images CMI and CRI. This SSIM calculation process is performed each time the blur strength β is changed by a fixed amount, and β that minimizes the difference in SSIM between the captured images CMI and CRI is stored in the image parameter storage unit 302 as a parameter indicating the blur strength. Note that, instead of SSIM, MSE (Mean Squared Error) or PSNR (Peak Signal to Noise Ratio) may also be used as the image quality evaluation index.

[0077] (2) Second calculation method FIG. 14 is a diagram used to explain the second calculation method. First, a formed aerial image MI is displayed in a mirror image space MS. As with the first calculation method, a chart CIM divided into two parts, black and white, is used as the displayed image. In this state, an area of ​​the formed aerial image MI including the chart CIM is photographed with a camera, and the photographed image is Fourier transformed to obtain MTF (Modulation Transfer Function) is obtained and saved.

[0078] Next, a direct-view aerial image RI is displayed in the mirror image space MS. In this case, a chart CIM divided into two parts, black and white, is used as the display image. Then, the area of ​​the direct-view aerial image including the chart CRI is photographed with a camera, and the photographed image is Fourier transformed to obtain MTF Obtain and save.

[0079] Next, the MTF obtained for the focused aerial image MI and the MTF obtained for the direct-view aerial image RI are normalized and compared to calculate the attenuation rate β of the spatial frequency component, and the calculated β is stored in the image parameter memory unit 302 as a parameter indicating the strength of blur.

[0080] When actually displaying an aerial image, if the aerial image to be displayed is an "imaged aerial image," then in step S13, image parameter control processing unit 103 outputs the display information read from display information storage unit 303 directly from input / output I / F unit 400 to display device DS. That is, the display image is displayed as is without adjusting the image parameters.

[0081] In contrast, if the aerial image to be displayed is a "direct-view aerial image," then in step S15, the image parameter control processing unit 103 adjusts the parameters of the display information read out from the display information storage unit 303 based on the control data for the image parameters, i.e., the control data for adjusting the blur strength β.

[0082] For example, a display image is blurred using a Gaussian filter. Alternatively, the display image is Fourier transformed to a frequency space image, and then this frequency space image is subjected to processing to attenuate the spatial frequency components. Then, the attenuated frequency space image is subjected to an inverse Fourier transform to return it to a two-dimensional display image.

[0083] Then, in step S13, the image parameter control processing unit 103 outputs the display image after adjusting the image parameters from the input / output I / F unit 400 to the display device DS, causing it to be displayed. That is, the image is displayed with reduced contrast by forcibly adding a blur β.

[0084] (effect) Therefore, according to the second embodiment, when a direct-view aerial image is presented, the display device DS displays an image with blur β added to it, i.e., with contrast previously reduced. As a result, the contrast of the direct-view aerial image RI can be made equivalent to that of the imaged aerial image MI, thereby homogenizing the image quality between the imaged aerial image MI and the direct-view aerial image RI. Therefore, the boundary between the mirror image space MS and the real space RS can be seamlessly crossed to reduce the sense of discomfort felt by the user due to the change in image quality of the aerial image when switching the display of the aerial image between the imaged aerial image MI and the direct-view aerial image RI, thereby enabling the display of an aerial image with high reproducibility.

[0085] (Variation) In the above explanation, the contrast (intensity of blur) is controlled as an image parameter. However, brightness or color may also be controlled as an image parameter. In this case, by adjusting the RGB of the displayed image, the brightness and color of the direct-view aerial image RI can be made equivalent to those of the focused aerial image MI, thereby making it possible to homogenize the image quality between the two aerial images.

[0086] [Third embodiment] (overview) Generally, the viewing area, i.e., the viewing angle, of the display screen of the display device DS is limited. Therefore, even when an aerial image is displayed by the display device DS, the viewing area of ​​the user US for the aerial image is naturally limited. As a result, as shown in FIG. 15 , depending on the viewing position of the user US, problems may arise, such as only being able to see the aerial image with reduced brightness, or the user US's viewing position being outside the viewing area of ​​the aerial image and therefore being unable to see the aerial image at all. This problem becomes particularly pronounced when the viewing area is limited by, for example, attaching a viewing-area-limiting film to the display screen of the display device DS to reduce stray light.

[0087] In order to solve the above problem, the third embodiment of the present invention detects the position of the user US in the real space RS and variably controls the display position and display direction of the display device DS according to the detected position of the user US, thereby enabling the user US to always view the aerial image from the front. Although the following description will be given taking as an example a case where both the display position and the display direction of the display device DS are variably controlled, it is also possible to variably control at least the display direction alone.

[0088] (Configuration example) 16 is a diagram showing an example of the configuration of an aerial image display system according to the third embodiment of the present invention. In this figure, the same parts as those in FIG. 1 are designated by the same reference numerals and detailed descriptions thereof will be omitted.

[0089] The system according to the third embodiment includes a display control device CS2 for controlling the display position and display direction of a display device DS. To enable control of the display position and display direction, the display device DS is provided with a rotation mechanism 55 and a movement mechanism 56. Furthermore, a camera CM is disposed in the optical system. The camera CM is, for example, a depth camera, and captures an image of the area in the real space RS where the user US is present, and outputs the captured image to the display control device CS2.

[0090] FIG. 17 is a block diagram showing the functional configuration of the display control device CS2 together with the display device DS and camera CM.

[0091] As in the second embodiment, the display control device CS2 is formed of, for example, a personal computer and includes a control unit 110 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 210 and a data storage unit 310, and an input / output I / F unit 410 are connected to the control unit 110 via a bus (not shown).

[0092] The display device DS and the camera CM are connected to the input / output I / F unit 410. More specifically, the display device main body 51 of the display device DS, the movement mechanism unit 56, the rotation mechanism unit 55, and the camera CM are each connected via a wireless interface such as a signal cable or a wireless LAN (Local Area Network).

[0093] The program storage unit 210 is configured by combining, for example, a nonvolatile memory such as an SSD as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a ROM, and stores middleware such as an OS as well as application programs required for control processing according to the third embodiment. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0094] The data storage unit 310 is, for example, a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from at any time with a volatile memory such as RAM, and the storage area thereof is provided with a display information storage unit 311. The display information storage unit 311 stores content information for displaying aerial images to the user US. This content information is obtained, for example, by reading it from an external storage medium or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0095] Control unit 110 includes, as processing functions necessary for implementing the third embodiment of the present invention, a user position acquisition processing unit 111, an aerial image display position acquisition processing unit 112, a movement position calculation processing unit 113, a rotation angle calculation processing unit 114, and a display position / direction control processing unit 115. All of these processing units 111 to 115 are realized by causing a hardware processor of control unit 110 to execute application programs stored in program storage unit 210.

[0096] Note that some or all of the processing units 111 to 115 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0097] The user position acquisition processing unit 111 acquires a captured image output from the camera CM via the input / output I / F unit 410, and calculates position information of the user US in the real space RS based on the acquired captured image. Note that if the user's viewing position is fixed, the user position acquisition processing unit 111 may acquire and store the fixed viewing position as a parameter in advance. In this case, the camera CM can be omitted.

[0098] The aerial image display position acquisition processing unit 112 acquires information indicating the display position of the aerial image to be displayed from the content information stored in the display information storage unit 311 .

[0099] Movement position calculation processing unit 113 calculates the display position of display device DS based on the position information of user US and the display position information of the aerial image.

[0100] The rotation angle calculation processor 114 calculates the rotation angle θ of the display device DS based on the position information of the user US, the display position information of the aerial image, and information indicating whether the aerial image to be displayed is an imaged aerial image or a direct-view aerial image. An example of the calculation process of this rotation angle θ will be described in the operation example.

[0101] The display position / orientation control processing unit 115 drives the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS according to the calculated display position and rotation angle θ of the display device DS, respectively, to control the display position and display angle of the display device DS.

[0102] (Example of operation) Next, the operation of the aerial image display system configured as above will be described.

[0103] FIG. 18 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 110 of the display control device CS2.

[0104] (1) Obtaining user location When displaying an aerial image, control unit 110 of display control device CS2 first acquires user position information by user position acquisition processing unit 111 in step S20. For example, user position acquisition processing unit 111 acquires a captured image output from camera CM via input / output I / F unit 410 and recognizes an image of user US from the acquired captured image. Then, based on the position coordinates of recognized user US in the image, it calculates position information of user US in real space RS. Note that, as mentioned above, if the user's viewing position is fixed, user position acquisition processing unit 111 may acquire and store the fixed viewing position as a parameter in advance.

[0105] (2) Acquisition of aerial image display position Next, in step S21, the control unit 110 of the display control device CS2 specifies the display position of the aerial image using the aerial image display position acquisition processing unit 112. For example, when the content information stored in the display information storage unit 311 includes information indicating the display target position of the aerial image, the control unit 110 acquires this display target position as is as display position information of the aerial image.

[0106] (3) Calculation of the display position of the display device DS Once the position information of the user US and the display position information of the aerial image are obtained, in step S22, the control unit 110 of the display control device CS2 calculates the display position of the display device DS using the movement position calculation processing unit 113. This display position of the display device DS can be found as a position that is plane-symmetric with respect to the display position information of the aerial image.

[0107] (4) Calculation of the display angle of the display device DS Next, in step S23, the control unit 110 of the display control device CS2 causes the rotation angle calculation processing unit 114 to calculate the rotation angle θ for controlling the display direction of the display device DS as follows.

[0108] 19 is a flowchart showing an example of the processing procedure and processing content of the calculation processing of the rotation angle θ. That is, in step S231, the rotation angle calculation processing unit 114 first calculates the orientation of the user US with respect to the aerial image, that is, the angle α formed by the straight line connecting the user US and the aerial image and the perpendicular line dropped from the user US to the first beam splitter 2, from information representing the display position of the aerial image and the position of the user US.

[0109] The rotation angle calculation processing unit 114 then calculates the rotation angle θ. The calculation process for this rotation angle θ differs depending on whether the aerial image is to be formed on the right side or the left side of the user US.

[0110] For example, when an aerial image is to be formed on the right side of the user US, the rotation angle calculation processing unit 114 first calculates (90-α)° from the angle α and the known angle 90°, and then calculates (45+α)° from the (90-α)° and the known angle 45°, as shown in Fig. 20. Then, using the (90-α)° and the known angle 90°, 180°=90°+90-α°+θ By calculating θ = |α| Calculate.

[0111] On the other hand, when the aerial image is to be formed on the left side of the user US, the rotation angle calculation processing unit 114 calculates the rotation angle θ using the calculated angle α and known angles 90° and 45°, as shown in Figure 21, for example.

[0112] Furthermore, in step S232, rotation angle calculation processor 114 determines whether the aerial image to be displayed is a "formed aerial image" or a "direct-view aerial image." If it is a "formed aerial image," then in step S233, the final rotation angle θ is calculated with -90° as the reference. On the other hand, if it is a "direct-view aerial image," then in step S234, the final rotation angle θ is calculated with 180° as the reference.

[0113] (5) Control of the moving position and display direction of the display device DS Once the movement position and rotation angle θ are calculated as described above, the control unit 110 of the display control device CS2 generates control signals for changing the position and angle by the calculated movement position and rotation angle θ in step S24 under the control of the display position / orientation control processing unit 115. Then, the display position / orientation control processing unit 115 outputs the generated movement position control signal and rotation angle control signal from the input / output I / F unit 410 to the movement mechanism unit 56 and rotation mechanism unit 55 of the display device DS, respectively.

[0114] In this way, the display position and display direction of the display device DS are controlled, and the imaging position and display direction of the aerial image are set so as to face the user US directly.

[0115] (effect) As described above, according to the third embodiment, the display position and display direction of the display device DS are controlled according to the position of the user US, thereby setting the imaging position and orientation of the aerial image so that it always faces the user US directly. This allows the user US to always view the aerial image reliably and with high brightness, regardless of their position, without having to adjust their own position according to the imaging position of the aerial image. This effect is particularly effective when the viewing zone is limited by, for example, attaching a viewing zone limiting film to the display screen of the display device DS to reduce stray light, for example.

[0116] (Variation) In the above explanation, the optical system is provided with the first beam splitter 2, but the first beam splitter 2 does not necessarily have to be installed. In this case, the same effect can be achieved.

[0117] [Fourth embodiment] (overview) Typically, in an optical system using a retroreflective member, when a user US faces the retroreflective member 1A, a virtual image VI of the display device DS is displayed as stray light behind the formed aerial image MI due to specular reflection from the surface of the retroreflective member 1A, as shown in Fig. 22. This stray light appears brighter and clearer than the formed aerial image MI, particularly when the retroreflective component is weaker than the specular reflected component, and therefore reduces the visibility of the formed aerial image MI to the user US.

[0118] In order to solve the above problem, the fourth embodiment of the present invention detects the position of the user US, as shown in Figure 23, and tilts the angle of the reflective surface of the retroreflective member 1A by x / 2 according to the detected position of the user US, so that the retroreflective member 1A does not face the user US directly, thereby causing the virtual image VI of the display device DS to deviate from the viewing direction of the user US relative to the aerial image MI.

[0119] (Configuration example) FIG. 24 shows an example of the configuration of an aerial image display system according to the fourth embodiment of the present invention.

[0120] The system according to the fourth embodiment has a rotation mechanism 55 and a movement mechanism 56 in the display device DS, and also has a movement mechanism 11 and a rotation mechanism 12 (shown in Figure 25) in the retroreflective member 1A for varying its position and orientation.

[0121] Furthermore, a display control device CS3 is provided to control the display position and display direction of the display device DS and the arrangement position and reflection direction of the retroreflective member 1A.

[0122] A camera CM is also arranged in the optical system. The camera CM is, for example, a depth camera, and captures an area in the real space RS where the user US is present and outputs the captured image to the display control device CS3. Note that if the user's viewing position is fixed, the camera CM can be omitted.

[0123] FIG. 25 is a block diagram showing the functional configuration of the display control device CS3 together with the display device DS, the retroreflective member 1A, and the camera CM.

[0124] The display control device CS3 is, for example, a personal computer, and includes a control unit 120 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 220 and a data storage unit 320, and an input / output I / F unit 420 are connected to the control unit 120 via a bus (not shown).

[0125] The display device DS, the mechanism of the retroreflective member 1A, and the camera CM are connected to the input / output I / F unit 420. More specifically, via a wireless interface such as a signal cable or a wireless LAN (Local Area Network), the display device main body 51, the movement mechanism 56, and the rotation mechanism 55 of the display device DS are connected, as well as the mechanism of the retroreflective member 1A and the camera CM.

[0126] The program storage unit 220 is configured by combining, for example, a nonvolatile memory such as an SSD as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a ROM, and stores middleware such as an OS as well as application programs required for control according to the fourth embodiment. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0127] The data storage unit 320 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time and a volatile memory such as a RAM as a storage medium, and a display information storage unit 321 is provided in the storage area.

[0128] The display information storage unit 321 stores content information for displaying an aerial image to the user US. This content information is obtained, for example, by reading it from an external storage medium or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0129] The control unit 120 has, as processing functions necessary for implementing the fourth embodiment, a user position acquisition processing unit 121, an aerial image display position acquisition processing unit 122, a display position and angle calculation processing unit 123, a retroreflective member position and angle calculation processing unit 124, a display position and direction control processing unit 125, and a retroreflective member position and direction control processing unit 126. These processing units 121 to 126 are all realized by causing a hardware processor of the control unit 120 to execute application programs stored in the program storage unit 210.

[0130] Note that some or all of the processing units 121 to 126 may be realized using hardware such as an LSI or an ASIC.

[0131] The user position acquisition processing unit 121 acquires a captured image output from the camera CM via the input / output I / F unit 420, and calculates position information of the user US in the real space RS based on the acquired captured image. Note that if the user's viewing position is fixed, the user position acquisition processing unit 121 may acquire and store the viewing position as a parameter. In this case, the camera CM can be omitted.

[0132] The aerial image display position acquisition processing unit 122 acquires information indicating the display position of the aerial image to be displayed from the content information stored in the display information storage unit 321.

[0133] The display position / angle calculation processor 123 calculates the display position of the display device DS based on the position information of the user US and the display position information of the aerial image. The display position / angle calculation processor 123 also calculates the rotation angle θ of the display device DS based on the position information of the user US, the display position information of the aerial image, and information indicating whether the aerial image to be displayed is a focused aerial image or a direct-view aerial image.

[0134] The retroreflective member position / angle calculation processing unit 124 calculates the position of the retroreflective member 1A based on the position information of the user US obtained by the user position acquisition processing unit 121, and also taking into consideration the display position of the display device DS and the known position of the first beam splitter 2, so as to minimize the optical path length between the retroreflective member 1A and the user US.

[0135] In addition, the retroreflective member position / angle calculation processing unit 124 calculates the rotation angle x / 2 of the reflective surface of the retroreflective member 1A to face the user US directly, based on the position information of the user US obtained by the user position acquisition processing unit 121, the display position of the display device DS, and the viewing zone (viewing angle) of the display device DS defined by the standard.

[0136] An example of the process for calculating the position of the retroreflective member 1A and the process for calculating the rotation angle will be described in the operation example.

[0137] The display position / orientation control processing unit 125 drives the movement mechanism unit 56 and the rotation mechanism unit 55 of the display device DS according to the calculated display position and rotation angle θ of the display device DS, respectively, to control the display position and display angle of the display device DS.

[0138] The retroreflective member position / direction control processing unit 126 drives the movement mechanism unit 11 and rotation mechanism unit 12 of the retroreflective member 1A in accordance with the position and rotation angle x / 2 of the retroreflective member 1A calculated by the retroreflective member position / angle calculation processing unit 124, thereby controlling the position and reflection direction of the retroreflective member 1A.

[0139] (Example of operation) Next, the operation of the aerial image display system configured as above will be described.

[0140] FIG. 26 is a flowchart showing an example of the processing procedure and processing content of the display control processing executed by the control unit 120 of the display control device CS3.

[0141] (1) Obtaining user location When displaying an aerial image, control unit 120 of display control device CS3 first acquires user position information by user position acquisition processing unit 121 in step S30. For example, user position acquisition processing unit 121 acquires a captured image output from camera CM via input / output I / F unit 420 and recognizes an image of user US from the acquired captured image. Then, based on the position coordinates of recognized user US in the image, it calculates position information of user US in real space RS. Note that, as mentioned above, if the user's viewing position is fixed, user position acquisition processing unit 121 may acquire and store the fixed viewing position as a parameter in advance.

[0142] (2) Acquisition of aerial image display position Next, in step S31, the control unit 120 of the display control device CS3 specifies the display position of the aerial image using the aerial image display position acquisition processing unit 122. For example, when the content information stored in the display information storage unit 321 includes information indicating the display target position of the aerial image, the control unit 120 acquires this display target position as is as the display position information of the aerial image.

[0143] (3) Calculation of the display position and display angle of the display device DS (3-1) Calculation of display position Once the position information of the user US and the display position information of the aerial image are obtained, in step S32, the control unit 120 of the display control device CS3 calculates the display position of the display device DS using the display position / angle calculation processing unit 123. This display position of the display device DS can be found as a position that is plane-symmetric with respect to the display position information of the aerial image.

[0144] (3-2) Calculation of rotation angle θ to set the display direction Next, in step S33, the display position / angle calculation processing unit 123 calculates the rotation angle θ for setting the display direction of the display device DS as follows: Note that the calculation process for the rotation angle θ is the same as the process described in Fig. 19, and therefore a description thereof will be omitted here.

[0145] (4) Calculate the position of the retroreflective member 1A and the angle x / 2 of the reflective surface. Next, under the control of the retroreflective member position / angle calculation processor 124, the control unit 120 of the display control device CS3 calculates the position of the retroreflective member 1A and the angle x / 2 of the reflective surface in steps S34 and S35, respectively.

[0146] (4-1) Calculation of the movement position of the retroreflective member 1A When presenting a formed aerial image MI to a user US, in order to present a high-quality formed aerial image MI, it is desirable to minimize the optical path length from the light of the displayed image output from the display device DS to the user US, until it is perceived by the user US.

[0147] Therefore, the retroreflective member position / angle calculation processor 124 calculates the movement position of the retroreflective member 1A so as to satisfy the following conditions: Fig. 27 is a diagram for explaining the calculation process of the movement position.

[0148] The retroreflective member position / angle calculation processor 124 first defines the retroreflective member 1A as moving on a straight line connecting the user US and the imaging position of the aerial image. It also sets safety areas E1 and E2 for the retroreflective member 1A and the display device DS to prevent collisions.

[0149] For example, the safety area E1 of the retroreflective member 1A is defined as a circle that includes the maximum width of the reflective surface of the retroreflective member 1A. In contrast, the safety area E2 of the display device DS is defined as a circle that includes the maximum width of the display device main body 51 or the base 52 and legs 53 that support it.

[0150] In this state, the retroreflective member position / angle calculation processor 124 calculates the optimal position of the retroreflective member 1A. For example, the retroreflective member position / angle calculation processor 124 first determines the position where the safety area E1 of the retroreflective member 1A maintains a safe distance from the safety area E2 of the display device DS and the second beam splitter 3. Then, the retroreflective member position / angle calculation processor 124 determines the position where the y-axis coordinate value of the circle representing the safety area E1 of the retroreflective member 1A is minimum, and calculates the position where the retroreflective member 1A is closest to the user US within a range smaller than the position where the y-axis coordinate value of the circle representing the safety area E2 of the display device DS is maximum. The calculated position is then set as the position to which the retroreflective member 1A should be moved.

[0151] (4-2) Calculation of the reflection angle of the retroreflective member 1A The retroreflective member position / angle calculation processing unit 124 then calculates the angle α calculated by the display position / angle calculation processing unit 12, i.e., the angle α formed by the direction in which the user US faces the first beam splitter 2 and the line connecting the user US and the aerial image MI, and also calculates a rotation angle x / 2 for further rotating the retroreflective member 1A.

[0152] 28 and 29 are diagrams used in the calculation process of the rotation angle x / 2. Here, the following values ​​are known parameters. a: Distance from the display position of the display device DS to the retroreflective member 1A b: Distance from the display device DS to the viewpoint of the user US t: Viewing area of ​​the display device DS.

[0153] The distance b from the display device DS to the viewpoint of the user US is calculated based on the position information of the user US obtained by the user position acquisition processing unit 121. Furthermore, the viewing zone t of the display device DS is expressed by the rating of a viewing zone limiting film, for example, if a viewing zone limiting film is attached to the display screen.

[0154] The retroreflective member position / angle calculation processing unit 124 first calculates the distance A from the viewpoint position of the user US to the retroreflective member 1A and the distance B as follows: Note that B is a symbol defined only in the formula as B=tan(t) for convenience.

[0155]

number

[0156] Next, the retroreflective member position / angle calculation processing unit 124 calculates the angle x using the calculated A and B as follows:

[0157]

number

[0158] In this way, the rotation angle α+x / 2 of the reflective surface of the retroreflective member 1A is calculated.

[0159] (5) Control of the moving position and display direction of the display device DS In step S36, the control unit 120 of the display control device CS3 generates control signals for adjusting the position and angle of the display device DS by the calculated movement position and rotation angle θ of the display device DS under the control of the display position / orientation control processing unit 115. Then, the display position / orientation control processing unit 125 outputs the generated movement position control signal and rotation angle control signal from the input / output I / F unit 420 to the movement mechanism unit 56 and rotation mechanism unit 55 of the display device DS, respectively. In this way, the display position and display direction of the display device DS are controlled.

[0160] (6) Control of the position and reflection direction of the retroreflective member 1A In step S37, the control unit 120 of the display control device CS3 generates control signals for adjusting the position and angle of the retroreflective member 1A by the calculated movement position and rotation angle α+x / 2 of the retroreflective member 1A under the control of the retroreflective member position / orientation control processing unit 126. The retroreflective member position / orientation control processing unit 126 then outputs the generated position control signal and rotation angle control signal from the input / output I / F unit 420 to the movement mechanism 11 and rotation mechanism 12 of the retroreflective member 1A, respectively. In this way, the position and reflection direction of the retroreflective member 1A are controlled.

[0161] (effect) As described above, according to the fourth embodiment, the display position and display angle of the display device DS are controlled according to the position of the user US and the display position of the aerial image, and the position and reflection direction of the retroreflective member 1A are controlled according to the display position of the display device DS and the position of the user US, thereby setting the retroreflective member 1A so that it does not face the user US directly.

[0162] As a result, it is possible to prevent the virtual image VI of the display device DS from entering the field of view of the user US viewing the aerial image MI, thereby improving the visibility of the aerial image MI for the user US.

[0163] (Variation) In the above description, the position and angle of the reflective surface of the retroreflective member 1A are controlled, but it is also possible to control only the angle of the reflective surface.Furthermore, although the position and display direction of the display device DS are also controlled, it is not necessary to control the position and display direction of the display device DS.

[0164] [Fifth embodiment] (overview) Displaying a background image behind an aerial image to enhance the sense of reality of the aerial image is extremely effective in practice. For example, displaying aerial images of characters or objects in conjunction with images representing their shadows makes it possible to create a more realistic display.

[0165] However, even if you try to display a background image, there may not be enough space to install a background display device in the real space where the first beam splitter on the user's side is located. Also, if you move the display device for displaying the aerial image to the mirror image space movement area, as shown in Figure 4, for example, to display a aerial image in the mirror image space, the moving device will block the background image. As a result, parts of the background image other than those corresponding to the display device will be perceived as black, hindering the display of the aerial image.

[0166] Therefore, in the fifth embodiment of the present invention, a second display device for background display is placed in the space on the opposite side of the second beam splitter from the placement position of the first display device for displaying the aerial image, and the background image displayed on this second display device is reflected by the second beam splitter and displayed in the direction of the user US.

[0167] (First Example) 30 is a diagram showing a first example of an optical system of an aerial image display system according to a fifth embodiment of the present invention. In this figure, the same parts as those in FIG. 1 are given the same reference numerals and detailed descriptions thereof will be omitted.

[0168] 30, a display device BD for background display is placed in the space on the opposite side of the second beam splitter 3 from the position of the display device DS for displaying the aerial image. The background image displayed on this display device BD for background display is reflected by the second beam splitter 3, then passes through the first beam splitter 2 and is presented to a user US present in the real space RS.

[0169] With this configuration, the display device BD for displaying the background can be installed even when there is no installation space available in the real space RS where the user US exists. Furthermore, as shown in Fig. 30, even when the display device DS for displaying the aerial image is moved into the mirror image space movement area ME to form an aerial image MI in the mirror image space MS, the display device DS does not interfere with the virtual image BI of the background image.

[0170] As a result, the user US is presented with a background-attached aerial image in which only the imaged aerial image MI is superimposed on the background video BI. An example of this display image is shown in Figure 31. This solves the problem of the placement space for the display device BD used to display the background, and also makes it possible to present the user US with a clear background-attached aerial image without interfering with the display of the aerial image MI displayed by the display device DS.

[0171] In order to resolve occlusion conflicts between the background image BI and the aerial image MI, the display of the background range that overlaps with the aerial image MI may be erased based on the position of the user US.

[0172] (Second Example) 32 is a diagram showing a second example of the optical system of the aerial image display system according to the fifth embodiment of the present invention, in which the same parts as those in FIG. 30 are denoted by the same reference numerals.

[0173] In the second embodiment, an optical element whose transmission and reflection change depending on the polarization direction, such as a reflective polarizer, is used as the second beam splitter 3. A phase difference film 4 is disposed on the reflective surface side of the retroreflective member 1A. The first beam splitter 2 is a half mirror whose reflection characteristics do not change depending on the polarization direction, for example, a transparent plate with metal deposition.

[0174] With this configuration, by matching the polarization characteristics of the second beam splitter 3 with the polarization direction of the display image output from the display device DS, it is possible to suppress attenuation of the brightness level of the aerial image, thereby making it possible to present a bright and clear aerial image with a background.

[0175] (Third Example) 33 is a diagram showing a third example of the optical system of the aerial image display system according to the fifth embodiment of the present invention. In this figure, the same parts as in FIG. 30 are denoted by the same reference numerals.

[0176] In the third embodiment, a display device VS for displaying a virtual image is used instead of the display device BD for displaying a background. Like the display device DS for displaying an aerial image, the display device VS for displaying a virtual image is equipped with a movement mechanism and a rotation mechanism for varying the display position and display direction, thereby enabling it to move within the mirror image space MS on the opposite side of the second beam splitter 3 from the position where the display device DS is placed.

[0177] With this configuration, for example, if the display device DS for aerial image display displays a front image of a character or an object, and the display device VS for virtual image display displays a rear image of the character or object in synchronization with this, then an aerial image equivalent to a virtual image of the formed aerial image corresponding to the front image of the character or object, and an aerial image corresponding to the rear image of the character or object, are simultaneously presented to the user US in real space RS.

[0178] Therefore, it is possible to display an aerial image displayed in real space using a virtual image representation, as if it were reflected in a mirror.

[0179] [Other embodiments] In addition, various modifications can be made without departing from the spirit and scope of the present invention with respect to the retroreflective members constituting the optical system, the materials and functions of the first and second beam splitters, their relative positions, the type and configuration of the display device, the type of content to be displayed on the aerial image, etc. Furthermore, various modifications can be made without departing from the spirit and scope of the present invention with respect to the functional configuration of the display control devices CS1, CS2, CS3 and the procedures and processing contents of their control processes.

[0180] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0181] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0182] US...Viewers (users) DS: Display device for mid-air images RS…Real space MS…mirror space RE: Real-space movement area ME…Mirror image space movement area RI…Direct aerial image MI...Aerial image formation BD: Display device for background display 1A, 1B...Retroreflective material 2...First beam splitter 3...Second beam splitter 4...Retardation film 11,56...Movement mechanism section 12,55...Rotation mechanism 51...Display device body 52...Foundation 53…legs 54...Caster 100, 110, 120...Control unit 101...Display instruction acquisition processing unit 102...Display position / direction control processing unit 103...Video parameter control processing unit 111, 121...User position acquisition processing unit 112, 122...Aerial image display position acquisition processing unit 113...Movement position calculation processing unit 114...rotation angle calculation processing unit 115...Display position / direction control processing unit 123...Display position / angle calculation processing unit 124...Retroreflective member position / angle calculation processing unit 125...Display position / direction control processing unit 126...Retroreflective member position / direction control processing unit 200, 210, 220...Program memory section 300, 310, 320...Data storage section 301...Display position / direction control data storage unit 302...Video parameter storage unit 303...Display information storage unit 311,321...display information storage section 400, 410, 420... Input / output interface section 500...input device

Claims

1. An aerial image display system comprising: a display device that displays a display image; and an optical system that includes a retroreflective member and an optical member having optical properties of reflecting part of incident light and transmitting part of the incident light, and that displays an aerial image corresponding to the display image toward a viewer, a location information acquisition unit that acquires location information of the viewer; a second position information acquisition unit that acquires position information of the display device; a display control unit that rotates the angle of the reflective surface of the retroreflective member by a specified angle with respect to the direction of the viewer based on the acquired position information of the viewer, and controls the position of the retroreflective member relative to the viewer and the angle of the reflective surface based on the acquired position information of the viewer and position information of the display device; An aerial image display system comprising:

2. A display control device used in an aerial image display system that displays an aerial image corresponding to a display image displayed on a display device to a viewer by passing the display image through an optical system that combines a retroreflective member and an optical member having an optical property of reflecting a portion of incident light and transmitting a portion of the light, a first location information acquisition unit that acquires location information of the viewer; a second position information acquisition unit that acquires position information of the display device; a display control unit that rotates the angle of the reflective surface of the retroreflective member by a specified angle with respect to the direction of the viewer based on the acquired position information of the viewer, and controls the position of the retroreflective member relative to the viewer and the angle of the reflective surface based on the acquired position information of the viewer and position information of the display device; A display control device comprising:

3. 3. The display control device according to claim 2, wherein the display control unit sets a position of the retroreflective member at a position that maintains a non-contact state with the display device and that minimizes a distance between the viewer and the retroreflective member on a straight line connecting the viewer and a display position of the aerial image.

4. The display control device according to claim 2, wherein the display control unit calculates the angle of the reflective surface of the retroreflective member based on the distance from the position of the display device to the retroreflective member, the distance from the position of the display device to the position of the viewer, and the viewing area of ​​the display device.

5. A display control method executed by a control device used in an aerial image display system, in which a display image displayed on a display device is passed through an optical system that combines a retroreflective member and an optical member having an optical property of reflecting a portion of incident light and transmitting a portion of the light, thereby displaying an aerial image corresponding to the display image to a viewer, comprising: acquiring location information of the viewer; acquiring position information of the display device; a step of rotating the angle of the reflective surface of the retroreflective member by a specified angle with respect to the direction of the viewer based on the acquired position information of the viewer, and controlling the position of the retroreflective member relative to the viewer and the angle of the reflective surface based on the acquired position information of the viewer and position information of the display device; A display control method comprising:

6. 5. A program causing a processor included in the display control device to execute processes performed by the position information acquisition unit and the display control unit included in the display control device according to claim 2.

Images