Aerial image display system and its display control device

Abstract

In an aspect of the present invention, a retroreflection member is disposed such that a reflection surface is orthogonal to or nearly orthogonal to the viewing direction of a viewer. In addition, a first optical member is disposed between the viewer and the retroreflection member such that an action surface is parallel to or nearly parallel to the reflection surface of the retroreflection member, to thereby form a mirror image space between the first optical member and the retroreflection member and form a real space in a space where the viewer is present. Further, in the mirror image space, a second optical member is disposed in such a state that an action surface is inclined to the viewing direction at a prescribed angle, to thereby form a mirror image space movement region between the second optical member and the retroreflection member. Furthermore, the display position of a movable device is set to be movable between the mirror image space movement region and a real space movement region, and the display direction is set to be changeable between a direction orthogonal to or a direction nearly orthogonal to the viewing direction and a direction parallel to or a direction nearly parallel to the viewing direction.

Description

Technical Field

[0001] One aspect of this invention relates to, for example, an aerial image display system for spatially displaying context information and the like, and a display control device used in this system.

Background Art

[0002] As a method for presenting context information and the like to a user, for example, an information presentation method using a mirror that a person normally uses has been proposed. For example, in Non-Patent Document 1, a special mirror having a function of transmitting part of incident light and reflecting part of it, for example, a magic mirror, is used, and context information is transmitted through the mirror from a display arranged on the back side of this mirror and displayed in the front direction, thereby presenting both the virtual image of the user himself and the real image of the above context information to the user.

[0003] Also, in Non-Patent Document 2, in the field of MR (Mixed Reality) for displaying digital information in the real world, for example, by simultaneously displaying the aerial image on the front surface and the aerial image on the back surface in the real space in front of the mirror and the mirror image space on the back side, a method for presenting highly realistic information is described.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, both Non-Patent Document 1 and Non-Patent Document 2 present information in a fixed location, either in real space or mirror space. In contrast, digital information such as contextual information is inherently independent of the presentation space. Therefore, there is a need for the development of information presentation methods with a higher degree of freedom that are not bound by physical phenomena in the real world.

[0006] This invention was made in view of the above circumstances, and aims to provide a technology that makes it possible to present an aerial image of digital information in either real space or mirror space, thereby improving the degree of freedom in information presentation. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the aerial image display system according to the present invention comprises a retroreflective member, first and second optical members that transmit a portion of incident light and reflect a portion of it, and a display device that displays display information. The retroreflective member is positioned such that its reflective surface is perpendicular or nearly perpendicular to the viewing direction from the viewer. The first optical member is positioned between the viewer and the retroreflective member such that its working surface is parallel or nearly parallel to the reflective surface of the retroreflective member, forming a mirror image space between itself and the retroreflective member, and forming a real space in the space where the viewer is located. The second optical member is positioned in the mirror image space such that its working surface has a predetermined inclination angle with respect to the viewing direction, forming a mirror image space movement region between itself and the retroreflective member. The display device comprises a moving mechanism and a rotating mechanism. The moving mechanism allows the display position of the display information to move between the mirror image space movement region and a real space movement region formed in the space opposite the second optical member via the mirror image space movement region. The rotating mechanism allows the display direction of the display information to be varied between a direction perpendicular or nearly perpendicular to the viewing direction and a direction parallel or nearly parallel to the viewing direction.

[0008] According to one aspect of this invention, if the display position of the display device is set to a real-space movement region and the display direction of the display device is set to a direction orthogonal to, for example, the first optical member, i.e., the direction of the second optical member, the display information displayed on the display device is reflected by the second optical member and then retroreflected by the retroreflecting member, and is projected as an aerial image of a real image in the real space where the viewer is located, passing through the second optical member and the first optical member. Furthermore, if the display position of the display device is moved to a mirror-image space movement region while the display direction of the display device is still oriented towards the second optical member, the display information displayed on the display device is reflected by the second optical member and then retroreflected by the retroreflecting member, and is projected as an aerial image in the mirror-image space. Moreover, if the display direction of the display device is oriented towards the viewer after moving the display position of the display device to a mirror-image space movement region, it becomes possible to present an aerial image to the viewer as they directly view the display screen of the display device through the first optical member.

[0009] In other words, by arbitrarily changing the display position and orientation of the display device, it becomes possible to seamlessly cross the boundary between real space and mirrored space and project an aerial image of the displayed information to any desired position. Furthermore, not only can the projection position of the aerial image of the displayed information be changed, but the aerial image of the displayed information can also be presented in a way that allows the viewer to directly see it. [Effects of the Invention]

[0010] According to one aspect of this invention, it is possible to provide a technology that allows an aerial image of digital information to be presented in either real space or mirrored space, thereby improving the degree of freedom in information presentation. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows a first embodiment of the optical system in an aerial image display system according to the first embodiment of the present invention. [Figure 2] Figure 2 is a perspective view showing an example of the configuration of a display device provided in the aerial image display system shown in Figure 1. [Figure 3]Figure 3 shows an example of the operation when an aerial image is projected onto real space in the aerial image display system shown in Figure 1. [Figure 4] Figure 4 shows an example of the operation when an aerial image is projected into the mirror image space in the aerial image display system shown in Figure 1. [Figure 5] Figure 5 shows an example of operation when a direct view of an aerial image is presented in the aerial image display system shown in Figure 1. [Figure 6] Figure 6 shows a second embodiment of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 7] Figure 7 shows a third embodiment of the optical system in the aerial image display system according to the first embodiment of this invention. [Figure 8] Figure 8 is a cross-sectional view showing an example of the structure of a beam splitter used in the aerial image display system shown in Figure 7. [Figure 9] Figure 9 shows a fourth embodiment of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 10] Figure 10 shows a fifth embodiment of the optical system in the aerial image display system according to the first embodiment of the present invention. [Figure 11] Figure 11 is a block diagram showing an example of the functional configuration of a display control device provided in an aerial image display system according to a second embodiment of the present invention. [Figure 12] Figure 12 is a flowchart showing an example of the processing procedure and processing content of the display control process executed by the control unit of the display control device shown in Figure 11. [Figure 13] Figure 13 is a diagram illustrating an example of a method for calculating video parameters in the display control process shown in Figure 12. [Figure 14] Figure 14 illustrates another example of the method for calculating video parameters in the display control process shown in Figure 12. [Figure 15] Figure 15 is a diagram illustrating an overview of an aerial image display system according to a third embodiment of the present invention. [Figure 16]FIG. 16 is a diagram showing an example of the configuration of an optical system of an aerial image display system according to a third embodiment of the present invention. [Figure 17] FIG. 17 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to the third embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart showing an example of the processing procedure and processing content of display control processing executed by the control unit of the display control device shown in FIG. 17. [Figure 19] FIG. 19 is a flowchart showing an example of the processing procedure and processing content of rotation angle calculation processing in the display control processing shown in FIG. 18. [Figure 20] FIG. 20 is a diagram for explaining a first example of the rotation angle calculation processing shown in FIG. 19. [Figure 21] FIG. 21 is a diagram for explaining a second example of the rotation angle calculation processing shown in FIG. 19. [Figure 22] FIG. 22 is a diagram for explaining the outline of an aerial image display system according to a fourth embodiment of the present invention. [Figure 23] FIG. 23 is a diagram for explaining the outline of an aerial image display system according to a fourth embodiment of the present invention. [Figure 24] FIG. 24 is a diagram showing an example of the configuration of an optical system of an aerial image display system according to a fourth embodiment of the present invention. [Figure 25] FIG. 25 is a block diagram showing an example of the functional configuration of a display control device provided in the aerial image display system according to a fourth embodiment of the present invention. [Figure 26] FIG. 26 is a flowchart showing an example of the processing procedure and processing content of display control processing executed by the control unit of the display control device shown in FIG. 25. [Figure 27] FIG. 27 is a diagram for explaining an example of processing for calculating the position of a retroreflective member with respect to a display device in the display control processing shown in FIG. 26. [Figure 28] FIG. 28 is a diagram for explaining an example of processing for calculating the angle of a retroreflective member in the display control processing shown in FIG. 27. [Figure 29]Figure 29 is a diagram illustrating an example of the process for calculating the angle of the retroreflective member, which is part of the display control process shown in Figure 27. [Figure 30] Figure 30 shows a first embodiment of the optical system in an aerial image display system according to a fifth embodiment of the present invention. [Figure 31] Figure 31 shows an example of an aerial image and background image displayed by the aerial image display system shown in Figure 30. [Figure 32] Figure 32 shows a second embodiment of the optical system in an aerial image display system according to the fifth embodiment of the present invention. [Figure 33] Figure 33 shows a third embodiment of the optical system in an aerial image display system according to the fifth embodiment of the present invention. [Modes for carrying out the invention]

[0012] Embodiments of this invention will be described below with reference to the drawings.

[0013] [First Embodiment] (First example) (Example configuration) Figure 1 shows a first embodiment of the aerial image display system according to the first embodiment of the present invention.

[0014] (1)Optical system In Figure 1, 1A is a retroreflective member, and this retroreflective member 1A is positioned so that its reflective surface is perpendicular to the viewing direction of the viewer (hereinafter also referred to as the user) US. A first beam splitter 2 is positioned between the user US and the retroreflective member 1A so that its operating surface is perpendicular to the viewing direction of the user US, that is, parallel to the reflective surface of the retroreflective member 1A. Furthermore, a second beam splitter 3 is positioned in the space between the retroreflective member 1A and the first beam splitter 2. The second beam splitter 3 is positioned so that its operating surface is oblique to the viewing direction of the user US at a predetermined angle, for example, 45°.

[0015] The first and second beam splitters 2 and 3 both have optical properties that transmit a portion of the incident light and reflect a portion of it. As a result, a real space RS is formed on the user US side of the first beam splitter 2, and a mirror image space MS is formed between the first beam splitter 2 and the retroreflective member 1A.

[0016] Furthermore, a mirror image space movement region ME is formed in the triangular region between the retroreflective member 1A and the second beam splitter 3 in the mirror image space MS. In addition, a real space movement region RE is formed in the region adjacent to the mirror image space movement region ME, on the extensions of each surface of the retroreflective member 1A and the first beam splitter 2. The boundary between the mirror image space movement region ME and the real space movement region RE is a virtual mirror surface VM.

[0017] (2)Display device DS The display device DS is positioned to be movable within the mirror image space movement region ME and the real space movement region RE.

[0018] Figure 2 is a perspective view showing an example of the configuration of a display device DS. The display device DS consists of a display device body 51, equipped with, for example, a liquid crystal panel, an organic EL panel, or an LED panel, mounted on a base 52. The lower surface of the display device body 51 is provided with legs 53 equipped with casters 54 to support the display device body 51 so that it can move. A rotating mechanism 55 is also installed on the upper surface of the base 52. The rotating mechanism 55 is used to variably set the display direction of the display device body 51 within a predetermined angular range, for example, a range of 180 degrees.

[0019] (Example of operation) With the above configuration, the aerial image display system operates as follows.

[0020] (1) When projecting an aerial image onto real space RS Figure 3 is a diagram illustrating the operation in this case. The display device DS is positioned in the real-space movement area RE, and its display direction is set to be, for example, perpendicular to the user US's viewing direction. The position and display direction of the display device DS may be set manually by the administrator or viewer, or automatically by the display control device shown in the figure.

[0021] Once the display position and orientation of the display device DS are set in this manner, the display information shown on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, and sequentially passes through the second beam splitter 3 and the first beam splitter 2 to be formed as an aerial image MI1 in the real space RS where the user US is located.

[0022] Therefore, the user US can perceive, for example, figures or photographs as aerial images in the real space RS in which they exist.

[0023] (2) When an aerial image is formed in the mirror space MS Figure 4 is a diagram illustrating the operation in this case. The display device DS is moved from the real-space movement region RE to the mirror-image space movement region ME. The display direction is set to be perpendicular to the user US's viewing direction. Once the display position and display direction of the display device DS are set in this way, the display information displayed on the display screen of the display device DS is reflected by the second beam splitter 3, then retroreflected by the retroreflective member 1A, passes through the second beam splitter 3, and is formed as an aerial image MI2 in the mirror-image space MS.

[0024] Therefore, the user US can use the first beam splitter 2 as a mirror to perform tasks such as brushing their teeth, applying makeup, or grooming themselves, while simultaneously checking news information such as traffic information or weather forecasts through the aerial image MI2 projected in the mirror image space MS.

[0025] (3) When presenting a direct view of the air to the user US Figure 5 is a diagram illustrating the operation in this case. The display device DS is moved from the real-space movement region RE to the mirror-image space movement region ME. The display direction is set to face the user US's viewing direction. Once the display position and display direction of the display device DS are set in this way, the display information shown on the display screen of the display device DS is sequentially transmitted through the second beam splitter 3 and the first beam splitter 2 and presented to the user US as a direct-view aerial image RI in real space RS.

[0026] Therefore, in real space RS, the user US can view the displayed information as if they were directly looking at the display screen of the display device DS.

[0027] (Second example) Figure 6 shows a second embodiment of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as in Figure 1 are denoted by the same reference numerals, and detailed descriptions are omitted.

[0028] In the second embodiment, the retroreflective member 1B is positioned in the mirror image space MS opposite to the virtual mirror surface VM with respect to the second beam splitter 3, and the reflective surface is oriented parallel to the user US's viewing direction.

[0029] With this configuration, if the display device DS is placed, for example, in the real-space movement area RE, and the display direction is set to be perpendicular to the user US's viewing direction, the display information displayed on the display device DS passes through the second beam splitter 3, is retroreflective by the retroreflective member 1B, is reflected by the second beam splitter 3, passes through the first beam splitter 2, and is formed as an aerial image MI3 in the real space RS where the user US is located.

[0030] Furthermore, when the display device DS is placed in the mirror image space movement region ME and the display direction is set to be perpendicular to the user US's viewing direction, the display information displayed on the display screen of the display device DS is transmitted through the second beam splitter 3 and then retroreflected by the retroreflective member 1B, forming an aerial image MI2 in the mirror image space MS.

[0031] Furthermore, when the display device DS is positioned in the mirror image space movement region ME and its display direction is set to face the user US's viewing direction, the display information shown on the display device DS is sequentially transmitted through the second beam splitter 3 and the first beam splitter 2 and presented to the user US as a direct-view aerial image RI in real space RS.

[0032] Therefore, the second embodiment also yields effects equivalent to those of the first embodiment.

[0033] (Third example) Figure 7 shows a third embodiment of the aerial image display system according to the first embodiment of the present invention. In this figure, the same parts as in Figures 1 and 6 are denoted by the same reference numerals, and detailed descriptions are omitted.

[0034] In the third embodiment, retroreflective members 1A and 1B are arranged on two mutually orthogonal sides of the mirror image space MS, with their respective reflective surfaces orthogonal to each other. Furthermore, as the second beam splitter 3, for example as shown in Figure 8, a reflective member 3a, such as a half-mirror with equal transmittance and reflectance, is sandwiched between two transparent members 3b and 3c, such as acrylic plates, with equal thickness and refractive index.

[0035] With this configuration, if the display device DS is placed in, for example, the real-space movement area RE, and the display direction is set to be perpendicular to the user US's viewing direction, the display information displayed on the display device DS is reflected by the second beam splitter 3, then retroreflective by the retroreflective member 1A, and passes sequentially through the second beam splitter 3 and the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is located. At the same time, the above display information passes through the second beam splitter 3, is retroreflective by the retroreflective member 1B, is reflected by the second beam splitter 3, and then passes through the first beam splitter 2 to be formed as an aerial image in the real space RS where the user US is located. The same applies when forming an aerial image in the mirror image space MS.

[0036] In other words, a composite aerial image MI4 is formed in real space RS and mirror image space MS, which is a combination of the aerial image retroreflective by retroreflective member 1A and the aerial image retroreflective by retroreflective member 1B.

[0037] Therefore, according to the third embodiment, it is possible to present a brighter imaged aerial image MI4 compared to when retroreflective members 1A and 1B are used individually. Furthermore, since the second beam splitter 3 has a structure in which both sides of the reflective member 3a are sandwiched between two transparent members 3b and 3c having the same thickness and refractive index, as illustrated in Figure 8, the optical path lengths corresponding to the first and second beam splitters 2 and 3 can be set to be equal, thereby preventing the formation of a double image of the imaged aerial image MI4.

[0038] (Fourth embodiment) Figures 9 and 10 show a fourth embodiment of the aerial image display system according to the first embodiment of the present invention.

[0039] In the fourth embodiment, a phase difference film 4 is placed on the reflective surface of the retroreflective member 1A. The phase difference film 4 is made of, for example, a 1 / 4 phase difference film and has optical properties that rotate the polarization direction of transmitted light by 45 degrees. Furthermore, as the first and second beam splitters 2 and 3, optical elements that switch between reflection and transmission depending on the polarization direction of the incident light are used. As these optical elements, for example, a reflective polarizer or a wire grid is used. In the following description, the operation when the elements are set up to reflect S-polarized light and transmit P-polarized light will be described as an example.

[0040] In contrast, the display screen of the display device DS is fitted with a depolarizing film, a diffuser, or a phase difference film so that the output light of the displayed information includes unpolarized light, S-polarized light, and P-polarized light. Alternatively, a polarizing plate or phase difference film is rotatably positioned facing the display screen of the display device DS, and the polarization direction of the output light is switched by rotation.

[0041] Given this configuration, when first projecting an aerial image onto the real space RS, the display device DS is placed in the real space movement region RE with its display direction orienting perpendicular to the viewing direction. Additionally, if a polarizing plate is present on the display screen of the display device DS, this polarizing plate is rotated so that the polarization component reflected from the display device DS to the beam splitter 3, for example, S-polarization, is output.

[0042] With this configuration, for example, as shown in Figure 9, the S-polarized light from the display device DS is reflected by the second beam splitter 3 and converted to circularly polarized light by passing through the phase difference film 4 placed in front of the retroreflective member 1A. The circularly polarized light is retroreflective by the retroreflective member 1A, reversing its direction of rotation, and is converted to P-polarized light by passing through the phase difference film 4 again. This retroreflective P-polarized light passes through the second beam splitter 3 and then through the first beam splitter 2 to form an aerial image MI5 in real space RS.

[0043] Next, when projecting an aerial image into the mirror image space MS, the display device DS is placed in the mirror image space movement region ME with its display direction orienting perpendicular to the viewing direction. At the same time, the polarizing plate placed on the display screen of the display device DS is rotated so that S-polarized light is output from the display device DS.

[0044] With this configuration, for example as shown in Figure 10, the S-polarized light from the display light output from the display device DS1 is reflected by the second beam splitter 3 and, as before, retroreflective as P-polarized light by the phase difference film 4 and retroreflective member 1A placed in front of the retroreflective member 1A. The retroreflective P-polarized light then passes through the second beam splitter 3 and is imaged as an aerial image MI6 in the mirror image space MS.

[0045] Furthermore, when presenting the display information of the display device DS as a direct-view aerial image RI, the display device DS is positioned with its display direction in the mirror image space movement region ME, and its display direction is oriented toward the user US. At the same time, the polarizing plate and phase difference plate placed on the display screen of the display device DS are rotated to set the display device DS to output P-polarized light.

[0046] With this configuration, the P-polarized light output from the display device DS is sequentially transmitted through the second beam splitter 3 and the first beam splitter 2, and presented to the user US in real space RS as a direct-view aerial image RI.

[0047] Therefore, according to the fourth embodiment, by selectively switching the display light output from the display device DS between S-polarized and P-polarized light, it becomes possible to suppress the attenuation of light by the beam splitter in the optical path and present a high-brightness aerial image.

[0048] (Fifth example) For example, as shown in Figure 10, the display devices DS1 and DS2 may be arranged in an L-shape, and each may be configured to output S-polarized and P-polarized light, respectively. With this configuration, it becomes possible to simultaneously and with high brightness present multiple aerial images in both the mirror space MS and the real space RS.

[0049] [Second Embodiment] (overview) In the system described in the first embodiment, the aerial image can be displayed seamlessly across the boundary between the mirror image space MS and the real space RS, thereby achieving extremely useful practical effects such as diversifying the display of the aerial image. However, the formed aerial image MI undergoes many reflections and transmissions by beam splitters 2,3 and retroreflective members 1A,1B along the optical path until the image is formed, resulting in a decrease in contrast and other characteristics, and uneven image quality between it and the directly viewed aerial image RI.

[0050] Therefore, in the second embodiment, when displaying the image-formed aerial image MI, the image parameters of the display image in the display device DS are adjusted in advance to homogenize the image quality between it and the directly viewed aerial image RI.

[0051] (Example configuration) Figure 11 is a block diagram showing the functional configuration of the display control device CS1, which is provided in the aerial image display system according to the second embodiment of this invention, together with the display device DS.

[0052] The display device DS is equipped with a movement mechanism 56 for moving its display position and a rotation mechanism 55 for changing the display direction. Both of these mechanisms 56 and 55 operate according to control signals output from the display control device CS1.

[0053] The display control device CS1 consists of, for example, a personal computer and includes a control unit 100 that uses a hardware processor such as a Central Processing Unit (CPU). A storage unit having a program storage unit 200 and a data storage unit 300, and an input / output interface (hereinafter referred to as I / F) unit 400 are connected to this control unit 100 via a bus (not shown).

[0054] Input devices 500, such as a keyboard and mouse, are connected to the input / output interface (I / F) section 400. Other devices such as a display device or external storage media like a USB (Universal Serial Bus) memory may also be connected to the I / F section 400. Furthermore, the I / F section 400 may be equipped with a communication interface function.

[0055] The program storage unit 200 is configured, for example, by combining a non-volatile memory that can be written to and read at any time, such as an SSD (Solid State Drive), and a non-volatile memory such as a ROM (Read Only Memory), and stores middleware such as an OS (Operating System), as well as application programs necessary for control according to the second embodiment. Hereafter, the OS and each application program will be collectively referred to as a program.

[0056] The data storage unit 300 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read at any time, and a volatile memory such as RAM (Random Access Memory), as a storage medium. The storage area is provided with a display position / direction control data storage unit 301, a video parameter storage unit 302, and a display information storage unit 303, which are the main storage units necessary for carrying out the second embodiment of this invention.

[0057] The display position / direction control data storage unit 301 stores control data necessary for controlling the display position and direction of the display device DS in response to input display instructions. The video parameter storage unit 302 stores control data necessary for controlling the video parameters of the display information according to the type of aerial image to be displayed. The display information storage unit 303 is used to store information to be displayed, such as content information.

[0058] The control unit 100 includes a display instruction acquisition processing unit 101, a display position / direction control processing unit 102, and a video parameter control processing unit 103 as processing functions necessary to carry out the second embodiment of this invention. All of these processing units 101 to 103 are realized by causing the hardware processor of the control unit 100 to execute an application program stored in the program storage unit 200.

[0059] Furthermore, some or all of the above-mentioned processing units 101 to 103 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0060] The display instruction acquisition processing unit 101 acquires the display instruction information when it is input to the input device 500. The display instruction information instructs whether to display the image-formed aerial image in real space RS, display the image-formed aerial image in mirror image space MS, or display the direct-view aerial image in real space RS.

[0061] The display position and direction control processing unit 102 reads control data from the display position and direction control data storage unit 301 according to the instructions in the display instruction information. Then, according to the read control data, it outputs position control signals and direction control signals from the input / output I / F unit 400 to the moving mechanism unit 56 and the rotating mechanism unit 55 of the display device DS, respectively, to set the display position and display direction of the display device DS.

[0062] The video parameter control processing unit 103 determines whether the aerial image to be presented is an "image-formed aerial image" or a "directly viewed aerial image" based on the setting data of the display position and display direction of the display device DS. Then, the video parameter control processing unit 103 controls the video parameters of the display image to be displayed on the display device DS based on the determination result and the video parameters stored in the video parameter storage unit 302. An example of this video parameter control process is described in the operation example.

[0063] (Example of operation) Next, the operation of the display control device CS1, configured as described above, will be explained.

[0064] Figure 12 is a flowchart showing an example of the processing procedure and processing content of the display control process executed by the control unit 100 of the display control device CS1.

[0065] The control unit 100 of the display control device CS1 monitors the input of display instruction information in step S10 under the control of the display instruction acquisition processing unit 101. In this state, when display instruction information is input to the input device 500, the control unit 100 of the display control device CS1 determines in step S11, under the control of the display position / direction control processing unit 102, whether the display instruction information is a display instruction for "image-formed aerial image" or a display instruction for "direct-view aerial image".

[0066] If the display instruction is "image in the air" as a result of the above determination, the display position / direction control processing unit 102 proceeds to step S12, where it generates a display position control signal based on the display position / direction control data stored in the display position / direction control data storage unit 301. The display position / direction control processing unit 102 then outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the real space movement area RE or the mirror image space movement area ME.

[0067] Furthermore, the display position and direction control processing unit 102 generates a display direction control signal based on the above display position and direction control data. The generated display direction control signal is then output to the rotation mechanism unit 55 of the display device DS, thereby setting the display direction of the display device DS to a direction perpendicular to the viewing direction.

[0068] In response to this, if the result of the above determination is that the display instruction is "direct view of the air", the display position / direction control processing unit 102 proceeds to step S14, where it generates a display position control signal based on the position / direction control data stored in the display position / direction control data storage unit 301. The display position / direction control processing unit 102 then outputs the generated display position control signal to the movement mechanism unit 56 of the display device DS, thereby moving the display position of the display device DS to the mirror image space movement region ME.

[0069] In addition, the display position / direction control processing unit 102 generates a display direction control signal based on the above position / direction control data. The generated display direction control signal is then output to the rotation mechanism unit 55 of the display device DS, thereby setting the display direction of the display device DS to the direction of the user US.

[0070] In the above explanation, the display instruction information is used to specify whether to display an "image-formed aerial image" or a "direct-viewed aerial image." However, if the content included in the display information contains information that specifies the type of aerial image to be displayed, that is, whether it is an "image-formed aerial image" or a "direct-viewed aerial image," or information that can determine this, then this information may be used instead.

[0071] Once the setting control of the display position and display direction of the display device DS is completed, the control unit 100 of the display control device CS1 then performs the following process to control the video parameters of the display information under the control of the video parameter control processing unit 103.

[0072] In other words, the video parameter control processing unit 103 receives the display instruction (whether it is an "image-formed aerial image" or a "directly viewed aerial image") from the display position / direction control processing unit 102 and sets the video parameters corresponding to the received display instruction.

[0073] Here, the video parameters used are control data for video parameters that have been previously stored in the video parameter storage unit 302. Two methods are possible for calculating the control data for the video parameters. Here, contrast (blur strength) is used as an example of the video parameter to be controlled.

[0074] (1) First calculation method Figure 13 is a diagram used to explain the first calculation method. First, the imaged aerial image MI is displayed in the mirror image space MS. At this time, a chart CIM divided into white and black is used as the display image. In this state, the region of the imaged aerial image MI that includes the chart CIM is photographed with a camera and the captured image CMI is saved.

[0075] Next, a direct-view aerial image RI is displayed in the mirror space MS. For this display, a chart CRI, divided into black and white sections with blur processing (β representing the degree of blurring) applied to the image, is used. Then, the region containing the chart CRI of the direct-view aerial image is captured by a camera, and the captured image CRI is saved. Blurring refers to image processing, for example, using a Gaussian filter. β is the standard deviation.

[0076] Next, for each of the above-mentioned captured image CMI and CRI, SSIM (Structural Similarity), which represents an image quality evaluation index, is calculated. This SSIM calculation process is performed each time the blur strength β is changed by a fixed amount, and the β that minimizes the difference in SSIM between each of the above-mentioned captured image CMI and CRI is stored in the video parameter storage unit 302 as a parameter indicating the blur strength. In addition to SSIM, MSE (Mean Squared Error) and PSNR (Peak Signal to Noise Ratio) may also be used as image quality evaluation indices.

[0077] (2) Second calculation method Figure 14 is a diagram used to explain the second calculation method. First, the imaged aerial image MI is displayed in the mirror image space MS. As with the first calculation method, a chart CIM divided into white and black is used as the display image. In this state, the region of the imaged aerial image MI that includes the chart CIM is photographed with a camera, and the captured image is Fourier transformed. MTF Get and save the (Modulation Transfer Function).

[0078] Next, the direct-view aerial image RI is displayed in the mirror image space MS. In this case as well, the chart CIM, which is divided into white and black, is used as the display image. Then, the region containing the chart CRI of the direct-view aerial image is photographed with a camera, and the captured image is Fourier transformed. MTF Retrieve and save it.

[0079] Next, the MTF obtained in the image-forming aerial image MI and the MTF obtained in the direct-view aerial image RI are normalized and compared to calculate the spatial frequency component attenuation rate β, and the calculated β is stored in the video parameter storage unit 302 as a parameter indicating the strength of blur.

[0080] When actually displaying an aerial image, if the aerial image to be displayed is a "formed aerial image", the video parameter control processing unit 103 outputs the display information read from the display information storage unit 303 directly to the display device DS via the input / output I / F unit 400 in step S13 for display. In other words, the display image is displayed as is without adjusting the video parameters.

[0081] In contrast, if the aerial image to be displayed is a "directly viewed aerial image," the video parameter control processing unit 103 adjusts the parameters in step S15 based on the control data for the video parameters, that is, the control data for adjusting the blur strength β, with respect to the display information read from the display information storage unit 303.

[0082] For example, a Gaussian filter is used to blur the displayed image. Alternatively, the displayed image is converted to a frequency-space image using a Fourier transform, and then the spatial frequency components of this frequency-space image are attenuated. Finally, an inverse Fourier transform is performed on the frequency-space image after the attenuation process to return it to a two-dimensional displayed image.

[0083] Then, in step S13, the video parameter control processing unit 103 outputs the display image after adjusting the video parameters from the input / output I / F unit 400 to the display device DS for display. In other words, it displays a display image with reduced contrast due to a process that forcibly adds blur β.

[0084] (effect) Therefore, according to the second embodiment, when a direct-view aerial image is presented, the displayed image will have a blurred β added to it, that is, it will be displayed on a display device DS with a pre-set low contrast. As a result, it becomes possible to make the contrast of the direct-view aerial image RI equivalent to that of the image-formed aerial image MI, thereby homogenizing the image quality between the image-formed aerial image MI and the direct-view aerial image RI. Consequently, it is possible to seamlessly cross the boundary between the mirror image space MS and the real space RS, minimizing the user's discomfort with the change in image quality of the aerial image when switching between the image-formed aerial image MI and the direct-view aerial image RI, thereby enabling the display of aerial images with high reproducibility.

[0085] (modified version) The above explanation used the case of controlling contrast (blur intensity) as an image parameter as an example. However, brightness or color can also be controlled as an image parameter. In this case, by adjusting the RGB of the displayed image, the brightness and color of the direct-view aerial image RI can be made equivalent to those of the image-formed aerial image MI, thereby enabling homogenization of image quality between the two aerial images.

[0086] [Third Embodiment] (overview) Generally, there are limits to the viewing area, or field of view, of the display screen of a display device DS. Therefore, even when an aerial image is displayed by the above-mentioned display device DS, the user's (US) field of view of the aerial image is naturally limited. As a result, problems arise such as the user only being able to see aerial images with reduced brightness depending on their viewing position, or the user being unable to see the aerial image at all if their viewing position is outside the field of view of the aerial image. This problem is particularly pronounced when the field of view is restricted by attaching, for example, a field of view limiting film to the display screen of the display device DS in order to reduce the generation of stray light.

[0087] A third embodiment of this invention solves the above problem by detecting the position of the user US in real space RS and variably controlling the display position and display direction of the display device DS according to the detected position of the user US, thereby enabling the user US to always view the aerial image from the front. In the following explanation, we will use the case where both the display position and display direction of the display device DS are variably controlled as an example, but it is also acceptable to variably control at least the display direction.

[0088] (Example configuration) Figure 16 shows an example of the configuration of an aerial image display system according to a third embodiment of the present invention. In this figure, the same reference numerals are used for parts that are the same as those in Figure 1, and detailed explanations are omitted.

[0089] The system according to the third embodiment includes a display control device CS2 for controlling the display position and display direction of the display device DS. In addition, to enable this control of the display position and display direction, the display device DS is provided with a rotation mechanism 55 and a movement mechanism 56. Furthermore, a camera CM is arranged in the optical system. The camera CM is, for example, a depth camera, and captures the area in real space RS where the user US is located and outputs the captured image to the display control device CS2.

[0090] Figure 17 is a block diagram showing the functional configuration of the display control device CS2 together with the display device DS and camera CM.

[0091] The display control device CS2, like the second embodiment, consists of, for example, a personal computer and includes a control unit 110 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 210 and a data storage unit 310, and an input / output I / F unit 410 are connected to this control unit 110 via a bus (not shown).

[0092] The input / output I / F unit 410 is connected to the display device DS and the camera CM. More specifically, the display device body 51, the moving mechanism 56, the rotating mechanism 55 of the display device DS, and the camera CM are connected, for example, via a signal cable or a wireless interface such as a wireless LAN (Local Area Network).

[0093] The program storage unit 210 is configured, for example, by combining a non-volatile memory such as an SSD that can be written to and read at any time as a storage medium, and a non-volatile memory such as ROM, and stores application programs necessary for the control processing according to the third embodiment, in addition to middleware such as the OS. Hereafter, the OS and each application program will be collectively referred to as a program.

[0094] The data storage unit 310 is, for example, a combination of non-volatile memory such as an SSD that can be written to and read at any time, and volatile memory such as RAM, and a display information storage unit 311 is provided in its storage area. The display information storage unit 311 stores content information for displaying an aerial image to the user US. This content information is obtained, for example, by reading from an external storage medium, or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0095] The control unit 110 includes, as processing functions necessary to implement the third embodiment of this invention, a user position acquisition processing unit 111, an aerial image display position acquisition processing unit 112, a movement position calculation processing unit 113, a rotation angle calculation processing unit 114, and a display position / direction control processing unit 115. All of these processing units 111 to 115 are realized by causing the hardware processor of the control unit 110 to execute an application program stored in the program storage unit 210.

[0096] Furthermore, some or all of the above-mentioned processing units 111 to 115 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0097] The user position acquisition processing unit 111 acquires the captured image output from the camera CM via the input / output interface unit 410, and calculates the user US position information in real space RS based on the acquired captured image. If the user's viewing position is fixed, the user position acquisition processing unit 111 may acquire and save the fixed viewing position as a parameter in advance. In this case, the camera CM can be omitted.

[0098] The aerial image display position acquisition processing unit 112 acquires information representing the display position of the aerial image to be displayed from the content information stored in the display information storage unit 311.

[0099] The movement position calculation processing unit 113 calculates the display position of the display device DS based on the user US's position information and the aerial image's display position information.

[0100] The rotation angle calculation processing unit 114 calculates the rotation angle θ of the display device DS based on the user US's position information, the display position information of the aerial image, and information indicating whether the aerial image to be displayed is an image-formed aerial image or a direct-view aerial image. An example of this rotation angle θ calculation process is described in the operation example.

[0101] The display position and direction control processing unit 115 controls the display position and display angle of the display device DS by driving the moving mechanism 56 and the rotating mechanism 55 of the display device DS, respectively, according to the calculated display position and rotation angle θ of the display device DS.

[0102] (Example of operation) Next, we will explain the operation of the aerial image display system configured as described above.

[0103] Figure 18 is a flowchart showing an example of the processing procedure and processing content of the display control process executed by the control unit 110 of the display control device CS2.

[0104] (1) Obtaining the user's location When displaying an aerial image, the control unit 110 of the display control device CS2 first acquires user position information in step S20 using the user position acquisition processing unit 111. For example, the user position acquisition processing unit 111 acquires the captured image output from the camera CM via the input / output I / F unit 410 and recognizes the user US image from the acquired captured image. Then, based on the position coordinates within the recognized user US image, it calculates the user US position information in real space RS. As mentioned earlier, if the user's viewing position is fixed, the user position acquisition processing unit 111 may acquire and save the fixed viewing position as a parameter in advance.

[0105] (2) Acquisition of the aerial image display position In step S21, the control unit 110 of the display control device CS2 then uses the aerial image display position acquisition processing unit 112 to determine the display position of the aerial image. If, for example, the content information stored in the display information storage unit 311 includes information indicating the display target position of the aerial image, this display target position is acquired as the aerial image display position information.

[0106] (3) Calculation of the display position of the display device DS Once the user US's location information and the aerial image's display position information are obtained, the control unit 110 of the display control device CS2 then calculates the display position of the display device DS using the movement position calculation processing unit 113 in step S22. This display position of the display device DS can be determined as a position that is symmetrical with respect to the aerial image's display position information.

[0107] (4) Calculation of the display angle of the display device DS Furthermore, in step S23, the control unit 110 of the display control device CS2 calculates the rotation angle θ for controlling the display direction of the display device DS using the rotation angle calculation processing unit 114 as follows.

[0108] Figure 19 is a flowchart showing an example of the processing procedure and content of the calculation process for the rotation angle θ. Specifically, in step S231, the rotation angle calculation processing unit 114 first calculates the angle α between the direction of the user US relative to the aerial image, that is, the straight line connecting the user US and the aerial image, and the perpendicular line drawn from the user US to the first beam splitter 2, based on information representing the display position of the aerial image and the position of the user US.

[0109] The rotation angle calculation processing unit 114 then calculates the rotation angle θ. This rotation angle θ calculation process differs depending on whether the aerial image is projected to the right or left of the user US.

[0110] For example, when projecting an aerial image to the right of the user US, the rotation angle calculation processing unit 114 first calculates (90-α)° from the angle α and the known angle 90°, as shown in Figure 20, and then calculates (45+α)° from the above (90-α)° and the known angle 45°. Then, using the above (90-α)° and the known 90°, 180° = 90° + 90 - α° + θ By calculating, θ = |α| Calculate.

[0111] On the other hand, when projecting an aerial image to the left of the user US, the rotation angle calculation processing unit 114 calculates the rotation angle θ using the calculated angle α and known angles of 90° and 45°, for example, as shown in Figure 21.

[0112] Furthermore, in step S232, the rotation angle calculation processing unit 114 determines whether the aerial image to be displayed is an "image-formed aerial image" or a "direct-viewed aerial image". If it is an "image-formed aerial image", the final rotation angle θ is calculated in step S233 using -90° as the reference. On the other hand, if it is a "direct-viewed aerial image", the final rotation angle θ is calculated in step S234 using 180° as the reference.

[0113] (5) Control of the movement position and display direction of the display device DS Once the movement position and rotation angle θ are calculated as described above, the control unit 110 of the display control device CS2, under the control of the display position / direction control processing unit 115, generates control signals in step S24 to change the position and angle by the calculated movement position and rotation angle θ. The display position / direction control processing unit 115 then outputs the generated movement position control signal and rotation angle control signal from the input / output I / F unit 410 to the movement mechanism unit 56 and rotation mechanism unit 55 of the display device DS, respectively.

[0114] Thus, the display position and orientation of the display device DS are controlled, thereby setting the image formation position and display orientation of the aerial image to face directly towards the user US.

[0115] (effect) As described above, according to the third embodiment, the display position and display direction of the display device DS are controlled according to the position of the user US, thereby setting the image formation position and orientation of the aerial image to always face the user US directly. Therefore, the user US can always reliably and brightly view the aerial image regardless of their position, without having to adjust their own position in accordance with the image formation position of the aerial image. This effect is particularly effective, for example, when the viewing area is restricted by attaching, for example, a viewing area limiting film to the display screen of the display device DS in order to reduce the generation of stray light.

[0116] (modified version) The above explanation uses the example of a case where the optical system is equipped with a first beam splitter 2, but the first beam splitter 2 does not necessarily have to be installed. In this case as well, the same effect can be achieved.

[0117] [Fourth Embodiment] (overview) Typically, in optical systems using retroreflective members, as shown in Figure 22, for example, when a user US faces the retroreflective member 1A directly, the specular reflection of the surface of the retroreflective member 1A causes the virtual image VI of the display device DS to appear as stray light behind the formed aerial image MI. This stray light appears brighter and more clearly than the formed aerial image MI, especially when the retroreflective component is weaker than the specular reflection component, thus reducing the visibility of the formed aerial image MI to the user US.

[0118] A fourth embodiment of this invention solves the above problem by, for example as shown in Figure 23, detecting the position of the user US and tilting the angle of the reflective surface of the retroreflective member 1A by x / 2 according to the detected position of the user US, so that the retroreflective member 1A does not face the user US directly, thereby causing the virtual image VI of the display device DS to be out of the viewing direction relative to the airborne image MI of the user US.

[0119] (Example configuration) Figure 24 shows an example of the configuration of an aerial image display system according to the fourth embodiment of this invention.

[0120] In the fourth embodiment, the display device DS is provided with a rotating mechanism 55 and a moving mechanism 56, and the retroreflective member 1A is provided with a moving mechanism 11 and a rotating mechanism 12 (shown in Figure 25) for changing its position and orientation.

[0121] Furthermore, the display control device CS3 is provided to control the display position and display direction of the display device DS, and the arrangement position and reflection direction of the retroreflective member 1A.

[0122] Furthermore, a camera CM is positioned within the optical system. The camera CM, for example, is a depth camera that captures the area in real space RS where the user US is located and outputs the captured image to the display control device CS3. Note that the camera CM can be omitted if the user's viewing position is fixed.

[0123] Figure 25 is a block diagram showing the functional configuration of the display control device CS3 together with the display device DS, retroreflective member 1A, and camera CM.

[0124] The display control device CS3 consists of, for example, a personal computer and includes a control unit 120 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 220 and a data storage unit 320, and an input / output I / F unit 420 are connected to this control unit 120 via a bus (not shown).

[0125] The input / output I / F section 420 is connected to the display device DS, the mechanism of the retroreflective member 1A, and the camera CM. More specifically, the display device body 51, the moving mechanism 56, and the rotating mechanism 55 of the display device DS are connected via a signal cable or a wireless interface such as a wireless LAN (Local Area Network), as are the mechanism of the retroreflective member 1A and the camera CM.

[0126] The program storage unit 220 is configured, for example, by combining a non-volatile memory such as an SSD that can be written to and read at any time as a storage medium, and a non-volatile memory such as ROM, and stores middleware such as an OS, as well as application programs necessary for control according to the fourth embodiment. Hereafter, the OS and each application program will be collectively referred to as a program.

[0127] The data storage unit 320 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read at any time, and a volatile memory such as RAM, and a display information storage unit 321 is provided in its storage area.

[0128] The display information storage unit 321 stores content information for displaying an aerial image to the user US. This content information is obtained, for example, by reading from an external storage medium or by downloading it via a network from a server device on the Web or cloud, or from another information terminal.

[0129] The control unit 120 includes, as processing functions necessary to implement the fourth embodiment, a user position acquisition processing unit 121, an aerial image display position acquisition processing unit 122, a display position / angle calculation processing unit 123, a retroreflective member position / angle calculation processing unit 124, a display position / direction control processing unit 125, and a retroreflective member position / direction control processing unit 126. All of these processing units 121 to 126 are realized by causing the hardware processor of the control unit 120 to execute an application program stored in the program storage unit 210.

[0130] Furthermore, some or all of the above-mentioned processing units 121 to 126 may be implemented using hardware such as LSIs or ASICs.

[0131] The user position acquisition processing unit 121 acquires the captured image output from the camera CM via the input / output interface unit 420, and calculates the user US position information in real space RS based on the acquired captured image. If the user's viewing position is fixed, the user position acquisition processing unit 121 may acquire and save the viewing position as a parameter. In this case, the camera CM can be omitted.

[0132] The aerial image display position acquisition processing unit 122 acquires information representing the display position of the aerial image to be displayed from the content information stored in the display information storage unit 321.

[0133] The display position / angle calculation processing unit 123 calculates the display position of the display device DS based on the user US's position information and the aerial image's display position information. The display position / angle calculation processing unit 123 also calculates the rotation angle θ of the display device DS based on the user US's position information, the aerial image's display position information, and information indicating whether the aerial image to be displayed is an image-formed aerial image or a direct-view aerial image.

[0134] The retroreflective member position and angle calculation processing unit 124 calculates the position of the retroreflective member 1A such that the optical path length between it and the user US is minimized, based on the user US position information obtained by the user position acquisition processing unit 121, and further considering the display position of the display device DS and the known placement position of the first beam splitter 2.

[0135] Furthermore, the retroreflective member position and angle calculation processing unit 124 calculates the rotation angle x / 2 of the reflective surface of the retroreflective member 1A in order to orient the reflective surface of the retroreflective member 1A directly toward the user US, based on the user US position information obtained by the user position acquisition processing unit 121, the display position of the display device DS, and the viewing range (field of view) of the display device DS as defined by the standard.

[0136] An example of the calculation process for the position of the retroreflective member 1A and the calculation process for the rotation angle will be explained in the operation example.

[0137] The display position and direction control processing unit 125 controls the display position and display angle of the display device DS by driving the moving mechanism 56 and the rotating mechanism 55 of the display device DS, respectively, according to the calculated display position and rotation angle θ of the display device DS.

[0138] The retroreflective member position and direction control processing unit 126 controls the position and reflection direction of the retroreflective member 1A by driving the movement mechanism 11 and rotation mechanism 12 of the retroreflective member 1A according to the position and rotation angle x / 2 of the retroreflective member 1A calculated by the retroreflective member position and angle calculation processing unit 124.

[0139] (Example of operation) Next, we will explain the operation of the aerial image display system configured as described above.

[0140] Figure 26 is a flowchart showing an example of the processing procedure and processing content of the display control process executed by the control unit 120 of the display control device CS3.

[0141] (1) Obtaining the user's location When displaying an aerial image, the control unit 120 of the display control device CS3 first acquires user position information in step S30 using the user position acquisition processing unit 121. For example, the user position acquisition processing unit 121 acquires the captured image output from the camera CM via the input / output I / F unit 420 and recognizes the user US image from the acquired captured image. Then, based on the position coordinates within the recognized user US image, it calculates the user US position information in real space RS. As mentioned earlier, if the user's viewing position is fixed, the user position acquisition processing unit 121 may acquire and save the fixed viewing position as a parameter in advance.

[0142] (2) Acquisition of the aerial image display position In step S31, the control unit 120 of the display control device CS3 then determines the display position of the aerial image using the aerial image display position acquisition processing unit 122. If the content information stored in the display information storage unit 321 includes information indicating the display target position of the aerial image, this display target position is acquired as the aerial image display position information.

[0143] (3) Calculation of display position and display angle of the display device DS (3-1) Calculation of display position Once the user US location information and the aerial image display position information are obtained, the control unit 120 of the display control device CS3 then calculates the display position of the display device DS using the display position / angle calculation processing unit 123 in step S32. This display position of the display device DS can be determined as a position that is symmetrical with respect to the aerial image display position information.

[0144] (3-2) Calculation of rotation angle θ for setting the display direction Furthermore, in step S33, the display position / angle calculation processing unit 123 calculates the rotation angle θ for setting the display direction of the display device DS as follows. Note that the calculation process for the rotation angle θ is the same as the process described in Figure 19, so the explanation is omitted here.

[0145] (4) Calculate the position of the retroreflective member 1A and the angle x / 2 of the reflective surface. The control unit 120 of the display control device CS3 then calculates the position of the retroreflective member 1A and the angle x / 2 of the reflective surface in steps S34 and S35, under the control of the retroreflective member position and angle calculation processing unit 124.

[0146] (4-1) Calculation of the movement position of the retroreflective member 1A When presenting an image-formed aerial image (MI) to a user (US), it is desirable to minimize the optical path length from the display device (DS) to the user (US) in order to present a high-quality image-formed aerial image (MI).

[0147] Therefore, the retroreflective member position and angle calculation processing unit 124 calculates the movement position of the retroreflective member 1A so as to satisfy the following conditions. Figure 27 is a diagram illustrating the process of calculating this movement position.

[0148] The retroreflective member position and angle calculation processing unit 124 first defines that the retroreflective member 1A moves along a straight line connecting the user US and the image formation position of the aerial image. Furthermore, safety areas E1 and E2 are set for both the retroreflective member 1A and the display device DS to prevent collisions.

[0149] For example, the safety area E1 of the retroreflective member 1A is defined as a circle that includes the maximum width of the reflective surface of the retroreflective member 1A. In contrast, the safety area E2 of the display device DS is defined as a circle that includes the part of the display device body 51 or the base 52 and legs 53 that support it that has the maximum width.

[0150] In this state, the retroreflective member position and angle calculation processing unit 124 calculates the optimal position of the retroreflective member 1A. For example, the retroreflective member position and angle calculation processing unit 124 first finds the position where the safety area E1 of the retroreflective member 1A maintains a safe distance from the safety area E2 of the display device DS and the second beam splitter 3. Then, it finds the position where the y-coordinate value of the circle representing the safety area E1 of the retroreflective member 1A is minimized, and calculates the position where the retroreflective member 1A is closest to the user US within a range where this position is smaller than the position where the y-coordinate value of the circle representing the safety area E2 of the display device DS is maximized. Then, it sets the calculated position as the position to which the retroreflective member 1A should be moved.

[0151] (4-2) Calculation of the reflection angle of retroreflective member 1A The retroreflective member position / angle calculation processing unit 124 then calculates the angle α calculated by the display position / angle calculation processing unit 12, that is, the angle α formed by the direction in which the user US faces the first beam splitter 2 and the straight line connecting the user US and the aerial image MI, in addition to the rotation angle x / 2 to further rotate the retroreflective member 1A.

[0152] Figures 28 and 29 are diagrams used in the calculation process for the rotation angle x / 2 described above. Here, the following values ​​are known parameters. a: Distance from the display position of the display device DS to the retroreflective member 1A b: Distance from the display device DS to the user's viewpoint US t: Viewing range of the display device DS.

[0153] The distance b from the display device DS to the user US's viewpoint is calculated based on the user US's position information obtained by the user position acquisition processing unit 121. Furthermore, the viewing range t of the display device DS is expressed by the rating of the viewing range limiting film, for example, if a viewing range limiting film is attached to the display screen.

[0154] The retroreflective member position and angle calculation processing unit 124 first calculates the distance A and B from the user US viewpoint to the retroreflective member 1A as follows. Note that B is a symbol defined only in the formula for convenience, as B = tan(t).

[0155]

number

[0156] Next, the retroreflective member position and angle calculation processing unit 124 uses the calculated A and B to calculate the angle x as follows.

[0157]

number

[0158] Thus, the rotation angle α+x / 2 of the reflective surface of the retroreflective member 1A is calculated.

[0159] (5) Control of the movement position and display direction of the display device DS The control unit 120 of the display control device CS3, under the control of the display position / direction control processing unit 115, generates control signals in step S36 to adjust the position and angle of the display device DS by the calculated movement position and rotation angle θ of the display device DS. The display position / direction control processing unit 125 then outputs the generated movement position control signal and rotation angle control signal to the movement mechanism 56 and rotation mechanism 55 of the display device DS, respectively, from the input / output I / F unit 420. Thus, the display position and display direction of the display device DS are controlled.

[0160] (6) Control of the position and reflection direction of the retroreflective member 1A Furthermore, the control unit 120 of the display control device CS3, under the control of the retroreflective member position / direction control processing unit 126, generates control signals in step S37 to adjust the position and angle of the retroreflective member 1A by the calculated movement position and rotation angle α+x / 2 of the retroreflective member 1A. The retroreflective member position / direction control processing unit 126 then outputs the generated position control signal and rotation angle control signal to the movement mechanism unit 11 and rotation mechanism unit 12 of the retroreflective member 1A, respectively, from the input / output I / F unit 420. Thus, the position and reflection direction of the retroreflective member 1A are controlled.

[0161] (effect) As described above, according to the fourth embodiment, the display position and display angle of the display device DS are controlled according to the position of the user US and the display position of the aerial image, and the position and reflection direction of the retroreflective member 1A are controlled according to the display position of the display device DS and the position of the user US, thereby setting the retroreflective member 1A so as not to face the user US directly.

[0162] As a result, it becomes possible to prevent the virtual image VI of the display device DS from entering the field of view of the user US while viewing the aerial image MI, thereby improving the visibility of the aerial image MI by the user US.

[0163] (modified version) In the above explanation, the position and angle of the reflective surface of the retroreflective member 1A are controlled, but it is also possible to control only the angle of the reflective surface. Furthermore, although the position and display direction of the display device DS are also controlled, it is not necessary to control the position and display direction of the display device DS.

[0164] [Fifth Embodiment] (overview) Displaying a background image behind an aerial image to enhance its sense of realism is an extremely effective technique in practice. For example, displaying an aerial image of a character or object in conjunction with an image representing its shadow allows for a more realistic display.

[0165] However, even when attempting to display a background image, there may not be enough space to install a background display device in the real space where the user's first beam splitter is located. Furthermore, if the display device for displaying the aerial image is moved to the mirrored space movement area, for example as shown in Figure 4, in order to display an aerial image in the mirrored space, this moving device will obscure the background image. As a result, areas other than those corresponding to the display device in the background image will also be perceived as black, hindering the display of the aerial image.

[0166] Therefore, in the fifth embodiment of this invention, a second display device for displaying a background is placed in the space opposite to the placement position of the first display device for displaying an aerial image, with the second beam splitter in between, and the background image displayed on this second display device is reflected by the second beam splitter and displayed in the direction of the user US.

[0167] (First example) Figure 30 shows a first embodiment of the optical system of an aerial image display system according to the fifth embodiment of the present invention. In this figure, the same reference numerals are used for parts that are the same as those in Figure 1, and detailed descriptions are omitted.

[0168] In Figure 30, a background display device BD is positioned in the space opposite the second beam splitter 3 to the position of the display device DS for displaying the aerial image. The background image displayed on this background display device BD is reflected by the second beam splitter 3 and then passed through the first beam splitter 2 to be presented to the user US in the real space RS.

[0169] Because of this configuration, the background display device BD can be installed even when there is no installation space available in the real space RS where the user US is located. Furthermore, as shown in Figure 30, even when the aerial image display device DS is moved into the mirror space movement region ME in order to project the aerial image MI onto the mirror space MS, the display device DS does not interfere with the virtual image BI of the background image.

[0170] As a result, the user US will be presented with an aerial image with a background, in which only the projected aerial image MI is superimposed on the background image BI. Figure 31 shows an example of this display image. Therefore, it is possible to present a clear aerial image with a background to the user US without solving the problem of placement space for the background display device BD, and without interfering with the display of the aerial image MI shown by the display device DS.

[0171] Furthermore, to resolve occlusion inconsistencies between the background image (BI) and the aerial image (MI), the display of the background area overlapping with the aerial image (MI) may be removed based on the user's (US) position.

[0172] (Second example) Figure 32 shows a second embodiment of the optical system of an aerial image display system according to the fifth embodiment of the present invention. In this figure, the same parts as in Figure 30 are denoted by the same reference numerals.

[0173] In the second embodiment, the second beam splitter 3 is an optical element such as a reflective polarizer whose transmission and reflection change depending on the polarization direction. In addition, a phase difference film 4 is placed on the reflective surface side of the retroreflective member 1A. The first beam splitter 2 is a half-mirror, for example, a transparent plate with metal deposition, whose reflection characteristics do not change depending on the polarization direction.

[0174] With this configuration, by matching the polarization characteristics of the second beam splitter 3 with the polarization direction of the display image output from the display device DS, it becomes possible to suppress the attenuation of the brightness level of the aerial image, thereby enabling the presentation of a bright and clear aerial image with a background.

[0175] (Third example) Figure 33 shows a third embodiment of the optical system of an aerial image display system according to the fifth embodiment of the present invention. In this figure, the same parts as in Figure 30 are denoted by the same reference numerals.

[0176] In the third embodiment, a virtual image display device VS is used instead of the background display device BD. The virtual image display device VS, like the aerial image display device DS, is equipped with a moving mechanism and a rotating mechanism for varying the display position and display direction, thereby allowing it to move in the mirror image space MS on the opposite side of the second beam splitter 3 from the position of the display device DS.

[0177] Given this configuration, for example, suppose a display device DS for displaying aerial images displays a front view of a character or object, and a display device VS for displaying virtual images displays a back view of the same character or object in synchronization with it. In this case, the user US in real space RS is simultaneously presented with an aerial image equivalent to the virtual image of the image formed in the aerial image corresponding to the front view of the character or object, and an aerial image corresponding to the back view of the character or object.

[0178] Therefore, it becomes possible to display an aerial image shown in real space using virtual image representation, as if it were reflected in a mirror.

[0179] [Other embodiments] Furthermore, the retroreflective members constituting the optical system, the materials, functions, and arrangement of the first and second beam splitters, the type and configuration of the display device, and the type of content for displaying the aerial image can be modified in various ways without departing from the spirit of this invention. Similarly, the functional configurations of the display control devices CS1, CS2, and CS3, as well as the procedures and content of their control processing, can also be modified in various ways without departing from the spirit of this invention.

[0180] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0181] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of Symbols]

[0182] US…Viewers (Users) DS... Display device for displaying aerial images RS... Real space MS…mirror space RE... Real-space movement area ME…Mirror space movement area RI…Direct aerial image MI...Image Formation Aerial Image BD... a display device for background display. 1A, 1B... Retroreflective material 2…First beam splitter 3…Second beam splitter 4…Phase difference film 11,56...Movement mechanism section 12.55... Rotating mechanism 51...Display device main body 52…Base 53...legs 54... Caster 100, 110, 120… Control Unit 101...Display instruction acquisition processing unit 102...Display position / direction control processing unit 103...Video parameter control processing unit 111,121...User position acquisition processing unit 112,122... Aerial image display position acquisition processing unit 113...Movement position calculation processing unit 114... Rotation Angle Calculation Processing Unit 115...Display position / direction control processing unit 123...Display position / angle calculation processing unit 124... Retroreflective member position and angle calculation processing unit 125...Display position / direction control processing unit 126... Retroreflective member position and direction control processing unit 200, 210, 220… Program memory 300, 310, 320… Data storage units 301...Display position / direction control data storage unit 302...Video parameter storage unit 303...Display information storage unit 311,321...display information storage section 400, 410, 420… Input / Output I / F section 500... Input devices

Claims

1. A retroreflective member whose reflective surface is positioned perpendicular or nearly perpendicular to the viewing direction from the viewer, A first optical member having a first optical property of transmitting a portion of incident light and reflecting a portion thereof, wherein a first working surface is positioned between the viewer and the retroreflective member in a state parallel or nearly parallel to the reflective surface of the retroreflective member, forming a mirror image space between the first working surface and the retroreflective member, and forming a real space in the space where the viewer is located, A second optical member having a second optical property of transmitting a portion of incident light and reflecting a portion thereof, wherein a second working surface is arranged in the mirror image space at a predetermined inclination angle with respect to the viewing direction, and a mirror image space movement region is formed between the second working surface and the retroreflective member, Display device that displays information and It is equipped with, The aforementioned display device is A movement mechanism that allows the display position to be moved between the mirror image space movement region and the real space movement region formed in the space opposite the second optical member via the mirror image space movement region, A rotating mechanism that allows the display direction to be varied between a direction perpendicular to or nearly perpendicular to the viewing direction and a direction parallel to or nearly parallel to the viewing direction. An aerial image display system equipped with the following features.

2. A retroreflective member whose reflective surface is positioned parallel or nearly parallel to the viewing direction from the viewer, A first optical member having a first optical property of transmitting a portion of incident light and reflecting a portion thereof, wherein the reflective surface of the retroreflective member and the first working surface are arranged to be perpendicular or nearly perpendicular, forming a mirror image space in the rectangular region between the first working surface and the retroreflective member, and forming a real space in the space where the viewer is located, A second optical member having a second optical property of transmitting a portion of incident light and reflecting a portion of it, wherein a second working surface is arranged in the mirror image space at a predetermined inclination angle with respect to the viewing direction, and a mirror image space movement region is formed in the triangular space on the back side of the second working surface, Display device that displays information and It is equipped with, The aforementioned display device is A movement mechanism that allows the display position to be moved between the mirror image space movement region and the real space movement region formed in the space opposite the second optical member via the mirror image space movement region, A rotating mechanism that allows the display direction to be varied between a direction perpendicular to or nearly perpendicular to the viewing direction and a direction parallel to or nearly parallel to the viewing direction. An aerial image display system equipped with the following features.

3. The retroreflective member further comprises a phase difference member that gives a phase difference with respect to the polarization direction of the incident light, The first optical member and the second optical member are composed of optical elements having optical properties in which transmission and reflection change depending on the polarization direction of the incident light. The aerial image display system according to claim 1 or 2.

4. The system further comprises a display control device connected to the aforementioned display device, The aforementioned display control device is A video parameter control unit that variably controls video parameters, including at least one of brightness, color, and contrast of the display information, in accordance with the movement of the display position and the variable movement of the display direction of the display device. An aerial image display system according to claim 1 or 2, comprising:

5. The aforementioned video parameter control unit is When the display direction is perpendicular to the viewing direction, the video parameter is set to a first value; when the display position is set in the mirror image space movement region and the display direction is parallel to the viewing direction, the video parameter is set to a second value which is lower than the first value. The aerial image display system according to claim 4.

Images