naked-eye AR display device

Abstract

This compound-eye AR display device comprises a half mirror, a structure like a dark room, a display, an illumination, and a video controller. The half mirror is disposed upright on a floor. The structure like a dark room is provided behind the half mirror. The display is disposed in the structure like a dark room obliquely to a surface of the half mirror. The illumination illuminates a user positioned in front of the half mirror opposite to the display. The video controller receives a video signal to cause the display to display the video.

Description

Technical Field

[0001] The present invention relates to a naked-eye AR (Augmented Reality) display device.

Background Art

[0002] A naked-eye AR display device is a device that displays an augmented reality world without using tools such as glasses. As one of the technologies using a naked-eye AR display device, there is, for example, a technology of a mirror-type video conference that synthesizes a local user and a remote conversation partner and displays them on a display, such as a space sharing system that realizes telepresence. For example, Patent Document 1 discloses an example of a mirror-type video conference.

[0003] In a mirror-type video conference, a user and their conversation partner are synthesized and displayed on a display made to look like a mirror that the user views with the naked eye. In a mirror-type video conference, it is known that by reproducing the situation of interacting with a partner in front of one's eyes, the feeling of being next to the partner and being in the same space is enhanced.

[0004] In recent years, a naked-eye AR display technology has also been proposed that synthesizes and displays images of the real world (such as users) and various digital information in the virtual world (such as avatars).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the future, it is expected that a naked-eye AR display device will be required to achieve higher-quality AR displays (such as reproducing a sense of spatial expansion and depth to enhance the feeling of being in the same room).

[0007] This invention was made in response to such needs, and its objective is to provide a glasses-free AR display device that achieves higher quality AR display. [Means for solving the problem]

[0008] One aspect of the present invention is a naked-eye augmented reality (AR) display device. The naked-eye AR display device comprises a half-mirror, a darkroom-like structure, a display, lighting, and a video controller. The half-mirror is positioned upright on the floor. The darkroom-like structure is provided behind the half-mirror. The display is positioned diagonally within the darkroom-like structure relative to the surface of the half-mirror. The lighting illuminates the user positioned in front of the half-mirror on the opposite side of the display. The video controller receives a video signal and displays the image on the display. [Effects of the Invention]

[0009] According to the present invention, a glasses-free AR screen display device is provided that can give users a high level of 3D spatial awareness. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic perspective view showing the configuration of a glasses-free AR display device according to an embodiment. [Figure 2] Figure 2 is a top view of the naked-eye AR display device shown in Figure 1. [Figure 3] Figure 3 is a diagram illustrating the configuration of a spatial sharing system in an example of joining real spaces using the naked-eye AR display device shown in Figures 1 and 2. [Figure 4] Figure 4 is a diagram illustrating the configuration of a spatial sharing system that connects real space and virtual space using the naked-eye AR display device shown in Figures 1 and 2. [Figure 5] Figure 5 is a perspective view showing the use of the naked-eye AR display device shown in Figures 1 and 2. [Figure 6] Figure 6 shows how a double image can be seen in the image displayed on the screen. [Figure 7]FIG. 7 is a diagram showing a person position sensor and a rotation mechanism. [Figure 8] FIG. 8 is a diagram showing a state where a double image of the video displayed on the display has disappeared. [Figure 9] FIG. 9 is a diagram showing a state where the video displayed on the display floats. [Figure 10] FIG. 10 is a diagram showing a state where furniture is prepared as "connection" and a sense of unity is achieved by common animation operations. [Figure 11] FIG. 11 is a plan view showing a state where a desk is placed between the half mirror and the display. [Figure 12] FIG. 12 is a perspective view showing the naked-eye AR display device shown in FIG. 11 as viewed from the user's side. [Figure 13] FIG. 13 is a perspective view of a naked-eye AR display device having a sub-display for an emotive effect. [Figure 14] FIG. 14 is a plan view of the naked-eye AR display device shown in FIG. 13 as viewed from above. [Figure 15] FIG. 15 is a diagram showing a state where the side-by-side video of the user and the interaction partner becomes planar. [Figure 16] FIG. 16 is a diagram showing a naked-eye AR display device provided with an additional display. [Figure 17] FIG. 17 is a perspective view showing a state when a desk is placed in front of the half mirror. [Figure 18] FIG. 18 is a plan view showing a state where an object is placed symmetrically with respect to the half mirror and a display is placed behind the sofa. [Figure 19] FIG. 19 is a perspective view showing the naked-eye AR display device with an object and a display placed as shown in FIG. 18 as viewed from the user's side. [Figure 20] FIG. 20 is a perspective view schematically showing a state where a real person is sitting on the sofa. [Figure 21] FIG. 21 is a perspective view showing the naked-eye AR display device with a desk placed as shown in FIG. 11 as viewed from the user's side. [Figure 22]FIG. 22 is a diagram schematically showing a state where the video of the interlocutor becomes transparent when the background behind the user is bright. [Figure 23] FIG. 23 is a perspective view showing how a naked-eye AR display device with a screen placed at a position symmetric to the display with respect to the surface of the half mirror is seen from above the user. [Figure 24] FIG. 24 is a perspective view showing how a naked-eye AR display device with two dark high-back chairs placed on the user's side with respect to the half mirror is seen from above. [Figure 25] FIG. 25 is a plan view showing the positional relationship among the display, the high-back chair, and the camera with respect to the half mirror in the naked-eye AR display device shown in FIG. 24. [Figure 26] FIG. 26 is a perspective view showing a state where a wide shadow is cast on the rear video reflected in the half mirror. [Figure 27] FIG. 27 is a diagram showing an operator wearing an HMD (head-mounted display) and operating an avatar using a hand controller. [Figure 28] FIG. 28 is a perspective view schematically showing a part of the virtual naked-eye AR display device shown in FIG. 4. [Figure 29] FIG. 29 is a diagram showing a state where the video of the interlocutor becomes transparent when the video of the user and the video of the interlocutor overlap. [Figure 30] FIG. 30 is a perspective view showing how a naked-eye AR display device with a robot placed next to the user is seen from the user's side. [Figure 31] FIG. 31 is a block diagram showing a control system for controlling the operation of the robot. [Figure 32] FIG. 32 is a perspective view showing how a naked-eye AR display device with a torso-type cushion placed next to the user is seen from the user's side. [Figure 33] FIG. 33 is a diagram showing an example of creating a local and remote fusion space using the naked-eye AR display device according to the embodiment. [Figure 34]Figure 34 shows an example of creating a fused space between the real world and the cyber world using a naked-eye AR display device according to this embodiment. [Figure 35] Figure 35 shows an example of a system that uses a glasses-free AR display device according to this embodiment to connect the real world (set construction) and the virtual world (in-world) as a pair. [Figure 36] Figure 36 shows an example of creating a fused space between the real world and the metaverse using a naked-eye AR display device according to the embodiment. [Figure 37] Figure 37 shows an example of a system that uses a glasses-free AR display device according to the embodiment to connect the real world (set construction) and the virtual world (in-world) as a pair. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings.

[0012] The naked-eye AR display device according to the embodiment will be described with reference to Figures 1 and 2. Figure 1 is a schematic perspective view showing the configuration of the naked-eye AR display device 10 according to the embodiment. Figure 2 is a plan view of the naked-eye AR display device 10 shown in Figure 1, viewed from above.

[0013] This section describes an example of a naked-eye augmented reality (AR) display device applied to a mirror-type video conferencing system.

[0014] The naked-eye AR display device 10 includes a half-mirror 11, a darkroom-like structure 12, a display 13, lighting 14, a camera 15, and a video controller 16.

[0015] The half-mirror 11 is positioned upright on the floor.

[0016] The darkroom-like structure 12 is located behind the half-mirror 11.

[0017] The display 13 is positioned at an angle to the surface of the half-mirror 11, in a corner of the darkroom-like structure 12. The display 13 is large enough to display a person at life size. The display 13 is not limited to this, but may be, for example, a large LED display.

[0018] When user U uses the naked-eye AR display device 10, they are positioned in front of the half-mirror 11 and facing the display 13. That is, user U is positioned in front of the half-mirror 11 on the opposite side of the display 13, directing their gaze towards the display 13 and viewing the image displayed on the display 13 through the half-mirror 11. The illumination 14 illuminates user U. As a result, user U's image is projected onto the half-mirror 11.

[0019] Camera 15 is positioned symmetrically with respect to the surface of the half-mirror 11 with respect to the display 13. Camera 15 captures the user U reflected in the half-mirror 11. That is, camera 15 is positioned toward the intersection of the line connecting the display 13 and the user U and the surface of the half-mirror 11.

[0020] The video controller 16 has the function of receiving a video signal S1 and displaying the video on the display 13, and the function of acquiring the video captured by the camera 15 and transmitting the video signal S2.

[0021] Camera 15 captures the user U reflected on the surface of the half-mirror 11. As a result, the image of user U in the video captured by camera 15 is horizontally inverted.

[0022] The video controller 16 performs image processing to horizontally flip the image of user U. Ideally, the image of user U should be taken in a location with a black background. Areas where the background is not black are separated using a technique to extract existing people. If extraction is not necessary, the image can be sent as is. The video controller 16 transmits the processed image as video signal S2.

[0023] Furthermore, the video signal includes both a video signal and an audio signal. In other words, the video captured by camera 15 includes both video and audio. The display 13 also has the function of playing back both the video signal and the audio signal.

[0024] The image displayed on display 13 is either a person or an avatar. It is preferable that camera 15 be positioned at the eye level of the person or avatar.

[0025] (Examples of joining real spaces) Next, with reference to Figure 3, we will describe a spatial sharing system for an example of joining real spaces using the naked-eye AR display device 10 shown in Figures 1 and 2. Figure 3 is a configuration diagram of the spatial sharing system for an example of joining real spaces using the naked-eye AR display device shown in Figures 1 and 2.

[0026] This spatial sharing system has two naked-eye AR display devices 10. The configuration of each naked-eye AR display device 10 is as described with reference to Figures 1 and 2.

[0027] The two naked-eye AR display devices 10 are connected to each other via a transmission network 21, with their respective video controllers 16 connected. This allows the two naked-eye AR display devices 10 to send and receive video signals to and from each other. Alternatively, the two naked-eye AR display devices 10 could be connected to each other via a transmission line instead of the transmission network 21.

[0028] The half-mirror 11 of the left-side naked-eye AR display device 10 reflects user U1, who is using the left-side naked-eye AR display device 10. Conversely, the half-mirror 11 of the right-side naked-eye AR display device 10 reflects user U2, who is using the right-side naked-eye AR display device 10.

[0029] The display 13 of the right-side naked-eye AR display device 10 shows the image of user U1 reflected in the half-mirror 11, which is captured by the camera 15 of the left-side naked-eye AR display device 10. Conversely, the display 13 of the left-side naked-eye AR display device 10 shows the image of user U2 reflected in the half-mirror 11, which is captured by the camera 15 of the right-side naked-eye AR display device 10.

[0030] As a result, user U1, using the left-side naked-eye AR display device 10, can see their own image (mirror image) reflected in the half-mirror 11 of the left-side naked-eye AR display device 10, and the image of user U2 displayed on the display 13 of the left-side naked-eye AR display device 10. Similarly, user U2, using the right-side naked-eye AR display device 10, can see their own image (mirror image) reflected in the half-mirror 11 of the right-side naked-eye AR display device 10, and the image of user U1 displayed on the display 13 of the right-side naked-eye AR display device 10.

[0031] For example, user U1 sees the image of user U2 displayed on display 13 next to their own image (mirror image) reflected in the half-mirror 11, allowing them to converse as if user U2 were in the same room and sitting next to them. In other words, user U1 can experience the sensation of conversing with user U2 who is sitting next to them. The same applies to user U2.

[0032] The spatial sharing system further includes a camera 22 for distribution and a transmitter 23.

[0033] The camera 22 for streaming is positioned in front of the half-mirror 11 of the left-hand naked-eye AR display device 10. For example, the camera 22 for streaming is positioned directly in front of the center of the half-mirror 11 of the left-hand naked-eye AR display device 10. Unlike face-to-face devices, the camera 22 for streaming can capture user U1 reflected in the half-mirror 11 and user U2 displayed on the display 13 without being obstructed by user U1. In other words, the camera 22 for streaming can capture user U1 and user U2 from a third-person perspective. Alternatively, the camera 22 for streaming may be positioned symmetrically with respect to the half-mirror 11.

[0034] The transmitter 23 transmits the video signal captured by the distribution camera 22 to the transmission network 21. The video signal is received by receiving devices 24 (e.g., computers) owned by multiple users U3, and the video of user U1 and user U2 is played back. This allows each user U3 to view the video of user U1 and user U2 in real time.

[0035] The spatial sharing system further includes a distribution server 25 and a distribution storage 26. The distribution server 25 stores the live-streamed video in the distribution storage 26. The distribution server 25 also reads the video from the distribution storage 26 in response to a viewing request from user U3 and transmits the video signal to user U3's receiving device 24. This allows user U3 to view archived videos from users U1 and U2.

[0036] (Example of joining real space and virtual space) Next, with reference to Figure 4, we will describe a spatial sharing system for an example of joining real space and virtual space using the naked-eye AR display device 10 shown in Figures 1 and 2. Figure 4 is a configuration diagram of the spatial sharing system for an example of joining real space and virtual space using the naked-eye AR display device shown in Figures 1 and 2.

[0037] This spatial sharing system comprises one naked-eye AR display device 10 and one virtual naked-eye AR display device 30. The configuration of the naked-eye AR display device 10 is as described with reference to Figures 1 and 2. The virtual naked-eye AR display device 30 is a device created in a virtual space that is similar to the naked-eye AR display device 10. The virtual naked-eye AR display device 30 has substantially the same functions as the naked-eye AR display device 10. The participant in the virtual naked-eye AR display device 30, corresponding to user U1 of the naked-eye AR display device 10, is avatar A. Avatar A is operated by a user not shown.

[0038] The virtual naked-eye AR display device 30 includes a virtual mirror 31, a darkroom-like virtual structure 32, a virtual display 33, a virtual light 34, a virtual camera 35, and a video controller 36. The virtual mirror 31, the darkroom-like virtual structure 32, the virtual display 33, the virtual light 34, and the virtual camera 35 are located in a virtual space. Furthermore, the virtual mirror 31, the virtual display 33, the virtual light 34, and the virtual camera 35 are located within the darkroom-like virtual structure 32.

[0039] The virtual mirror 31 is positioned upright on the floor of the darkroom-like virtual structure 32. The virtual mirror 31 completely reflects light incident from one side and completely transmits light incident from the other side. The side of the virtual mirror 31 that completely transmits incident light is positioned facing the back of the darkroom-like virtual structure 32.

[0040] The virtual display 33 is positioned on the side of the virtual mirror 31 that completely transmits incoming light. Furthermore, the virtual display 33 is positioned at an angle to the surface of the virtual mirror 31, at the back of the darkroom-like virtual structure 32.

[0041] Avatar A is positioned on the side of the virtual mirror 31 that completely reflects the incident light when using the virtual naked-eye AR display device 30. Furthermore, Avatar A is positioned in front of the virtual mirror 31 and facing the virtual display 33. The virtual mirror 31 is positioned so as not to obstruct the field of view of Avatar A as it looks at the virtual display 33. As a result, Avatar A can see the images displayed on the virtual display 33 without any obstruction. The virtual illumination 34 illuminates Avatar A. As a result, the image of Avatar A is projected onto the virtual mirror 31.

[0042] The virtual camera 35 is positioned in front of the virtual display 33. That is, the virtual camera 35 is positioned on the side of the virtual mirror 31, which completely transmits incoming light. Furthermore, the virtual camera 35 is positioned facing the avatar A that uses the virtual naked-eye AR display device 30. The virtual camera 35 photographs the avatar A through the virtual mirror 31. Since the virtual mirror 31 is completely transparent from the side of the virtual camera 35, the virtual camera 35 photographs the avatar A without any obstruction.

[0043] The video controller 36 has the function of receiving a video signal and displaying the video on the virtual display 33, and the function of acquiring a video signal from the virtual camera 35 and transmitting it.

[0044] The virtual display 33 shows user U1 of the naked-eye AR display device 10. Since the virtual camera 35 is not a physical object, it does not obstruct the image on the virtual display 33 that avatar A sees. For this reason, the virtual camera 35 can preferably be positioned at the eye level of user U1 displayed on the display 13.

[0045] The video controller 16 of the naked-eye AR display device 10 and the video controller 36 of the virtual naked-eye AR display device 30 are connected via a transmission network 21. This allows the naked-eye AR display device 10 and the virtual naked-eye AR display device 30 to send and receive video signals to and from each other.

[0046] The half-mirror 11 of the naked-eye AR display device 10 reflects the user U1 who is using the naked-eye AR display device 10. The virtual mirror 31 of the virtual naked-eye AR display device 30 reflects avatar A, which is located in front of the virtual mirror 31.

[0047] The virtual display 33 of the virtual naked-eye AR display device 30 displays the image of user U1 reflected in the half-mirror 11, which is captured by the camera 15 of the naked-eye AR display device 10. The display 13 of the naked-eye AR display device 10 displays the image of avatar A, which is captured by the virtual camera 35 of the virtual naked-eye AR display device 30.

[0048] As a result, user U1 using the naked-eye AR display device 10 can see their own image (mirror image) reflected in the half-mirror 11 of the naked-eye AR display device 10, and the image of avatar A displayed on the display 13 of the naked-eye AR display device 10. On the other hand, avatar A can see its own image reflected in the virtual mirror 31 of the virtual naked-eye AR display device 30, and the image of user U1 displayed on the virtual display 33 of the virtual naked-eye AR display device 30.

[0049] For example, user U1 sees the image of avatar A displayed on display 13 next to their own image (mirror image) reflected in the half-mirror 11, giving them the feeling that avatar A is in the same room and next to them. The same applies to avatar A.

[0050] For example, Avatar A could be a Vtuber (Virtual YouTuber), and User U1 could be a fan of the Vtuber. In this case, User U1 can experience the feeling that the Vtuber, who does not have a physical body, is sitting next to them in the same room in the real world. In other words, User U1 can get the feeling of participating in a fan meeting in the real world with the Vtuber from the virtual world.

[0051] The spatial sharing system shown in Figure 4 also includes a camera 22 for distribution and a transmitter 23, similar to the system shown in Figure 3. The camera 22 for streaming is positioned in front of the half-mirror 11 of the naked-eye AR display device 10. For example, the camera 22 for streaming is positioned directly in front of the center of the half-mirror 11 of the naked-eye AR display device 10. The camera 22 for streaming can capture the user U1 reflected in the half-mirror 11 and the avatar A displayed on the virtual display 33. In other words, the camera 22 for streaming can capture the user U1 and the avatar A from a third-person perspective.

[0052] The transmitter 23 transmits the video signal captured by the distribution camera 22 to the transmission network 21. The video signal is received by receiving devices 24 (e.g., computers) owned by multiple users U3, and the video of user U1 and avatar A is played back. This allows each user U3 to view the video of user U1 and avatar A in real time.

[0053] Furthermore, the spatial sharing system shown in Figure 4 also includes a distribution server 25 and a distribution storage 26, similar to the system shown in Figure 3. The functions and operation of the distribution server 25 and the distribution storage 26 are the same as those of the system shown in Figure 3.

[0054] Figure 5 shows how the naked-eye AR display device 10 shown in Figures 1 and 2 is used. In Figure 5, the actual user U1 is depicted in the foreground, and the user U1 reflected in the half-mirror 11 and the conversation partner, i.e., avatar A, displayed on the display 13 are depicted in the background. In front of the half-mirror 11, there is a chair-like structure with at least two seats. In Figure 5, the chair-like structure is depicted as a sofa 48. The sofa 48 is positioned in front of the half-mirror 11 so that the direction of the seating is parallel to the surface of the half-mirror 11. A long table 44 is also positioned between the half-mirror 11 and the sofa 48. Both long tables 44 are positioned so that their longitudinal directions are parallel to the surface of the half-mirror 11. User U1 is sitting in the rightmost seat of sofa 48, facing the display 13. The leftmost seat of sofa 48 is empty and is positioned symmetrically with respect to the surface of the half-mirror 11 and the display 13. Therefore, avatar A displayed on the display 13 appears to be sitting in the leftmost seat of sofa 48. As a result, user U1 can interact with avatar A while maintaining eye contact, as shown by the dashed line in Figure 5. Here, a chair-like structure having at least two seats is described as a sofa 48. However, a chair-like structure having at least two seats may also be a collection of two or more individual chairs. In this case, the user sits in one chair, the other chairs are empty, and one of the empty chairs is positioned symmetrically with respect to the surface of the half-mirror 11 and the display 13.

[0055] Furthermore, the standing positions and depth distances of user U1 and avatar A are reproduced using binocular parallax, enabling user U1 to experience a strong sense of being in the same room.

[0056] (Concerns and solutions regarding AR display) The following describes, in relation to the naked-eye AR display device 10 according to this embodiment, the issues that are generally a concern regarding AR display, measures to improve them, and ways to enhance the sense of being in the same room. In other words, the issues described below are not limited to the naked-eye AR display device 10 according to this embodiment.

[0057] (I see a double image) The half-mirror 11 has a higher reflectivity than ordinary glass. As a result, the image displayed on the display 13 is reflected by the half-mirror 11, and the reflected image may be projected onto the display 13, as shown by the dashed line in Figure 6. Consequently, a double image of the image (for example, avatar A) displayed on the display 13 may be visible.

[0058] If the screen of display 13 is non-glare, the double image itself is reduced, but if it is a matte non-glare, the screen is slightly illuminated by diffused light, so at a close distance of 1-2 meters, the unlit black parts of the aerial image appear as flat surfaces to the naked eye, and it does not become an aerial image.

[0059] As an improvement measure to reduce double images, the display 13 has an anti-reflective film 13a attached to its surface, as shown in Figure 2. The anti-reflective film 13a is not limited to this, but may be, for example, an anti-reflective film having a moth-eye structure.

[0060] As another improvement to reduce double images, the half-mirror 11 has an anti-reflective film 11a attached to its back surface. The anti-reflective film 11a is not limited to this, but may be, for example, an anti-reflective film having a moth-eye structure. Alternatively, the anti-reflective film 11a may be a circularly polarizing film.

[0061] Another improvement to reduce double images is to change the orientation of the display 13 according to the position of the user U. For this reason, as shown in Figure 7, for example, the naked-eye AR display device 10 has a rotation mechanism 41 for rotating the display 13 and a person position sensor 42 for sensing the position of the user U.

[0062] Here, let a be the distance between user U and the interaction partner (e.g., avatar A) displayed on display 13 in a direction parallel to the surface (front or back) of the half mirror 11, let b be the distance between the half mirror 11 and user U in a direction perpendicular to the surface of the half mirror 11, and let Θ be the angle between the normal vector drawn on the surface (e.g., front) of the half mirror 11 and the line extending from the interaction partner to the user. In other words, the angle between the normal vector drawn on the surface of the half mirror 11 and the normal vector drawn on the surface of display 13 is Θ / 2. Also, assume that the interaction partner is displayed in the center of display 13. The rotation axis 13b of display 13 passes through the center of display 13 and extends vertically.

[0063] The person position sensor 42 senses the position of user U and provides this position information to the rotation mechanism 41. The rotation mechanism 41 rotates the display 13 around the rotation axis 13b by Θ / 2 so as to maintain tanΘ = a / 2b with respect to distance a, distance b and angle Θ. For example, if a = 1.2m and 2b = 2.4m, the rotation mechanism 41 rotates the display 13 around the rotation axis 13b by Θ / 2 so as to satisfy tanΘ = 0.5, thereby adjusting the angle Θ.

[0064] Through this rotation mechanism 41 control, from the user U's position, the image of avatar A displayed on the display 13 and the reflected image of avatar A projected onto the display 13 due to reflection appear to overlap. In other words, the double image of avatar A is no longer visible. In addition, because avatar A on the display 13 is posed as if viewed from an angle of Θ / 2 to U, rather than directly in front of U, avatar A appears more three-dimensional.

[0065] Here, we have described an example where the double image of Avatar A is no longer visible from the user U's position. However, the orientation of the display 13 may be controlled by the rotation mechanism 41 so that the double image of Avatar A is no longer visible when viewed from the streaming camera 22. Alternatively, the orientation of the display 13 may be fixed, and the position where the double image of Avatar A is no longer visible may be determined in advance using this formula. User U may then be guided and positioned at the position where the same condition is met.

[0066] By implementing at least one of the improvement measures described above, the double image of Avatar A displayed on the display 13 will disappear, as shown in Figure 8.

[0067] (The image floats) Because the half-mirror 11 illuminates the user or object in front of it with spotlighting, the image of the person interacting with (e.g., avatar A) displayed on the display 13 inside the darkroom-like structure 12 appears to float in the dark space, as shown in Figure 9. As a result, AR does not blend in naturally.

[0068] To address this issue, common furniture items with shared animations are provided and displayed as "connectors," and a sense of unity is created through these shared animations. Figure 10 shows an example of this. In Figure 10, a desk 44 and a wine glass 45A are provided as common furniture items, and a mirrored image of the wine glass 45B is displayed. In addition, actions such as sitting on a sofa and drinking wine are shown as common animations. This makes the AR more natural and familiar. Note that in Figure 10, for the sake of explaining the above situation, the illustration of the user U1 and sofa, which would actually be present in the foreground, has been omitted.

[0069] However, as another improvement measure, an object is actually placed between the display 13 and the user U. Figure 11 shows a desk 47B placed between the display 13 and the sofa 48 where the user U is sitting, and in front of the half mirror 11. Also, a desk 47A is placed between the display 13 and the half mirror 11. Desks 47A and 47B are arranged symmetrically with respect to the half mirror 11. That is, desks 47A and 47B are structures that are symmetrical with respect to the half mirror 11. Furthermore, desks 47A and 47B are placed at equal distances from the half mirror 11.

[0070] Furthermore, Figure 12 is a perspective view of the naked-eye AR display device 10, with desks 47A and 47B placed as shown in Figure 11, as seen from the user's side. In Figure 12, the desk visible in the background is a reflected image of desk 47B, which is placed between the half-mirror 11 and the sofa 48, while desk 47A (see Figure 11), which is placed between the half-mirror 11 and the display 13, is actually only faintly visible from the sofa 48 side.

[0071] In this way, by placing desks 47B and 47A between the display 13 and the sofa 48 where user U is sitting, occlusion (where an object in front hides an object behind it) occurs. As a result, motion parallax makes it apparent that desks 47B and 47A (which have physical presence in the room) are in front of the image on display 13, and that the display 13 is behind them, thus preventing the image on display 13 from appearing to float.

[0072] (Uniformity of sharpness) The half-mirror 11 has a higher reflectivity than ordinary glass. As a result, user U is clearly reflected in the half-mirror 11, but the image displayed on the display 13 (for example, avatar A) is less clear due to the half-mirror 11. Consequently, the clarity of the mirrored image of user U that user U sees and the avatar A may differ.

[0073] To avoid such problems, in the naked-eye AR display device 10 shown in Figures 11 and 12, the half-mirror 11 has a higher reflectivity and lower transmittance in the portion facing the user U than in the portion facing the display 13. For example, the half-mirror 11 has a first half-mirror 11A with high reflectivity and low transmittance, and a second half-mirror 11B with low reflectivity and high transmittance, and the first half-mirror 11A and the second half-mirror 11B are joined to each other at the boundary 11C.

[0074] As a result, the image of user U1 is clearly reflected in the first half-mirror 11A. In addition, user U1 can see a clear image of avatar A displayed on the display 13 through the second half-mirror 11B, which has low reflectivity and high transmittance.

[0075] (Emote effect / Sense of movement within the same space) Figure 13 is a perspective view of a glasses-free AR display device 10 having a sub-display 17 for displaying emote effects (strong emotional expressions) and speech bubble chat text above the user's head. Figure 14 is a top view of the glasses-free AR display device 10 shown in Figure 13.

[0076] The naked-eye AR display device 10 shown in Figure 13 includes a mobile cart 18A that movably supports the display 13, a mobile cart 18B that movably supports the camera 15, a sub-display 17 for emote effects, and a mobile cart 18C that movably supports the sub-display 17.

[0077] The sub-display 17 is provided for emote effects and may, for example, display shapes, text, or particles that represent User U's emotions such as joy, anger, sadness, or happiness, or display effects that highlight User U. It may also display chat text in speech bubbles. Furthermore, it may display User U's name, avatar name, display name, affiliated group, title, online status, etc. These displayed characters may include translated or mirrored characters. It may also display comments from viewers of the stream. In addition, as shown in Figure 13, the sub-display 17 may display clothing that User U is virtually wearing. In this case, User U chooses bright clothing, and the parts corresponding to the clothing displayed on the sub-display 17 are made dark. This makes the video captured by the camera 15 appear as if User U is wearing the clothing displayed on the sub-display 17. By changing the clothing displayed on the sub-display 17 in various ways, User U can get the impression of trying on various clothes.

[0078] The mobile cart 18A supporting the display 13 moves back and forth and left and right on the floor in response to the movement of the interaction partner (person, avatar, etc.) displayed on the display 13. The mobile cart 18B supporting the camera 15 moves back and forth and left and right on the floor so as to maintain the camera 15 in a position symmetrical to the display 13 with respect to the surface of the half mirror 11. The mobile cart 18C supporting the sub-display 17 moves back and forth and left and right on the floor so as to maintain the sub-display 17 in a position symmetrical to the user U in response to the user U's movement.

[0079] This makes it possible to create visuals that give the impression that user U and their conversation partner (a person, avatar, etc.) are moving within the same space.

[0080] (The image becomes flat.) Because the half-mirror 11 illuminates the user or object in front of it with spotlighting, and the display 13 inside the darkroom-like structure 12 is flat, the side-by-side image of the conversation partner (e.g., avatar A) and user U (e.g., co-star) appears flat, as shown in Figure 15. For this reason, AR does not feel natural.

[0081] As a solution to this problem, an additional display is provided in addition to display 13. Figure 16 shows a naked-eye AR display device 10 with additional displays 13A and 13B in addition to display 13. Additional display 13A is provided between display 13, which displays avatar A, and the half mirror 11. Additional display 13B is provided behind user U1, for example, on the wall behind user U1. In Figure 16, the additional display 13B is shown reflected in the half mirror 11.

[0082] In other words, the additional displays 13A and 13B are positioned at a different distance from the half-mirror 11 than the display 13. The additional displays 13A and 13B each display an image appropriate to their distance from the half-mirror 11. To put it another way, the video controller 16 (see Figure 1) causes the additional displays 13A and 13B to display an image appropriate to their distance from the half-mirror 11.

[0083] In this way, additional displays 13A and 13B are provided at positions different from the display 13 at a distance from the half-mirror 11, and images appropriate to the distance from the user U1 are displayed on each of the additional displays 13A and 13B. As a result, the image reflected in the half-mirror 11 becomes three-dimensional, like a pop-up book, with multiple displays at different depths. For example, in 13A, a CG image of a new product under design could be displayed for discussion, or sweets or desserts could be displayed. In 13B, presentation materials could be displayed mirrored for meetings or collaborative work among multiple people, videos could be shown for viewing by multiple people, or landscapes could be displayed for meetings or events based on the premise of being virtually present there. Furthermore, the sub-display 17 may display, for example, a string of characters representing the user U1's emotions. In Figure 16, the string "Yes!" is displayed in the form of a speech bubble on the first line, and a mirrored version of "Yes!" is displayed on the second line. Here, a mirrored string is a string of characters that appears when a string is reflected in a mirror. Simply put, it is a string of characters that has been horizontally flipped. This allows the string of characters representing the user U1's emotions to be easily read from both the user U1's side and the avatar A's side (in other words, the operator of avatar A). This enables communication through means other than video. The string of characters may be, for example, chat messages. In the case of chat messages, automatically entered text via keyboard or voice recognition may be displayed. It may also be displayed in multiple languages ​​using automatic translation or the like. Subtitles may also be displayed. Of course, the sub-display 17 may display something other than strings of characters.

[0084] (Occlusion is incorrect) When an object is placed in front of the half-mirror 11, the object is reflected in the half-mirror 11, but since there is no object there in reality, the image displayed on the display 13 becomes transparent, and the occlusion (front-to-back relationship) is incorrect. Figure 17 shows the situation when a desk 51 is placed in front of the half-mirror 11. As shown in Figure 17, the legs of the woman in avatar A displayed on the display 13 are shown in front of the desk 51 reflected in the half-mirror 11, and the occlusion (front-to-back relationship) is incorrect.

[0085] As a solution to this problem, it is recommended to place clone-shaped objects symmetrically with respect to the half-mirror 11 within a distance of 2m from the half-mirror 11 (up to a maximum distance of 4m from the user when viewed in the mirror), and to place a display showing the room, scenery, etc., at least 1m behind the user U within a distance of 2m or more from the half-mirror 11 (more than 4m from the user when viewed in the mirror).

[0086] Figure 18 is a plan view showing a pair of desks 51A and 51B placed symmetrically with respect to the half-mirror 11, as described above, with a display 54 placed behind the sofa 48 where user U is seated. A pair of wine glasses 52A and 52B are placed on the pair of desks 51A and 51B, respectively, symmetrically with respect to the half-mirror 11.

[0087] Figure 19 is a perspective view of the naked-eye AR display device 10, as seen from the user U's side, with objects (desks 51A, 51B and wine glasses 52A, 52B) and a display 54 placed on it, as shown in Figure 18.

[0088] As explained in relation to Figures 11 and 14, in Figure 19, the desk and wine glass visible in the background are reflected images of desk 51A and wine glass 52A placed between the half mirror 11 and sofa 48, while desk 51B and wine glass 52B placed between the half mirror 11 and display 13 are actually only vaguely visible from the side of sofa 48.

[0089] Within a distance of 2m from the half-mirror 11 (up to 4m from user U when viewed in the mirror), the sense of depth due to binocular parallax is sensitive. Therefore, when a desk 51A and a wine glass 52A are actually placed between the half-mirror 11 and the sofa 48, a strong sense of 3D depth is created. This is because, in conventional mirror-type video conferencing, objects are arranged as images within a single screen of the display, resulting in a flat AR display. However, in this configuration, the actual images of the desk 51A and wine glass 52A in front of user U, and the images of the desk 51A and wine glass 52A reflected in the half-mirror 11, are perceived by user U through binocular parallax as actually existing at that distance, thus creating a high sense of 3D depth and spatial awareness.

[0090] Furthermore, from the perspective of motion parallax, the table 51A and wine glass 52A placed between user U and half mirror 11, and the table 51B and wine glass 52B placed between half mirror 11 and display 13, make it appear as if Avatar A is hidden behind the table 51A and wine glass 52A reflected in the half mirror 11. Occlusion occurs due to motion parallax caused by the movement of user U's head, creating a strong sense of 3D space.

[0091] To user U, images of rooms, scenery, etc., displayed on display 54, which is positioned at a distance of 1 meter or more behind user U, are reflected in the half-mirror 11. At distances of 2 meters or more from the half-mirror 11, the mirror causes the object to appear more than 4 meters away, and the binocular parallax weakens. Therefore, the motion parallax that occurs between the near view and the distant view of the room, scenery, etc., displayed on display 54, as the user moves their head, enhances the sense of 3D depth.

[0092] (A 3D-like discomfort occurs in the knee.) In the naked-eye AR display device 10 according to this embodiment, the distance from user U to the display 13 that displays the image is approximately 2.68 m, which is closer than 4 m. Therefore, the user U is aware of the 3D distance perception using both eyes. Figure 20 is a schematic perspective view showing person P sitting on sofa 48. In reality, when person P is sitting on sofa 48, their knees are about 30-50 cm in front of their torso. However, when an image of the person sitting on sofa 48 is displayed on the display 13, because the display 13 is flat, the feet are positioned at the depth of the backrest of sofa 48, resulting in a strong sense of 3D dissonance.

[0093] As a solution to this problem, we will adopt a method similar to that described in relation to Figure 11. That is, as shown in Figure 11, we will place a desk 47A between the half mirror 11 and the display 13, and a desk 47B will be placed symmetrically with respect to the half mirror 11.

[0094] Figure 21 is a perspective view of the naked-eye AR display device with a desk placed on it, as shown in Figure 11, as seen from the user's side. The desk 47A placed between the half-mirror 11 and the display 13 prevents user U from seeing avatar A's knees. In other words, the desk-like structure 47A positioned behind the half-mirror 11 blocks user U's view of the area around the knees of the person they are interacting with, as displayed on the display 13. Moreover, to user U, avatar A's knees appear to be hidden by the desk 47B reflected in the half-mirror 11. User U no longer experiences any 3D dissonance regarding avatar A's knees. At this time, although the distance to avatar A's shins and toes is not accurate due to binocular parallax, occlusion (where an object in front hides an object behind) occurs between the lower body of avatar A and the top surface and legs of the desks 47A and 47B, creating a strong 3D effect due to motion parallax. Furthermore, people P, who are standing around desk 47B and sofa 48, can also see avatar A's knees with a strong sense of 3D without any 3D distortion, just like user U. In addition to directly concealing the knees, even when the knees are crossed, exposing the knees while concealing a portion of the waist and groin area, thus blocking the continuity of the upper and lower body's field of view, it is possible to eliminate the 3D distortion caused by binocular parallax and create a strong sense of 3D through motion parallax.

[0095] This makes it possible to display the person you are interacting with (e.g., Avatar A) at a closer distance, enabling a more immersive and impactful naked-eye AR display. Furthermore, not only user U, but also people P around user U, and even those watching the stream, can all naturally see each other.

[0096] (The image is transparent) If the background behind user U is bright, the background scenery will be displayed, and the conversation partner (e.g., avatar A) displayed on display 13 will appear transparent. Figure 22 schematically illustrates how the image of avatar A displayed on display 13 becomes transparent when the background behind user U is bright. In Figure 22, the bright background behind user U is reflected in the half mirror 11, causing the face portion of the image of avatar A to become transparent.

[0097] As a solution to this problem, a dark-colored screen can be placed on the surface of the half-mirror 11 at a position symmetrical to the display 13. Figure 23 is a perspective view of the naked-eye AR display device 10, as seen from above by user U, with a dark-colored screen 61 placed on the surface of the half-mirror 11 at a position symmetrical to the display 13. A camera 15 is mounted on the screen 61 at a position symmetrical to the display 13 with respect to the surface of the half-mirror 11. The screen 61 may also be human-shaped. By placing the screen 61, the background behind user U reflected in the half-mirror 11 becomes dark, preventing the image of avatar A displayed on the display 13 from becoming transparent.

[0098] Alternatively, instead of using the partition 61, the transparency of the image can be prevented by providing an electronic curtain 62 on the surface (front or back) of the half mirror 11, as shown by the dashed line. Alternatively, instead of using the partition 61, the transparency of the image can be prevented by applying a vertical polarizing film 63 to the surface (front or back) of the half mirror 11, as shown by the dashed line. The electronic curtain 62 and the vertical polarizing film 63 may also be used in combination with the partition 61.

[0099] Another improvement is to place a dark-colored high-back chair in a position roughly symmetrical to the display 13 with respect to the surface of the half-mirror 11. In this case, another dark-colored high-back chair should be placed next to that one, and user U should sit in the dark-colored high-back chair and interact. Figure 24 is a perspective view from above of the naked-eye AR display device 10 with two dark-colored high-back chairs 65A and 65B placed on the user U side with respect to the half-mirror 11. Figure 25 is a plan view showing the arrangement of the display, high-back chairs and camera with respect to the half-mirror in the naked-eye AR display device shown in Figure 24.

[0100] The high-back chair 65A is positioned so that its backrest is symmetrical with respect to the surface of the half-mirror 11 and the display 13. The backrest of the high-back chair 65A has a hole 65Ah at the eye level of user U, and the camera 15 is placed inside this hole 65Ah. Here, the eye level of user U is assumed to be the eye level of a person with a standard build sitting in the high-back chair 65A.

[0101] More specifically, as shown in Figure 25, for example, the high-back chair 65A is positioned such that a line perpendicular to the surface of the half-mirror 11 passing through the center of the lens at the tip of the camera 15, which is placed in the hole 65Ah in its backrest, passes through the center of the display 13. Furthermore, the distance from the center of the lens at the tip of the camera 15 to the half-mirror 11 is equal to the distance from the center of the display 13 to the half-mirror 11, for example, both being d1.

[0102] The high-back chair 65B is placed next to the high-back chair 65A, in a direction parallel to the surface of the half-mirror 11. The high-back chair 65B is positioned on a straight line extending from the center of the display 13 at an angle Θ with respect to the normal to the surface of the half-mirror 11, such that the distance from the backrest to the half-mirror 11 is equal to that of the high-back chair 65A, and equal to the distance from the center of the display 13 to the half-mirror 11. Furthermore, the high-back chair 65B is positioned so that the backrest faces the display 13 at an angle Θ / 2 (see Figure 7) with respect to the normal to the surface of the half-mirror 11. For example, the high-back chairs 65A and 65B are positioned on the floor in a plane symmetrical with respect to a plane perpendicular to the half-mirror 11. User U uses the naked-eye AR display device 10 while seated in the high-back chair 65B.

[0103] Furthermore, as another improvement measure, as explained in relation to Figure 18, a display may be placed behind the user U and an image with a broad shadow may be displayed on the display. Figure 26 shows the rear image 67 reflected in the half mirror 11 with a broad shadow 68. As shown in Figure 18, a display 54 is placed behind the user U. The display 54 displays the rear image 67 reflected in the half mirror 11. Furthermore, the rear image 67 displayed by the display 54 has a broad shadow 68 applied by image processing at positions on the surface of the half mirror 11 corresponding to the user U and the person they are interacting with (e.g., avatar A). This reduces the brightness of the background behind the user U and prevents the image of the person they are interacting with (e.g., avatar A) displayed on the display 13 from becoming transparent.

[0104] (Unable to view third-person perspective footage) The operator controlling Avatar A may wear an HMD (Head-Mounted Display) to operate Avatar A. Figure 27 schematically shows operator A1 wearing an HMD 71 and using a hand controller 72 to operate Avatar A. Because operator A1 is wearing an HMD 71, the image operator A1 sees is a first-person view of Avatar A that operator A1 is controlling. Therefore, operator A1 can see the other person's image displayed on the virtual display 33 (see Figure 4), but cannot see the third-person view image in which operator A1 and the other person are adjacent.

[0105] As a solution to this problem, it is advisable to provide a virtual mirror 31 in the virtual space (metaverse), as shown in Figure 28. Figure 28 is a schematic perspective view showing a part of the virtual naked-eye AR display device 30 shown in Figure 4. Figure 4 is as described above. The virtual mirror 31 displays an image of avatar A as if it were reflected in a mirror. Therefore, when operator A1 places the virtual mirror 31 within the field of view of the avatar A that they are controlling, they can see avatar A reflected in the virtual mirror 31. As a result, operator A1, wearing the HMD 71, can see avatar A reflected in the virtual mirror 31 and the person displayed on the virtual display 33. In other words, operator A1 can view adjacent avatars A and the person in question from a third-person perspective. In addition, operator A1 can confirm the facial expressions, emotes, and speech bubble chats that they have instructed avatar A to express through avatar A reflected in the virtual mirror 31.

[0106] (The overlapping parts of the image are transparent / the tactile experience is not possible) When the image of user U reflected in the half-mirror 11 and the image of the conversation partner displayed on the display 13 overlap on the surface of the half-mirror 11, the image of the conversation partner becomes transparent. Figure 29 shows how the image of the conversation partner becomes transparent when the image of user U and the image of the conversation partner overlap. In Figure 29, the part of the image of the conversation partner (avatar A) that overlaps with the legs of user U becomes transparent.

[0107] Furthermore, while user U perceives their conversation partner as being right beside them, they are unable to experience tactile interactions such as shaking hands, toasting, high-fiving, or hugging. In other words, the conversation partner is visible but becomes a transparent ghost without any physical collision. Therefore, although user U perceives their conversation partner as being right next to them, user U is actually unable to communicate with them physically.

[0108] As a solution to this problem, a robot can be placed next to user U in a position symmetrical to the display 13 with respect to the surface of the half-mirror 11, and the robot's movements can be controlled in synchronization with the interaction partner displayed on the display 13. Figure 30 is a perspective view of a naked-eye AR display device with robot 81 placed next to user U, as seen from user U's side. A torso-shaped cushion 89 may be placed instead of robot 81. The torso-shaped cushion 89 will be described later. Figure 31 is a block diagram showing a control system 84 that controls the movements of robot 81 and torso-shaped cushion 89.

[0109] In Figure 30, Avatar A is depicted as the conversation partner for User U. Robot 81 is positioned next to User U, in a position symmetrical to the display 13 with respect to the surface of the half-mirror 11. For example, Robot 81 is positioned adjacent to User U on the sofa 48 where User U is sitting. For example, robot 81 has a skeletal structure similar to that of a human and actuators that drive the skeletal structure. Robot 81 may further have a stretchable membrane-like material covering the skeletal structure and a shape control mechanism that deforms the shape of the membrane-like material into an arbitrary shape. The membrane-like material mimics human skin and expresses body shape by its shape. The shape control mechanism also constitutes a body shape reproduction means (see Figure 31) that reproduces the body shape of an interaction partner by controlling the shape of the membrane-like material. For example, the membrane-like material may be made of stretchable rubber, and the shape control mechanism may be able to control the shape of the membrane-like material by air pressure. Robot 81 is painted, for example, black, so as not to impair the image of avatar A displayed on the display 13 when reflected in the half-mirror 11. Alternatively, robot 81 is made of a transparent material.

[0110] Robot 81 operates in imitation of avatar A by the control system 84 shown in Figure 31. More specifically, the robot 81's movement is controlled so that it overlaps with avatar A on the half-mirror 11. In other words, robot 81 is controlled to operate symmetrically with respect to avatar A. For this reason, in order to accurately determine the position of each part of robot 81, position detection markers 82 are attached to multiple locations on robot 81. Furthermore, in order to detect the position of the markers 82 attached to robot 81 in three-dimensional space, a 3D grid sensor, for example, is provided in the space where robot 81 and user U are located.

[0111] As shown in Figure 31, the control system 84 includes motion analysis means 74, shape data acquisition means 75, body shape reproduction means 76, collision feedback means 77, motion acquisition means 85, transmission means 86, motion reproduction means 87, and video reproduction means 88. The control system 84 receives information from the camera 37 that films the operator A1 of avatar A, the video of avatar A displayed on the screen, the HMD 71 and hand controller 72 worn by operator A1 of avatar A, and the motion tracker 73, and controls the robot 81 (and the torso-shaped cushion described later). Here, the HMD 71 and hand controller 72 are worn by operator A1 of avatar A, and the movements (motions) of operator A1 of avatar A are detected. Motion information is received from the HMD 71 and hand controller 72 by the motion acquisition means 85 and passed directly to the transmission means 86. Here, eye tracking information, vital data, voice recognition, facial recognition emotes, etc. may also be sent along with the motion information. Here, the motion tracker 73 can be anything that can detect the movements (motion) of operator A1 of avatar A. For example, it could be clothing worn by operator A1 that has multiple detectable targets, and a system that detects those multiple targets. Furthermore, the control system 84 transmits physical collisions and contact sensations as vibrations to the interaction partner via a vibration reproduction device 78 installed on the interaction partner's side, based on the movements of the robot 81.

[0112] The motion analysis means 74, when the conversation partner is a person, receives a video signal from the camera 15 that films the conversation partner and analyzes the conversation partner's movements (motion). Alternatively, the video signal of the conversation partner displayed on the display may be used as input, either directly or by capturing the camera image again. The motion acquisition means 85 receives information from the motion analysis means 74 and acquires the conversation partner's movements (motion). The shape data acquisition means 75 receives the video signal from the camera 15 and passes it directly to the transmission means 86, while also acquiring shape data related to the conversation partner's body shape. The motion analysis means 74, when interacting with an avatar, receives video footage from the camera 37 that films the avatar's operator and video footage of avatar A displayed on the screen, and analyzes the avatar's movements (motion). The shape data acquisition means 75 receives the video footage from the camera 37 and passes it directly to the transmission means 86, while also acquiring shape data related to the avatar operator's body shape. The motion acquisition means 85 receives information from the motion analysis means 74 and acquires the avatar operator's movements (motion). The shape data acquisition means 75 receives the video footage from the camera 37 and acquires shape data related to the avatar operator's body shape. Of course, the avatar's shape data may also be obtained directly from an external program. For example, the motion analysis means 74 and the shape data acquisition means 75 may be provided inside the video controller 16 shown in Figures 1, 3, and 4, and the transmission means 86 may consist of the transmission network 21 shown in Figures 3 and 4. In other words, the video analysis by the motion analysis means 74 and the shape data acquisition means 75 may be performed on the side where the camera 15 or camera 37 is installed.

[0113] The motion acquisition means 85 transmits the acquired movements (motions) of the conversation partner to the transmission means 86. The shape data acquisition means 75 also transmits the acquired shape data regarding the conversation partner's body shape to the transmission means 86.

[0114] The transmission means 86 transmits the video signals acquired by the cameras 15 and 37, the motion information of the conversation partner acquired by the motion acquisition means 85, and the shape data related to the conversation partner's body shape acquired by the shape data acquisition means 75 to the user U's naked-eye AR display device 10. The transmission means 86 may be included in the video controller 16 of the conversation partner's naked-eye AR display device 10 or the video controller 36 of the virtual naked-eye AR display device 30.

[0115] The video reproduction means 87 receives the video signal transmitted from the transmission means 86 and displays the video of the person being interacted with on the display 13. The video reproduction means 87 may be included in the video controller 16 of the user-side glasses-free AR display device 10. Furthermore, the body shape reproduction means 88 receives shape data transmitted from the transmission means 86 and controls the shape of the membrane-like material covering the skeletal structure of the robot 81 based on the shape data. As a result, the external shape of the robot 81 becomes close to the body shape of the operator of avatar A. Therefore, user U can feel as if they are directly touching the operator of avatar A.

[0116] The motion reproduction means 88 receives motion information of the interaction partner transmitted from the transmission means 86 and controls the movement of the robot 81. At the same time, the motion reproduction means 88 also uses position detection markers 82 attached to the robot 81 to acquire position information of each part of the robot 81 and controls the movement of the robot 81 so that the image of the robot 81 reflected in the half mirror 11 completely overlaps with the image of the interaction partner on the half mirror 11. In other words, the motion reproduction means 88 controls the robot 81 so that each part of the robot 81 is positioned symmetrically with respect to the surface of the half mirror 11 with respect to each part of the interaction partner (for example, avatar A in Figure 30) and performs movements symmetrically with respect to avatar A.

[0117] With this configuration, for example, when user U puts their arm behind robot 81, user U's arm is blocked by the black-painted robot 81 and is not reflected in the half-mirror 11. This prevents the image of the person being interacted with from being visible through the half-mirror 11.

[0118] Furthermore, the robot 81 performs plane-symmetrical movements with respect to the surface of the half-mirror 11, in relation to the interaction partner (for example, avatar A in Figure 30). Therefore, when user U touches the interaction partner in the image of the half-mirror 11, the robot 81 can provide a sense of physical contact. This allows user U to feel as if they are having tactile experiences such as shaking hands, toasting, high-fiving, or hugging with the interaction partner. Figure 30 shows an example of user U holding hands with avatar A. User U is able to engage in physical communication with the interaction partner, albeit in a simulated manner. In addition, the collision feedback means 77 acquires contact information from the robot 81 and transmits it to the vibration reproduction device 78 on the interaction partner's side, and the vibration reproduction device 78 expresses the sense of contact as vibration.

[0119] As another simple improvement, for example, as shown in Figure 32, a cushion, such as a torso-shaped cushion 89, may be placed next to the user U in a position symmetrical to the display 13 with respect to the surface of the half-mirror 11. Figure 32 is a perspective view of a naked-eye AR display device with a torso-shaped cushion 89 placed next to the user U, as seen from the user U's side.

[0120] The torso-shaped cushion 89 is an object that mimics the torso of a human being, excluding the head, arms, and legs. The torso-shaped cushion 89 is placed on the seat next to user U, who is sitting on the sofa 48, in a position symmetrical to the display 13 with respect to the surface of the half-mirror 11. The torso-shaped cushion 89 may also have a stretchable membrane-like material and a shape control mechanism that can deform the shape of the membrane-like material into an arbitrary shape, similar to the robot 81. The torso-shaped cushion 89 is painted black, for example, so as not to impair the image of avatar A displayed on the display 13 when reflected in the half-mirror 11. The torso-shaped cushion 89 may also be divided into three blocks, for example, a head, a torso, and a waist, with the head and waist being joined to the torso by joints, and moving forward, backward, left, right, up, and down with actuators. Furthermore, the head may have a mouth, nostrils, tongue, voice-generating mechanism, heat-generating and sweat-generating mechanism, fragrance-generating mechanism, wig, etc.

[0121] With this configuration, for example, when user U puts their arms behind the torso-shaped cushion 89, user U's arms are blocked by the black-painted torso-shaped cushion 89 and are not reflected in the half-mirror 11. This prevents the image of avatar A on the half-mirror 11 from being visible through the cushion. In addition, user U can feel physical contact with the torso-shaped cushion 89. This allows user U to feel as if they are having a tactile experience with avatar A. In other words, user U can engage in physical communication with their conversation partner, albeit a simulated one.

[0122] (effect) As described above, in the naked-eye AR display device 10 and the spatial sharing system using the naked-eye AR display device 10 according to the embodiment, the user can converse with the other person while maintaining eye contact with them using the naked eye. Furthermore, even at close range, the user is less likely to perceive the other person as flat, is more likely to obtain a correct sense of depth, and is more likely to feel as if the other person is in the same room next to them.

[0123] Furthermore, since the user sees their own image (mirror image) reflected in the half-mirror 11, there is no delay in the user's image. The naked-eye AR display device 10 has a simple structure and can be constructed at a low cost. Not only the user, but also people around user U and people watching the stream can naturally see each other with the correct depth and position.

[0124] (Example of use of the naked-eye AR display device 10) The following describes some examples of spatial sharing that can be realized using the naked-eye AR display device 10, with reference to the drawings.

[0125] (Space sharing example 1) Figure 33 shows an example of creating a fused space of local and remote locations using a naked-eye AR display device 10 according to an embodiment. The two naked-eye AR display devices 10 are able to communicate with each other via a wide-area network 91. Each naked-eye AR display device 10 is equipped with high-back chairs 65A and 65B to prevent the images of users U1 and U2 from appearing transparent. The distributed video 92 is equipped with connecting furniture such as a table, chairs, and a national flag to prevent the images of users U1 and U2 from appearing detached.

[0126] (Space sharing example 2) Figure 34 shows an example of creating a fused space between the real world and the cyber world using the naked-eye AR display device 10 according to the embodiment, as another example of spatial sharing. The naked-eye AR display device 10 is installed in the real world. A virtual naked-eye AR display device 30 is installed in the cyber world. In other words, a real set (actual room) that serves as an entrance from the real world is created as a venue. A world containing a clone space of the room is created in the cyber world on the metaverse. The real set in the real world (naked-eye AR display device 10) and the world on the metaverse (virtual naked-eye AR display device 30) can communicate with each other via a high-speed network 95.

[0127] In the real world, multiple real-world audience members RUn (n=1,2,…) are waiting in line to participate in the real-world set (analogless AR display device 10). In the cyber world, multiple in-world audience members MUn (n=1,2,…) and their respective multiple avatars MAn (n=1,2,…) are waiting in line to participate in the metaverse world (virtual analogless AR display device 30).

[0128] Multiple real-world audience members (RUn) take turns participating in the real-world set (analogless AR display device 10) one at a time. Similarly, multiple in-world audience members (MUn) take turns participating in the metaverse world (virtual analogless AR display device 30) using their own avatar (MAn) one at a time.

[0129] Figure 34 illustrates, as an example, how Real Participant RU1, one of the Real Mass Audience RUns, participates in the Real Set (Analogless AR Display Device 10), and Avatar MA1 of In-World Audience MU1, one of the In-World Audience MUns, participates in the Metaverse World (Virtual Analogless AR Display Device 30). Real Participant RU1 is displayed on the virtual display 33 of the Virtual Analogless AR Display Device 30, and Avatar MA1 is displayed on the display 13 of the Analogless AR Display Device 10, both virtually synchronized in real time by the high-speed network 95.

[0130] This allows real participants RU1 and in-world audience MU1 to share time together while maintaining a sense of being in the same room through their avatars MA1.

[0131] (Space sharing example 3) Figure 35 shows another example of spatial sharing, illustrating a system that uses the naked-eye AR display device 10 according to the embodiment to connect the real world (set construction) and the virtual world (in-world) as a pair.

[0132] In the real world (set construction), a glasses-free AR display device 10 is installed, and in the virtual world (in-world), a virtual glasses-free AR display device 30 is installed. The glasses-free AR display device 10 and the virtual glasses-free AR display device 30 can communicate with each other via a high-speed network (not shown).

[0133] In the real world (set construction), the components of the naked-eye AR display device 10 include a half-mirror 11, a display 13, lighting 14, and a camera 15, as well as a person position sensor 42 (see Figure 7), desks 47A and 47B and a sofa 48 (both see Figure 11), and further displays 101 and 102.

[0134] Display 101 is located on desk 47A and is for object AR, displaying objects for AR purposes. Display 102 is located behind sofa 48 and is for information sharing, displaying images for sharing information to be projected onto half mirror 11. Similar to the spatial sharing example 2 described above, one of several real mass audience members RUn (n=1,2,…) participates in the real world (set construction) as a real participant. Part of the illumination light from lighting 14 is blocked by a light-shielding member (not shown), and a shadow spot 103 is formed on the surface of the half mirror 11 at a position symmetrical to the display 13, thereby preventing the image of avatar A displayed on display 13 from being visible through it. In Figure 35, the virtual image of sofa 48 is drawn with a dashed line.

[0135] Meanwhile, in the virtual world (in-world), in addition to the virtual mirror 31 and virtual display 33, the virtual desks 47AV, 47BV and virtual sofas 48V1, 48V2, as well as virtual displays 105, 106, are provided as components of the virtual naked-eye AR display device 30.

[0136] The virtual desks 47AV and 47BV and the virtual sofas 48V1 and 48V2 are virtual objects corresponding to desks 47A and 47B and sofa 48, respectively. The virtual displays 105 and 106 are positioned behind the virtual sofas 48V1 and 48V2, respectively. Both virtual displays 105 and 106, like display 102, are for information sharing and display images of the information to be shared. Similar to the spatial sharing example 2 described above, the configuration allows one of multiple in-world spectators MUn (n=1,2,…) to participate in the virtual world (in-world) using their own avatar MAn.

[0137] In this way, by constructing a pair of real worlds (set construction) and virtual worlds (in-world), it becomes possible for one of multiple real-world audience members RUn (n=1,2,…) and one of multiple in-world audience members MUn (n=1,2,…) to virtually experience a dialogue in the same room in both the real world (set construction) and the virtual world (in-world).

[0138] (Space sharing example 4) Figure 36 shows another example of spatial sharing, illustrating an example of creating a fused space between the real world and the metaverse using the naked-eye AR display device 10 according to the embodiment. The naked-eye AR display device 10 is provided in the real world. The metaverse 111 is also provided. The naked-eye AR display device 10 and the metaverse 111 generation device (not shown) are able to communicate via a network 112. Data is transmitted from the metaverse 111 generation device (not shown) to the naked-eye AR display device 10 via the network 112, and the naked-eye AR display device 10's distribution camera 22 (see Figure 4) captures the image displayed on the half-mirror 11, thereby creating a fused space between the real world and the metaverse 113.

[0139] (Space sharing example 5) Figure 37 shows an example of a system that uses the naked-eye AR display device 10 according to the embodiment to connect the real world (set construction) and the virtual world (in-world) as a pair, as yet another example of spatial sharing. The basic configuration of the system shown in Figure 37 is the same as the basic configuration of the system shown in Figure 35. That is, in Figure 37, the components indicated by the same reference numerals as in Figure 35 are the same components as in Figure 35. Here, in order to avoid duplication of explanation, the explanation of these components will be omitted. The differences specific to the system shown in Figure 37 will be described below.

[0140] In the system shown in Figure 37, a virtual sensor 121 is provided in the virtual world (in-world) to acquire information about the avatar MA1. The information about the avatar MA1 is not limited to this, but includes, for example, the actions of the avatar MA1 (e.g., sitting, clapping, reactions, etc.) and the environment surrounding the avatar MA1 (e.g., wind, light particles, scent, etc.). On the other hand, in the real world (set construction), for example, an action reproduction device 123 that reproduces the actions of the avatar MA1 and an environment reproduction device 124 that reproduces the environment surrounding the avatar MA1 are provided. The virtual sensor 121 and the action reproduction device 123 and environment reproduction device 124 can communicate with each other via a high-speed network 122, and the information from the virtual sensor 121 is instantly transmitted to the action reproduction device 123 and environment reproduction device 124.

[0141] The action reproduction device 123 includes, for example, a vibration device, and transmits the actions of avatar MA1 (e.g., sitting, clapping, reacting, etc.) to the real participant RU1 as body sonics. The environment reproduction device 124 includes, for example, an air conditioning device and a light source device, and provides the environment surrounding avatar MA1 (e.g., wind, light particles, etc.) to the real participant RU1. This makes it possible for the real participant RU1 to perceive the actions of avatar MA1 and the environment surrounding avatar MA1 through their five senses. Furthermore, the transmission of avatar MA1's actions and the provision of the environment surrounding avatar MA1 may be performed for all real mass audience members RUn, including the real participant RU1.

[0142] Furthermore, when real participants RU1 enter and exit, that is, from the time they enter until they are seated, and from the time they leave their seats until they exit, the images that the in-world audience MUn sees through the avatar MAn appear unnatural. To avoid this, when real participants RU1 enter and exit, it would be good to turn off the lights, or use point cloud data processing or hologram processing to create the effect of the real participants RU1 gradually appearing and gradually disappearing, like in a science fiction movie.

[0143] (Examples of applications of the naked-eye AR display device 10) This section describes an example of applying the naked-eye AR display device 10 according to the embodiment to a mirror-type video conferencing system. However, the application of the naked-eye AR display device 10 according to the embodiment is not limited to mirror-type video conferencing. For example, the naked-eye AR display device 10 according to the embodiment may be applied to any AR technology. For example, it may be applied to AR broadcasting or AR distribution, in which the user and a remote conversation partner (person or avatar) are AR-combined and the resulting image projected onto a half-mirror is captured by a distribution camera and broadcast or distributed. For example, it may be applied to AR theater, in which the user and a remote conversation partner (person or avatar) create an image through AR synthesis that makes it appear as if they are performing together in a play.

[0144] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention. [Explanation of Symbols]

[0145] 10...Naked-eye AR display device 11… Half mirror 11a... Anti-reflective film 12…Darkroom-like structures 13…Display 13A, 13B… Additional displays 13a... Anti-reflective film 13b... Rotation axis 14…Lighting 15…Camera 16…Video controller 17…Sub-display 18A,18B,18C...Moving trolley 21…Transmission Network 22... Camera for streaming 23... Transmitter 24… Receiving equipment 25…Distribution server 26…Distribution Storage 30…Virtual naked-eye AR display device 31…Virtual Mirror 32...A darkroom-like virtual structure 33…Virtual Display 34…Virtual lighting 35…Virtual Camera 36…Video controller 37... Camera 41... Rotation mechanism 42...Person position sensor 44...desk 45A, 45B… Wine Glasses 47A,47B…desk 47AV, 47BV… Virtual Desk 48... Sofa 48V1, 48V2… virtual sofa 51...desk 51A,51B…desk 52A, 52B… Wine Glasses 54…Display 61... Dark-colored screen 62... Electronic Curtain 63…Vertical polarizing film 65A, 65B… Dark-colored high-back chairs 65Ah…hole 67... Rear view 68... Shadow 71…HMD 72... Hand controller 73…Motion tracker 74…Motion analysis means 75…Means for acquiring shape data 76… Body shape reproduction means 77...Collision feedback means 78... Vibration reproduction means 81... Robot 82… Marker 84...Control System 85…Method for acquiring motion 86...Transmission means 87…Methods for reproducing video 88...Means for reproducing the operation 89... Torso-shaped cushion 91…Wide-area network 92...Streaming video 95…High-speed network 101... Display 102…Display 103...Shadow Spot 105…Virtual Display 106…Virtual Display 111... Metaverse 112…Network 113…Fusion space 121…Virtual Sensor 122… High-speed network 123... Action Reproduction Device 124…Environmental reproduction device S1,S2…Video signal U1, U2, U3... Users A... Avatar A1…Operator RUn…Multiple real mass audience RU1…Real participants MUn…Multiple In-World Spectators MU1…In-world audience MAn... Multiple avatars MA1... Avatar

Claims

1. A half-mirror positioned upright on the floor, A darkroom-like structure is provided behind the half-mirror, A display positioned diagonally within the darkroom-like structure with respect to the surface of the half-mirror, A light fixture positioned in front of the half-mirror and illuminating the user facing the display, It includes a video controller that receives a video signal and displays the image on the display, Naked-eye AR display device.

2. The system further includes a camera positioned symmetrically with respect to the surface of the half-mirror with respect to the display, which captures the user reflected in the half-mirror. The aforementioned video controller acquires and transmits a video signal from the camera. The naked-eye AR display device according to claim 1.

3. The system further comprises at least one pair of identically shaped desk-like structures arranged symmetrically with respect to the surface of the half-mirror, wherein the desk-like structures positioned behind the half-mirror obstruct the user's view of the knees, waist, or groin area of ​​the person they are talking to, as displayed on the screen. The naked-eye AR display device according to claim 1.

4. The system further comprises a chair-like structure having at least two seats, the chair-like structure being positioned in front of the half-mirror such that the direction of the arrangement of the at least two seats is parallel to the surface of the half-mirror, the user sitting in one seat and the other seats being empty, one of the empty seats being positioned symmetrically with respect to the surface of the half-mirror and the display. The naked-eye AR display device according to claim 1.

5. An anti-reflective film having a moth-eye structure is attached to the surface of the display, or the back surface of the half-mirror, or both. The naked-eye AR display device according to claim 1.

6. It also has multiple additional displays, At least one of the plurality of additional displays is positioned within the darkroom-like structure at a distance from the half-mirror that is different from the display. At least one of the aforementioned additional displays is positioned behind the user, in front of the half-mirror. The naked-eye AR display device according to claim 1.

7. Within the darkroom-like structure, an additional display is positioned symmetrically with respect to the surface of the half-mirror and the user's position, The system further includes a movable stand that moves the additional display in accordance with changes in the user's position, The naked-eye AR display device according to claim 1.

8. The device further comprises a rotation mechanism that rotates the display such that tanΘ = a / 2b, where a is the distance between the user and the person they are interacting with, in a direction parallel to the surface of the half-mirror; b is the distance between the surface of the half-mirror and the user; and Θ / 2 is the angle between the normal vector drawn on the surface of the half-mirror and the normal vector drawn on the surface of the display. The naked-eye AR display device according to claim 1.

9. The system further comprises a dark-colored screen or a dark-colored high-back chair having a high backrest, wherein the dark-colored screen or the backrest of the high-back chair is positioned symmetrically with respect to the surface of the half-mirror with respect to the display, and a hole is formed in the screen or the backrest at the user's eye level, and the camera is positioned in the hole. The naked-eye AR display device according to claim 2.

10. The present invention further comprises a dark-colored high-back chair with a high backrest, wherein the angle between the normal vector drawn on the surface of the half-mirror and the normal vector drawn on the surface of the display is Θ / 2, and the high-back chair is positioned such that the distance from the backrest to the half-mirror is equal to the distance from the center of the display to the half-mirror, and further, the backrest faces the display at an angle of Θ / 2 with respect to the normal vector drawn on the surface of the half-mirror. The naked-eye AR display device according to claim 1.

11. The system further includes a dark-colored cushion placed on an empty seat positioned symmetrically with respect to the surface of the half-mirror and the display, or a dark-colored robot that operates symmetrically with respect to the person being interacted with displayed on the display. The naked-eye AR display device according to claim 4.

Images