Augmented reality collaboration system

Abstract

A system includes a user device including a sensor configured to sense data related to a physical environment of the user device, a display; a hardware processor; and a non-transitory machine-readable storage medium encoded with instructions executable by the hardware processor to: place a virtual object in a 3D scene displayed by a second user device, determine a gesture of the user device relative to a physical location in a physical environment of the user device, and store the virtual object in the 3D scene. And generate an image of the virtual content based on the gesture of the user device relative to the placed virtual object, wherein the image of the virtual content is projected by one or more displays of the user device at a predetermined location relative to a physical location in a physical environment of the user device.

Description

[0001] This application is a divisional application of an application with a filing date of September 16, 2021, application number 202180076991.X, and name “Augmented Reality Collaboration System”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 17 / 023,220, filed on September 16, 2020, the entire contents of which are incorporated herein by reference. Background Art

[0004] The disclosed technology relates generally to augmented reality systems, and more particularly, some embodiments relate to collaboration using such systems. Summary of the invention

[0005] A claimed solution based on computer technology overcomes problems arising specifically in the field of computer technology.

[0006] In general, one aspect of the disclosure features a system that includes: a system configured to display virtual content, the system displaying virtual content comprising: a user device comprising: one or more sensors configured to sense data related to a physical environment of the user device, and one or more displays; one or more hardware processors; a non-transitory machine-readable storage medium encoded with instructions executable by the one or more hardware processors to: in response to user input received by the user device, place a virtual object in a virtual location in a 3D scene displayed by a second user device, the virtual location corresponding to a physical location in the physical environment of the user device, determine a posture of the user device relative to the physical location in the physical environment of the user device, and generate an image of the virtual content based on the posture of the user device relative to the placed virtual object, wherein the image of the virtual content is projected by the one or more displays of the user device at a predetermined location relative to the physical location in the physical environment of the user device.

[0007] Embodiments of the system may include one or more of the following features. Some embodiments include a second virtual content system comprising: a structure including a plurality of radiation emitters arranged in a predetermined pattern, and a second user device, wherein the second user device comprises: one or more sensors configured to sense outputs of the plurality of radiation emitters, and one or more displays; one or more hardware processors; and a non-transitory machine-readable storage medium encoded with instructions executable by the one or more hardware processors to: determine a posture of the second user device relative to the structure based on the sensed outputs of the plurality of radiation emitters, and generate a second image of virtual content based on the posture of the second user device relative to the structure, wherein the second image of the virtual content is projected by the one or more displays of the second user device at a predetermined position relative to the structure. In some embodiments, the first virtual content system is remote from the second virtual content system. In some embodiments, the first virtual content system is proximate to the second virtual content system. In some embodiments, the instructions may also be executable by the one or more hardware processors to: generate a virtual agent of the first user device: and add the virtual agent to the second image based on (i) the posture of the first user device relative to the virtual object, wherein the image is projected by the one or more displays of the second user device. In some embodiments, the instructions are further executable by one or more hardware processors to: generate a virtual agent for the second user device; and based on (i) a posture of the second user device relative to the structure, and (ii) a posture of the placed virtual object relative to the structure, add the virtual agent to the first image, wherein the image is projected by one or more displays of the first user device. In some embodiments, the plurality of radiation emitters are configured to emit infrared light; the one or more sensors on the second user device are configured to sense the output of the plurality of radiation emitters; and the instructions are further executable by one or more hardware processors in the second user device to determine the posture of the second user device relative to the structure based on the sensed output of the plurality of radiation emitters. In some embodiments, the instructions are further executable by one or more hardware processors to perform at least one of the following in response to user input received by the user device: selecting virtual content, and modifying the virtual content. In some embodiments, the second user device can be worn on the user's head. Some embodiments include a handheld electronic device; wherein the instructions are further executable by one or more hardware processors to perform at least one of the following in response to user input received by the handheld electronic device: selecting virtual content, and modifying the virtual content. In some embodiments, the second user device further includes at least one optical element that facilitates viewing the second image combined with a view of the physical environment. In some embodiments, the structure is attached to the physical object.Some embodiments include a second system configured to display virtual content, comprising: a second user device, the second user device comprising: a second set of one or more sensors configured to sense data related to a physical environment of the second user device, and a second set of one or more displays; wherein the instructions are further executable by one or more hardware processors to: in response to user input received by the second user device, place a second virtual object in a 3D scene displayed by the second user device at a virtual location corresponding to a physical location in the physical environment of the second user device, determine a posture of the second user device relative to the physical location in the physical environment of the second user device, and generate a second image of the virtual content based on the posture of the second user device relative to the placed virtual object, wherein the second image of the virtual content is displayed by the one or more displays of the second user device at a predetermined location relative to the placed second virtual object. In some embodiments, the first virtual content system is remote from the second virtual content system. In some embodiments, the first virtual content system is proximate to the second virtual content system.

[0008] In general, one aspect of the disclosure features a non-transitory machine-readable storage medium encoded with instructions executable by one or more hardware processors to, for a user device in a system configured to display virtual content: sense a representation of a physical environment of the user device; in response to user input received by the user device, place a virtual object in a 3D scene displayed by a second user device at a virtual location that corresponds to a physical location in the physical environment of the user device; determine a posture of the user device relative to the physical location in the physical environment of the user device; and generate an image of the virtual content based on the posture of the user device relative to the placed virtual object, wherein the image of the virtual content is projected by one or more displays of the user device at a predetermined location relative to the physical location in the physical environment of the second user device.

[0009] Embodiments of the non-transitory machine-readable storage medium may include one or more of the following features. In some embodiments, the instructions may be further executed by one or more hardware processors to, for a second user device in a second system configured to display virtual content: sense outputs of a plurality of radiation emitters arranged in a predetermined pattern in a structure; determine a posture of the second user device relative to the structure based on the sensed outputs of the plurality of radiation emitters, and generate a second image of the virtual content based on the posture of the second user device relative to the structure, wherein the second image of the virtual content is projected by one or more displays of the second user device at a predetermined position relative to the structure. In some embodiments, the instructions may also be executed by one or more hardware processors to: generate a virtual agent for the first user device; add the virtual agent to the second image based on the posture of the first user device relative to the virtual object, wherein the image is projected by one or more displays of the second user device. In some embodiments, the instructions may be further executed by one or more hardware processors to: generate a virtual agent for the second user device; add the virtual agent to the first image based on the posture of the second user device relative to the structure, wherein the image is displayed by one or more displays of the first user device. In some embodiments, the plurality of radiation emitters are configured to emit infrared light; the instructions may be further executed by one or more hardware processors of the second user device to: sense outputs of the plurality of radiation emitters, and determine a posture of the second user device relative to the structure based on the sensed outputs of the plurality of radiation emitters. In some embodiments, the instructions may be further executed by one or more hardware processors to perform at least one of the following in response to user input received by the user device: selecting virtual content; and modifying the virtual content, wherein the second user device includes at least one optical element that facilitates viewing the second image in conjunction with a view of the second user's physical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present invention, according to one or more different embodiments, will be described in detail with reference to the following drawings. The drawings are provided for illustration purposes only and depict only typical or example embodiments.

[0011] Figure 1 An augmented reality collaboration system featuring a physical holographic board according to some embodiments of the disclosed technology is shown.

[0012] Figure 2 The use of an augmented reality collaboration system featuring a physical holographic plate in conjunction with a computer-aided design (CAD) system according to some embodiments of the disclosed technology is shown.

[0013] Figure 3 is a perspective view of the interior of a physical holographic plate according to some embodiments of the disclosed technology.

[0014] Figure 4is a perspective view of a head device according to some embodiments of the disclosed technology.

[0015] Figure 5 is a perspective view of an add-on device according to some embodiments of the disclosed technology.

[0016] Figure 6 is a perspective view of a user device with additional devices attached according to some embodiments of the disclosed technology.

[0017] Figure 7 An augmented reality collaboration system featuring a virtual holographic board according to some embodiments of the disclosed technology is shown.

[0018] Figure 8 Some example networked systems according to embodiments of the disclosed technology are shown.

[0019] Fig. 9 is a flow chart of a process for a physical holographic system according to some embodiments of the disclosed technology.

[0020] Fig.10 is a flow chart of a process of a virtual holographic system according to some embodiments of the disclosed technology.

[0021] Fig.11 is a flow chart of a process for providing a virtual agent of a user in a first holographic system to a second holographic system user according to an embodiment of the disclosed technology.

[0022] Fig.12 are example computing components that can be used to implement various features of the embodiments described in this disclosure.

[0023] The drawings are not exhaustive and do not limit the invention to the precise form disclosed. DETAILED DESCRIPTION

[0024] As computing and networking technologies continue to advance, computer systems are providing better ways for people to collaborate virtually, whether in the same place or in remote locations. At the same time, augmented reality systems are providing better ways for people to view and modify three-dimensional content. However, attempts to combine these systems have proven unsuccessful. Current systems do not provide adequate visual quality, often offering low image quality and an inadequate field of view. These systems also often lack the ability to provide close-up views and are not suitable for users who wear glasses. And these systems do not allow users to see each other's eyes, further weakening the collaborative experience.

[0025] In addition, these systems provide users with inadequate input and control features. Current systems are often complex and ergonomically challenging, making text input difficult, for example, and new users face a steep learning curve. In addition, current systems require extensive preparation before use. For example, many of these systems typically require data preparation, software development, gesture recognition technology, and / or time-consuming room and surface scanning before and during each use—as well as computing power to perform all of these functions. Finally, many of these systems are incompatible with current collaboration systems, such as existing collaborative conferencing solutions.

[0026] In response to these challenges, some embodiments of the disclosed technology provide an augmented reality collaboration system, characterized by a physical device that specifies the projection location for virtual content displayed by the collaboration system, also referred to herein as a physical "holographic board". In these embodiments, images of virtual content are displayed to users at predetermined locations relative to the physical holographic board, so that the virtual content resembles a hologram projected by the holographic board. Users can view and interact with virtual content using a variety of user devices, including head devices, tablet computers, smart phones, controllers, gestures, and gesture recognition coupled to a virtual control system. Each physical holographic board not only provides position and orientation information to the system, but also provides an intuitive physical reference point for the user. While traditional two-dimensional collaboration systems clearly indicate where virtual content will appear, shared holographic experiences can appear anywhere, so it is not clear where users should look. Through the disclosed holographic board, the disclosed technology acts as a shared 3D monitor, making it clear to users of the collaboration system where virtual content will appear, thereby contributing to the purpose and effectiveness of the system. In some embodiments, the disclosed technology allows 3D video conferencing with holographic boards at each location. These embodiments make users distributed in different regions feel that they are all collaborating at the same table.

[0027] In some embodiments, each physical holographic plate can be a structure including multiple radiation emitters, such as infrared emitters, arranged in a known pattern. The holographic plate can be placed in a central location where users can gather in a single location, such as a table in a conference room. Sensors on each user device can detect emissions from the emitters to determine the pose of the head device relative to the holographic plate. As used herein, the "pose" of an object refers to its spatial relationship to another object, including position and orientation.

[0028] In other embodiments, sensors on the user device may use object or image recognition techniques to determine the pose of the head device relative to the physical holographic plate. In such embodiments, the physical holographic plate need not include a radiation emitter.

[0029] In any embodiments, determining posture may include processing the output of an inertial monitoring unit (IMU) in a user device. For example, a user device in the form of a head device may include an IMU. Sensors such as cameras typically allow for positioning at a relatively low frequency of approximately 60 Hz. In contrast, an IMU can provide position changes at a very high rate, such as at a frequency of 200 Hz or more. The fusion of these two technologies and their associated outputs enables the collaborative system to achieve highly accurate positioning while also being able to make highly accurate predictions of the user's future position.

[0030] The system can generate an image of the virtual content for each user device based on the posture of each user device relative to the physical holographic plate. The user device can display the image at a predetermined position relative to the physical holographic plate. For example, the virtual content may appear to the user to be hovering above the holographic plate, as if the holographic plate is projecting a hologram of the virtual content.

[0031] Some embodiments of the disclosed technology provide an augmented reality collaboration system featuring a virtual holographic board. In these embodiments, users can view and interact with virtual content using various user devices (including head devices, tablets, smart phones, etc.) without using a physical holographic board. Instead of placing a physical structure on the surface, users can place a virtual holographic board on a surface using a user device and surface tracking software (such as ARKit or ARCore). These embodiments are ideal for environments where physical holographic boards may not be available or inconvenient to use, such as hotel rooms. Like a physical holographic board, a virtual holographic board is a visual reference point for the user and a means to keep the coordinate system of the user environment consistent with the coordinate system of a remote system including a physical or virtual holographic board. Like a physical holographic board, the system generates an image of the virtual content to be displayed on the user device at a predetermined position relative to the virtual holographic board. For example, the virtual content may appear to the user to be hovering above the virtual holographic board, as if the virtual holographic board is projecting a hologram of the virtual content. Regardless of the user's position, and whether a physical or virtual holographic board is being used, all users see the virtual content at the same position relative to the holographic board, creating the system perception of presenting a shared 3D / holographic monitor.

[0032] In some embodiments, one or more of the disclosed augmented reality systems may be interconnected to share virtual content for communication or collaboration. For example, the interconnected systems may be networked and may include one or more physical holographic systems and one or more virtual holographic systems. In such embodiments, each user may have one or more permissions. For example, permissions may include permissions to view or edit virtual content. Users may change permissions during collaboration based on user input and / or specific parameters that may be set on the system at the beginning or during a collaborative session. Any system may be remote from one or more other systems. Any virtual holographic system may be collocated with other virtual or physical holographic systems.

[0033] Figure 1 An augmented reality collaboration system 100 featuring a physical holographic board according to some embodiments of the disclosed technology is shown. Figure 1 , the system 100 includes a physical holographic projection board 102 and a plurality of user devices used by three users 104a, b, c to view and interact with virtual content 106, which in this example represents a building. In this example, the users 104a, b use head devices 108a, b to view the virtual content 106, while the user 104c uses a tablet computer 110 to view the virtual content 106.

[0034] exist Figure 1 In the example of FIG. 1 , users 104b, c are local and physically present in the same room as the holographic panel 102, while user 104a is remote and in another location, as indicated by the dashed line. A virtual agent of remote user 104a can be displayed to local users 104b, c as part of the virtual content 106, such as described in detail below. Figure 1 In the example of FIG. 1 , the virtual agent represents the head device 108 a of the remote user 104 a .

[0035] As described above, the posture of each user device relative to the physical holographic plate 102 can be determined by sensing emitters on the holographic plate or by object or image recognition. To facilitate the use of emitters, the tablet 110 can include sensors, or can be equipped with additional devices including sensors, such as described below.

[0036] The system may include one or more computer systems 116, each having a memory and one or more processors that execute instructions stored in the memory to generate an image of the virtual content 106 for display to the user 104 on a user device that may be connected to the computer system 116 for power and communication. For example, a head device may be connected to a computer system, while a mobile device may run on its own battery and computing power. In some embodiments, some or all of the processing may occur at a user device that may be battery powered and include any and all processors and communication devices required to operate in a collaborative system. In some embodiments, the user device may have a wired connection to a computer, for example to provide high image quality, computing power, and / or power. In some embodiments, the user device may have a wireless connection to a computer, such as Bluetooth, WiFi, etc., for example to allow increased mobility of the user 104.

[0037] Users 104 may use their user devices or peripheral devices directly or indirectly connected to their user devices to interact with virtual content 106. For example, a smartphone may include an application that provides tools for interacting with virtual content. Figure 1 In the example of FIG. 1 , user 104 b is shown using smart phone 112 b to interact with content using beam 114. In some embodiments, beam 114 may be a virtual beam generated by the system and visible on the user device. Interaction with virtual content may include selection of virtual content, modification of virtual content, annotation of virtual content, etc.

[0038] In some embodiments, other users 104a, c may also interact with the virtual content 106, either sequentially or simultaneously. For example, user 104a may use smartphone 112a to interact with the virtual content 106, while user 104c may use tablet 110. In some embodiments, users 104a, b may use their head devices 108a, b to interact with the virtual content 106. In such embodiments, the head device may include a user interface including tactile controls, gaze controls, voice commands, etc. As described below, in a gesture control system, the head device 108 may include sensors for detecting gestures.

[0039] In some embodiments, the disclosed system can be used in combination with other systems. For example, the system can include an application for creating virtual content using 3D data from existing 3D tools such as computer-aided design (CAD) systems. Figure 2 The use of an augmented reality collaboration system 200 featuring a physical holographic plate in conjunction with a CAD system is shown in accordance with some embodiments of the disclosed technology. Figure 2 , system 200 includes a physical holographic plate 202 and a head device 208 for viewing virtual content 206, which in this example represents a rocket engine. In this example, system 200 also includes a CAD system 220. The view generated by the CAD system 220 can be coordinated with the view of the virtual content 206. For example, the manipulation of one view can affect both views, such as in real time.

[0040] Although in Figure 2 The physical holographic plate 202 is shown as a stand-alone unit, but the disclosed physical holographic plate can take any physical form and can be attached to or integrated with one or more physical objects. Figure 2 , three examples are shown. Physical holographic plate 202 is a cross-shaped design. Physical holographic plate 210 is attached to or integrated with the frame of a computer monitor. Physical holographic plate 212 is attached to or integrated with bracket 214 for fixing head device 208.

[0041] Figure 3 300 is a perspective view of the interior of a physical holographic plate 300 according to some embodiments of the disclosed technology. In these embodiments, the holographic plate has a cross-shaped design and has four radiation emitters 302a, b, c, d arranged in corresponding arms of the physical holographic plate 300. However, as described above, the disclosed physical holographic plate can take any physical shape. In some embodiments, the emitters 302 are arranged in a predetermined pattern known to the system. These arrangements help determine the posture of the user device relative to the physical holographic plate 300.

[0042] Figure 4 is a perspective view of a head device 400 according to some embodiments of the disclosed technology. Figure 4 , the head device 400 may include a sensor 402 configured to capture the emission of the emitter of the physical holographic plate. For example, when the emitter is an infrared emitter, the sensor 402 may include an infrared camera configured to capture an image of the emitter.

[0043] Head device 400 may include one or more optical elements. The optical element may be implemented as any reflective semi-transparent optical device, such as shown at 404. For example, optical element 404 may be implemented as an off-axis optical device, a basin optical device, a prismatic optical device, a planar optical device (i.e., a "waveguide"), etc. In some of these embodiments, the optical element is designed to allow other juxtaposed users to see the wearer's eyes. The semi-transparent optical device can be replaced by the user with a reflective opaque optical device to obtain a virtual reality experience.

[0044] In some embodiments, optical element 404 can be replaced or enhanced by a display to allow for a direct augmented reality or virtual reality experience. In such embodiments, the display can be an LCD, LED, OLED, or other display in a user device located in front of the user's eyes. In such embodiments, the display facilitates viewing images of virtual content combined with the physical environment. In some of these embodiments, head device 400 includes a camera for capturing images of the physical environment, and these images are added to the images of the virtual content so the wearer sees the virtual content integrated with the physical environment. In such embodiments, the front of the head device can be opaque.

[0045] The head device 400 may include a head attachment mechanism 406. The head attachment mechanism 406 may be implemented in any manner. For example, the head attachment mechanism 406 may be implemented as a headband, etc. In some embodiments, the optical element 404 is positioned so that the head device 400 provides sufficient space to accommodate a user wearing glasses.

[0046] In some embodiments, the user device for viewing virtual content may be implemented as a handheld computer, such as a smart phone, tablet computer, etc. In some embodiments, these handheld user devices may include sensors capable of detecting the emitters of the physical holographic board. In some embodiments, where the user device may not be capable of such sensing, some embodiments may provide or be configured to accept additional devices capable of performing sensing.

[0047] Figure 5 is a perspective view of an additional device 500 according to some embodiments of the disclosed technology. Figure 5 , the additional device 500 may include an attachment device 502 configured to attach the additional device 500 to the user device. Figure 5 In the example of FIG. 5 , the attachment device 502 is implemented as a clamp. However, the attachment device 502 can be attached to the user device in any manner.

[0048] Figure 6 is a perspective view of a user device 600 with an additional device 500 attached according to some embodiments of the disclosed technology. Figure 6 In the example of FIG. 6 , the user device is implemented as a smart phone. However, the user device 600 may be any device capable of performing the functions described herein.

[0049] In some embodiments, the additional device 500 may be configured to communicate wirelessly with the user device 600. For example, the additional device 500 may include a Bluetooth transceiver, an NFC device, a WiFi device, etc. In some embodiments, the additional device 500 may have a wired communication connection to the user device 600. For example, the additional device 500 may have a connector configured to connect to a connector of the user device 600. In some embodiments, there is no direct wired or wireless communication connection between the additional device 500 and the user device 600. Instead, the additional device 500 may communicate to a computer network to which the user device 600 is also connected, such as via a cellular or Internet connection of the user device (e.g., via WiFi). In some embodiments, the additional device 500 may include an NFC tag to pair the additional device 500 and the user device 600 for direct communication. For example, a user may enter a conference room, pick up the additional device 500, which is paired with the user device 600, identify himself to the user device 600, and the user device 600 automatically opens the controller application on the user device 600.

[0050] In some embodiments, the additional device 500 can have a power source that is independent of the power source of the user device 600. In some embodiments, the additional device 500 can be configured to draw power from the user device 600. In some embodiments, the additional device 500 can be configured to provide power to the user device 600. All combinations of the described configurations are contemplated.

[0051] Figure 7 An augmented reality collaboration system 700 featuring a virtual holographic board 702 is shown according to some embodiments of the disclosed technology. Figure 7 , system 700 includes a user device that a user can use to view and interact with virtual content 706, which in this example represents a rocket engine. Figure 7 In the example of FIG. 7 , the user device is implemented as a tablet computer 710. However, the user device may be any device capable of performing the functions described herein. For example, the user device may be implemented as a head device, a smart phone, a notebook computer, etc.

[0052] The user can use the tablet computer 710 to place virtual objects in the 3D scene displayed by the tablet computer 710 in virtual locations that correspond to physical locations in the user's physical environment. In this example, the user has placed the virtual holographic plate 702 in a virtual location that corresponds to the surface of the table 720. Of course, the user can also place the virtual holographic plate 702 in virtual locations that correspond to other surfaces in the user's physical environment.

[0053] Like a physical holographic plate, the virtual holographic plate serves as a reference point for the user. In some embodiments, the system can determine the posture of the tablet computer 710 relative to the physical surface indicated by the virtual holographic plate 702. For example, an application executing on the tablet computer 710 can use one or more sensors to scan and track surfaces in the physical environment and determine the physical position indicated by the virtual holographic plate relative to the tablet computer 710.

[0054] Based on the posture of the tablet 710 relative to the surface of the table 720 selected by the user and enabled by the surface tracking software implemented by the tablet 710, the system generates an image of the virtual content 706 and the virtual holographic plate 702. The tablet 710 can then display the image at a predetermined position relative to the virtual holographic plate 702. For example, the virtual content may appear to the user to be hovering above the virtual holographic plate, as if the holographic plate is projecting a hologram of the virtual content. Figure 1 The image may include an image of the virtual holographic plate 702. The image may include an image of the physical environment, such as Figure 7 As shown, the image includes an image of table 720.

[0055] Figure 8 Some example networking systems according to embodiments of the disclosed technology are shown. One or more systems may include a wireless router to connect one or more components of the system. The wireless router may use any wireless technology, including, for example, WiGig, WiFi6, 5G millimeter wave, etc.

[0056] Several example system configurations are shown, interconnected by a network 812. However, it should be understood that other system configurations may be employed and interconnected in any number. Although the user devices are shown as head devices, any type of user device may be employed. At 804, a configuration including two users and a physical holographic board in a room is shown. In configuration 804, the two users are viewing the same virtual content. At 806, a configuration including two users in a room and a first physical holographic board and one user in a second room and a second physical holographic board is shown. In configuration 806, all three users are viewing the same virtual content. At 808, a configuration including one user in a room and a first physical holographic board and one user in a second room and a second physical holographic board is shown. In configuration 808, the two users are viewing the same virtual content. At 810, a configuration including one user and a virtual holographic board (indicated by a dotted line) in a room is shown.

[0057] Fig. 9 is a flow chart of a process 900 for a physical holographic system according to some embodiments of the disclosed technology. For example, the process 900 may be Figure 1For clarity, process 900 is described for one user device. However, process 900 may be performed for each user device in the physical holographic system.

[0058] Reference Fig. 9 , process 900 may include, at 902, sensing outputs of a plurality of radiating emitters arranged in a structure. Figure 1 In the example of FIG. 1 , each of the head devices 108 may include one or more sensors, such as Figure 4 Sensor 402. In a gesture control system, the head device 108 may include additional sensors for detecting gestures. Each user device may include a sensor, such as integrated in the user device or as a component such as Figure 5 The additional device 500 is a part of the additional device. The emitter may be an infrared emitter, and the sensor may be a camera that captures an image of the infrared emitter.

[0059] Refer again Fig. 9 , process 900 may include, at 904, determining a posture of the user device relative to the structure based on the sensed output of the radiating emitter and the output of the IMU. Figure 1 For example, the computer system(s) 116 can determine a posture of the user device 108 relative to the physical holographic plate 102.

[0060] Refer again Fig. 9 , process 900 may include, at 906, generating an image of the virtual content based on the posture of the user device relative to the structure. Figure 1 For example, the computer system(s) 116 may generate an image of the virtual content 106 for the user device 108 based on a posture of the user device 108 relative to the holographic panel 102 .

[0061] Each user device can have a different posture relative to the holographic panel 102. Therefore, each image can be different, and virtual content viewed from the corresponding posture can be presented. The system can fix the orientation of the virtual content relative to the holographic panel 102, and the image can be updated frequently. For example, complex algorithms (such as fusion of cameras and inertial sensors and predictions based on head motion models) can be used to refresh the image at a high rate. Therefore, the virtual content may appear to remain stationary, so that the user can move around the room to view the virtual content from different perspectives.

[0062] For user devices that allow stereoscopic viewing, the system can generate a pair of images. Such user devices can simultaneously display the pair of images as a stereoscopic image of the virtual content.

[0063] Refer again Fig. 9, process 900 may include, at 908, displaying an image on an optical element of a user device in a predetermined position relative to the structure. Figure 1 In an example of , computer system 116 can transmit the image to the user device, and in response to receiving the image, the user device can display the image on its display or optical element. Process 900 can be repeated to provide a seamless viewing experience for the user.

[0064] Fig.10 is a flow chart of a process 1000 of a virtual holographic system according to some embodiments of the disclosed technology. For example, the process 1000 may be Figure 7 For the sake of clarity, process 1000 is described for one user device. However, process 1000 may be performed for each user device in the virtual holographic system.

[0065] Reference Fig.10 , process 1000 may include capturing data related to a physical environment at 1002. Figure 7 In the example of , the tablet computer 710 may include a sensor for capturing an image. The sensor may be implemented as a camera, a depth sensor, etc. or a combination thereof.

[0066] Refer again Fig.10 ,exist Figure 7 In the example of , the tablet computer 710 may include a sensor for capturing an image. The sensor may be implemented as a camera, a depth sensor, etc. or a combination thereof.

[0067] Refer again Fig.10 , process 1000 may include determining a posture of the user device relative to the physical environment at 1003, and placing a virtual object in a 3D scene displayed by the user device at a virtual location corresponding to a physical location in the physical environment in response to user input received by the user device at 1004. Figure 7 For example, a user can use a tablet computer 710 to place a virtual holographic plate 702 on a table 720. Then, even if the tablet computer 710 is moved by the user, the virtual holographic plate 702 appears to remain in that position.

[0068] Refer again Fig.10 , process 1000 may include determining, at 1006, a posture of the user device relative to the physical location of the placed virtual object. Figure 7In an example of FIG. 7 , tablet computer 710 may determine a posture of tablet computer 710 relative to virtual holographic plate 702. That is, tablet computer 710 may determine a posture of tablet computer 710 relative to a physical location on table 720, where virtual holographic plate 702 has been placed in the virtual environment. The posture may be determined in any suitable manner. For example, the posture may be determined by a commercial application such as ARKit or ARCore that creates a virtual map of a surface and other significant physical features of a user's physical environment, and the application is programmed to recognize the user's physical environment.

[0069] Refer again Fig.10 , process 1000 may include, at 1008, generating an image of the virtual content based on a posture of the user device relative to the physical position of the virtual object. Figure 7 For example, tablet computer 710 can generate an image of virtual content 706 for the user based on the posture of tablet computer 710 relative to the physical location corresponding to where the user places the virtual holographic plate. The image presents the virtual content 706 viewed from that posture. The system can fix the orientation of the virtual content relative to the virtual holographic plate 702, and can frequently update the image. Thus, the virtual content appears to remain stationary, allowing the user to move around the room to view the virtual content from different angles.

[0070] For user devices that allow stereoscopic viewing, the system can generate a pair of images. Such user devices can simultaneously display the pair of images as a stereoscopic image of the virtual content.

[0071] Refer again Fig.10 , process 1000 may include, at 1010, displaying an image on a display or optical element of a user device at a predetermined location relative to a physical location of a placed virtual object. Figure 7 For example, the image can be displayed as Figure 7 The image may be displayed on a tablet computer 710 as shown. In other embodiments, the image may be displayed on other user devices, such as a head device, a smart phone, a laptop computer, etc. The process 1000 may be repeated to provide a seamless viewing experience for the user.

[0072] In some embodiments, users can share their virtual content images with other users. For example, when explaining a specific view of virtual content, a presenter can share the presenter's view of the virtual content so that other users see the same view.

[0073] In some embodiments, each networked system may include representations of users and / or user devices in other systems in the image provided to its local user. These representations, referred to herein as "virtual agents," may take any form. For example, a user's virtual agent may simply represent the user device currently being used by the user. However, the virtual agent may be represented in any manner. In some embodiments, the user's movement may be tracked to provide a more realistic virtual agent. For example, the system may track the movement of the user's head, hands, body, eyes, etc.

[0074] Fig.11 is a flow chart of a process 1100 for providing a virtual agent of a user in a first holographic system to a second holographic system user according to an embodiment of the disclosed technology. Either of the first and second holographic systems may be a virtual holographic system or a physical holographic system.

[0075] Reference Fig.11 , process 1100 may include generating a virtual proxy of a user device in a physical holographic system at 1102. The virtual proxy may include one or more three-dimensional objects. The virtual proxy may represent one or more user devices, users, control devices being used by users, or any combination thereof.

[0076] Refer again Fig.11 , process 1100 may include adding a virtual agent to an image of the virtual content for the second holographic system at 1104. In some embodiments, the virtual agent is added to the image of the virtual content based on a gesture of the user device relative to the holographic plate in the first holographic system. The use of such gestures preserves the spatial relationship between the user and the virtual content, so that the relationship is replicated by the virtual agent in the second holographic system.

[0077] In some embodiments, the first holographic system may add the virtual agent to images of the virtual content and provide those images to the second holographic system. Alternatively, the first holographic system may transmit the virtual agent or information representing the virtual agent to the second holographic system along with information describing the posture of the user device relative to the structure, thereby enabling the second holographic system to generate images of the virtual content including the virtual agent.

[0078] Finally, process 1100 may include displaying the image in the second holographic system at 1106. For example, the image may be displayed on one or more user devices used by a user of the second holographic system to view virtual content.

[0079] In some embodiments, the disclosed systems may collaborate to present virtual content in the same geographic orientation at each system. In such an embodiment, each system may use compass directions to orient the virtual content. For example, virtual content representing a building may be oriented so that the north side of the building faces north. In such an embodiment, the physical holographic plate may include a compass. In some embodiments, the compass is visible to the user so the user can rotate the holographic plate to face north. In some embodiments, the compass may include a transmitter to send directional information to a device that generates an image of the virtual content, which may use the directional information to rotate the virtual content accordingly. In some embodiments, the physical holographic plate may include design elements that indicate how the holographic plate should be placed in a room to provide the audience with a desired viewing angle of the virtual content. For example, the top of the holographic plate may have an arrow with the word "audience" to indicate the correct orientation.

[0080] These techniques can also be used for positioning of virtual agents. For example, when a user of a first holographic panel is located to the west of the virtual content, a virtual agent can be presented in a second holographic system to the west of the virtual content. In some cases, users in multiple systems may occupy the same position relative to the virtual content. Superimposing virtual agents of these users may be disconcerting to the users. Therefore, in some embodiments, the system can display virtual agents so that they do not appear on top of each other.

[0081] It may be desirable to match the lighting conditions of the virtual content and virtual agent to the lighting conditions where the virtual content will be displayed. Thus, in some embodiments, the physical holographic panel and / or the user device may include a camera for detecting ambient lighting conditions. The camera may have a wide field of view, such as with a fisheye lens. The computing system and / or device that generates the image of the virtual content may then adjust the lighting of the virtual content accordingly.

[0082] Fig.12 A block diagram of an example computer system 1200 is depicted in which embodiments described herein may be implemented. Computer system 1200 includes a bus 1202 or other communication mechanism for communicating information, one or more hardware processors 1204 coupled to bus 1202 for processing information. Hardware processor 1204 may be, for example, one or more general purpose microprocessors.

[0083] The computer system 1200 also includes a main memory 1206, such as a random access memory (RAM), cache, and / or other dynamic storage device, coupled to the bus 1202 for storing information and instructions to be executed by the processor 1204. The main memory 1206 may also be used for storing temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1204. These instructions, when stored in a storage medium accessible to the processor 1204, present the computer system 1200 as a special-purpose machine that is customized to perform the operations specified in the instructions.

[0084] The computer system 1200 also includes a read only memory (ROM) 1208 or other static storage device coupled to the bus 1202 for storing static information and instructions for the processor 1204. A storage device 1210, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 1202 for storing information and instructions.

[0085] The computer system 1200 may be coupled to a display 1212, such as a liquid crystal display (LCD) (or touch screen), via the bus 1202 for displaying information to a computer user. An input device 1214, including alphanumeric and other keys, is coupled to the bus 1202 for communicating information and command selections to the processor 1204. Another type of user input device is a cursor control 1216, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1204 and for controlling cursor movement on the display 1212. In some embodiments, the same direction information and command selections as cursor control may be achieved via receiving touches on a touch screen without a cursor.

[0086] The computing system 1200 may include a user interface module for implementing a GUI, which may be stored in a mass storage device as executable software code executed by the computing device. This module and other modules may include, for example, components, such as software components, object-oriented software components, class components, and task components, procedures, functions, properties, programs, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0087] In general, as used herein, the terms "component", "engine", "system", "database", "data store", etc. may refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly with entry and exit points, written in a programming language such as Java, C, C++, and Python. Software components may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as BASIC, Perl, or Python. It should be understood that software components may be called from other components or from themselves, and / or may be called in response to detected events or interrupts. Software components configured for execution on a computing device may be provided on a computer-readable medium, such as a compact disc, digital video disc, flash drive, disk, or any other tangible medium, or as a digital download (and may be initially stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored in part or in whole on a memory device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It should also be understood that hardware components may include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0088] The computer system 1200 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, causes or programs the computer system 1200 to become a special purpose machine. According to an embodiment, the techniques herein are performed by the computer system 1200 in response to the processor 1204 executing one or more sequences of one or more instructions contained in the main memory 1206. Such instructions may be read into the main memory 1206 from another storage medium such as the storage device 1210. Execution of the sequences of instructions contained in the main memory 1206 causes the processor 1204 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0089] As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 1210. Volatile media include dynamic memory, such as main memory 1206. Common forms of non-transitory media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes, and network versions of the same.

[0090] Non-transitory media are distinct from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between non-transitory media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up bus 1202. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0091] The computer system 1200 also includes a communication interface 1218 coupled to the bus 1202. The network interface 1218 provides a two-way data communication coupled to one or more network links, which are connected to one or more local networks. For example, the communication interface 1218 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the network interface 1218 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or a WAN component that communicates with a WAN). Wireless links can also be implemented. In any such implementation, the network interface 1218 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0092] A network link typically provides data communication to other data devices through one or more networks. For example, a network link may provide a connection to a host computer or to data equipment operated by an Internet Service Provider (ISP) through a local network. The ISP, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network links and the signals through the communication interface 1218 are example forms of transmission media that carry digital data to and from the computer system 1200.

[0093] Computer system 1200 can send messages and receive data, including program code, through the network, network link and communication interface 1218. In the Internet example, a server may transmit the requested code for an application program through the Internet, ISP, local network and communication interface 1218.

[0094] The received code may be executed by processor 1204 as it is received, and / or stored in storage device 1210 or other non-volatile storage for later execution.

[0095] Each process, method, and algorithm described in the preceding section may be embodied in a code component executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by the code component. One or more computer systems or computer processors may also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). These processes and algorithms may be implemented in part or in whole in a dedicated circuit system. The various features and processes described above may be used independently of each other, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of the present invention, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be executed in other suitable orders, or may be executed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The execution of certain operations or processes may be distributed between computer systems or computer processors, not only residing in a single machine, but deployed on multiple machines.

[0096] As used herein, any form of hardware or a combination of hardware and software may be used to implement a circuit. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms may be implemented to form a circuit. In implementation, the various circuits described herein may be implemented as discrete circuits, or the functions and features described may be partially or completely shared between one or more circuits. Although various features or functional elements may be described or declared as separate circuits separately, these features and functions may be shared between one or more common circuits, and such descriptions should not require or imply that separate circuits are needed to implement such features or functions. In the case where the circuit is implemented in whole or in part using software, such software may be implemented to operate together with a computing or processing system capable of performing the functions described therein, such as computer system 1200.

[0097] As used herein, the term "or" may be interpreted as inclusive or exclusive. In addition, descriptions of resources, operations, or structures in the singular should not be understood to exclude the plural. Conditional language, such as "can / could / might / may", unless otherwise specifically stated or otherwise understood in context, is generally intended to convey that some embodiments include, while other embodiments do not include, certain features, elements, and / or steps.

[0098] Unless expressly stated otherwise, the terms and phrases used in this document and variations thereof should be construed as open ended and not limiting. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be construed to limit the items described in a given time period to those available at a given time, but should be construed to include conventional, traditional, normal, or standard technology available or known at any time now or in the future. In some cases, there are broadening terms and phrases such as "one or more," "at least," "but not limited to," or other similar phrases, which should not be construed as intending or requiring a narrower context where these broadening phrases may not be present.

Claims

1. A system comprising: A virtual content system configured to display virtual content, the virtual content system comprising: User equipment, comprising: one or more sensors configured to sense data related to the physical environment of the user device, and one or more displays; one or more hardware processors; and a non-transitory machine-readable storage medium encoded with instructions executable by the one or more hardware processors to: In response to user input received by the user device, placing a virtual object in the 3D scene displayed by the second user device in a virtual position corresponding to a physical position in the physical environment of the user device, determining a posture of the user device relative to the physical location in the physical environment of the user device, and An image of virtual content is generated based on the posture of the user device relative to the placed virtual object, wherein the image of the virtual content is projected by the one or more displays of the user device in a predetermined position relative to the physical location of the physical environment of the user device.

Images