Multi-Location Augmented Reality

Abstract

A method and corresponding device for synchronizing the simultaneous rendering of the same augmented reality environment at multiple locations. The method enables multiple devices to obtain camera views of their respective local environments and maintain the augmented reality environment by integrating the camera views with virtual objects. At at least one of the multiple devices, information regarding the integration of the virtual objects, including any modifications or interactions, is transmitted to at least one other device of the multiple devices. At the receiving device, the corresponding object can be modified or updated according to the received information. In some embodiments, the device can generate a virtual representation of an object detected in the camera view, such as a person, and enable other devices to represent and update the object in their respective augmented reality environments.

Description

[Technical field]

[0001] The present invention relates to the synchronization of augmented reality content between several devices, allowing users who are not co-located to experience and interact with the same augmented reality content on their respective devices. [Background technology]

[0002] Augmented reality (AR) content is experienced by users by accessing AR content and viewing this content through a mobile app or AR glasses. Typically, each user accesses the AR content independently and therefore sees and interacts with their own specific AR content, and other users cannot affect the AR environment. A first user's interaction with the AR content on one mobile device is not reflected in other users' experience and interaction with the same AR content on their respective devices. However, shared AR is the concept of multiple users viewing the same AR content from multiple devices at the same time. That is, users are not viewing their own specific instance of the AR content through their own devices, but are experiencing the same unique instance of the AR content viewed through their respective devices. A further extension of this is multiplayer AR, which allows multiple users to not only view but also interact with the same AR content at the same time.

[0003] Software development kits (SDKs) are available that allow developers to create multiplayer AR apps, and such apps are commercially available. However, existing technologies have significant limitations and are therefore not widely used. For example, the latency associated with loading and interacting with AR content must be very small to provide a satisfying experience for the user. Other shortcomings relate to the manner in which real and virtual elements are overlaid to create a seamless AR environment. For example, because the parts of an AR scene that represent the actual reality are different in different locations, it may be desirable to represent objects that are part of the real environment in one location but not in another as virtual objects in another location. New solutions that address these needs are needed to create a more satisfying AR user experience in a synchronized manner across multiple locations. Summary of the Invention [Means for solving the problem]

[0004] The above-mentioned needs are addressed by a method for synchronizing the simultaneous rendering of the same augmented reality environment at multiple locations. According to the method, a camera view of a local environment is obtained. Objects of a predetermined type are detected in the camera view of the local environment and a virtual representation of the detected object is generated. The augmented reality environment is maintained by integrating the camera view of the local environment with the generated virtual representation of the detected object. Information regarding the generated virtual representation of the detected object and the integration of the virtual object is transmitted to at least one other device, enabling the other device to create a corresponding augmented reality environment based on the corresponding integration of its local camera view and the virtual object.

[0005] The information regarding the integration of the virtual objects may include information describing a change in at least one of a position and an orientation of the virtual objects.

[0006] At least one of the multiple devices may detect an object of a predetermined type within a camera view of the local environment and may generate a virtual representation of the detected object, which may be transmitted to at least one other device of the multiple devices to enable integration at the other device of a virtual object that corresponds to a real object in the environment of the first device.

[0007] The predetermined type of object may be a person in some embodiments, and the information relating to the integration of the virtual representation of the person may include a description of movements performed by the person.

[0008] The information transmitted to the at least one other device may be used by the at least one other device to integrate or change the integration of the corresponding virtual object with a camera view of the local environment.

[0009] Information from at least one device may in some embodiments be transmitted to a server configured to forward the received information to at least one other device. Processing of information regarding detection of objects, movements, interactions, etc. may be performed locally, to some extent by the recipient, or by an intermediate server or device in the cloud or at the edge. In this manner, information regarding the generation or update of information may be distributed between at least one device, the server, and at least one other device.

[0010] A device according to the invention may include modules and functions configured to obtain a camera view of a local environment, detect objects of a predetermined type within the camera view of the local environment, generate a virtual representation of the detected objects, and maintain a description of the augmented reality environment by integrating the camera view of the local environment with the generated virtual representation of the detected objects. Furthermore, such a device may be capable of transmitting the generated virtual representation of the detected objects and information regarding their integration into the augmented reality environment to at least one other device. Furthermore, such a device may be configured to receive information regarding the integration of the virtual objects from the at least one other device and update the augmented reality environment according to the received information. [Brief description of the drawings]

[0011] The invention will now be explained in more detail with reference to the drawings.

[0012] [Figure 1] 1 illustrates a system that can be configured to operate in accordance with the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating modules within a device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention introduces a new multi-location AR that provides techniques and tools that enable multiple users in different locations to interact with the same AR content simultaneously in an immersive manner.

[0014] Figure 1 illustrates a system configured for multi-location AR in accordance with the present invention. The multi-location AR functionality is provided by a mobile AR app installed on a mobile device 101 and configured to synchronize AR content with a multi-location AR backend 102 having a multi-location AR controller, or with other devices 101 running a mobile AR app with similar functionality.

[0015] The multi-location AR controller may be hosted in the cloud 103 or on the edge and provides tools and methods that enable the establishment of the necessary connections between devices.

[0016] The system allows users in different locations to access the same AR object or AR environment and experience each other's immersion at the same time. For example, if two users located in two different locations load the same AR content and see the same object on their mobile devices 101, the following is possible:

[0017] First, object synchronization ensures that objects are synchronized in all participating AR devices 101. This means that when a first user interacts with an object and thus changes the state of the object on his device 101A, the new state is communicated to the other device 101B, and the state is changed accordingly on device 101B so that the view that a second user has of the same object is updated. According to one aspect of the invention, multi-location AR includes an AR format that allows tracking of several parameters related to the AR object, such that not only the position and orientation can be tracked, but also features related to lighting, position relative to other objects, etc.

[0018] Second, scene synchronization provides improved immersion. When objects shared between two or more users are synchronized, they are displayed in a consistent manner on all devices. If one of the users brings himself or any real object into the AR scene, this is reflected in the camera view of the other users. That is, user A can stand in front of the camera on his mobile device 101A and adjacent to a virtual object. The camera on user A's mobile device 101A captures the real part of the AR scene including the user, and at the same time, the state of the virtual AR object is recorded. The functionality provided by the present invention includes detection algorithms configured to detect specific objects. Although embodiments of the present invention may be configured to detect various types of objects, the exemplary embodiments described herein focus on detecting humans. However, other embodiments may be configured to detect other types of objects, such as, for example, automobiles, game equipment, boxing gloves, etc.

[0019] The human detection algorithm may be configured to detect aspects of the human body, such as joint positions and depth sensing information. This information may be forwarded to User B's device 101B. User B's device 101B may then render an AR version of the detected human body in the camera view of the remote device 101A and place it next to the virtual object according to state information received from the remote mobile device 101A regarding both the position of the human body and the position of the virtual AR object in the scene determined by the human detection algorithm. Thus, the view of the AR scene provided on User B's device 101B is the same as that presented on User A's device 101A, except for the fact that on the first device 101A the human body is displayed as captured by the device camera, while on the second device 101B the human body is shown as a virtual AR object. Similarly, when a human enters the scene captured by the camera on User B's mobile device 101B, User A's device renders the scene with the human captured by User B's device 101B rendered as a virtual AR object.

[0020] FIG. 1 further illustrates a database 104, which may be a repository of AR content.

[0021] The various modules included in a system operating according to the present invention will now be described in more detail with reference to Figure 2. Although this example includes two mobile devices 101, embodiments of the present invention may be configured to include several devices. No loss of generality is made in the description of two devices, and those skilled in the art will readily understand how the present invention can be configured for additional devices.

[0022] A first multi-location AR app 201A is installed on a first mobile device 101A. The AR app 201A includes a camera module 202A configured to access, control, and capture images of a camera that is part of the device 101A. The multi-location AR app 201A further includes a multi-location AR module 203A that receives virtual AR objects from a remote device, as described in more detail below. The real environment captured by the camera module 202A and the virtual objects provided by the multi-location AR module 203A are combined in an AR content integration module 204A. The virtual AR content received by the integration module 204A includes metadata that describes the position and orientation of the virtual objects in the AR environment. This description may be absolute, relative to other objects, or determined by features detected in the scene received by the camera module 202A based on, for example, surface detection or image recognition techniques.

[0023] 2, the camera module 202A of the first device 101A captures a scene including a person. This person is a real part of the scene at the location of the first device 101A. In order for this person to be represented in the scene presented by the second device 101B, object recognition is performed and a virtual representation of the person is generated and transmitted to the multi-location AR controller 205 running on the backend server 102.

[0024] A similar process is executed in the second device 101B where the multi-location AR module 201B is running. In this example, a second person is present in the real scene and is captured by the camera module 202B of the second device 101B. A virtual representation of the second person is generated with respect to the first device and sent to the AR controller 205 running on the backend server 102.

[0025] The back-end server can maintain representations of all virtual objects in the shared AR environment, which in this example includes two virtual representations of real people, as well as a purely virtual object in the form of a cake. In the figure, the virtual objects are all represented in black, while the real objects are represented in white. In the environment maintained by the AR controller, all objects are virtual.

[0026] In some embodiments, the backend may not maintain a representation of the virtual object, but rather simply acts as an intermediary that distributes data to participating devices as soon as the data is available, but does not maintain this data in the form of any stored representation of the virtual object, hi other embodiments, information regarding synchronization is transmitted directly between participating devices in a peer-to-peer manner.

[0027] Virtual objects are shared between participating devices 101. However, objects that are actually present in the local environment do not need to be presented as virtual objects in the local presentation of the AR environment. Thus, in the first device 101A, the multi-location AR module 203A maintains only a representation of a virtual object that is entirely virtual (in this example, a cake) or a virtual representation of a real object in a different location (in this example, a person captured by a camera module on a remote device). As a result, the multi-location AR app 201A on the first device 101A uses its AR content integration module 204A to generate an AR scene with a representation of a present person provided by the camera module 202A and a virtual representation of the cake and the remote person. This scene is presented by the local device 101A on its display 206A or connected AR goggles or some similar device.

[0028] The same applies to the second device 101B, except that the cake and the first person are given a virtual representation while the other person is captured by the camera and provided and presented by the camera module 202B.

[0029] Various techniques may be used, such as surface detection, object recognition, etc., which are well known in the art and will not be described in detail herein.

[0030] The actual sharing of the AR environment can be facilitated in several ways known in the art and not specific to augmented reality. For example, a unique QR code or link shared by one user to another via message, or in an app consisting of a list of friends, the user can simply invite friends to share / view the AR content, which is then activated. Once sharing is activated, any AR content is synced with participating users.

[0031] When a connection between two or more devices is established, any changes to objects in the scene or the scene itself are synchronized. This synchronization is provided as a service by the multi-location AR controller 205. When multi-location AR sharing is activated, the synchronization function provides state synchronization between objects seen by the user. In the case of object synchronization, the synchronization process allows objects and interactions with objects to be synchronized with other devices using information rigid body information, shaders, vector positions, etc.

[0032] The synchronization of the scenes themselves is based on object recognition algorithms. As already mentioned, different embodiments can be configured to recognize and synchronize different objects. This example describes the recognition of human bodies, but the principles are similar for the detection and recognition of other kinds of objects.

[0033] The human body recognition algorithm detects and tracks human bodies in the scene. In some embodiments, the multi-location AR app 201 detects, tracks, and forwards live data. This means that human bodies detected in the images provided by the camera module 202 are tracked and segmentation is performed in near real-time. The resulting data is sent to the multi-location AR controller 205, from where it is distributed to the participating devices. (In some embodiments, the devices operate in peer-to-peer mode and information is sent directly to other participating devices.) The data forwarded to the participating devices typically includes vector position information.

[0034] Upon receiving the resulting tracking data and segmented human data, the participating devices render it on the display 206 of the receiving device 101. In the process of rendering, the device can apply Gaussian blur to generate a 3D human figure and properly position it in the scene according to the received vector data. Thus, in this method, the processing is done in near real-time and is subject only to processing and transmission latency.

[0035] Instead of performing segmentation on each device, the devices can send tracked human body information to an edge system or cloud where the multi-location AR controller 205 is located for processing there. The multi-location AR controller 205 then performs segmentation and generation of 3D humanoids in near real-time and forwards the results to the participating devices for rendering there. In principle, the degree of distribution of tasks between the originating device, the backend, and the rendering device may vary among different embodiments, as can be easily understood by those skilled in the art.

[0036] The synchronization methods described above can provide high accuracy and low data loss. However, these methods require a relatively large processing load and data volume, and therefore may require higher battery usage and greater bandwidth because of the larger amount of data being processed and transferred. In embodiments where the multi-location AR controller 205 performs the processing, this may increase costs associated with cloud and / or edge processing.

[0037] In another embodiment of the present invention, partial tracking is performed. In an embodiment implementing this method, a user can share an avatar picture with other users prior to scene synchronization. When the shared AR environment is activated, a humanoid version of each user's uploaded picture is already available and can be loaded into memory. For example, upon activation of an AR environment shared between users operating device 101A and device 101B, respectively, the device 101A of the first user loads an avatar representing the second user, and the device 101B of the second user loads an avatar representing the first user. The avatar may be a humanoid representation generated from a 2D image that the users are sharing between them. The processing of the 2D image to generate a 3D humanoid representation may be performed by the originating device, by the receiving device, or as a service by the multi-location AR controller 205 in various embodiments.

[0038] When a user (or any person) enters the scene and is captured by the device camera, the human figure can be detected, its position and pose in the scene recognized and transferred to the other device, and a 3D humanoid avatar can be rendered on the other device. As the person moves, its motion is tracked (joints, rotations, etc.) and the tracked motion data is transmitted to the remote device. The multi-location AR app 201 on the remote device uses the received motion data to animate the representation of the other user already rendered on the device's display. This synchronization method can reduce the amount of data transferred between devices and does not require the AR controller to perform intermediate processing. Thus, depending on the accuracy that can be achieved in capturing the joints, rotations, and other physical body aspects, this method can reduce battery usage, processing load, and bandwidth requirements at the expense of reduced accuracy.

[0039] The two synchronization methods described herein may of course be combined in some embodiments such that the method selected depends on the user configuration, choices made for a given shared environment, or the type of objects tracked within the scene.

Claims

1. A method for synchronizing the simultaneous rendering of the same augmented reality environment in multiple locations, Steps to obtain the camera view of the local environment, The steps include detecting an object of a predetermined type in the camera view of the local environment, The steps include generating a virtual representation of the detected object, The steps of maintaining an augmented reality environment by integrating the camera view of the local environment with the generated virtual representation of the detected object, The steps include transmitting the generated virtual representation of the detected object and information regarding its integration into the augmented reality environment to at least one other device. A method that includes this.

2. The method according to claim 1, wherein the information relating to the integration of the virtual objects includes information describing a change in at least one of the position and orientation of the virtual objects.

3. The method according to claim 1 or 2, wherein the predetermined type of object is a person.

4. The step of generating information regarding the integration of the virtual representation of the person, including a description of the movements performed by the person. The method according to claim 3, further comprising:

5. The method according to claim 1 or 2, wherein the information transmitted to the at least one other device is used by the at least one other device to integrate or modify the integration of the corresponding virtual object with the camera view of the local environment.

6. The method according to claim 1 or 2, wherein information from at least one device is transmitted to a server configured to transfer received information to at least one other device.

7. The method according to claim 6, wherein information processing related to the generation or updating of information is distributed between the at least one device, the server, and the at least one other device.

8. Get the camera view of the local environment, The camera view of the local environment detects a predetermined type of object, A virtual representation of the detected object is generated, The camera view of the local environment is integrated with the generated virtual representation of the detected object. The generated virtual representation of the detected object and information regarding its integration into the augmented reality environment are transmitted to at least one other device. A device configured to maintain a description of an augmented reality environment by doing so.

9. The device according to claim 8, further configured to receive information regarding the integration of the virtual objects from at least one other device and to update the augmented reality environment in accordance with the received information.

Images