Multi-location augmented reality

Abstract

The invention provides a method for synchronously and simultaneously rendering the same augmented reality environment at multiple positions and corresponding equipment. The method allows multiple devices to obtain camera views of their respective local environment and maintain an augmented reality environment by integrating the camera views and virtual objects. On at least one of the plurality of devices, information relating to integration of virtual objects, including any changes or interactions, is transmitted to at least one other of the plurality of devices. The corresponding object on the receiving device can be changed or updated according to the received information. In some implementations, a device can generate a virtual representation of an object (e.g., a person) detected in a camera view, and enable other devices to represent and update the object in their respective augmented reality environment.

Description

Technical Field

[0001] The present invention relates to synchronization of augmented reality content between several devices, enabling users who are not in the same location to experience and interact with the same augmented reality content on their respective devices. Background Art

[0002] Users experience augmented reality (AR) content by accessing and viewing AR content through mobile apps or AR glasses. Typically, each user accesses AR content independently, and can therefore view and interact with their own AR content without any other user being able to affect the AR environment. The first user's interaction with AR content on one mobile device does not affect any other user's experience and interaction with the same AR content on their respective devices. However, shared AR is the concept of multiple users viewing the same AR content from multiple devices at the same time. That is, instead of viewing their own instance of AR content through their devices, users experience the same instance of AR content only through their respective devices. A further extension is multi-role AR, which allows multiple users to not only view the same AR content, but also interact with the same AR content at the same time.

[0003] There are software development kits (SDKs) available that allow developers to create multi-role AR applications, and such applications are commercially available. However, there are significant limitations associated with the existing technology, so it has not yet been widely used. For example, in order to provide a satisfactory experience for the user, the delay associated with loading and interacting with AR content must be very small. Other shortcomings are related to the way in which real and virtual elements overlap to create a seamless AR environment. For example, because the part of the AR scene that represents actual reality will be different at different locations, it may be necessary to represent an object that is part of the real environment at one location but not at another location as a virtual object at another location. In order to create a more satisfactory AR user experience in a synchronized manner between multiple locations, new solutions are needed to address these needs. Summary of the invention

[0004] The above needs are solved by a method for synchronously rendering the same augmented reality environment in multiple locations. According to the method, a camera view of a local environment is obtained. Objects of predefined types are detected in the camera view of the local environment, and virtual representations of the detected objects are generated. The augmented reality environment is maintained by integrating the camera view of the local environment with the generated virtual representations of the detected objects. The generated virtual representations of the detected objects and information related to the integration of the virtual objects are transmitted to at least one other device, so that the other device can produce a corresponding augmented reality environment based on its local camera view and the corresponding integration of the virtual objects.

[0005] The information related to the integration of the virtual object may include information describing a change in at least one of a position and a direction of the virtual object.

[0006] On at least one of the plurality of devices, an object of a predefined type may be detected in a camera view of a local environment, and a virtual representation of the detected object may be generated. The virtual representation may be transmitted to at least one other of the plurality of devices, thereby enabling the other device to integrate a virtual object corresponding to a real object in the environment of the first device.

[0007] In some implementations, the predefined type of object may be a person, and the information related to the integration of the virtual representation of the person may include a description of movements performed by the person.

[0008] The information transmitted to the at least one other device may be used by the at least one other device to integrate or change the integration of a corresponding virtual object with a camera view of the local environment.

[0009] In some embodiments, information from at least one device may be transmitted to a server that is configured to forward the received information to at least one other device. Processing of information related to detection of objects, movements, interactions, etc. may be performed locally, or to some extent by the recipient, or by an intermediate server or device in the cloud or edge. Thus, information related to the generation or update of information may be distributed between at least one device, the server, and at least one other device.

[0010] Devices according to the present invention may include modules and functions configured to maintain a description of an augmented reality environment by obtaining a camera view of a local environment, detecting objects of a predefined type in the camera view of the local environment, generating a virtual representation of the detected object, and integrating the camera view of the local environment and the generated virtual representation of the detected object. Such a device will also be capable of transmitting the generated virtual representation of the detected object and information related to its integration into the augmented reality environment to at least one other device. Such a device may also be configured to receive information related to the integration of the virtual object from at least one other device, and update the augmented reality environment based on the received information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will now be described in more detail with reference to the accompanying drawings, in which:

[0012] Figure 1 shows a system that may be configured to operate according to the present invention; and

[0013] Figure 2 is a block diagram illustrating modules in a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention introduces new multi-location AR, which provides techniques and tools that allow multiple users in different locations to interact with the same AR content simultaneously in an immersive way.

[0015] Figure 1 A system configured for multi-position AR according to the present invention is shown. The multi-position AR function is provided by a mobile AR application installed on a mobile device 101, and the multi-position AR function is configured to synchronize AR content with a multi-position AR backend 102 having a multi-position AR controller, or with other devices 101 running mobile AR applications with similar functions.

[0016] The multi-position AR controller may be hosted in the cloud 103 or at the edge, and provides tools and methods that allow the necessary connections to be established between devices.

[0017] The system allows users at different locations to access the same AR object or AR environment and experience each other's immersion at the same time. For example, if two users at two different locations load the same AR content and view the same object on their respective mobile devices 101, the following operations will be implemented.

[0018] First, object synchronization will ensure that the object is synchronized in all participating AR devices 101. This means that when a first user interacts with an object and thus changes the state of the object on his or her device 101A, the new state is transmitted to the other devices 101B, and the state changes accordingly on the other devices to update the second user's view of the same object. According to one aspect of the present invention, multi-position AR includes an AR format that is capable of tracking multiple parameters related to AR objects, so that not only position and orientation can be tracked, but also characteristics related to lighting, position relative to other objects, etc.

[0019] Secondly, scene synchronization provides an improved sense of immersion. When objects shared between two or more users are synchronized, they will appear in a consistent manner on all devices. If one of the users now brings himself or a real object into the AR scene, this will be reflected in the camera view of other users. That is, user A can stand in front of the camera on his mobile device 101A and be in a position adjacent to the virtual object. The camera on user A's mobile device 101A will capture the real part of the AR scene, including the user, and simultaneously record the state of the virtual AR object. The functionality provided by the present invention includes a detection algorithm configured to detect a specific object. Embodiments of the present invention may be configured to detect different types of objects, but the exemplary embodiments described herein will focus on the detection of the human body. However, other embodiments may be configured to detect other types of objects, such as cars, game parts, boxing gloves, and the like.

[0020] The human body detection algorithm may be configured to detect various aspects of the human body, such as the location of joints, depth sensing information, etc. This information may be communicated to user B's device 101B. User B's device 101B may then render an AR version of the human body detected in the camera view of the remote device 101A and place it next to the virtual object based on the state information received from the remote mobile device 101A regarding the location of the human body in the scene (determined by the human body detection algorithm) and the location of the virtual AR object. Thus, the view of the AR scene provided on user B's device 101B is the same as the view of the AR scene presented on user A's device 101A, except that the human body captured by the device camera is displayed on the first device 101A, while the human body is shown as a virtual AR body on the second device 101B. Similarly, if a human body enters a scene captured by the camera on user B's mobile device 101B, user A's device will render the scene, but the human body captured by user B's device 101B will be rendered as a virtual AR object.

[0021] Figure 1 Also shown is a database 104 which may be a repository for AR content.

[0022] Now refer to Figure 2 , to discuss in more detail the various modules included in a system operating according to the present invention. This example includes two mobile devices 101, but embodiments of the present invention may be configured to include several devices. No loss of generality is made in the description of two devices, and those skilled in the art will readily understand how to configure the present invention for other devices.

[0023] A first multi-position AR application 201A is installed on a first mobile device 101A. The AR application 201A includes a camera module 202A, which is configured to access, control, and obtain images from a camera that is part of the device 101A. As will be described in further detail below, the multi-position AR application 201A also includes a multi-position AR module 203A that receives virtual AR objects from a remote device. The real environment captured by the camera module 202A and the virtual objects provided by the multi-position AR module 203A are combined in an AR content integration module 204A. The virtual AR content received by the integration module 204A includes metadata describing the position and orientation of the virtual objects in the AR environment. The description can be absolute, relative to other objects, or determined by features detected in the scene received by the camera module 202A, such as based on surface detection or image recognition techniques.

[0024] exist Figure 2 In the example shown, the camera module 202A of the first device 101A captures a scene including a person. The person is a part of the actual reality in the scene at the location of the first device 101A. In order for the person to appear in the scene presented by the second device 101B, object recognition needs to be performed, and a virtual representation of the person is generated and transmitted to the multi-position AR controller 205 running on the backend server 102.

[0025] A similar process is performed on the second device 101B running the multi-position AR module 201B. In this example, the second person appears in the real scene and is captured by the camera module 202B in the second device 101B. For the first device, a virtual representation of the second person is generated and transmitted to the AR controller 205 running on the backend server 102.

[0026] The backend server can maintain representations of all virtual objects in the shared AR environment. In this example, there are two virtual representations of real people and a purely virtual object in the form of a cake. In the accompanying figure, the virtual objects are represented in all black, while the real objects are represented in white. In the environment maintained by the AR controller, all objects are virtual.

[0027] In some embodiments, the backend may not maintain representations of virtual objects, but may simply serve as an intermediary for distributing data to participating devices as it becomes available, but without preserving that data in the form of any stored representations of virtual objects. In other embodiments, synchronization-related information is sent directly between participating devices in a peer-to-peer manner.

[0028] Virtual objects are shared between participating devices 101. However, objects that actually exist in the local environment do not have to be presented as virtual objects in the local presentation of the AR environment. Therefore, on the first device 101A, the multi-position AR module 203A only maintains representations of completely virtual virtual objects (cakes in this example) or virtual representations of real objects in different locations (people captured by the camera module on the remote device in this example). Therefore, the multi-position AR application 201A on the first device 101A uses its AR content integration module 204A to generate an AR scene with representations of the people present provided by the camera module 202A and virtual representations of the cake and remote people. The scene is presented by the local device 101A on the display 206A of the local device or connected AR goggles, or some similar device.

[0029] The situation is the same on the second device 101B, except that here it is another person who is presented as captured by the camera and provided by the camera module 202B, while the cake and the first person are given virtual representations.

[0030] Different techniques may be used, such as surface detection, object recognition, etc., but as these techniques are well known in the art, they will not be described in detail herein.

[0031] The actual sharing of the AR environment can be facilitated in a variety of ways known in the art and not specific to augmented reality. For example, a unique QR code or link can be shared by one user to another user via a message, or in an application containing a friend list, the user can simply invite friends to share / view AR content and then activate it. Once sharing is activated, any AR content will be synchronized with the participating users.

[0032] When a connection has been established between two or more devices, any changes to the objects in the scene or the scene itself are synchronized. This synchronization is provided as a service by the multi-position AR controller 205. Once multi-position AR sharing is activated, the synchronization function provides synchronization of the state between the objects seen by the user. In the case of object synchronization, the synchronization process uses information rigid body information, shaders, vector positions, etc. to enable objects and interactions with objects to be synchronized with other devices.

[0033] The synchronization of the scene itself is based on an object recognition algorithm. As already mentioned, different implementations can be configured to recognize and synchronize different objects. This example will describe human recognition, but the principles are similar for detection and recognition of other types of objects.

[0034] The human body recognition algorithm detects and tracks human bodies in the scene. In some embodiments, the multi-position AR application 201 detects, tracks, and delivers real-time data. This means that human bodies detected in the images provided by the camera module 202 are tracked and segmentation is performed in near real time. The resulting data is sent to the multi-position AR controller 205, and then distributed to participating devices from the controller. In some embodiments, the devices operate in a point-to-point mode and information is sent directly to other participating devices. The data delivered to the participating devices will typically include vector position information.

[0035] After receiving the incoming tracking data and segmented human body data, the participating device will render it on the display 206 of the receiving device 101. During the rendering process, the device can apply Gaussian blur to generate a 3D human figure and place it appropriately in the scene based on the received vector data. Therefore, in this method, the processing is completed in near real time, only affected by processing and transmission delays.

[0036] As an alternative to performing segmentation on each device, the device can instead send the tracked human body information to the edge system or cloud where the multi-position AR controller 205 is located for processing on the edge system or cloud. The multi-position AR controller 205 then performs segmentation and 3D human figure generation in near real time and forwards the results to the participating devices for rendering on the participating devices. In principle, as will be readily appreciated by those skilled in the art, the degree to which tasks are distributed between the starting device, the backend, and the rendering device can vary between different implementations.

[0037] The above synchronization methods can provide high accuracy and less data loss. However, these methods require relatively large processing loads and data volumes, and may result in higher battery usage and greater bandwidth requirements due to the large amount of data processed and transferred. In embodiments where the multi-position AR controller 205 performs processing, this may increase costs associated with cloud and / or edge processing.

[0038] In other embodiments of the present invention, partial tracking is performed. In an embodiment implementing the method, a user can share an avatar picture with other users before the scene is synchronized. When a shared AR environment is activated, a humanoid version of an uploaded picture of a response user that is already available can be loaded into a memory. For example, when an AR environment shared between users operating device 101A and device 101B respectively is activated, the device 101A of the first user will load an avatar representing the second user, and the device 101B of the second user will load an avatar representing the first user. The avatar can be a humanoid representation generated from a 2D image that the user has shared between them. In different embodiments, processing a 2D image to generate a 3D humanoid representation can be performed by a starting device, by a receiving device, or as a service by a multi-position AR controller 205.

[0039] When a user (or anyone) enters the scene and is captured by the device camera, a human figure is detected and its position and pose in the scene are identified and passed to another device that renders a 3D humanoid avatar. As the person moves, their movements are tracked (joints, rotations, etc.) and the tracked motion data is sent to the remote device. The multi-position AR application 201 on the remote device uses the received motion data to animate a representation of another user that has been rendered on the device's display. This synchronization method can reduce the amount of data transferred between devices and does not require the AR controller to perform any intermediate processing. Therefore, this method can reduce battery usage, processing load, and bandwidth requirements, but depends on the accuracy that can be achieved when capturing joints, rotations, and other body aspects, at the expense of reduced accuracy.

[0040] Of course, in some implementations, the two synchronization methods described herein may be combined, such that the method selected depends on user configuration, selections made for a given sharing environment, or on the types of objects tracked in the scene.

Claims

1. A method for synchronously rendering the same augmented reality environment at multiple locations, the method comprising: Get the camera view of the local environment; detecting objects of predefined types in the camera view of the local environment; generating a virtual representation of the detected object; maintaining the augmented reality environment by integrating the camera view of the local environment with a generated virtual representation of the detected object; and The generated virtual representation of the detected object and information related to integration of the generated virtual representation into the augmented reality environment are transmitted to at least one other device.

2. The method according to claim 1, wherein: The information related to the integration of the virtual object includes information describing a change in at least one of a position and an orientation of the virtual object.

3. The method according to claim 1 or 2, wherein: The predefined type of object is a person. 4 . The method of claim 3 , further comprising generating information related to the integration of the virtual representation of the person, the information comprising a description of movements performed by the person.

5. The method according to one of the preceding claims, wherein: The information transmitted to the at least one other device is used by the at least one other device to integrate or change the integration of a corresponding virtual object with the camera view of the local environment.

6. The method according to one of the preceding claims, wherein: Information from at least one device is transmitted to a server, which is configured to forward the received information to the at least one other device.

7. The method according to claim 6, wherein: Processing of information related to generation or updating of information is distributed between the at least one device, the server, and the at least one other device.

8. A device configured to maintain a description of an augmented reality environment by: Get the camera view of the local environment; detecting objects of predefined types in the camera view of the local environment; generating a virtual representation of the detected object; integrating the camera view of the local environment with a generated virtual representation of the detected object; and The generated virtual representation of the detected object and information related to integration of the generated virtual representation into the augmented reality environment are transmitted to at least one other device.

9. The device according to claim 8, further configured to: receive information related to the integration of the virtual object from the at least one other device, and update the augmented reality environment according to the received information.

Images