Marker-based Shared Augmented Reality Session Creation

Abstract

A method for creating a marker-based shared augmented reality (AR) session begins with initializing a shared AR session via a first device and via a second device. The first device displays a marker on a display. The second device uses a camera device included in the second device to detect the marker and uses the camera device to capture an image of the marker. The second device uses the image of the marker to determine a transformation between the first device and the second device. Then, the transformation is used to determine a common coordinate system, the common coordinate system is used to generate a shared AR session, and the shared AR session is displayed by the first device and by the second device. Other embodiments are described herein.

Description

[0001] Priority Claim

[0002] This application claims priority to U.S. Patent Application No. 16 / 729,078, filed on December 27, 2019, which is incorporated herein by reference in its entirety. Background Art

[0003] Augmented Reality (AR) is a modification of the virtual environment. For example, in Virtual Reality (VR), the user is completely immersed in a virtual world, while in AR, the user is immersed in a world where virtual objects are combined or superimposed on the real world. AR systems are designed to generate and present virtual objects that interact realistically with the real-world environment and with each other. Examples of AR applications can include single or multi-player video games, instant messaging systems, etc. Brief Description of the Drawings

[0004] In the drawings (not necessarily drawn to scale), the same reference numerals may describe similar components in different views. Similar reference numerals with different alphabetical suffixes may represent different instances of similar components. In the figures of the drawings, some embodiments are shown by way of example and not limitation, in which:

[0005] Figure 1 is a graphical representation of a networked environment in which the present disclosure may be deployed, according to some example embodiments.

[0006] Figure 2 is a graphical representation of a messaging client application, according to some example embodiments.

[0007] Figure 3 is a graphical representation of a data structure maintained in a database, according to some example embodiments.

[0008] Figure 4 is a graphical representation of a message, according to some example embodiments.

[0009] Figure 5 is a flowchart of a process for creating a marker-based shared AR session, according to some example embodiments.

[0010] Figure 6 is a block diagram showing a software architecture in which the present disclosure may be implemented, according to some example embodiments.

[0011] Figure 7is a graphical representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein.

[0012] Figure 8 is a graphical representation of a processing environment according to some example embodiments. Detailed Description

[0013] The following description includes systems, methods, techniques, instruction sequences, and computer program products that embody illustrative embodiments of the present disclosure. In the following description, numerous specific details are set forth for purposes of explanation in order to provide an understanding of the various embodiments of the inventive subject matter. However, it will be apparent to those skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0014] In addition, embodiments of the present disclosure improve the functionality of augmented reality (AR) creation software and systems by establishing a common coordinate system for a marker-based shared AR session. In one embodiment, the system hosts a shared AR session that multiple users can participate in via client devices. For example, the shared AR session can be an AR racing game where each user controls a different car. Each of the users is viewing the same shared AR scene displayed on the client device at the same time. The experience can be synchronized and shared among all users. In such an example, the actions of one user in the shared AR session can be synchronized and broadcast to all the other users. Thus, the shared AR session is a shared virtual space, but in AR.

[0015] One challenge in generating a shared AR session is establishing the origin of the shared AR scene and how the shared AR scene is aligned relative to the surrounding environment. For example, when users are tracking the shared AR scene, each of the client devices knows its position within the shared AR scene as it moves. However, the client devices may not be able to agree on the location of the origin of the shared AR scene or how the shared AR scene is aligned relative to the surrounding environment. Thus, although each of the client devices is presenting the same virtual content (e.g., cars, race tracks, etc.), the virtual content may not appear in the same position within the shared AR scene on each device. For example, the virtual content may not be rotated in the same way on the display screens of each client device to be aligned.

[0016] In one embodiment, the system implements the following method: the method is for all client devices in an AR session to agree on the origin center of a shared AR scene (or world), and to agree on the rotation of the shared AR scene in a specific direction. In one embodiment, the system uses markers to create a shared AR session to establish the origin center of the shared AR scene.

[0017] Figure 1 FIG. 4 is a block diagram illustrating an example system 100 for exchanging data (e.g., messages and associated content) over a network. System 100 includes multiple instances of client devices 102, each instance hosting multiple applications including a messaging client application 104 and an AR session client controller 124. Each messaging client application 104 is communicatively coupled via a network 106 (e.g., the Internet) to other instances of the messaging client application 104 and to a messaging server system 108. Each AR session client controller 124 is communicatively coupled via the network 106 to other instances of the AR session client controller 124 and to an AR session server controller 126 in the messaging server system 108.

[0018] The messaging client application 104 is capable of communicating and exchanging data with another messaging client application 104 and the messaging server system 108 via the network 106. The data exchanged between the messaging client applications 104 and between the messaging client application 104 and the messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0019] The messaging server system 108 provides server-side functionality to specific messaging client applications 104 via the network 106. Although certain functions of system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, the location of certain functions within the messaging client application 104 or within the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client application 104 where the client device 102 has sufficient processing power.

[0020] The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to the messaging client application 104, receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. As an example, the data may include message content, client device information, geographical location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled through functions available via the user interface (UI) of the messaging client application 104.

[0021] The AR session client controller 124 is capable of communicating with and exchanging data with another AR session client controller 124 and the AR session server controller 126 via the network 106. Data exchanged between AR session client controllers 124 and between the AR session client controller 124 and the AR session server controller 126 may include: a session identifier identifying a shared AR session, a transformation between a first device and a second device (e.g., the plurality of client devices 102 includes the first device and the second device), a common coordinate system, functions (e.g., commands to invoke functions), and other payload data (e.g., text, audio, video, or other multimedia data).

[0022] Turning now specifically to the messaging server system 108, the application programming interface (API) server 110 is coupled to the application server 112 and provides a programming interface to the application server 112. The application server 112 is communicatively coupled to the database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0023] The Application Programming Interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the Application Programming Interface (API) server 110 provides a collection of interfaces (e.g., routines and protocols) that can be invoked or queried by the messaging client application 104 to invoke the functions of the application server 112. The Application Programming Interface (API) server 110 exposes various functions supported by the application server 112, including account registration, login functionality, sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114, setting up a collection of media data (e.g., stories) for possible access by another messaging client application 104, retrieving the friend list of the user of the client device 102, retrieving such collections, retrieving messages and content, adding and deleting friends to and from the social graph, the location of friends within the social graph, and opening application events (e.g., involving the messaging client application 104).

[0024] The application server 112 hosts multiple applications and subsystems, including the messaging server application 114, the image processing system 116, the social networking system 122, and the AR session server controller 126. The messaging server application 114 implements many message processing techniques and functions, particularly related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or libraries). The messaging server application 114 then makes these collections available to the messaging client application 104. Given the hardware requirements for such processing, the messaging server application 114 may also perform other processor- and memory-intensive data processing on the server side.

[0025] The application server 112 also includes an image processing system 116 dedicated to performing various image processing operations, typically on images or videos received within the payloads of messages at the messaging server application 114.

[0026] The social networking system 122 supports various social networking functionality services and makes these functions and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses the entity graph 304 within the database 120 (as Figure 3as shown). Examples of functions and services supported by the social networking system 122 include identification of other users of the messaging system 100 with whom a particular user has a relationship or "follows", as well as identification of other entities and particular users of interest.

[0027] The application server 112 also includes an AR session server controller 126 that can communicate with an AR session client controller 124 in the client device 102 to establish a separate or shared AR session. The AR session server controller 126 can also be coupled to the messaging server application 114 to establish an electronic group communication session (e.g., group chat, instant messaging) for the client device in the shared AR session. The electronic group communication session can be associated with a session identifier provided by the client device 102 to obtain access to the electronic group communication session and the shared AR session. In one embodiment, the client device first obtains access to the electronic group communication session and then obtains a session identifier in the electronic group communication session that enables the client device to access the shared AR session. In some embodiments, the client device 102 is capable of accessing the shared AR session without the assistance of or communication with the AR session server controller 126 in the application server 112.

[0028] The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 that stores data associated with messages processed by the messaging server application 114.

[0029] Figure 2 is a block diagram showing additional details regarding the system 100 according to an example embodiment. Specifically, the system 100 is shown to include a messaging client application 104 and an application server 112, which in turn includes a plurality of some subsystems, namely a transient timer system 202, a collection management system 204, and an annotation system 206.

[0030] The transient timer system 202 is responsible for implementing transient access to the content allowed by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 incorporates a plurality of timers that selectively display messages and associated content and enable access to messages and associated content via the messaging client application 104 based on the duration and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0031] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image, video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories". Such collections can be made available for a specified period of time (e.g., the duration of the event to which the content pertains). For example, content related to a concert can be made available as a "story" during the duration of that concert. The collection management system 204 can also be responsible for publishing an icon that notifies the user interface of the messaging client application 104 of the existence of a particular collection.

[0032] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., delete inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In certain implementations, compensation can be paid to users to include user-generated content in a collection. In such cases, the curation interface 208 operates to automatically pay such users for the use of their content.

[0033] The annotation system 206 provides various functions that enable a user to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to the generation and publication of a media overlay for a message processed by the system 100. The annotation system 206 operably supplies a media overlay or supplement (e.g., an image filter) to the messaging client application 104 based on the geographical location of the client device 102. In another example, the annotation system 206 operably supplies a media overlay to the messaging client application 104 based on other information (e.g., the social network information of the user of the client device 102). The media overlay may include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects may be applied to a media content item (e.g., a photo) at the client device 102. For example, the media overlay may include text that can be overlaid on a photo taken by the client device 102. In another example, the media overlay includes a location identifier (e.g., Venice Beach) overlay, a name of a live event, or a business name (e.g., Beach Café) overlay. In another example, the annotation system 206 uses the geographical location of the client device 102 to identify a media overlay that includes the name of a business at the geographical location of the client device 102. The media overlay may include other markers associated with the business. The media overlay may be stored in the database 120 and accessed via the database server 118.

[0034] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables a user to select a geographical location on a map and upload content associated with the selected geographical location. The user may also specify the circumstances under which a particular media overlay should be provided to other users. The annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographical location.

[0035] In another example embodiment, the annotation system 206 provides a business-based publishing platform that enables a business to select a particular media overlay associated with a geographical location via a bidding process. For example, the annotation system 206 associates the media overlay of the highest-bidding business with the corresponding geographical location within a predefined amount of time.

[0036] Figure 3 is a schematic diagram showing a data structure 300 that may be stored in the database 120 of the messaging server system 108 according to certain example embodiments. Although the contents of the database 120 are shown as including multiple tables, it should be understood that the data may be stored in other types of data structures (e.g., as an object-oriented database).

[0037] The database 120 includes message data stored within a message table 314. An entity table 302 stores entity data, including an entity graph 304. Entities whose records are maintained within the entity table 302 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of type, any entity for which the messaging server system 108 stores data can be an identified entity. Each entity is set with a unique identifier, as well as an entity type identifier (not shown).

[0038] The entity graph 304 also stores information regarding relationships and associations between entities. By way of example only, such relationships can be social relationships based on interests or activities, professional relationships (e.g., working at a common company or organization).

[0039] The database 120 also stores annotation data in the form of an example of a filter within an annotation table 312. The filter whose data is stored within the annotation table 312 is associated with a video (whose data is stored in a video table 310) and / or an image (whose data is stored in an image table 308) and is applied to the video and / or the image. In one example, the filter is an overlay that is displayed as being overlaid on the image or video during presentation to a recipient user. Filters can be of various types, including user-selected filters from a library of filters presented to a sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geo-filters), which can be presented to the sending user based on a geographic location. For example, based on geographic location information determined by a GPS unit of the client device 102, the messaging client application 104 can present a geographic location filter specific to a neighborhood or a specific location within a user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other input or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0040] Other annotation data that can be stored within the image table 308 is so-called "lens" data. A "lens" can be real-time special effects and sounds that can be added to an image or a video.

[0041] As mentioned above, the video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained in the message table 314. Similarly, the image table 308 stores image data associated with messages whose message data is stored in the entity table 302. The entity table 302 can associate various annotations from the annotation table 312 with the various images and videos stored in the image table 308 and the video table 310.

[0042] The story table 306 stores data about a collection of messages and associated image, video, or audio data, where the messages and associated image, video, or audio data are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose records are maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the messaging client application 104 can include icons that a user can select to enable the sending user to add specific content to his or her personal story.

[0043] Collections can also constitute "Live Stories", which are collections of content from multiple users that are created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and are at a common location event at a particular time can be presented, for example via the user interface of the messaging client application 104, with the option to contribute content to a specific Live Story. Live Stories can be identified to users by the messaging client application 104 based on the user's location. The end result is a "Live Story" told from a community perspective.

[0044] Another type of content collection is called a "Location Story", which enables users whose client devices 102 are located within a particular geographical location (e.g., on a college or university campus) to contribute to a specific collection. In some embodiments, contributing to a Location Story may require a second level of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0045] The database 120 can also store data related to individual and shared AR sessions in the AR session table 316. The data in the AR session table 316 can include data communicated between an AR session client controller 124 and another AR session client controller 124, as well as data communicated between the AR session client controller 124 and the AR session server controller 126. The data can include data for establishing a common coordinate system for a shared AR scene, transformations between devices, session identifiers, etc.

[0046] Figure 4 FIG. is a schematic diagram showing the structure of a message 400 according to some embodiments, the message 400 being generated by a messaging client application 104 for transmission to another messaging client application 104 or a messaging server application 114. The content of a particular message 400 is used to populate a message table 314 stored in a database 120 accessible to the messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data of the client device 102 or the application server 112. The message 400 is shown to include the following components:

[0047] · Message identifier 402: A unique identifier that identifies the message 400.

[0048] · Message text payload 404: Text to be generated by a user via a user interface of the client device 102 and included in the message 400.

[0049] · Message image payload 406: Image data captured by a camera device component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400.

[0050] · Message video payload 408: Video data captured by a camera device component or retrieved from a memory component of the client device 102 and included in the message 400.

[0051] · Message audio payload 410: Audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400.

[0052] · Message annotation 412: Annotation data (e.g., filters, stickers, or other enhancements) representing an annotation to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400.

[0053] · Message duration parameter 414: A parameter value indicating, in seconds, the amount of time that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented or made accessible to the user via the messaging client application 104.

[0054] · Message geographical location parameter 416: Geographical location data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geographical location parameter 416 values may be included in the payload, each of these parameter values being associated with a content item included in the content (e.g., a particular image within the message image payload 406 or a particular video within the message video payload 408).

[0055] · Message story identifier 418: An identifier value that identifies one or more content collections (e.g., “stories”) associated with a particular content item in the message image payload 406 of message 400. For example, multiple images within message image payload 406 can each be associated with multiple content collections using identifier values.

[0056] · Message tag 420: Each message 400 can be tagged with multiple tags, where each tag in the multiple tags indicates a topic of the content included in the message payload. For example, in a case where a particular image included in message image payload 406 depicts an animal (e.g., a lion), the tag value can be included within message tag 420 that indicates the relevant animal. The tag value can be manually generated based on user input or can be automatically generated using, for example, image recognition.

[0057] · Message sender identifier 422: An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of client device 102 on which message 400 is generated and from which message 400 is sent.

[0058] · Message recipient identifier 424: An identifier (e.g., a messaging system identifier, an email address, or a device identifier) that indicates the user of client device 102 to which message 400 is addressed.

[0059] The content (e.g., value) of each component of message 400 can be a pointer to a location in a table in which content data values are stored. For example, the image value in message image payload 406 can be a pointer to a location (or address) within image table 308. Similarly, the value within message video payload 408 can point to data stored within video table 310, the value stored within message annotation 412 can point to data stored within annotation table 312, the value stored within message story identifier 418 can point to data stored within story table 306, and the values stored within message sender identifier 422 and message recipient identifier 424 can point to user records stored within entity table 302.

[0060] Figure 5 is a flowchart of a process for creating a tagged-based shared AR session according to some example embodiments. Although the flowchart may depict the operations as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process terminates when its operations are complete. The process can correspond to a method, a program, etc. The steps of the method can be performed in whole or in part, can be combined with part or all of the steps in other methods, and can be performed by any number of different systems (such as Figure 1 、Figure 7 and / or Figure 8 is performed by the system described in) or any part of the system, such as a processor included in any system).

[0061] At operation 502, the first device and the second device initialize a shared augmented reality (AR) session. The AR session client controller 124 in each of the first client device and the second client device (e.g., client device 102) can independently initialize the shared AR session. In one embodiment, during the initialization of the shared AR session, the first device and the second device can be in independent, active Simultaneous Localization And Mapping (SLAM) sessions with respect to each other, and these SLAM sessions need to be aligned with each other to establish the shared AR session.

[0062] Initializing the shared AR session can include determining, by the first device, a first device transformation (e.g., referred to as transformation T A ). The first device transformation can be based on the current pose of the first device and the origin of the first device. The current pose of the first device can be the position and orientation of the first device with respect to the real world along the x-axis, y-axis, and z-axis. The origin of the first device is the origin of the coordinate system tracked by the first device. Similarly, initializing the shared AR session can also include determining, by the second device, a second device transformation (e.g., referred to as transformation T B ). The second device transformation can be based on the current pose of the second device and the origin of the second device. The current pose of the second device can be the position and orientation of the second device with respect to the real world along the x-axis, y-axis, and z-axis. The origin of the second device is the origin of the coordinate system tracked by the second device. The origin of the coordinate system tracked by the first device can be different from the origin of the coordinate system tracked by the second device.

[0063] At operation 504, the first device displays a marker on the display device of the first device. The AR session client controller 124 in the first device may cause the first device to display the marker. In one embodiment, the marker may be continuously displayed for multiple frames or may be displayed periodically once (e.g., once every 30 frames). The marker may be a Quick Response (QR) code, a natural feature marker, or a user interface of an application displayed on the display of the first device. In some embodiments, the marker may include a session identifier associated with the shared AR session. For example, the marker may encode the session identifier. The marker may also be a representation of the session identifier. The session identifier may be selected and generated by the AR session client controller 124 of the first device or by the AR session server controller 126. In one embodiment, the appearance of the marker (or specific pixels in the marker) is known to the device that wants to join the AR sharing session and / or the AR session server controller 126. When the marker is the user interface of an application, the application may be an application related to the shared AR session. For example, the application may be a shared AR video game such that the marker is the user interface of the shared AR video game. One advantage of the marker being the user interface of an application is that the user of the first device does not have to stop using the application to display a different marker such as a QR code.

[0064] In one embodiment, the first device uses the session identifier to access a communication session. The first device may use the session identifier to establish a communication connection or coupling with a real-time messaging service. The real-time messaging service may be an electronic group communication session established by the AR session server controller 126.

[0065] At operation 506, the second device uses a camera device included in the second device to detect the marker. For example, the second user may point the second device at the marker displayed on the display of the first device such that the marker is within the field of view of the camera device included in the second device. The AR session client controller 124 in the second device may detect the marker by analyzing the image within the field of view of the camera device. In one embodiment, the AR session client controller 124 may implement a marker detection algorithm to detect the marker.

[0066] At operation 508, the second device uses the camera device included in the second device to capture an image of the marker. In one embodiment, when the AR session client controller 124 detects the marker, the AR session client controller 124 causes the camera device to capture an image of the marker.

[0067] In one embodiment, the AR session client controller 124 of the second device extracts the session identifier from the marked image and uses the extracted session identifier to access the communication session. For example, the second device may use the session identifier to establish or couple a communication connection with a real-time messaging service. The real-time messaging service may be an electronic group communication session established by the AR session server controller 126.

[0068] At operation 510, the AR session client controller 124 of the second device uses the marked image to determine the transformation (T C ) between the first device and the second device. The transformation (T C ) may be a transformation matrix that describes the transformation between the first device and the second device.

[0069] In one embodiment, the second device receives the first device transformation (T A ) from the first device and also uses the first device transformation (T A ) to determine the transformation (T C ) between the first device and the second device. In this embodiment, the transformation (T C ) is a transformation matrix that describes the transformation between the first device transformation (T A ) and the second device transformation (T B ).

[0070] The AR session client controller 124 of the second device may also determine a second offset based on the first device transformation (T A ). The second offset may be the offset between the current pose of the second device and the origin of the first device. In this embodiment, the second offset may be chained with the offsets calculated by other devices in the shared AR session to ensure that the devices in the shared AR session can agree on the origin of the AR shared session (e.g., the shared AR world origin).

[0071] In one embodiment, the AR session client controller 124 of the second device determines the transformation (T C ) between the first device and the second device by using the position of the marker in the image, the scale of the marker in the image, the rotation of the marker in the image, or any combination thereof. For example, to determine the transformation (T C ), the AR session client controller 12 of the second device may use basic computer vision, image processing, geometry, translation, visual representation, angles, distances, etc.

[0072] In one embodiment, the AR session client controller 124 of the second device applies the transformation (T C) is sent to the AR session client controller 124 and / or the AR session server controller 126 of the first device. The AR session client controller 124 of the second device may also send the transformation (T C ) in a (group) communication session.

[0073] At operation 512, the AR session client controller 124 of the second device uses the transformation (T C ) to determine a common coordinate system. In one embodiment, the AR session client controller 124 of the first device may use the transformation (T C ) received from the second device to determine the common coordinate system.

[0074] In one embodiment, the AR session client controller 124 of the second device sends the common coordinate system to the AR session client controller 124 and / or the AR session server controller 126 of the first device. The AR session client controller 124 of the second device may also send the common coordinate system in a (group) communication session.

[0075] At operation 514, the AR session client controller 124 of the second device uses the common coordinate system to generate a shared AR session. In one embodiment, the AR session client controller 124 of the first device may use the common coordinate system received from the second device to generate a shared AR session.

[0076] At operation 516, the AR session client controller 124 in the first device causes the shared AR session to be displayed by the first device, and the AR session client controller 124 in the second device causes the shared AR session to be displayed by the second device. In one embodiment, the shared AR scene included in the shared AR session is displayed by the first device and the second device.

[0077] In one embodiment, the marker may be the first device itself, rather than the display marker of the first device on the display of the first device at operation 504. In this embodiment, the three-dimensional (3-D) appearance of the first device may be the marker, such that the AR session client controller 124 of the second device can detect the first device at any angle or position within the field of view of the camera device of the second device as the marker and cause the camera device to capture an image of the first device. In this embodiment, instead of using a one-time coordinate setting using the marker, the relative transformation (e.g., transformation T C ) can be continuously updated by periodically capturing an image of the first device and updating the common coordinates accordingly. In this embodiment, the database 120 may store a database of images of different electronic devices at different angles and positions that can be used as markers.

[0078] In another embodiment, to further refine the determination of the common coordinate system, it is necessary to address the time delay between the moment when the second device determines the transformation (T C ) and the moment when the first device sends the first transformation (T A ) to the second device. At the precise moment when scanning or tracking occurs, the second device requires the transformation (T C ) and the first transformation (T A ). In this embodiment, instead of sending the first transformation (T A ) as a single data point, the first device sends a stream of the first transformation (T A ) over time. The AR session client controller 124 of the second device needs to establish a global time for alignment to be performed. The global time can be obtained from the application server 112, but there is still some delay introduced. To refine the time provided by the application server 112, the second device can capture a video or multiple frames of the marker. For example, the first device can display a visual transformation of the marker (e.g., rotation, scaling, blinking, moving in different directions on the display screen, using several markers and tracking the movement and orientation of the markers relative to each other). Then, the second device can determine the time (e.g., in milliseconds) when the first device sends the first transformation (T A ) and obtain the first transformation (T A ) from the stream of the first transformation (T A ).

[0079] In one example, the first device can display a rotating marker captured by the second device. The AR session client controller 124 of the second device can use the video or multiple frames of the captured rotating marker to refine the time code received from the AR session server controller 126 or the application server 112. Specifically, the known rotation speed of the marker can be used to resolve the uncertainty regarding which frame is the precise frame on which the common coordinate system should be based.

[0080] In one embodiment, the AR session client controller 124 of the second device can filter out the images of the markers that result in a transformation (T C ) opposite to gravity. Although the frames of the captured markers are rotationally offset in their positions along the ground plane, both the first device and the second device can agree on the position of the ground (e.g., which direction is up or down). The AR session client controller 124 of the second device performs a gravity check by determining whether the observed first current pose (or the image of the captured marker) is opposite to gravity.

[0081] In this embodiment, the AR session client controller 124 of the second device receives information including the gravity axis angle and uses this information to compare with the tracking data of the marker (e.g., the frames of the marker captured by the imaging device in the second device). Then, the AR session client controller 124 of the second device can check whether the first device and the second device agree on how the pose of the first device is oriented relative to the gravity position. Then, the AR session client controller 124 of the second device can filter out the data (or the captured frames of the marker) that are in disagreement in the direction relative to the gravity position (e.g., lower quality data).

[0082] In another embodiment, the AR session client controller 124 of the second device can further refine the common coordinate system using reprojection. When the AR session client controller 124 of the second device detects the marker and captures an image of the marker using the imaging device, the AR session client controller 124 of the second device reprojects the image of the marker onto the display screen space and analyzes the pixels of the detected marker. Based on the analysis of the reprojected pixels, the AR session client controller 124 of the second device can cause the second device to display a notification indicating whether the marker is captured sufficiently (e.g., "too far from the marker", "too fast", etc.). In another example, the AR session client controller 124 of the second device captures multiple images of the marker to average out potential errors in the capture.

[0083] In one embodiment, the AR session client controller 124 of the second device can capture an image or multiple images of the marker before connecting to the AR session server controller 126 or the application server 112. The first device can maintain a trace window of the pose of the first device that is known in terms of the position (angle of the gravity axis) and orientation of the first device relative to the origin of the coordinate system tracked by the first device. Once the marker is detected within the field of view of the imaging device of the second device, the second device can start recording or capturing images of the marker (e.g., pose history). Once connected to the application server 112, the first device can send the trace window of the pose to the second device. The second device can use visual time alignment addition to match the two streams in time (e.g., the trace window from the first device and the pose history from the second device).

[0084] In one embodiment, when using the user interface of the application displayed on the first device as the marker instead of capturing the user interface through the second device, the first device can send each frame of the user interface to the second device. Then, the second device can perform stream alignment using the received stream of frames.

[0085] In another embodiment, the tag may also encode one or more poses of the first device into a code. The second device may decode the code from the image of the tag and obtain one or more poses of the first device.

[0086] Figure 6 FIG. 600 is a block diagram showing a software architecture 604 that may be installed on any one or more of the devices described herein. The software architecture 604 is supported by hardware, such as a machine 602 including a processor 620, a memory 626, and I / O components 638. In this example, the software architecture 604 may be conceptually viewed as a stack of layers, where each layer provides a specific function. The software architecture 604 includes the following layers, such as an operating system 612, libraries 610, frameworks 608, and applications 606. In operation, the application 606 activates an API call 650 through the software stack and receives a message 652 in response to the API call 650.

[0087] The operating system 612 manages hardware resources and provides common services. The operating system 612 includes, for example, a kernel 614, services 616, and drivers 622. The kernel 614 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 614 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. The services 616 may provide other common services for other software layers. The drivers 622 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 622 may include a display driver, a camera device driver, or a low-power driver, a flash memory driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), drivers, an audio driver, a power management driver, etc.

[0088] The library 610 provides low-level common infrastructure used by the application 606. The library 610 may include a system library 618 (e.g., a C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the library 610 may include an API library 624, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), High Efficiency Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., the OpenGL framework for 2D and 3D rendering in graphical content on a display), a database library (e.g., SQLite for providing various relational database functions), a network library (e.g., WebKit for providing web browsing functions), etc. The library 610 may also include various other libraries 628 to provide many other APIs to the application 606.

[0089] The framework 608 provides high-level common infrastructure used by the application 606. For example, the framework 608 provides various Graphical User Interface (GUI) functions, advanced resource management, and advanced location services. The framework 608 may provide a wide range of other APIs that can be used by the application 606, and some of these other APIs may be specific to a particular operating system or platform.

[0090] In an example embodiment, the application 606 may include a home application 636, a contacts application 630, a browser application 632, a book reader application 634, a location application 642, a media application 644, a messaging application 646, a gaming application 648, and a wide variety of other applications such as third-party applications 640. The application 606 is a program that executes functions defined in the program. One or more of the applications 606 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language). In a particular example, a third-party application 640 (e.g., an application developed using an ANDROID TM or IOS TM software development kit (SDK)) can be an application on a mobile operating system such as IOS TM 、ANDROID TM 、 Mobile software running on a Phone or other mobile operating system. In this example, third-party application 640 can activate API call 650 provided by operating system 612 to facilitate the functions described herein.

[0091] Figure 7 is a graphical representation of a machine 700 within which instructions 708 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 700 to perform any one or more of the methods discussed herein. For example, instructions 708 can cause the machine 700 to perform any one or more of the methods described herein. Instructions 708 transform the general unprogrammed machine 700 into a particular machine 700 programmed to perform the described and illustrated functions in the described manner. Machine 700 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 700 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 700 can include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular telephones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 708 specifying actions to be taken by machine 700. Further, although only a single machine 700 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 708 to perform any one or more of the methods discussed herein.

[0092] Machine 700 can include a processor 702, a memory 704, and I / O components 742, which can be configured to communicate with each other via a bus 744. In an example embodiment, processor 702 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 706 and a processor 710 that execute instructions 708. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 7A number of processors 702 are shown, but machine 700 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0093] Memory 704 includes main memory 712, static memory 714, and storage unit 716, all of which are accessible by processor 702 via bus 744. Main memory 704, static memory 714, and storage unit 716 store instructions 708 embodying any one or more of the methods or functions described herein. Instructions 708 may also reside, completely or partially, within main memory 712, within static memory 714, within machine-readable medium 718 within storage unit 716, within at least one of the processors 702 (e.g., within a cache memory of the processor), or in any suitable combination thereof during execution by machine 700.

[0094] I / O components 742 can include various components that receive input, provide output, generate output, transfer information, exchange information, capture measurements, and the like. The specific I / O components 742 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O components 742 can include Figure 7 many other components not shown. In various example embodiments, I / O components 742 can include output components 728 and input components 730. Output components 728 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), auditory components (e.g., a speaker), tactile components (e.g., a vibration motor, a resistance mechanism), other signal generators, and the like. Input components 730 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), pointing-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides the location and / or force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0095] In other example embodiments, the I / O component 742 may include a biometric component 732, a motion component 734, an environmental component 736, or a positioning component 738, as well as a variety of other components. For example, the biometric component 732 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so on. The motion component 734 includes acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes), and the like. The environmental component 736 includes, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety reasons or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 738 includes position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers for detecting the air pressure from which altitude can be obtained), orientation sensor components (e.g., magnetometers), and the like.

[0096] Various techniques can be used to implement communication. The I / O component 742 also includes a communication component 740, which is operable to couple the machine 700 to the network 720 or the device 722 via the couplings 724 and 726. For example, the communication component 740 may include a network interface component or other suitable device to interface with the network 720. In other examples, the communication component 740 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power), components, and other communication components that provide communication via other forms. The device 722 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0097] In addition, communication component 740 can detect an identifier or include components operable to detect an identifier. For example, communication component 740 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: one-dimensional barcodes, such as Universal Product Code (UPC) barcodes; multi-dimensional barcodes, such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an auditory detection component (e.g., a microphone for identifying an audio signal of a marker). Additionally, various information can be obtained via communication component 740, such as a location via Internet Protocol (IP) geolocation, a location via signal triangulation, a location via detecting an NFC beacon signal that can indicate a specific location, etc.

[0098] Various memories (e.g., memory 704, main memory 712, static memory 714, and / or the memory of processor 702) and / or storage unit 716 can store one or more sets of instructions and data structures (e.g., software) that embody any one or more of the methods or functions described herein or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 708), when executed by processor 702, cause various operations to implement the disclosed embodiments.

[0099] Instructions 708 can be sent or received over network 720 using a transmission medium, via a network interface device (e.g., the network interface component included in communication component 740), and using any one of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 708 can be sent or received via a coupling 726 (e.g., a peer-to-peer coupling) to device 722 using a transmission medium.

[0100] Now turning to Figure 8 , a graphical representation of a processing environment 800 is shown, which includes processors 806, 808, and 802 (e.g., a GPU, a CPU, or a combination thereof).

[0101] Processor 802 is shown coupled to power supply 804 and includes a module (either permanently configured or temporarily instantiated), namely an AR session client controller component 810. The AR session client component 810 can, in operation, for example, initialize a shared AR session, cause a marker to be displayed, use a camera device to capture an image of the marker, generate a transformation (TC ) and a common coordinate system, and enables the display of a shared AR session. Although not shown, the processor 802 may alternatively include an AR session server controller component that can perform the operations of the AR session server controller 126. As shown, the processor 802 is communicatively coupled to both the processor 806 and the processor 808.

[0102] In cases where phrases similar to “at least one of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, or C,” or “one or more of A, B, and C” are used, it is intended that the phrase be interpreted to mean that A can exist alone in an embodiment, B can exist alone in an embodiment, C can exist alone in an embodiment, or any combination of elements A, B, and C can exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C.

[0103] Changes and modifications can be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A method, comprising: Initializing a shared augmented reality (AR) session via a first device and via a second device; Displaying a marker on a display of the first device; Detecting the marker using a camera device via the second device, the second device including the camera device; Capturing an image of the marker using the camera device via the second device; Reprojecting the image of the marker into a display screen space via the second device to determine a quality of the captured image of the marker, wherein reprojecting the image of the marker includes causing the second device to display the image of the marker on the display screen space; Determining a transformation between the first device and the second device using the image of the marker via the second device; Determining a common coordinate system using the transformation; Generating the shared AR session using the common coordinate system; and Causing the shared AR session to be displayed by the first device and by the second device.

2. The method according to claim 1, wherein Initializing the shared AR session via the first device and via the second device includes: Determining a first device transformation via the first device, wherein the first device transformation is based on a current pose of the first device and a first device origin, wherein the first device origin is an origin of a coordinate system tracked by the first device; Determining a second device transformation via the second device, wherein the second device transformation is based on a current pose of the second device and a second device origin, wherein the second device origin is an origin of a coordinate system tracked by the second device.

3. The method according to claim 2, wherein, Determining the transformation between the first device and the second device using the image of the marker via the second device further includes: Receiving, via the second device, the first device transformation from the first device.

4. The method according to claim 3, wherein Determining the transformation between the first device and the second device using the image of the marker via the second device further includes: Determining a second offset via the second device based on the first device transformation, wherein the second offset is an offset between the current pose of the second device and the first device origin.

5. The method according to claim 4, wherein Determining the transformation between the first device and the second device using the image of the marker via the second device further includes: Determining the transformation using a position of the marker in the image, a scale of the marker in the image, or a rotation of the marker in the image.

6. The method according to claim 1, wherein The marker includes a session identifier.

7. The method according to claim 6, wherein The marker is a Quick Response (QR) code, a natural feature marker, or a user interface of an application displayed on the display of the first device.

8. The method according to claim 7, further comprising: Extracting the session identifier from the image of the marker via the second device.

9. The method according to claim 8, further comprising: Accessing a communication session via the first device using the session identifier included in the marker displayed on the display of the first device; And Accessing the communication session via the second device using the session identifier extracted from the image of the marker.

10. The method according to claim 5, further comprising: Perform a gravity check by the second device, wherein performing the gravity check includes: Determine whether the direction of gravity transformed based on the first device corresponds to the direction of gravity based on the transformation between the first device and the second device.

11. The method according to claim 1, further comprising: Cause the second device to display a notification indicating the quality of the image of the captured marker.

12. A system, comprising: A processor; And A memory component having instructions stored thereon that, when executed by the processor, cause the processor to perform operations including the following: Use a camera device included in a second device to detect a marker displayed by a first device; Use the camera device to capture an image of the marker; Reproject the image of the marker by the second device onto a display screen space to determine the quality of the captured image of the marker, wherein reprojecting the image of the marker includes causing the second device to display the image of the marker on the display screen space; Use the image of the marker to determine the transformation between the first device and the second device; Use the transformation to determine a common coordinate system; Use the common coordinate system to generate a shared AR session; and Cause the shared AR session to be displayed by the first device and by the second device.

13. The system according to claim 12, wherein, The operations further include: Receive a first device transformation from the first device, wherein the first device transformation is based on a first device current pose and a first device origin, and wherein the first device origin is the origin of a coordinate system tracked by the first device.

14. The system according to claim 13, wherein, Using the image of the marker to determine the transformation between the first device and the second device further includes: Determine a second offset based on the first device transformation, wherein the second offset is an offset between the second device current pose and the first device origin.

15. The system according to claim 14, wherein Using the image of the marker to determine the transformation between the first device and the second device further includes: Use the position of the marker in the image, the scale of the marker in the image, or the rotation of the marker in the image to determine the transformation.

16. The system according to claim 12, wherein, The marker includes a session identifier.

17. The system according to claim 16, wherein The marker is a Quick Response (QR) code, a natural feature marker, or a user interface of an application displayed on a display of the first device.

18. The system according to claim 16, wherein The operations further include: Extract the session identifier from the image of the marker.

19. The system according to claim 18, wherein, The operations further include: Provide access to a communication session to the first device based on the session identifier included in the marker displayed on the display of the first device; and Provide access to the communication session to the second device based on the session identifier extracted from the image of the marker.

20. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations including the following: Use a camera device included in a second device to detect a marker displayed by a first device; Use the camera device to capture an image of the marker; Reproject the marked image onto the display screen space by means of the second device to determine the quality of the captured image of the marker, wherein reprojecting the image of the marker includes causing the second device to display the image of the marker on the display screen space; Use the image of the marker to determine the transformation between the first device and the second device; Use the transformation to determine a common coordinate system; Use the common coordinate system to generate a shared AR session; and Cause the shared AR session to be displayed by the first device and by the second device.

Images