Inter-device co-located AR using hand tracking

Abstract

A method for aligning a coordinate system from a separate handheld device is described. In one aspect, the method includes accessing first pose data of a first handheld device, receiving second pose data of a second handheld device, detecting hand tracking data of a second user holding the second handheld device from the first handheld device, and transmitting the hand tracking data to the first handheld device. And aligning a first coordinate system of the first handheld device with a second coordinate system of the second handheld device based on the first pose data, the second pose data and hand tracking data of a second user holding the second handheld device.

Description

[0001] CLAIM OF PRIORITY

[0002] This application claims the benefit of priority of Greek Patent Application Serial No. 20230100308, filed April 11, 2023, and U.S. Patent Application Serial No. 18 / 357,050, filed July 21, 2023, each of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The subject matter disclosed herein relates generally to handheld devices. Specifically, the present disclosure relates to systems and methods for aligning coordinate systems of handheld devices. BACKGROUND

[0004] Augmented reality (AR) devices enable users to observe a scene while seeing relevant virtual content that can be aligned with items, images, objects, or environments in the field of view of the device. In multiple AR devices, each device has its own 6 degrees of freedom (6DoF) tracker. Sharing an AR experience between multiple AR devices can be difficult because the reference coordinate system (coordinate system) of each device is different. As a result, virtual objects displayed in AR devices can look different. BRIEF DESCRIPTION OF DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the highest three digits in a figure number refer to the figure number in which that element is first introduced. Some non-limiting examples are illustrated in the drawings, in which:

[0006] Figure 1 is a block diagram illustrating a network environment for sharing an augmented reality experience, according to one example implementation.

[0007] Figure 2 is a block diagram illustrating a handheld device, according to one example implementation.

[0008] Figure 3 is a block diagram illustrating components of a hand tracking system, according to one example implementation.

[0009] Figure 4 is a block diagram illustrating components of a handheld device pairing application, according to one example implementation.

[0010] Figure 5 is a flow diagram illustrating a method for aligning coordinate systems of handheld devices, according to one example implementation.

[0011] Figure 6is a flowchart showing a method for displaying a virtual object based on an aligned reference coordinate system according to one example embodiment.

[0012] Figure 7 is showing aligning two coordinate systems of two handheld devices according to one example embodiment.

[0013] Figure 8 is a diagrammatic representation of a networked environment according to some examples in which the present disclosure can be deployed.

[0014] Figure 9 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality according to some examples.

[0015] Figure 10 is a diagrammatic representation of a data structure maintained in a database according to some examples.

[0016] Figure 11 is a diagrammatic representation of a message according to some examples.

[0017] Figure 12 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein, according to some examples.

[0018] Figure 13 is a block diagram showing a software architecture in which examples can be implemented. DETAILED DESCRIPTION

[0019] The following description describes systems, methods, techniques, instruction sequences, and computing machine program products illustrative of example embodiments of the present subject matter. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the present subject matter. It will be apparent, however, to one skilled in the art that embodiments of the present subject matter can be practiced without some or all of these specific details. Examples are merely representative of possible variants. Structures (e.g., structural components such as modules) are optional, and can be combined or subdivided, and operations (e.g., in procedures, algorithms, or other functions) can vary in sequence or be combined or subdivided unless otherwise indicated. Unless otherwise specified, operations (e.g., in procedures, algorithms, or other functions) can be implemented by one or more processors, hardware, software, firmware, or combinations thereof, and can be implemented by one or more of the components of the system.

[0020] The term “augmented reality” (AR) is used herein to refer to an interactive experience of a real-world environment where physical objects residing in the real world are “augmented” or enhanced by computer-generated digital content (also referred to as virtual content or synthetic content). AR can also refer to a system that enables a combination of real and virtual worlds, real-time interaction, and 3D registration of virtual objects and real objects. A user of an AR system perceives virtual content that appears to be connected with or interacting with physical objects of the real world.

[0021] The term “virtual reality” (VR) is used herein to refer to a simulated experience of a virtual world environment that is completely different from the real-world environment. Computer-generated digital content is displayed in the virtual world environment. VR also refers to a system that enables a user of the VR system to be fully immersed in the virtual world environment and to interact with virtual objects presented in the virtual world environment.

[0022] The term “AR application” is used herein to refer to a computer-operated application that implements an AR experience. The term “VR application” is used herein to refer to a computer-operated application that implements a VR experience. The term “AR / VR application” refers to a computer-operated application that implements a combination of an AR experience or a VR experience.

[0023] The term “visual tracking system” is used herein to refer to a computer-operated application or system that enables the system to track visual features identified in images captured by one or more cameras of the visual tracking system. The visual tracking system constructs a model of a real-world environment based on the tracked visual features. Non-limiting examples of visual tracking systems include: visual simultaneous localization and mapping systems (VSLAM) and visual-inertial odometry (VIO) systems. VSLAM can be used to construct a map of a target from an environment or scene based on one or more cameras of the visual tracking system. A VIO system (also referred to as a visual-inertial tracking system) determines a latest pose (e.g., position and orientation) of a device based on data acquired from multiple sensors of the device (e.g., optical sensors, inertial sensors).

[0024] The term “inertial measurement unit” (IMU) is used herein to refer to a device that can report inertial states of a moving body, including acceleration, velocity, orientation, and position of the moving body. An IMU enables tracking of movement of a body by integrating acceleration and angular velocity measured by the IMU. An IMU can also refer to a combination of an accelerometer and a gyroscope that can determine and quantify linear acceleration and angular velocity, respectively. Values from the gyroscope of the IMU can be processed to obtain pitch, roll, and heading of the IMU, and thus of a body associated with the IMU. Signals from the accelerometer of the IMU can also be processed to obtain velocity and displacement of the IMU.

[0025] The term "three-degree-of-freedom tracking system" (3DOF tracking system) is used herein to refer to a device that tracks rotational movement. For example, a 3DOF tracking system can track whether a user of a head-worn device is looking left or right, rotating their head up or down, and turning left or right. However, a head-worn device cannot use a 3DOF tracking system to determine whether the user is moving around a scene by moving in the physical world. Therefore, a 3DOF tracking system may not be accurate enough to be used for positioning signals. A 3DOF tracking system can be part of an AR / VR display device that includes an IMU sensor. For example, a 3DOF tracking system uses sensor data from sensors such as accelerometers, gyroscopes, and magnetometers.

[0026] The term "six-degree-of-freedom tracking system" (6DOF tracking system) is used herein to refer to a device that tracks rotational and translational motion. For example, a 6DOF tracking system can track whether a user has turned their head and moved forward or backward, laterally or vertically, and up or down. A 6DOF tracking system may include a simultaneous localization and mapping (SLAM) system and / or a VIO system that relies on data acquired from multiple sensors, such as depth cameras and inertial sensors. The 6DOF tracking system analyzes the data from the sensors to accurately determine the pose of the display device.

[0027] The term "handheld device" is used herein to refer to a computing device that can be held in a user's hand. Handheld devices include a display, a camera, and a computing unit that operates AR applications. Handheld devices also include a 6DOF tracking system that tracks rotational and translational self-motion.

[0028] Each handheld device may include its own 6DOF tracking system, which generates its own reference coordinate system. Therefore, two or more handheld devices can have two or more different reference coordinate systems, which need to be aligned to represent the pose of any handheld device in a common coordinate system. Standard solutions involve area scanning and using reference markers (e.g., predefined 2D images) to align the coordinate system of each handheld device.

[0029] In one example, the present application describes a system for aligning the reference coordinate systems (CS) of two or more handheld devices' 6DOF trackers. For example, one handheld device tracks the other handheld device to align the coordinate systems of both devices to share an AR experience. Based on hand tracking of the hand holding the other device by another user, the relationship from the tracked point to the coordinate origin of the other device is implicitly found. Thus, the presently described approach establishes a common coordinate system by observing the other device only, rather than by detecting markers, surrogate objects, or a scene to establish a common coordinate system. The poses of the two handheld devices can be expressed in a common CS, or in two different but aligned CSs. The common CS is referred to as the reference CS and the world CS. The system determines the relative pose between the handheld devices and uses the relative pose to align the handheld devices' VIO reference coordinate systems in 3D space for sharing an AR experience.

[0030] In one example implementation, a method for aligning coordinate systems from two or more separate handheld devices is described. In one aspect, the method includes accessing first pose data of a first handheld device, receiving second pose data of a second handheld device, detecting hand tracking data of a second user holding the second handheld device from the first handheld device, and aligning a first coordinate system of the first handheld device with a second coordinate system of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device.

[0031] Accordingly, one or more of the methods described herein help to address the technical problem of resource management due to aligning coordinate systems of separate augmented reality (AR) devices. The methods described herein provide improvements to the functional operation of a computer by reducing power consumption. Other improvements include eliminating environmental mapping, fiducial markers (e.g., QR codes), or specific gestures. Accordingly, one or more of the methods described herein can avoid the need for certain working or computational resources. Examples of such computational resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0032] Figure 1 is a network diagram illustrating a network environment 100 suitable for operating handheld device B 110, handheld device A 114, and server 108, in accordance with some example implementations. Network environment 100 includes handheld device B 110, handheld device A 114, and optionally server 108 communicatively coupled to each other via network 104 (or via other wireless communication means). Handheld device B 110, handheld device A 114, and server 108 can each be wholly or partially in a cloud computing environment, as described below with respect to FIG. 1. Figure 12The described computer system is implemented in. The server 108 can be part of a network-based system. For example, the network-based system can be or include a cloud-based server system that provides additional information, such as alignment data for handheld device B 110 and handheld device A 114.

[0033] User B 106 holds handheld device B 110 with his / her hand 120. User B 106 can be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with handheld device B 110), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). User B 106 holds handheld device B 110 and aligns a rear-facing camera (not shown) of handheld device B 110 with handheld device A 114 and user A 112 in real-world environment 102. A field of view 116 of the rear-facing camera of handheld device B 110 captures an image of a hand 122 of user A 112 holding handheld device A 114.

[0034] User A 112 holds handheld device A 114 with his / her hand 122. User A 112 aligns a rear-facing camera (not shown) of handheld device A 114 with handheld device B 110 and user B 106 in real-world environment 102. A field of view 118 of the rear-facing camera of handheld device A 114 captures an image of a hand 120 of user B 106 holding handheld device B 110.

[0035] Handheld device B 110 includes a tracking system (not shown). The tracking system uses a combination of 6DoF systems or optical sensors (e.g., image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track a pose (e.g., position and orientation) of handheld device B 110 relative to real-world environment 102 to determine a location of handheld device B 110 within real-world environment 102. In one example, handheld device B 110 uses its 6DoF system to generate pose data for handheld device B 110.

[0036] Handheld device A 114 also includes its own tracking system (not shown). The tracking system uses a combination of its own 6DoF system or optical sensors (e.g., image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track the pose (e.g., position and orientation) of handheld device A 114 relative to real-world environment 102 to determine the location of handheld device A 114 within real-world environment 102. In one example, handheld device A 114 uses its 6DoF system to generate pose data for handheld device A 114.

[0037] In one example implementation, handheld device A 114 communicates its pose data to handheld device B 110 (e.g., via network 104 and server 108, or directly to handheld device B 110). In one example, handheld device B 110 computes the relative pose between handheld device B 110 and handheld device A 114 and communicates the relative pose to handheld device A 114 via server 108. In another example, the computation of the relative pose can be performed on handheld device B 110, server 108, or a combination of handheld device B 110 and server 108.

[0038] Figure 1 Any of the illustrated machines, databases, or devices can be implemented in a general-purpose computer modified (e.g., configured or programmed) by software to be a special-purpose computer to perform one or more of the functions described herein for that machine, database, or device. For example, the following discussion regarding Figure 5 and Figure 6 computer systems capable of implementing any one or more of the methodologies described herein. As used herein, a “database” is a data storage resource and can store data structured as text files, tables, spreadsheets, relational databases (e.g., object- relational databases), triple stores, hierarchical data stores, or any suitable combination thereof. Furthermore, Figure 1 Any two or more of the illustrated machines, databases, or devices can be combined into a single machine, and the functions described herein for any single machine, database, or device can be subdivided among multiple machines, databases, or devices.

[0039] The network 104 can be any network that enables communication among or between machines (e.g., the server 108), databases, and devices (e.g., the handheld device B 110, the handheld device A 114). Thus, the network 104 can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 104 can include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0040] Figure 2 is a block diagram illustrating modules (e.g., components) of the handheld device B 110, in accordance with some example embodiments. The handheld device B 110 includes the sensor 202, the display 204, the processor 208, the graphics processing unit 214, the display controller 216, and the storage device 206. Examples of the handheld device B 110 include a tablet computer or a smartphone.

[0041] The sensor 202 includes an optical sensor 228 and an IMU 212 (inertial motion unit). The optical sensor 228 includes a camera. The IMU 212 includes a combination of a gyroscope, an accelerometer, a magnetometer. Other examples of the sensor 202 include a proximity sensor or a location sensor (e.g., near field communication, GPS, Bluetooth, Wifi), an audio sensor (e.g., a microphone), or any suitable combination thereof. Note that the sensor 202 described herein is for the purpose of illustration, and thus the sensor 202 is not limited to the sensors described above. Other examples of the sensor 202 include a depth sensor, such as a structured light sensor, a time-of-flight sensor, a passive stereo sensor, and an ultrasonic device, a time-of-flight sensor.

[0042] The display 204 includes a screen or monitor configured to display images generated by the processor 208. In one example embodiment, the display 204 includes a (non-transparent) touchscreen display.

[0043] The processor 208 includes an AR application 210, a 6DOF tracker 226, a hand tracking system 224, and a handheld device pairing application 222. The AR application 210 uses computer vision to detect and recognize physical environments and items (e.g., physical objects) in the physical environments. The AR application 210 retrieves virtual objects (e.g., 3D object models) based on the recognized items or physical environments. The display 204 displays the virtual objects. The AR application 210 includes a local rendering engine that generates visualizations of the virtual objects overlaid (e.g., superimposed or otherwise displayed contemporaneously with) images of the items captured by the optical sensor 228. The AR application 210 displays the virtual objects so that the virtual objects appear anchored to the items in the physical environment. The visualization of the virtual objects can be manipulated by adjusting the positioning of the items (e.g., their physical location, orientation, or both) relative to the optical sensor 228. Similarly, the visualization of the virtual objects can be manipulated by adjusting the pose of the handheld device B 110 relative to the items.

[0044] The 6DOF tracker 226 estimates the pose of the handheld device B 110. For example, the 6DOF tracker 226 uses image data from the optical sensor 228 and corresponding inertial data from the inertial sensors 212 to track the position and pose of the handheld device B 110 relative to a frame of reference (e.g., the real-world environment 102). In one example, the 6DOF tracker 226 uses the sensor data to determine a three-dimensional pose of the handheld device B 110. The three-dimensional pose is a determined orientation and location of the handheld device B 110 relative to the user’s real-world environment 102. For example, the handheld device B 110 can use images of the user’s real-world environment 102, as well as other sensor data, to identify the relative position and orientation of the handheld device B 110 to physical objects in the real-world environment 102 surrounding the handheld device B 110. The 6DOF tracker 226 continuously collects and uses updated sensor data describing movement of the handheld device B 110 to determine updated three-dimensional poses of the handheld device B 110 that indicate changes in the relative position and orientation of the handheld device B 110 to physical objects in the real-world environment 102. The 6DOF tracker 226 provides the three-dimensional pose of the handheld device B 110 to the hand tracking system 224 and the handheld device pairing application 222.

[0045] The hand tracking system 224 allows users to interact with a computer using their hands without the need for touch, controllers, or devices. The hand tracking system can estimate a sparse hand skeleton by using the following operations:

[0046] Acquire input images: Use a camera module that captures the motion of the hand with high precision.

[0047] Image correction: apply image processing techniques for lens distortion removal, etc.

[0048] Hand detection: use color and depth data (e.g., sparse point cloud) to locate and segment hand regions from background.

[0049] Feature extraction: extract and match features across frames and construct optical flow field.

[0050] Hand pose estimation: use skeleton model to estimate positions and orientations of hand joints and bones.

[0051] Gesture recognition: use machine learning algorithms to classify hand poses into predefined gestures.

[0052] The hand tracking system 224 can provide data about the position and rotation of each finger, the whole palm, and gesture data. Example components of the hand tracking system 224 are further described below with respect to Figure 3

[0053] The handheld device pairing application 222 accesses pose data from the 6DOF tracker 226, receives pose data from handheld device A 114, accesses hand tracking information for the hand 122 of user A 112, and pairs the coordinate system of handheld device B 110 with handheld device A 114 based on the pose data from the 6DOF tracker 226, the pose data from handheld device A 114, and the hand tracking information. Example components of the handheld device pairing application 222 are further described below with respect to Figure 4

[0054] The AR application 210 uses the relative pose to enable sharing of the AR experience between handheld device B 110 and handheld device A 114. For example, the correct position / perspective of a virtual object is accurately presented in both handheld device B 110 and handheld device A 114 (e.g., user B 106 uses handheld device B 110 to point to a country on a virtual globe, and handheld device A 114 displays the virtual globe such that user B 106 appears to be aligned to the same country as perceived from the perspective of handheld device A 114). Example components of the handheld device pairing application 222 are further described below with respect to Figure 4

[0055] ​​​The graphics processing unit 214 includes a rendering engine (not shown) configured to render frames of a 3D model of a virtual object based on virtual content provided by the AR application 210 and a pose of the handheld device B 110 (relative to the handheld device A 114). In other words, the graphics processing unit 214 uses the three-dimensional pose of the handheld device B 110 to generate frames of virtual content to be presented on the display 204. For example, the graphics processing unit 214 uses the three-dimensional pose to render frames of virtual content such that the virtual content is presented in a certain orientation and position in the display 204 to appropriately augment the real environment of the user B 106. As an example, the graphics processing unit 214 can use the three-dimensional pose data to render frames of virtual content such that, when presented on the display 204, the virtual content overlaps / looks anchored to a physical object in the real-world environment 102 of the user B 106. The graphics processing unit 214 generates updated frames of virtual content based on updated three-dimensional poses of the handheld device B 110 that reflect changes in the position and orientation of the user B 106 relative to the physical object in the real-world environment 102 of the user B 106.

[0056] The graphics processing unit 214 transmits the rendered frames to the display controller 216. The display controller 216 is positioned as an intermediary between the graphics processing unit 214 and the display 204, receives image data (e.g., rendered frames) from the graphics processing unit 214, and provides the rendered frames to the display 204.

[0057] The storage device 206 stores virtual object content 218, relative pose data 220 (e.g., relative pose between handheld device B 110 and handheld device A 114, common frame of reference), and metadata 230. The virtual object content 218 includes, for example, a database of visual references (e.g., images, QR codes) and corresponding virtual content (e.g., three-dimensional models of virtual objects). The relative pose data 220 indicates a relative pose between a reference coordinate frame of the handheld device B 110 and a reference coordinate frame of the handheld device A 114.

[0058] The metadata 230 includes, for example, factory calibration parameters of the handheld device B 110, calibration parameters based on a position of the optical sensor 228 of the handheld device B 110 relative to the display 204, and time information. In other examples, the metadata 230 includes VIO calibration parameters involving IMU intrinsics, image sensor intrinsics, IMU image sensor extrinsics, and IMU image sensor time alignment parameters.

[0059] For example, the calibration parameters can include intrinsic calibration parameters of the IMU 212 (sometimes referred to herein as "IMU intrinsics"). Examples of calibration parameters that are considered IMU intrinsics include gyroscope scale, gyroscope skew, accelerometer scale, accelerometer skew, accelerometer misalignment, gyroscope misalignment, a rotation quaternion between the gyroscope and the accelerometer (e.g., a "gyroscope-accelerometer rotation quaternion"), gyroscope bias, and accelerometer bias.

[0060] The calibration parameters can include intrinsic calibration parameters of the optical sensor 228 (sometimes referred to herein as "camera intrinsics"). Examples of calibration parameters that are considered camera intrinsic include a focal length and an optical center of the optical sensor 228.

[0061] The calibration parameters can include extrinsic calibration parameters corresponding to the IMU 212 and the optical sensor 228 (sometimes referred to herein as "IMU-camera extrinsics"). In general, there is an unknown transformation between the frame of reference of the IMU 212 ("IMU reference coordinate system") and the frame of reference of the image sensor ("camera coordinate system"), and this transformation can be represented by a rotation quaternion corresponding to a rotation from the optical reference coordinate system to the IMU reference coordinate system, and a translation vector corresponding to a translation from a 3D position derived from the IMU of the handheld device B 110 to the optical reference coordinate system. This rotation quaternion and translation vector are examples of calibration parameters that are considered IMU-camera extrinsics.

[0062] The calibration parameters can include calibration parameters for performing time alignment between the IMU 212 and the optical sensor 228 (sometimes referred to herein as "IMU-camera time alignment parameters"). Examples of calibration parameters that are considered IMU-camera time alignment parameters include a time offset between the IMU 212 and the optical sensor 228 (e.g., an offset between timestamps generated, respectively).

[0063] Any one or more of the modules described herein can be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein can configure a processor to perform the operations described herein for that module. Moreover, any two or more of these modules can be combined into a single module, and the functions described herein for a single module can be divided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0064] Figure 3is a block diagram illustrating components of a hand tracking system 224, in accordance with one example embodiment. The hand tracking system 224 detects a hand pose of a user A 112 holding a handheld device A 114. For example, the hand tracking system 224 uses one or more hand tracking cameras 302 to capture tracking video frame data of the hand pose of the user A 112. The hand tracking cameras 302 transmit the tracking video frame data to hand tracking components 304 of a hand tracking pipeline 308.

[0065] The hand tracking components 304 receive the tracking video frame data and generate hand tracking data 310 based on the tracking video frame data. The hand tracking data 310 includes skeleton model data of one or more skeleton models of the hand 122 of the user A 112 in a 3D coordinate system based on landmark features extracted from the tracking video frame data, and hand pose classification data of a hand pose made by the hand of the user A 112. The skeleton model includes skeleton model features corresponding to identified visual landmarks of portions of the hand 122 of the user A 112. In some examples, the hand tracking data 310 includes landmark data, such as landmark identification, physical locations of landmarks, links between joints of the user’s fingers, and classification information of one or more landmarks associated with the hand 122 of the user A 112. In some examples, the hand pose classification data includes an indication of a hand pose or gesture made by the hand 122 of the user A 112.

[0066] For example, the hand tracking components 304 identify landmark features on portions of the hand 122 of the user A 112 captured in the tracking video frame data. In some examples, the hand tracking components 304 extract landmarks of the hand 122 of the user A 112 from the tracking video frame data using computer vision methods including, but not limited to, Harris corner detection, Shi-Tomasi corner detection, Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Features from Accelerated Segment Test (FAST), Oriented FAST and Rotated BRIEF (ORB), etc.

[0067] In other examples, the hand tracking component 304 generates the hand pose classification data and series of skeletal models of the hand tracking data 310 based on landmarks extracted from the tracking video frame data using an artificial intelligence method and a ML hand tracking model 306 previously generated using a machine learning method. In some examples, the ML hand tracking model 306 includes, but is not limited to, a neural network, a learning vector quantization network, a logistic regression model, a support vector machine, a random decision forest, a naive Bayes model, a linear discriminant analysis model, and a K- nearest neighbor model. In some examples, the machine learning method used to generate the ML hand tracking model 306 can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, dimensionality reduction, self-learning, feature learning, sparse dictionary learning, and anomaly detection.

[0068] In some examples, the hand tracking component 304 generates the series of skeletal models and the hand pose classification data of the hand tracking data 310 based on landmarks extracted from the tracking video frame data using a geometric method.

[0069] Figure 4 FIG. 4 is a block diagram illustrating a handheld device pairing application 222, according to one example implementation. The handheld device pairing application 222 includes a device VIO pose module 402, a device-to-other-user-hand pose module 406, an other-camera-to-other-user-hand pose module 408, an other-device VIO pose module 404, and a coordinate alignment engine 410.

[0070] The device VIO pose module 402 accesses 6DOF pose data from the 6DOF tracker 226. The other-device VIO pose module 404 receives 6DOF pose data from the handheld device A 114. The device-to-other-user-hand pose module 406 accesses hand tracking data (of the hand 122 of user A 112) from the hand tracking system 224. For example, the device-to-other-user-hand pose module 406 identifies 2D observations of hand joints of the handheld device A 114 as seen by the handheld device B 110. Any hand joint can be used.

[0071] The other camera to other user hand pose module 408 identifies a coordinate transformation between hand tracking data and 6DOF pose data of handheld device A 114. In one example, the other camera to other user hand pose module 408 transforms a coordinate frame transformation between a hand 122 and a device coordinate frame reference (e.g., an IMU of handheld device A 114). In other examples, an estimate of the transformation can be performed using soft constraints on the lever arm. In one example, the hand tracking component 304 estimates the transformation (hand to camera) by tracking the hand over multiple frames. The hand tracking component 304 assumes there is a rigid offset between the camera and the hand (e.g., the user does not hold their handheld device differently during a pairing). This unknown vector is then added to a solver (further described below with respect to Figure 7 that solves (a) the alignment between devices (e.g., handheld device A 114 and handheld device B 110) and (b) the unknown lever arm between cameras (e.g., a camera of handheld device A 114 and the hand 122). Note that the hand tracking system 224 uses the displacement (movement) of handheld device A 114 to solve the problem.

[0072] The coordinate alignment engine 410 aligns coordinate frames of handheld device B 110 and handheld device A 114 based on the 6DOF pose of handheld device B 110, the 6DOF pose of handheld device A 114, the hand tracking data, and a transformation between the hand tracking data and a coordinate frame of handheld device A 114. In one example, the coordinate alignment engine 410 determines a relative pose between handheld device A 114 and handheld device B 110. The coordinate alignment engine 410 sends the relative pose data to the AR application 210. The AR application 210 uses this relative pose data to enable accurate sharing of the AR experience between handheld device B 110 and handheld device A 114 by "synchronizing" / "aligning" the reference frames of each device. For example, the position / point of view of a virtual object is rendered within a common frame of reference in both handheld device B 110 and handheld device A 114.

[0073] AR applications 210 are aligned using the relative pose in the VIO reference coordinate frame of each AR device. Note that once alignment is performed, handheld device B 110 and handheld device A 114 are considered “paired” because they share a common reference frame. Once paired, the AR devices (e.g., handheld device B 110 and handheld device A 114) do not need to be synchronized again. Thus, the relative pose computation is only performed while handheld device B 110 and handheld device A 114 are connected during a joint collaborative AR session (e.g., each AR device is viewing “the same” virtual object in real-world environment 102). In another example, coordinate alignment engine 410 provides alignment data (e.g., the relative pose of handheld device B 110 with respect to handheld device A 114) to graphics processing unit 214 to accurately place / display virtual objects.

[0074] Figure 5 is a flowchart illustrating a method for aligning coordinate frames of handheld devices, according to one example implementation. Two or more user devices use an alignment process to align themselves based on a common reference coordinate frame. The operations in method 500 can be performed by handheld device B 110 using the components (e.g., modules, engines) described above with respect to Figure 2 、 Figure 3 and Figure 4 . Thus, method 500 is described by way of example with reference to handheld device B 110. However, it should be understood that at least some of the operations of method 500 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0075] At block 502, handheld device B 110 accesses pose data from 6DOF tracker 226. In one example, device VIO pose module 402 accesses pose data from 6DOF tracker 226.

[0076] At block 504, handheld device B 110 receives pose data from handheld device A 114. In one example, other device VIO pose module 404 receives pose data from handheld device A 114.

[0077] At block 506, handheld device B 110 tracks the hand of user A 112 holding handheld device A 114. In one example, hand tracking system 224 performs the hand tracking process.

[0078] At block 508, handheld device B 110 transforms coordinate frames between the hand of user A 112 and the coordinate frame of handheld device A 114 (based on the pose data of handheld device A 114). In one example, device-to-other-user-hand pose module 406 performs the operations of block 508.

[0079] At block 510, handheld device B 110 aligns the coordinate system of handheld device B 110 with handheld device A 114 based on the pose data of both handheld device B 110 and handheld device A 114 and based on the hand tracking-based coordinate transformation. In one example, coordinate alignment engine 410 performs the operations of block 510. For example, coordinate alignment engine 410 aligns itself to handheld device A 114 based on the relative position data. For example, coordinate alignment engine 410 uses the relative position data to create a transformation matrix that transforms a position in the coordinate system of handheld device A 114 to a position in the coordinate system of handheld device B 110. In subsequent operations, handheld device B 110 applies the transformation matrix to subsequent 6DoF data received from IMU 212 to compute the position and orientation (pose) of handheld device B 110.

[0080] Note that other implementations can complete similar functions using different ordering, additional or fewer operations, and different nomenclature or terminology. In some implementations, various operations can be performed in parallel with other operations, in a synchronous or asynchronous manner. The operations described herein were selected to illustrate some principles of operation in a simplified form.

[0081] Figure 6 is a flowchart showing a method for displaying virtual objects based on an aligned reference coordinate system, according to one example implementation. The operations in method 600 can be performed by handheld device B 110 using the components (e.g., modules, engines) described above with respect to Figure 2 to Figure 4 FIG. 2. Thus, method 600 is described by way of example with reference to handheld device B 110. However, it should be appreciated that at least some of the operations of method 600 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0082] According to some examples, the method includes aligning a reference coordinate system between the two devices at block 602. The operations at block 602 can be performed by handheld device pairing application 222.

[0083] According to some examples, the method includes providing the aligned reference coordinate system to the device(s) at block 604. The operations at block 604 can be performed by handheld device pairing application 222.

[0084] According to some examples, the method includes displaying virtual objects at the device based on the aligned reference coordinate system at block 606. The operations at block 606 can be performed by AR application 210.

[0085] Note that other implementations can use different ordering, additional or fewer operations, and different nomenclature or terminology to accomplish similar functions. In some implementations, various operations can be performed in parallel with other operations, in a synchronous or asynchronous manner. The operations described herein were chosen to illustrate some of the principles in a simplified form.

[0086] Figure 7 is a block diagram illustrating aligning two coordinate systems of two handheld devices, according to one example implementation. In one example, Figure 7 An environment is shown in which multiple users are collaborating. Each user has an AR device for viewing the environment and AR elements or objects. User B 106 and user A 112 are facing each other.

[0087] Each user device (e.g., handheld device B 110 and handheld device A 114) captures images of the hands from their perspective, with handheld device B 110 capturing images of user A’s 112 hand 122 and handheld device A 114 capturing images of user B’s 106 hand 120.

[0088] P A (t) represents the relative 6DOF pose at time (t). P A (t) is identified on handheld device A 114 (using the 6DOF tracker of handheld device A 114) and passed / communicated to handheld device B 110.

[0089] P B (t) represents the relative 6DOF pose at time (t). On handheld device B 130, the 6DoF tracker 226 determines the 6DoF pose (using the 6DOF tracker 226).

[0090] P B->HA (t) represents the 2D observation of the hand joints of handheld device A 134 as seen by handheld device B 110. This operation can be performed by the hand tracking system 224 and the device-to-other-user-hand-pose module 406.

[0091] P HA->A (t) represents the transform between the hand 122 and the coordinate system reference of the device (e.g., the IMU of handheld device A 114). This operation can be performed by the other-camera-to-other-user-hand-pose module 408.

[0092] The alignment of handheld device B 110 and handheld device A 114 involves taking a common coordinate system and mapping the positions and measurements in CS A of (handheld device A 114) to CS BThe pose of each handheld device is now measured in the common coordinate system, and users can collaborate in the AR / VR environment.

[0093] Networked computing environment

[0094] Figure 8 is a block diagram illustrating an example interaction system 800 for facilitating interactions (e.g., exchanging text messages, making text, audio, and video calls, or playing games) over a network. The interaction system 800 includes a plurality of user systems 802, each of which hosts a plurality of applications including an interaction client 804 and other applications 806. Each interaction client 804 is communicatively coupled to other instances of the interaction client 804 (e.g., hosted on respective other user systems 102), an interaction server system 810, and third-party servers 812 via one or more communication networks including a network 808 (e.g., the Internet). The interaction client 804 can also communicate with locally-hosted applications 806 using an application program interface (API).

[0095] Each user system 802 can include a plurality of user devices, such as a mobile device 814, a head-mounted device 816, and a computer client device 818, which are communicably connected to exchange data and messages.

[0096] The interaction client 804 interacts with other interaction clients 804 and with the interaction server system 810 via the network 808. Data exchanged between interaction clients 804 (e.g., interactions 820) and between interaction clients 804 and the interaction server system 810 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0097] The interaction server system 810 provides server-side functionality to the interaction clients 804 via the network 808. While certain functions of the interaction system 800 are described herein as being performed by the interaction client 804 or by the interaction server system 810, it can be a design choice as to whether a particular technique and function is located within the interaction client 804 or within the interaction server system 810. For example, it can be technically preferable to initially deploy a particular technique or function within the interaction server system 810, but to later migrate that technique or function to the interaction client 804 of a user system 802 that has sufficient processing power.

[0098] The interaction server system 810 supports various services and operations provided to the interaction clients 804. Such operations include sending data to the interaction clients 804, receiving data from the interaction clients 804, and processing data generated by the interaction clients 804. This data can include message content, client device information, geolocation information, media augmentations and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchange within the interaction system 800 is activated and controlled through functionality available via the user interfaces (UIs) of the interaction clients 804.

[0099] Turning now to the interaction server system 810 in particular, an application program interface (API) server 822 is coupled to, and provides a programmatic interface to, an interaction server 824 that exposes functions of the interaction server 824 to the interaction clients 804, other applications 806, and third-party servers 812. The interaction server 824 is communicatively coupled to a database server 826 that facilitates access to a database 828 in which is stored data associated with interactions processed by the interaction server 824. Similarly, a web server 830 is coupled to the interaction server 824 and provides a web-based interface to the interaction server 824. To this end, the web server 830 processes incoming network requests over the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0100] The application program interface (API) server 822 receives and transmits interaction data (e.g., commands and message payloads) between the interaction server 824 and the user systems 802 (and, e.g., the interaction clients 804 and other applications 806) as well as the third-party servers 812. In particular, the application program interface (API) server 822 provides a set of interfaces (e.g., routines and protocols) that the interaction clients 804 and other applications 806 can invoke or query to activate the functionality of the interaction server 824. The application program interface (API) server 822 exposes various functions supported by the interaction server 824, including account registration; login functionality; sending interaction data from a particular interaction client 804 to another interaction client 804 via the interaction server 824; transferring media files (e.g., images or videos) from the interaction client 804 to the interaction server 824; setting a collection of media data (e.g., a story); retrieving a list of friends of a user of a user system 802; retrieving messages and content; adding and deleting entities (e.g., friends) to / from an entity relationship graph (e.g., the entity graph 1010); pinning friends in the entity relationship graph; and opening application events (e.g., related to the interaction client 804).

[0101] The interaction server 824 hosts a number of systems and subsystems, described below with reference to Figure 9 the interaction server 824.

[0102] Linked applications

[0103] Returning to the interaction client 804, features and functionality of an external resource (e.g., a linked application 806 or a widget) are made available to the user via an interface of the interaction client 804. In this context, "external" refers to the fact that the application 806 or widget is external to the interaction client 804. External resources are typically provided by a third party, but can also be provided by the creator or provider of the interaction client 804. The interaction client 804 receives a user selection of an option to launch or access features of such an external resource. The external resource can be an application 806 installed on the user system 802 (e.g., a "native app"), or a scaled-down version of an application (e.g., a "widget") hosted on the user system 802 or located remotely from the user system 802 (e.g., on a third-party server 812). The scaled-down version of an application includes a subset of the features and functionality of the application (e.g., of a full-scale native version of the application), and is implemented using markup language documents. In some examples, the scaled-down version of an application (e.g., a "widget") is a web-based markup language version of the application, and is embedded in the interaction client 804. In addition to using markup language documents (e.g., HTML files), a widget can include script language (e.g., JavaScript® files or files) and style sheets (e.g., CSS files).

[0104] In response to receiving a user selection of an option to launch or access features of an external resource, the interaction client 804 determines whether the selected external resource is a web-based external resource or a locally installed application 806. In some cases, an application 806 that is locally installed on the user system 802 can be launched independently of and separately from the interaction client 804, e.g., by selecting an icon corresponding to the application 806 on a home screen of the user system 802. A scaled-down version of such an application can be launched or accessed via the interaction client 804, and in some examples, no part of the scaled-down application can be accessed outside of the interaction client 804 or only limited parts of the scaled-down application can be accessed outside of the interaction client 804. A scaled-down application can be launched by the interaction client 804 receiving, e.g., a markup language document associated with the scaled-down application from a third-party server 812 and processing such a document.

[0105] ​​​In response to determining that the external resource is a locally installed application 806, the interaction client 804 instructs the user system 802 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 804 communicates with the third-party server 812, for example, to obtain a markup language document corresponding to the selected external resource. The interaction client 804 then processes the obtained markup language document to present the web-based external resource within the user interface of the interaction client 804.

[0106] The interaction client 804 can notify a user of the user system 802 or other users (e.g., “friends”) associated with such a user of activity occurring in one or more external resources. For example, the interaction client 804 can provide participants in a conversation (e.g., a chat conversation) in the interaction client 804 with notifications related to one or more members of a group of users currently or recently using an external resource. One or more users can be invited to join an active external resource or launch a recently used but not currently active external resource (in a group of friends). The external resource can provide participants in a conversation each using a respective interaction client 804 with the ability to share an item, condition, state, or location in the external resource with one or more members of a group of users in a chat session. The shared item can be an interactive chat card that members of the chat can utilize to interact with, for example, to launch the corresponding external resource, view particular information within the external resource, or bring the members of the chat to a particular location or state within the external resource. Within a given external resource, response messages can be sent to users on the interaction client 804. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0107] The interaction client 804 can present a list of available external resources (e.g., applications 806 or widgets) to a user to launch or access a given external resource. The list can be presented in the manner of a contextually relevant menu. For example, icons representing different applications (or widgets) of applications 806 (or widgets) can vary based on how the user launched the menu (e.g., from a conversation interface or from a non-conversation interface).

[0108] System Architecture

[0109] Figure 9is a block diagram illustrating additional details regarding the interaction system 904, according to some examples. In particular, the interaction system 904 is shown to include an interaction client 908 and an interaction server 930. The interaction system 904 includes a number of subsystems that are supported on the client side by the interaction client 908 and on the server side by the interaction server 930. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) can have components that enable the microservice subsystem to operate independently and communicate with other services. Example components of a microservice subsystem can include:

[0110] • Function logic: The function logic implements the functionality of the microservice subsystem, representing a particular capability or function provided by the microservice

[0111] • API interface: The microservice can communicate with each other component using a well-defined API or interface, such as a lightweight protocol like REST or messaging. The API interface defines the inputs and outputs of the microservice subsystem and how the microservice subsystem interacts with other microservice subsystems of the interaction system 904.

[0112] • Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanism (e.g., with the database server 826 and the database 828). This enables the microservice subsystem to operate independently of other microservices of the interaction system 904.

[0113] • Service discovery: The microservice subsystem can find and communicate with other microservice subsystems of the interaction system 904. The service discovery mechanism enables the microservice subsystem to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0114] • Monitoring and logging: The microservice subsystem can need to be monitored and logged to ensure availability and performance. The monitoring and logging mechanism enables the health and performance of the microservice subsystem to be tracked.

[0115] In some examples, the interaction system 904 can employ a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture:

[0116] An example subsystem is discussed below.

[0117] The image processing system 902 provides various functionality that enables a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0118] The camera system 906 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 802 (e.g., directly or via operating system controls) to modify and enhance live images captured and displayed via the interactive client 908.

[0119] The augmentation system 910 provides functionality related to the generation and publication of augmentations (e.g., media overlays) for images captured in real-time by the camera of the user system 802 or images retrieved from the memory of the user system 802. For example, the augmentation system 910 operatively selects, presents, and displays media overlays (e.g., image filters or image lenses) for the interactive client 908 for augmenting live images received via the camera system 906 or stored images retrieved from the memory 1206 of the user system 802. These augmentations are selected by the augmentation system 910 based on some inputs and data, for example,

[0120] • the geographic location of the user system 802; and

[0121] • entity relationship information of the user of the user system 802.

[0122] Augmentations can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to a media content item (e.g., a photo or a video) at the user system 802 for communication in a message or to video content, e.g., a video content stream or feed sent from the interactive client 908. Thus, the image processing system 902 can interact with and support various subsystems of the communication system 912, e.g., the messaging system 914 and the video communication system 916.

[0123] Media overlays can include text or image data that can be overlaid on a photo taken by the user system 802 or a video stream made by the user system 802. In some examples, the media overlays can be location overlays (e.g., Venice Beach), name of a live event, or a business name overlay (e.g., Beachside Café). In other examples, the image processing system 902 uses the geographic location of the user system 802 to identify a media overlay that includes the name of a business at the geographic location of the user system 802. The media overlay can include other indicia associated with the business. The media overlays can be stored in the database 828 and accessed by the database server 826.

[0124] The image processing system 902 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify circumstances under which particular media overlays should be provided to other users. The image processing system 902 generates media overlays that include the uploaded content and associate the uploaded content with the selected geographic location.

[0125] The augmentation creation system 918 supports an augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive client 908. The augmentation creation system 918 provides a library of built-in features and tools for content creators, including, for example, custom shaders, tracking techniques, and templates.

[0126] In some examples, the augmentation creation system 918 provides a merchant-based publishing platform that enables merchants to select particular augmentations associated with geographic locations via a bidding process. For example, the augmentation creation system 918 associates the media overlays of the highest bidding merchants with corresponding geographic locations for a predefined amount of time.

[0127] The communication system 912 is responsible for enabling and handling various forms of communication and interaction within the interactive system 904 and includes a messaging system 914, an audio communication system 920, and a video communication system 916. The messaging system 914 is responsible for enforcing temporary or time-limited access to content by the interactive client 908. The messaging system 914 includes a plurality of timers (e.g., in a transient timer system) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive client based on the duration and display parameters associated with a message or collection of messages (e.g., a story). The audio communication system 920 enables and supports audio communication (e.g., real-time audio chat) between a plurality of interactive clients 908. Similarly, the video communication system 916 enables and supports video communication (e.g., real-time video chat) between a plurality of interactive clients 908.

[0128] The user management system 922 is operationally responsible for managing user data and profiles and maintaining entity information (e.g., stored in the entity table 1110, the entity graph 1010, and the profile data 1002) about users and relationships between users of the interactive system 904.

[0129] The collection management system 924 is operationally responsible for managing collections or sets of media (e.g., sets of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified period of time (e.g., the duration of an event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 924 can also be responsible for publishing icons to the user interfaces of the interactive clients 908 that provide notifications of particular collections. The collection management system 924 includes curation functionality that enables a curator to manage and curate particular collections of content. For example, a curation interface enables an event organizer to curate a collection of content related to a particular event (e.g., to delete inappropriate content or redundant messages). In addition, the collection management system 924 employs machine vision (or image recognition technology) and content rules to automatically curate collections of content. In certain examples, users can be paid compensation for including user-generated content into a collection. In such cases, the collection management system 924 operates to automatically pay such users for use of their content.

[0130] The map system 926 provides various geographic location (e.g., geolocation) functionality and supports the presentation of map-based media content and messages by the interactive clients 908. For example, the map system 926 enables the display of user icons or avatars (e.g., stored in the profile data 1002) on a map to indicate the current or past locations of a user's "friends" within the context of a map, as well as media content (e.g., collections of messages including photos and videos) generated by these friends. For example, on a map interface of the interactive client 908, a message posted by a user from a particular geographic location to the interactive system 904 can be displayed to a particular user's "friends" within the context of that particular location on the map. A user can also share his or her location and status information with other users of the interactive system 904 via the interactive client 908 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of a map interface of the interactive client 908.

[0131] The game system 928 provides various game functionality within the context of the interaction client 908. The interaction client 908 provides a game interface that provides a list of available games that can be launched by a user within the context of the interaction client 908 and played with other users of the interaction system 904. The interaction system 904 also enables a particular user to invite other users to participate in playing a particular game by issuing invitations from the interaction client 908 to the other users. The interaction client 908 also supports voice, video, and text messaging (e.g., chat) within the context of playing games, provides leaderboards for games, and also supports providing in-game rewards (e.g., game currency and items).

[0132] The external resource system 932 provides an interface for the interaction client 908 to communicate with remote servers (e.g., third-party servers 812) to launch or access external resources (i.e., applications or applets). Each third-party server 812 hosts an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5), for example. The interaction client 908 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the third-party server 812 associated with the web-based resource. The application hosted by the third-party server 812 is programmed in JavaScript with a software development kit (SDK) provided by the interaction server 930. The SDK includes an application program interface (API) with functionality that can be called or activated by the web-based application. The interaction server 930 hosts a JavaScript library that provides given external resource access to particular user data of the interaction client 908. HTML5 is an example of a technology to program games, but applications and resources programmed based on other technologies can be used.

[0133] To integrate the functionality of the SDK into the web-based resource, the SDK is downloaded by the third-party server 812 from the interaction server 930 or otherwise received by the third-party server 812. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functionality of the SDK to integrate features of the interaction client 908 into the web-based resource.

[0134] The SDK stored on the interaction server system 810 effectively provides a bridge between external resources (e.g., applications 806 or widgets) and the interaction client 908. This gives the user a seamless experience of communicating with other users on the interaction client 908 while also preserving the look and feel of the interaction client 908. To bridge the communication between external resources and the interaction client 908, the SDK facilitates the communication between the third party servers 812 and the interaction client 908. A bridge script running on the user system 802 establishes two one-way communication channels between the external resources and the interaction client 908. Messages are sent asynchronously between the external resources and the interaction client 908 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identification and sending a message with the callback identification.

[0135] By using the SDK, not all information from the interaction client 908 is shared with the third party servers 812. The SDK limits which information is shared based on the needs of the external resources. Each third party server 812 provides the interaction server 930 with an HTML5 file corresponding to the web-based external resource. The interaction server 930 can add a visual representation (e.g., a box design or other graphic) of the web-based external resource in the interaction client 908. Once the user selects the visual representation or indicates through the GUI of the interaction client 908 that the interaction client 908 access a feature of the web-based external resource, the interaction client 908 obtains the HTML5 file and instantiates the resource for accessing the feature of the web-based external resource.

[0136] The interaction client 908 presents a graphical user interface for the external resource (e.g., a landing page or title screen). During, before, or after presenting the landing page or title screen, the interaction client 908 determines whether the launched external resource has been previously authorized to access user data of the interaction client 908. In response to determining that the launched external resource has been previously authorized to access user data of the interaction client 908, the interaction client 908 presents another graphical user interface of the external resource that includes functionality and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access user data of the interaction client 908, after displaying the landing page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interaction client 908 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access user data. The menu identifies types of user data that the external resource will be authorized to use. In response to receiving a user selection of an accept option, the interaction client 908 adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client 908. The external resource is authorized by the interaction client 908 to access user data under an OAuth 2 framework.

[0137] The interaction client 908 controls types of user data shared with external resources based on types of the external resources that are authorized. For example, external resources that include full-scale applications (e.g., the application 806) are provided access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics). As another example, external resources that include small-scale versions of applications (e.g., web-based versions of the application) are provided access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0138] The advertising system 934 is operable to enable third parties to purchase advertisements to be presented to end users via the interaction client 908, and also handles the delivery and presentation of these advertisements.

[0139] The artificial intelligence and machine learning system 936 provides various services to different subsystems within the interaction system 904. For example, the artificial intelligence and machine learning system 936 operates with the image processing system 902 and the camera system 906 to analyze images and extract information, such as objects, text, or faces. This information can then be used by the image processing system 902 to enhance, filter, or manipulate the images. The artificial intelligence and machine learning system 936 can be used by the augmentation system 910 to generate augmented content and augmented reality experiences, such as adding virtual objects or animations to real-world images. The communication system 912 and the messaging system 914 can use the artificial intelligence and machine learning system 936 to analyze communication patterns and provide insights into how users interact with each other, as well as to provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic. The artificial intelligence and machine learning system 936 can also provide chatbot functionality to the message interactions 820 between the user system 802 and the interaction server system 810. The artificial intelligence and machine learning system 936 can also cooperate with the audio communication system 920 to provide speech recognition and natural language processing capabilities, enabling users to interact with the interaction system 904 using voice commands.

[0140] Data Architecture

[0141] Figure 10 FIG. 10 is a schematic diagram illustrating a data structure 1000 that can be stored in the database 1004 of the interaction server system 810, in accordance with certain examples. While the contents of the database 1004 are illustrated as including a plurality of tables, it will be appreciated that data can be stored in other types of data structures (e.g., object-oriented databases).

[0142] The database 1004 includes message data stored within a message table 1006. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Reference is made below to FIG. 11 for a more detailed description of the message data. Figure 10 Further details regarding information that can be included in a message and included within the message data stored in the message table 1006 are described below.

[0143] The entity table 1008 stores entity data and is linked (e.g., by reference) to the entity graph 1010 and the profile data 1002. Entities for which records are maintained within the entity table 1008 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of the entity type, any entity for which the interaction server system 810 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0144] The entity graph 1010 stores information about relationships and associations between entities. Such relationships can be social, professional (e.g., working at a common company or organization), interest-based, or activity-based, merely as examples. Certain relationships between entities can be one-way, such as a personal user's subscription to digital content of a business or publication user (e.g., a newspaper or other digital media channel or brand). Other relationships can be two-way, such as a "friend" relationship between individual users of the interaction system 800.

[0145] Certain permissions and relationships can be attached to each relationship, and can also be attached to each direction of a relationship. For example, a two-way relationship (e.g., a friend relationship between personal users) can include authorization for publication of digital content items between the personal users, but can impose certain restrictions or filters on publication of such digital content items (e.g., based on content characteristics, location data, or time-of-day data). Similarly, a subscription relationship between a personal user and a business user can impose varying degrees of restriction on publication of digital content from the business user to the personal user, and can significantly restrict or prevent publication of digital content from the personal user to the business user. As an example of an entity, a particular user can record certain restrictions in the record for that entity within the entity table 1008 (e.g., through privacy settings). Such privacy settings can apply to all types of relationships in the context of the interaction system 800, or can selectively apply to certain types of relationships.

[0146] The profile data 1002 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 1002 can be selectively used and presented to other users of the interaction system 800. In the case of an entity being a person, the profile data 1002 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 800 and on map interfaces displayed by the interaction client 804 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to transmit at a particular time.

[0147] In the case of an entity being a group, the profile data 1002 for the group can similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the relevant group.

[0148] The database 1004 also stores augmentation data, such as overlays or filters, in an augmentation table 1012. The augmentation data is associated with video (data for which is stored in a video table 1014) and images (data for which is stored in an image table 1016) and is applied to the video and images.

[0149] In some examples, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. The filter can be of various types, including a user-selected filter from a set of filters presented to a sending user by the interaction client 804 when the sending user is composing a message. Other types of filters include a geo-location filter (also referred to as a geofilter), which can be presented to the sending user based on a geo-location. For example, a geo-location filter specific to a nearby or special location can be presented by the interaction client 804 within a user interface based on geo-location information determined by a global positioning system (GPS) unit of the user system 802.

[0150] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 804 based on other input or information gathered by the user system 802 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the user system 802, or a current time.

[0151] Other augmentation data that can be stored within the image table 1016 includes augmented reality content items (e.g., corresponding to an application "lens" or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0152] A collection table 1018 stores data regarding collections of messages and associated image, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in the entity table 1008). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the interaction client 908 can include a user-selectable icon to enable a sending user to add particular content to his or her personal story.

[0153] Collections can also constitute "live stories," which are collections of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via a user interface of the interaction client 804. A live story can be identified to a user by the interaction client 804 based on the user's location. The end result is a "live story" told from a group perspective.

[0154] Yet another type of content collection is referred to as a "location story," which enables users of the user system 802 who are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some examples, contributions to a location story can employ secondary authentication to verify that an end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0155] As mentioned above, the video table 1014 stores video data that, in some examples, is associated with messages for which records are maintained within the message table 1006. Similarly, the image table 1016 stores image data that is associated with messages for which message data is stored in the entity table 1008. The entity table 1008 can associate various augmentations from the augmentation table 1012 with various images and videos stored in the image table 1016 and the video table 1014.

[0156] Data communication architecture

[0157] Figure 11 FIG. 10 is a schematic diagram illustrating the structure of a message 1100 generated by the interaction client 804 for transmission to another interaction client 804 via the interaction server 824, according to some examples. The content of a particular message 1100 is used to populate the message table 1006 stored within the database 828 accessible by the interaction server 824. Similarly, the content of the message 1100 is stored in memory as "in-transit" or "in-flight" data for the user system 802 or the interaction server 824. The message 1100 is shown to include the following example components:

[0158] • Message identifier 1102: a unique identifier that identifies the message 1100.

[0159] • Message text payload 1104: text to be generated by a user via a user interface of the user system 802 and included in the message 1100.

[0160] • Message image payload 1106: image data captured by a camera component of user system 802 or retrieved from a memory component in user system 802 and included in message 1100. Image data for a sent or received message 1100 can be stored in image table 1122.

[0161] • Message video payload 1108: video data captured by a camera component or retrieved from a memory component of user system 802 and included in message 1100. Video data for a sent or received message 1100 can be stored in image table 1122.

[0162] • Message audio payload 1112: audio data captured by a microphone or retrieved from a memory component of user system 802 and included in message 1100.

[0163] • Message augmentation data 1114: augmentation data (e.g., filters, stickers, or other annotations or augmentations) that represent augmentations to be applied to message image payload 1106, message video payload 1108, or message audio payload 1112 of message 1100. Augmentation data for a sent or received message 1100 can be stored in augmentation table 1116.

[0164] • Message duration parameter 1118: a parameter value indicating, in seconds, an amount of time that content (e.g., message image payload 1106, message video payload 1108, message audio payload 1112) of a message is to be presented to or made accessible to a user via interaction client 804.

[0165] • Message geolocation parameter 1120: geolocation data (e.g., latitude and longitude coordinates) associated with a content payload of a message. Multiple message geolocation parameter 1120 values can be included in a payload, each of which is associated with a content item included in the content (e.g., a particular image within message image payload 1106 or a particular video in message video payload 1108).

[0166] • Message story identifier 1124: an identifier value that identifies one or more content collections (e.g., "stories" identified in collection table 1126) associated with a particular content item in message image payload 1106 of message 1100. For example, multiple images within message image payload 1106 can each be associated with multiple content collections using the identifier value.

[0167] • Message tags 1128: Each message 1100 can be tagged with multiple tags, each of which indicates a subject of content included in the message payload. For example, where a particular image included in the message image payload 1106 depicts an animal (e.g., a lion), a tag value can be included within the message tags 1128 that indicates the relevant animal. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition.

[0168] • Message sender identifier 1130: An identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of the user system 802 on which the message 1100 was generated and from which the message 1100 was sent.

[0169] • Message receiver identifier 1132: An identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of the user system 802 to which the message 1100 is addressed.

[0170] The content (e.g., values) of the various components of the message 1100 can be pointers to locations in tables where the content data values are stored. For example, the image value in the message image payload 1106 can be a pointer (or address) to a location within the image table 1122. Similarly, the value within the message video payload 1108 can point to data stored within the image table 1122, the value stored within the message augmentation data 1114 can point to data stored in the augmentation table 1116, the value stored within the message story identifier 1124 can point to data stored in the collection table 1126, and the values stored within the message sender identifier 1130 and the message receiver identifier 1132 can point to user records stored within the entity table 1110.

[0171] Machine architecture

[0172] Figure 12is a diagrammatic representation of the machine 1200 within which instructions 1202 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1200 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 1202 can cause the machine 1200 to execute any one or more of the methods described herein. The instructions 1202 transform the general, non-programmed machine 1200 into a particular machine 1200 programmed to carry out the described and illustrated functions in the manner described. The machine 1200 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1200 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 1200 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile telephone, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1202, sequentially or otherwise, that specify actions to be taken by machine 1200. Further, while a single machine 1200 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 1202 to perform any one or more of the methodologies discussed herein. For example, the machine 1200 can include any of the user system 802 or multiple server devices forming part of the interaction server system 810. In some examples, the machine 1200 can further include a client system and a server system, where certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0173] The machine 1200 can include processors 1204, memory 1206, and I / O components 1248, which can be configured to communicate with each other via a bus 1210. In an example, the processors 1204 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 1212 and a processor 1214 that execute instructions 1202. The term “processor” is intended to include multiple processors 1204 that can be present in a Figure 12 Multiple processors 1204 are shown, but the machine 1200 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0174] The memory 1206 includes a main memory 1216, a static memory 1218, and a storage unit 1220 each of which can be accessed via the bus 1210 by the processors 1204. The main memory 1216, static memory 1218, and storage unit 1220 store the instructions 1202 embodying any one or more of the methodologies or functions described herein. The instructions 1202 can also reside completely, or a portion thereof, within the main memory 1216, within the static memory 1218, within the storage unit 1220, within at least one of the processors 1204 (e.g., within a cache of the processors), or any suitable combination thereof, during execution thereof by the machine 1200.

[0175] The I / O components 1208 can include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. Specific I / O components 1208 included in the machine 1200 will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 1208 can include Figure 12Many other components not shown in FIG. 12. In various examples, the I / O components 1208 can include output components 1224 and input components 1226. The output components 1224 can include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 1226 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0176] In further examples, the I / O components 1208 can include biometric components 1228, motion components 1230, environmental components 1232, or position components 1234, among a wide array of other components. For example, the biometric components 1228 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body poses, or eye

[0177] Example types of BMI technology include:

[0178] • Electroencephalography (EEG)-based BMIs, which utilize electrodes placed on the scalp to record electrical activity in the brain.

[0179] • Invasive BMIs, which use electrodes implanted surgically in the brain.

[0180] • Optogenetic BMIs, which utilize light to control the activity of specific neural cells in the brain.

[0181] Any biometric data collected by the biometric components is captured and stored only with user approval and deleted upon user request. Furthermore, such biometric data can be used for very limited purposes (e.g., identity verification). To ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to such data is limited to authorized personnel (if access to such data occurs). Any use of biometric data can be strictly limited to identity verification purposes and the data cannot be shared or sold to any third party without explicit consent of the user. Furthermore, appropriate technical and organizational measures are also implemented to ensure the security and confidentiality of this sensitive information.

[0182] Motion components 1230 include, e.g., acceleration sensor components (e.g., accelerometers), gravitation sensor components, rotation sensor components (e.g., gyroscopes).

[0183] Environmental components 1232 include, e.g., one or more cameras (with still and / or video capabilities), illumination sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to a physical environment.

[0184] With respect to cameras, user system 802 can have a camera system that includes, e.g., a front-facing camera on a front surface of user system 802 and a rear-facing camera on a rear surface of user system 802. The front-facing camera can be used, e.g., to capture still images and videos of a user of user system 802 (e.g., “selfies”), which can then be enhanced with the enhancement data (e.g., filters) described above. The rear-facing camera can be used, e.g., to capture still images and videos in a more conventional camera mode, which are similarly enhanced with enhancement data. In addition to front- and rear-facing cameras, user system 802 can also include a 360° camera for capturing 360° photos and videos.

[0185] Additionally, the camera system of the user system 802 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even a triple, quad, or penta- rear-facing camera configuration on the front and back sides of the user system 802. For example, these multi-camera systems can include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0186] The positioning components 1234 include location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0187] Communication can be implemented using a wide variety of technologies. The I / O ® components and other communication components to provide communication via other modalities. The devices 1240 can be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

[0188] Moreover, the communication components 1236 can detect identifiers or include components operable to detect identifiers. For example, the communication components 1236 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one- dimensional bar codes such as Universal Product Code (UPC) bar codes, multidimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify acoustic signals indicating a specific location). Additionally, a variety of information can be derived via the communication components 1236, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that can indicate a specific location, and so forth.

[0189] Various memories, such as main memory 1216, static memory 1218, and memory of processor 1204, and storage unit 1220 can store one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 1202), when executed by processor 1204, cause various operations to implement the disclosed examples.

[0190] The instructions 1202 can be transmitted or received over the network 1238 via the network interface device (e.g., network interface component included in communication components 1236) using a transmission medium and any one of a number of well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, instructions 1202 can be transmitted or received using a transmission medium via the coupling (e.g., peer-to-peer coupling) to device 1240.

[0191] Software Architecture

[0192] Figure 13 is a block diagram 1300 illustrating a software architecture 1302, which can be installed on any one or more of the devices described herein. The software architecture 1302 is provided for example by hardware, such as machine 1304 that includes processors 1306, memory 1308, and I / O components 1310. In this example, the software architecture 1302 can be conceptualized as a stack of layers, where each layer provides particular functionality. The software architecture 1302 includes layers such as an operating system 1312, libraries 1314, frameworks 1316, and applications 1318. Operationally, the applications 1318 invoke API calls 1320 through the software stack and receive messages 1322 in response to the API calls 1320.

[0193] The operating system 1312 manages hardware resources and provides common services. The operating system 1312 includes, for example, a kernel 1324, services 1326, and drivers 1328. The kernel 1324 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 1324 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 1326 can provide other common services for the other software layers. The drivers 1328 are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 1328 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, and so forth.

[0194] The libraries 1314 provide common low-level infrastructure used by the applications 1318. The libraries 1314 can include system libraries 1330 (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries 1314 can include API libraries 1332 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render two dimensional (2D) and three dimensional (3D) graphics on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 1314 can also include a wide variety of other libraries 1334 to provide many other APIs to the applications 1318.

[0195] The frameworks 1316 provide common high-level infrastructure used by the applications 1318. For example, the frameworks 1316 provide various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The frameworks 1316 can provide a broad spectrum of other APIs that can be used by the applications 1318, some of which can be specific to a particular operating system or platform.

[0196] In an example, the applications 1318 include a home application 1336, a contacts application 1338, a browser application 1340, a book reader application 1342, a location application 1344, a media application 1346, a messaging application 1348, a game application 1350, and a broad assortment of other applications such as a third party application 1352. The applications 1318 are programs that execute functions defined in the programs. Programs may

[0197] Examples

[0198] Example 1 is a method comprising: accessing first pose data of a first handheld device; receiving second pose data of a second handheld device; detecting, from the first handheld device, hand tracking data of a second user holding the second handheld device; and aligning a first coordinate frame of the first handheld device with a second coordinate frame of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device.

[0199] In Example 2, the subject matter of Example 1 includes: mapping the first coordinate frame to the second coordinate frame by aligning the first coordinate frame with the second coordinate frame; and displaying content in a display of the first handheld device based on the mapping.

[0200] In Example 3, the subject matter of Examples 1-2 includes: a first 6 degrees of freedom tracker configured to identify the first pose data at time t, wherein the second handheld device includes a second 6 degrees of freedom tracker configured to identify the second pose data at time t, wherein the second handheld device is configured to wirelessly send the second pose data at time t to the first handheld device.

[0201] In Example 4, the subject matter of Examples 1-3 includes: wherein the first handheld device includes a first camera aimed at the second handheld device and a hand of the second user holding the second handheld device, wherein the second handheld device includes a second camera aimed at the first handheld device.

[0202] In Example 5, the subject matter of Examples 1-4 includes: wherein the second handheld device is within a field of view of the first camera of the first handheld device.

[0203] In Example 6, the subject matter of Examples 1-5 includes: wherein detecting, from the first handheld device, the hand tracking data of the second user holding the second handheld device includes: capturing, using the first camera of the first handheld device, images of one or more fingers of the hand of the second user; and identifying, using a hand tracking identification process, positions of hand joints based on the images of the one or more fingers of the hand of the second user.

[0204] In Example 7, the subject matter of Example 6 includes: identifying a first coordinate transformation between the positions of the hand joints and a second pose of the second handheld device, the second pose based on the second pose data; and identifying a second coordinate transformation between the positions of the hand joints and a first pose of the first handheld device, the first pose based on the first pose data.

[0205] In Example 8, the subject matter of Example 7 includes, wherein aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device is further based on a first coordinate transformation and a second coordinate transformation.

[0206] In Example 9, the subject matter of Example 8 includes: detecting initiation of a session of an augmented reality application at the first handheld device; in response to detecting the initiation of the session, calibrating the first handheld device by aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device for a predetermined number of image frames generated using the first handheld device; and displaying content in a display of the first handheld device based on the alignment of the first coordinate frame with the second coordinate frame.

[0207] In Example 10, the subject matter of Examples 1-9 includes: determining a relative pose between the first handheld device and the second handheld device by: identifying a first reference coordinate frame based on the first pose data; identifying a second reference coordinate frame based on the second pose data; and forming a world reference coordinate frame based on the first reference coordinate frame and the second reference coordinate frame.

[0208] Example 11 is a server comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the server to: access first pose data of a first handheld device; receive second pose data of a second handheld device; detect, from the first handheld device, hand tracking data of a second user holding the second handheld device; and align a first coordinate frame of the first handheld device with a second coordinate frame of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device.

[0209] In Example 12, the subject matter of Example 11 includes, wherein the instructions further configure the server to: map the first coordinate frame to the second coordinate frame by aligning the first coordinate frame with the second coordinate frame; and display content in a display of the first handheld device based on the mapping.

[0210] In Example 13, the subject matter of Examples 11-12 includes: a first 6- degree-of-freedom tracker configured to identify the first pose data at time t, wherein the second handheld device includes a second 6-degree-of-freedom tracker configured to identify the second pose data at time t, wherein the second handheld device is configured to wirelessly transmit the second pose data at time t to the first handheld device.

[0211] In Example 14, the subject matter of Examples 11-13 includes, wherein the first handheld device includes a first camera aimed at the second handheld device and a hand of a second user holding the second handheld device, wherein the second handheld device includes a second camera aimed at the first handheld device.

[0212] In Example 15, the subject matter of Examples 11-14 includes, wherein the second handheld device is within a field of view of the first camera of the first handheld device.

[0213] In Example 16, the subject matter of Examples 11-15 includes, wherein detecting, from the first handheld device, hand tracking data of a second user’s hand holding the second handheld device includes: capturing, using a first rear-facing camera of the first handheld device, an image of one or more fingers of the second user’s hand; and identifying, using a hand tracking recognition process, a position of a joint of the hand based on the image of the one or more fingers of the second user’s hand.

[0214] In Example 17, the subject matter of Example 16 includes, wherein the instructions further configure the server to: identify a first coordinate transform between the position of the joint of the hand and a second pose of the second handheld device, the second pose based on the second pose data; and identify a second coordinate transform between the position of the joint of the hand and a first pose of the first handheld device, the first pose based on the first pose data.

[0215] In Example 18, the subject matter of Example 17 includes, wherein aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device is further based on the first coordinate transform and the second coordinate transform.

[0216] In Example 19, the subject matter of Example 18 includes, wherein the instructions further configure the server to: detect initiation of a session of an augmented reality application at the first handheld device; in response to detecting initiation of the session, calibrate the first handheld device by aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device for a predetermined number of image frames generated with the first handheld device; and display content in a display of the first handheld device based on the alignment of the first coordinate frame with the second coordinate frame.

[0217] Example 20 is a non-transitory computer-readable storage medium comprising instructions that, when executed by a server, cause the server to: access first pose data of a first handheld device; receive second pose data of a second handheld device; detect, from the first handheld device, hand tracking data of a second user’s hand holding the second handheld device; and align a first coordinate frame of the first handheld device with a second coordinate frame of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user’s hand holding the second handheld device.

[0218] Example 21 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0219] Example 22 is an apparatus comprising means to implement any of the examples of 1-20.

[0220] Example 23 is a system to implement any of the examples of 1-20.

[0221] Example 24 is a method to implement any of the examples of 1-20.

[0222] While implementations have been described with reference to particular embodiments, it will be apparent to those of ordinary skill in the art that various modifications and changes can be made to these implementations without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments of the present application and together with the description, serve to explain the principles of the present application. The implementations shown are intended to be illustrative of the present teachings and not limiting thereof. Other implementations can be apparent to those of ordinary skill in the art and can be made without departing from the scope of the present teachings. Accordingly, the particular implementations are not to be taken as limiting in scope and the scope of the present teachings is only to be limited as by the appended claims and their equivalents.

[0223] Such embodiments of the inventive subject matter can be referred to herein, individually and / or collectively, by the term "application" merely for convenience and without intending to voluntarily limit the scope of this application to any single application or inventive concept. Thus, although specific implementations have been illustrated and described herein, it will be appreciated that any arrangement can be substituted for the specific implementations shown. This disclosure is intended to cover any and all changes or modifications within the scope of the various implementations. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

[0224] The Abstract of the Disclosure is provided to allow a reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that the Abstract is not intended to be used to interpret or limit the scope or the meaning of the claims. Additionally, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, where each claim stands as a separate embodiment.

[0225] Glossary

[0226] “Carrier signal” means, for example, any intangible medium that facilitates communication or transfer of computer program instructions between hardware devices or software programs, such as a signal, carrier wave, or other tangible medium. The instructions can be transmitted or received over a network via the network interface device utilizing a transmission medium.

[0227] “Client device” means, for example, any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0228] “Communication network” means, for example, one or more portions of a network that can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other

[0229] A "component" refers to a logical, physical, or device that has boundaries that are defined by function or subroutine calls, branches points, APIs, or other techniques that provide a partitioning or modularization of a particular processing or control function. Components can be combined via their interfaces to create a machine process. A component can be a packaged functional hardware unit designed for use with other components and typically provides a particular function or set of functions. Components can be software or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be appreciated that theWhere multiple hardware components are present, communication can be achieved through signal transmission among or between two or more of the hardware components (e.g., through appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can for example be achieved through storage of information in memory structures to which the multiple hardware components have access, and retrieval of that information. For example, one hardware component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware component can then, at a later time, access the memory device to retrieve and process the stored output. Hardware components can also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information). The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor- implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented components. Moreover, a processor or processors can also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some examples, the processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components can be distributed across a number of geographic locations.

[0230] “Computer-readable storage medium” refers to both machine-storage media and transmission media. Thus, the terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure.

[0231] A "ephemeral message" refers to a message that is accessible, for example, for a time-limited duration. An ephemeral message can be text, image, video, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0232] "Machine-storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid state memories, as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine-storage medium," "device-storage medium," "computer-storage medium," and "device-storage media" mean the same thing and can be used interchangeably in this disclosure. The terms "machine-storage medium," "computer-storage medium," and "device-storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are, by necessity, excluded from the term "signal medium."

[0233] "Non-transitory computer-readable storage medium" refers to, for example, a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0234] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term "signal medium" shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

[0235] "User device" refers to a device that is accessed by, controlled by, or owned by a user and with which a user interacts to perform actions or with which a user interacts, including interactions with other users or computer systems.

Claims

1. A method comprising: accessing first pose data of a first handheld device; receiving second pose data of a second handheld device; detecting, from the first handheld device, hand tracking data of a second user holding the second handheld device; and aligning a first coordinate frame of the first handheld device with a second coordinate frame of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device.

2. The method of claim 1, further comprising: mapping the first coordinate frame to the second coordinate frame by aligning the first coordinate frame with the second coordinate frame; and displaying content in a display of the first handheld device based on the mapping. the first handheld device comprises a first 6 degrees of freedom tracker configured to identify the first pose data at time t, wherein the second handheld device comprises a second 6 degrees of freedom tracker configured to identify the second pose data at time t, 3. The method of claim 1, wherein, wherein the second handheld device is configured to wirelessly transmit the second pose data at time t to the first handheld device. the first handheld device comprises a first camera aimed at the second handheld device and a hand of the second user holding the second handheld device, wherein the second handheld device comprises a second camera aimed at the first handheld device.

4. The method of claim 1, wherein, the second handheld device is within a field of view of the first camera of the first handheld device. detecting, from the first handheld device, hand tracking data of a second user holding the second handheld device comprises:

5. The method of claim 1, wherein, capturing, using a first camera of the first handheld device, images of one or more fingers of a hand of the second user; and 6. The method of claim 1, wherein, identifying, using a hand tracking identification process, positions of hand joints based on the images of the one or more fingers of the hand of the second user.

7. The method of claim 6, further comprising: identifying a first coordinate transformation between the positions of the hand joints and a second pose of the second handheld device, the second pose based on the second pose data; and identifying a second coordinate transformation between the positions of the hand joints and a first pose of the first handheld device, the first pose based on the first pose data. aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device is further based on the first coordinate transformation and the second coordinate transformation.

9. The method of claim 8, further comprising: detecting initiation of a session of an augmented reality application at the first handheld device; 8. The method of claim 7, wherein, in response to detecting initiation of the session, calibrating the first handheld device by aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device for a predetermined number of image frames generated using the first handheld device; and displaying content in a display of the first handheld device based on the alignment of the first coordinate frame with the second coordinate frame. ​ ​ ​ ​ 10. The method of claim 1, further comprising: determining a relative pose between the first handheld device and the second handheld device by: identifying a first reference coordinate frame based on the first pose data; identifying a second reference coordinate frame based on the second pose data; and forming a world reference coordinate frame based on the first reference coordinate frame and the second reference coordinate frame.

11. A server comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the server to: access first pose data of a first handheld device; receive second pose data of a second handheld device; detect hand tracking data of a second user holding the second handheld device from the first handheld device; and align a first coordinate frame of the first handheld device with a second coordinate frame of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device. the instructions further configure the server to:

12. The server of claim 11, wherein, map the first coordinate frame to the second coordinate frame by aligning the first coordinate frame with the second coordinate frame; and display content in a display of the first handheld device based on the mapping. the first handheld device comprises a first 6 degrees of freedom tracker configured to identify the first pose data at time t, 13. The server of claim 11, wherein, wherein the second handheld device comprises a second 6 degrees of freedom tracker configured to identify the second pose data at time t, wherein the second handheld device is configured to wirelessly transmit the second pose data at time t to the first handheld device. the first handheld device comprises a first camera aimed at the second handheld device and a hand of the second user holding the second handheld device, 14. The server of claim 11, wherein, wherein the second handheld device comprises a second camera aimed at the first handheld device. the second handheld device is within a field of view of the first camera of the first handheld device.

15. The server of claim 11, wherein, detecting hand tracking data of the second user holding the second handheld device from the first handheld device comprises:

16. The server of claim 11, wherein, capturing an image of one or more fingers of the hand of the second user using a first rear-facing camera of the first handheld device; and identifying a position of a hand joint based on the image of the one or more fingers of the hand of the second user using a hand tracking identification process. the instructions further configure the server to:

17. The server of claim 16, wherein, identify a first coordinate transformation between the position of the hand joint and a second pose of the second handheld device, the second pose based on the second pose data; and identify a second coordinate transformation between the position of the hand joint and a first pose of the first handheld device, the first pose based on the first pose data. aligning the first coordinate frame of the first handheld device with the second coordinate frame of the second handheld device is further based on the first coordinate transformation and the second coordinate transformation. the instructions further configure the server to:

18. The server of claim 17, wherein, ​ 19. The server of claim 18, wherein, ​ detecting initiation of a session of an augmented reality application at the first handheld device; in response to detecting the initiation of the session, calibrating the first handheld device by aligning a first coordinate system of the first handheld device with a second coordinate system of the second handheld device for a predetermined number of image frames generated using the first handheld device; and displaying content in a display of the first handheld device based on the alignment of the first coordinate system with the second coordinate system.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a server, cause the server to: access first pose data of a first handheld device; receive second pose data of a second handheld device; detect, from the first handheld device, hand tracking data of a second user holding the second handheld device; and align a first coordinate system of the first handheld device with a second coordinate system of the second handheld device based on the first pose data, the second pose data, and the hand tracking data of the second user holding the second handheld device. ​ ​

Images