Fast AR device pairing using depth prediction

Abstract

A method of aligning coordinate systems of different augmented reality (AR) devices is described. In one aspect, the method includes generating a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera of a first augmented reality (AR) device and a first sparse 3D point generated by a first SLAM system at the first AR device; generating a predicted depth of the second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of the second AR device and a second sparse 3D point generated by a second SLAM system at the second AR device; a relative pose between the first AR device and the second AR device is determined by registering the first point cloud with the second point cloud.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to Greek patent application serial number 20220100478 filed on June 8, 2022 and U.S. patent application serial number 17 / 893,723 filed on August 23, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein generally relates to augmented reality (AR) devices. Specifically, the present disclosure proposes systems and methods for pairing AR devices using depth prediction of a monocular camera device. Background Art

[0004] Augmented reality (AR) devices enable a user to observe a scene while seeing related virtual content that can be aligned with items, images, objects, or environments in the device's field of view. Virtual reality (VR) devices provide a more immersive experience than AR devices. VR devices occlude the user's field of view with virtual content displayed based on the positioning and orientation of the VR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0006] Figure 1 is a block diagram illustrating a network environment for collaborative augmented reality experiences according to one example implementation.

[0007] Figure 2 is a block diagram illustrating a network environment for collaborative augmented reality experience according to another example implementation.

[0008] Figure 3 is a block diagram illustrating an AR device according to an example embodiment.

[0009] Figure 4 is a block diagram illustrating a process for reconstructing a dense point cloud according to one example implementation.

[0010] Figure 5 The training and use of a machine learning program according to some example embodiments is shown.

[0011] Figure 6 is a block diagram illustrating a shared device application according to one example implementation.

[0012] Figure 7 is a block diagram illustrating a server according to an example implementation.

[0013] Figure 8 is an interaction diagram illustrating an alignment operation according to one example implementation.

[0014] Fig. 9 is an interaction diagram illustrating a collaborative AR experience according to one example implementation.

[0015] Fig.10 is a flow chart illustrating a method for aligning a reference coordinate system according to an example implementation.

[0016] Fig.11 is a flow chart illustrating a method for sharing an AR experience according to one example implementation.

[0017] Fig.12 A routine according to an exemplary embodiment is shown.

[0018] Fig.13 is a diagram illustrating aligning coordinate systems from two AR devices according to one embodiment.

[0019] Fig.14 A network environment in which a head wearable device may be implemented according to an example embodiment is shown.

[0020] Fig.15 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to an example implementation.

[0021] Fig.16 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein, according to an example embodiment. DETAILED DESCRIPTION

[0022] The following description describes the system, method, technology, instruction sequence and computing machine program product of the example implementation of this theme. In the following description, for the purpose of illustration, many specific details are set forth in order to provide an understanding of the various implementations of this theme. However, it will be apparent to those skilled in the art that the implementation of this theme can be put into practice without some or other details in these specific details. Examples represent possible variations only. Unless otherwise clearly stated, structures (e.g., structural components such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms or other functions) can be changed in order or combined or subdivided.

[0023] The term "augmented reality" (AR) is used herein to refer to an interactive experience of a real-world environment in which physical objects residing in the real world are "augmented" or enhanced by computer-generated digital content (also referred to as virtual content or synthetic content). AR may also refer to a system that enables the combination of the real and virtual worlds, real-time interaction, and 3D registration of virtual and real objects. Users of AR systems perceive virtual content that appears to be connected to or interacting with physical objects in the real world.

[0024] The term "virtual reality" (VR) is used herein to refer to a simulated experience of a virtual world environment that is completely different from the real world environment. Computer-generated digital content is displayed in a virtual world environment. VR also refers to a system that enables a user of a VR system to be fully immersed in a virtual world environment and interact with virtual objects presented in the virtual world environment.

[0025] The term "AR application" is used herein to refer to a computer-operated application that implements an AR experience. The term "VR application" is used herein to refer to a computer-operated application that implements a VR experience. The term "AR / VR application" refers to a computer-operated application that implements a combination of an AR experience or a VR experience.

[0026] The term "visual tracking system" is used herein to refer to a computer-operated application or system that enables the system to track visual features identified in images captured by one or more cameras of the visual tracking system. The visual tracking system builds a model of the real-world environment based on the tracked visual features. Non-limiting examples of visual tracking systems include: visual simultaneous localization and mapping systems (VSLAM) and visual inertial odometry (VIO) systems. VSLAM can be used to construct targets from an environment or scene based on one or more cameras of the visual tracking system. A VIO system (also known as a visual inertial tracking system) determines the latest posture (e.g., position and orientation) of a device based on data acquired from multiple sensors of the device (e.g., optical sensors, inertial sensors).

[0027] The term "inertial measurement unit" (IMU) is used herein to refer to a device that can report the inertial state of a mobile subject, including the acceleration, velocity, orientation, and location of the mobile subject. The IMU tracks the movement of the subject by integrating the acceleration and angular velocity measured by the IMU. The IMU may also refer to a combination of an accelerometer and a gyroscope, which can determine and quantify linear acceleration and angular velocity, respectively. The values ​​obtained from the IMU gyroscope can be processed to obtain the pitch, roll, and heading of the IMU, thereby obtaining the pitch, roll, and heading of the subject associated with the IMU. The signal from the accelerometer of the IMU can also be processed to obtain the velocity and displacement of the IMU.

[0028] The term "three degrees of freedom tracking system" (3DOF tracking system) is used herein to refer to a device that tracks rotational movement. For example, a 3DOF tracking system can track whether a user of a head wearable device is looking left or right, rotating their head up or down, and turning left or right. However, a head wearable device cannot use a 3DOF tracking system to determine whether the user is moving around a scene by moving in the physical world. Therefore, a 3DOF tracking system may not be accurate enough to be used for positioning signals. A 3DOF tracking system can be part of an AR / VR display device that includes an IMU sensor. For example, a 3DOF tracking system uses sensor data from sensors such as accelerometers, gyroscopes, and magnetometers.

[0029] The term "six degrees of freedom tracking system" (6DOF tracking system) is used herein to refer to a device that tracks rotational and translational motion. For example, a 6DOF tracking system can track whether a user rotates their head and moves forward or backward, sideways or vertically, and up or down. A 6DOF tracking system may include a simultaneous localization and mapping (SLAM) system and / or a VIO system that relies on data acquired from multiple sensors (e.g., depth cameras, inertial sensors). The 6DOF tracking system analyzes the data from the sensors to accurately determine the pose of the display device.

[0030] Each AR device may include its own 6DOF tracking system that generates its own reference coordinate system / frame. Therefore, two or more AR devices may have two or more different reference coordinate systems that need to be aligned to represent the posture of any AR device in a common coordinate system. Each AR device generates a dense point cloud based on its corresponding reference coordinate system. However, generating a dense point cloud using a conventional depth sensor (e.g., using a stereoscopic camera) can be very time-consuming. High-resolution depth is calculated by processing visual information, which is a computationally intensive process. Typically, an AR device estimates a depth map for the entire image area of ​​each processed frame. However, due to limited computing resources and power constraints, depth estimation in portable AR devices may not be performed for every frame. Similarly, pre-building a map of an existing physical environment can be time-consuming and may raise privacy issues. Other standard solutions include using markers (e.g., predefined 2D images) to synchronize the coordinate systems of each AR device.

[0031] This application describes a system that enables two or more AR devices to share an AR experience by aligning the different coordinate systems of each AR device using single-view depth prediction. In order to share the AR experience, the coordinate systems are aligned so that the pose (e.g., 3D positioning + orientation) of any device is represented in a common coordinate system. The system predicts depth using deep SLAM (VI-SLAM) and reconstructs a dense point cloud from a single image of each AR device. Each AR device perceives the same scene from a different perspective. The system determines the relative pose of the AR devices (e.g., the relative pose between the VIO reference frames) by aligning / registering the overlapping areas of the point clouds. The system uses the relative pose to align the VIO reference coordinate systems of the AR devices in 3D to achieve a shared AR experience.

[0032] In one example implementation, a method for aligning coordinate systems from separate augmented reality (AR) devices is described. In one aspect, the method includes: generating a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera of a first augmented reality (AR) device and a first sparse 3D point generated by a first SLAM system at the first AR device; generating a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of a second AR device and a second sparse 3D point generated by a second SLAM system at the second AR device; determining a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud based on the corresponding predicted depths; and providing the relative pose to at least one of the first AR device or the second AR device.

[0033] Therefore, one or more of the methods described herein help solve the technical problem of resource management caused by aligning the coordinate systems of different augmented reality (AR) devices. The method currently described provides an improvement in the operation of the functions of the computer by providing a reduction in power consumption. Therefore, one or more of the methods described herein can eliminate the need for certain workloads or computing resources. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0034] Figure 1 1 is a network diagram illustrating a network environment 100 suitable for operating an AR device A 112, an AR device B 114, and a server 110 according to some example embodiments. The network environment 100 includes the AR device A 112, the AR device B 114, and the server 110 communicatively coupled to each other via a network 104. The AR device A 112, the AR device B 114, and the server 110 can each be fully or partially connected to the network 104 as described below. Fig.16 The server 110 may be implemented in the computer system described. The server 110 may be part of a network-based system. For example, the network-based system may be or include a cloud-based server system that provides additional information, such as reference frame alignment data of the AR device A 112 and the AR device B 114.

[0035] The user 106 operates the AR device A 112. The user 106 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the AR device A 112), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 106 operates the AR device A 112 by pointing the AR device A 112 at a physical object 108 in the real-world environment 102.

[0036] The user 116 operates the AR device B 114. The user 116 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the AR device B 114), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 116 operates the AR device B 114 by pointing the AR device B 114 at the physical object 108 in the real-world environment 102.

[0037] AR device A 112 and AR device B 114 may each be a computing device with a display, such as a smartphone, a tablet computer, or a wearable computing device (e.g., a watch or glasses). The computing device may be handheld or may be removably mounted to the head of a user (e.g., user 106, user 116). In one example, the display may be a screen that displays content captured using a camera of AR device A 112 / AR device B 114. In another example, the display of the device may be transparent, such as a lens of wearable computing glasses, which enables a user to view content presented on the display while viewing real-world objects visible through the display.

[0038] The user 106 operates an AR application at the AR device A 112. The AR application may be configured to provide the user 106 with an AR experience triggered by a physical object 108, such as a two-dimensional physical object (e.g., a picture), a three-dimensional physical object (e.g., a statue), a location (e.g., at a factory), or any reference in the real-world environment 102 (e.g., the perceived corner of a wall or furniture). For example, the user 106 may direct the camera of the AR device A 112 to capture an image of the physical object 108.

[0039] The user 116 operates an AR application at AR device B 114. The AR application may be configured to provide the user 106 with an AR experience triggered by a physical object 108, such as a two-dimensional physical object (e.g., a picture), a three-dimensional physical object (e.g., a statue), a location (e.g., at a factory), or any reference in the real-world environment 102 (e.g., the perceived corner of a wall or furniture). For example, AR device B 114 may direct a camera of AR device B 114 to capture an image of the physical object 108 from a different perspective (relative to AR device A 112). Thus, the images captured by AR device A 112 and AR device B 114 include overlapping areas.

[0040] The AR device A 112 includes a tracking system (not shown). The tracking system uses optical sensors (e.g., image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track the pose (e.g., position and orientation) of the AR device A 112 relative to the real-world environment 102 to determine the position of the AR device A 112 within the real-world environment 102. In one example, the tracking system of the AR device A 112 predicts a dense depth map using a single image and sparse 3D points from a monocular camera of the AR device A 112 and reconstructs a corresponding dense point cloud.

[0041] The AR device B 114 includes a tracking system (not shown). The tracking system tracks the pose (e.g., position and orientation) of the AR device B 114 relative to the real-world environment 102 using optical sensors (e.g., image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to determine the position of the AR device B 114 within the real-world environment 102. In one example, the tracking system of the AR device B 114 predicts a dense depth map using a single image and sparse 3D points from a monocular camera of the AR device B 114 and reconstructs a corresponding dense point cloud.

[0042] In one example implementation, the server 110 receives dense point clouds from the AR device A 112 and the AR device B 114, and aligns the dense point clouds between the AR device A 112 and the AR device B 114 to obtain a relative pose between the VIO reference frames of the AR device A 112 and the AR device B 114. The server 110 provides alignment data (e.g., relative pose data) to the AR device A 112 and the AR device B 114. In another example, the alignment of the dense point cloud can be performed on the AR device A 112, the AR device B 114, or the server 110, or a combination of the AR device A 112, the AR device B 114, and the server 110.

[0043] Figure 1 Any of the machines, databases, or devices shown may be implemented in a general-purpose computer that is modified (e.g., configured or programmed) by software to be a special-purpose computer to perform one or more of the functions described herein for the machine, database, or device. Fig.10 A computer system capable of implementing any one or more of the methods described herein is discussed. As used herein, a "database" is a data storage resource and may store data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, or any suitable combination thereof. In addition, Figure 1 Any two or more of the machines, databases, or devices shown in the may be combined into a single machine, and the functionality described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

[0044] The network 104 may be any network that enables communication between or among machines (e.g., server 110), databases, and devices (e.g., AR device A 112, AR device B 114). Thus, the network 104 may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 104 may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0045] Figure 2 is a block diagram illustrating a network environment 200 for collaborative augmented reality experience according to another example implementation. The network environment 200 includes an AR device A 112 and an AR device B 114. One AR device receives a dense point cloud from another AR device and performs alignment. For example, AR device B 114 provides a dense point cloud to AR device A 112. AR device A 112 aligns the dense point cloud from AR device A 112 with the dense point cloud from AR device B 114 to obtain a relative pose between the VIO reference systems of AR device A 112 and AR device B 114. AR device A 112 uses the relative pose to align the VIO reference coordinate system of AR device A 112 with the VIO reference coordinate system of AR device B 114 in 3D. AR device A 112 uses the aligned VIO reference coordinate system to display a virtual object anchored to a physical object 108 or a real-world environment 102.

[0046] Figure 3is a block diagram illustrating modules (e.g., components) of an AR device A 112 according to some example implementations. The AR device A 112 includes a sensor 302, a display 304, a processor 308, a graphics processing unit 318, a display controller 320, and a storage device 306. Examples of the AR device A 112 include a wearable computing device, a tablet computer, or a smartphone.

[0047] The sensor 302 includes an optical sensor 314 and an inertial sensor 316. The optical sensor 314 includes a monocular camera. The inertial sensor 316 includes a combination of a gyroscope, an accelerometer, and a magnetometer. Other examples of the sensor 302 include a proximity sensor or a position sensor (e.g., near field communication, GPS, Bluetooth, Wifi), an audio sensor (e.g., a microphone), or any suitable combination thereof. Note that the sensor 302 described herein is for illustrative purposes, and therefore the sensor 302 is not limited to the sensors described above. In an example embodiment, the AR device A112 does not include a depth sensor, such as a structured light sensor, a time-of-flight sensor, a passive stereo sensor, and an ultrasonic device, a time-of-flight sensor.

[0048] The display 304 includes a screen or monitor configured to display images generated by the processor 308. In one example implementation, the display 304 may be transparent or translucent so that the user 106 can see through the display 304 (in an AR use case). In another example, the display 304 (e.g., an LCOS display) presents each frame of the virtual content in multiple renderings.

[0049] The processor 308 includes an AR application 310, a 6DOF tracker 312, a depth system 324, and a shared device application 328. The AR application 310 uses computer vision to detect and identify a physical environment or physical object 108. The AR application 310 retrieves a virtual object (e.g., a 3D object model) based on the identified physical object 108 or physical environment. The display 304 displays the virtual object. The AR application 310 includes a local rendering engine that generates a visualization of a virtual object superimposed on (e.g., superimposed on or otherwise displayed simultaneously with) an image of the physical object 108 captured by the optical sensor 314. The visualization of the virtual object can be manipulated by adjusting the positioning of the physical object 108 relative to the optical sensor 314 (e.g., its physical position, orientation, or both). Similarly, the visualization of the virtual object can be manipulated by adjusting the posture of the AR device A 112 relative to the physical object 108.

[0050] The 6DOF tracker 312 estimates the pose of the AR device A 112. For example, the 6DOF tracker 312 uses image data from the optical sensor 314 and corresponding inertial data from the inertial sensor 316 to track the position and pose of the AR device A 112 relative to a reference frame (e.g., the real-world environment 102). In one example, the 6DOF tracker 312 uses the sensor data to determine a three-dimensional pose of the AR device A 112. The three-dimensional pose is a determined orientation and position of the AR device A 112 relative to the user's real-world environment 102. For example, the AR device A 112 can use an image of the user's real-world environment 102 as well as other sensor data to identify the relative position and orientation of the AR device A 112 and physical objects in the real-world environment 102 surrounding the AR device A 112. The 6DOF tracker 312 continuously collects and uses updated sensor data describing the movement of the AR device A 112 to determine an updated three-dimensional pose of the AR device A 112 that indicates changes in the relative position and orientation of the AR device A 112 and physical objects in the real-world environment 102. The 6DOF tracker 312 provides the three-dimensional pose of the AR device A 112 to 324 and the shared device application 328.

[0051] The depth system 324 accesses a single image from an optical sensor 314 (e.g., a monocular camera) and sparse 3D points from the 6DOF tracker 312 to predict depth and generate a dense point cloud. In one example implementation, the AR device A112 does not include a depth sensor or a stereo sensor. The depth system 324 uses a trained model based on a single image and sparse 3D points to predict depth and generate a dense point cloud. Figure 4 Depth system 324 is described in more detail.

[0052] The shared device application 328 accesses the dense point cloud from the AR device A 112 and the dense point cloud from the AR device B 114, and performs alignment of the dense point clouds based on the partially overlapping areas of the corresponding dense point clouds. The shared device application 328 identifies the relative posture between the AR device A 112 and the AR device B 114. The AR application 310 uses the relative posture to enable sharing of the AR experience between the two AR devices. For example, the correct position / perspective of the virtual object is accurately presented in both the AR device A 112 and the AR device B 114 (for example, the user 116 points to a country on the virtual globe, and the AR device A 112 displays the virtual globe so that the user 106 can see the same country that the user 116 points to (perceived from the perspective of the user 106)). The following will refer to Figure 6 Example components of the shared device application 328 are further described.

[0053] The graphics processing unit 318 includes a rendering engine (not shown) configured to render frames of 3D models of virtual objects based on the virtual content provided by the AR application 310 and the pose of the AR device A 112 (relative to the AR device B 114). In other words, the graphics processing unit 318 uses the three-dimensional pose of the AR device A 112 to generate frames of virtual content to be presented on the display 304. For example, the graphics processing unit 318 uses the three-dimensional pose to render the frames of the virtual content so that the virtual content is presented in a certain orientation and position in the display 304 to appropriately enhance the realism of the user. As an example, the graphics processing unit 318 can use the three-dimensional pose data to render the frames of the virtual content so that when presented on the display 304, the virtual content overlaps with the physical objects in the user's real-world environment 102. The graphics processing unit 318 generates an updated frame of the virtual content based on the updated three-dimensional pose of the AR device A 112, which reflects the change in the position and orientation of the user relative to the physical objects in the user's real-world environment 102.

[0054] The graphics processing unit 318 transmits the rendered frames to the display controller 320 . The display controller 320 is positioned as an intermediary between the graphics processing unit 318 and the display 304 , receives image data from the graphics processing unit 318 , and provides the rendered frames to the display 304 .

[0055] The storage device 306 stores virtual object content 322, relative pose data 326 (e.g., relative pose between AR device A 112 and AR device B 114), and a pre-trained model 330. The virtual object content 322 includes, for example, a database of visual references (e.g., images, QR codes) and corresponding virtual content (e.g., three-dimensional models of virtual objects). The relative pose data 326 indicates the relative pose between a first reference coordinate system of a first AR device and a second reference coordinate system of a second AR device by registering a first point cloud with a second point cloud based on corresponding predicted depths. The pre-trained model 330 includes a machine learning model trained using (monocular) images and corresponding pose data provided by multiple AR devices.

[0056] Any one or more modules in the modules described herein can be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein can configure a processor to perform the operation described herein for the module. In addition, any two or more modules in these modules can be combined into a single module, and the function described herein for a single module can be subdivided among multiple modules. In addition, according to various example embodiments, the modules described herein as implemented in a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0057] Figure 4 is a block diagram illustrating a process for reconstructing a dense point cloud according to an example implementation. The 6DOF tracker 312 includes a VI-SLAM 406. The VI-SLAM 406 can be used to identify sparse 3D points from the real-world environment 102. The VI-SLAM 406 (also known as a visual inertial-tracking system) determines the latest pose (e.g., position and orientation) of the AR device A112 based on data acquired from multiple sensors (e.g., optical sensors, inertial sensors) of the AR device A112. In one example, the 6DOF tracker 312 provides the sparse 3D points and a single image from a monocular camera of the AR device A112 to the depth system 324. The sparse 3D points, referred to as 3D points, are tracked and 3D reconstructed by the VI-SLAM 406.

[0058] The depth system 324 receives a single image from the optical sensor 314 (e.g., a monocular camera) and sparse 3D points from the 6DOF tracker 312. The depth system 324 generates a dense point cloud by predicting the depth in the single image using a trained machine learning model (e.g., pre-trained model 402). In one example, the depth system 324 includes a deep neural network that provides depth from a single image.

[0059] The depth system 324 includes a pre-trained model 402 and a depth prediction module 404. The pre-trained model 402 is trained using images generated by the AR device and gesture data corresponding to the images. Therefore, the AR device A 112 does not include a pre-map construction or pre-construction of the real world environment 102. In other words, the AR device A 112 does not build a detailed model of the real world environment 102. Figure 5 An example describing a machine learning training procedure in more detail.

[0060] The depth prediction module 404 applies the single image and sparse 3D points to the pre-trained model 402 to predict the depth in the single image and generate a dense 3D point cloud (e.g., point cloud data A408) based on the predicted depth. The depth system 324 provides the point cloud data A408 to the shared device application 328 to align the VIO reference coordinate system of the AR device.

[0061] Figure 5 The training and use of a machine learning program 500 is shown according to some example embodiments. In some example embodiments, a machine learning program (MLP) (also referred to as a machine learning algorithm or tool) is used to perform operations associated with dense point cloud depth prediction.

[0062] Machine learning is a field of study that gives computers the ability to learn without explicit programming. Machine learning explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data and make predictions about new data. Such machine learning tools operate by building models from example training data 504 to make data-driven predictions or decisions represented as outputs or evaluations (e.g., evaluations 512). Although example implementations are given for some machine learning tools, the principles presented herein can be applied to other machine learning tools.

[0063] In some example embodiments, different machine learning tools may be used. For example, logistic regression (LR), naive Bayes, random forest (RF), neural network (NN), matrix factorization, and support vector machine (SVM) tools may be used to classify or score job positions.

[0064] Two common types of problems in machine learning are classification problems and regression problems. Classification problems, also known as categorization problems, aim to classify an item into one of several categorical values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify some item (e.g., by providing real-valued values)

[0065] The machine learning algorithm uses features 502 to analyze the data to generate estimates 512. Each of the features 502 is a separate measurable attribute of the observed phenomenon. The concept of a feature is related to the concept of an explanatory variable used in statistical techniques such as linear regression. Selecting features that are informative, discriminative, and independent is important for the effective operation of MLPs in pattern recognition, classification, and regression. Features can be of various types, such as numeric features, strings, and graphics.

[0066] In one example implementation, features 502 may be of different types and may include one or more of content 514, concepts 516, attributes 518, historical data 520, and / or user data 522, just to name a few.

[0067] The machine learning algorithm uses the training data 504 to discover correlations between the identified features 502 that influence the outcome or evaluation 512. In some example embodiments, the training data 504 includes labeled data, which is known data for one or more identified features 502 and one or more outcomes (e.g., detecting a deep pattern).

[0068] Using training data 504 and identified features 502, the machine learning tool is trained at machine learning program training 506. The machine learning tool evaluates the value of features 502 associated with training data 504. The result of the training is a trained machine learning program 510.

[0069] When performing an evaluation using the trained machine learning program 510, new data 508 is provided as input to the trained machine learning program 510, and the trained machine learning program 510 generates an evaluation 512 as output.

[0070] Figure 6 6 is a block diagram illustrating a shared device application 328 according to an example implementation. The shared device application 328 includes an AR device interface 602, a point cloud alignment application 604, a relative pose module 606, and a coordinate system alignment module 608. The AR device interface 602 is configured to communicate with the AR device B 114 (e.g., via Bluetooth, wireless network). In one example, the AR device interface 602 accesses point cloud data B 610 from the AR device B 114.

[0071] The point cloud alignment application 604 accesses the point cloud data A 408 from the depth system 324 and the point cloud data B 610 from the AR device interface 602. The point cloud alignment application 604 performs a joint alignment of the point cloud data A 408 and the point cloud data B 610. In one example, the point cloud alignment application 604 performs a joint multi-point set registration (JRMPC) algorithm on the point cloud data A 408 and the point cloud data B 610. In another example, the point cloud alignment application 604 performs an iterative closest point (ICP) algorithm on the point cloud data A 408 and the point cloud data B 610. One of ordinary skill in the art will recognize that the point cloud alignment application 604 may use other joint point cloud alignment algorithms.

[0072] The relative pose module 606 determines a relative pose based on the aligned point clouds determined at the point cloud alignment application 604. In one example, the relative pose module 606 determines a relative pose between the VIO reference frames of each AR device based on the aligned point clouds.

[0073] The coordinate system alignment module 608 uses the relative pose to align the VIO reference coordinate system of each AR device in 3D. Note that once the alignment of the point cloud alignment application 604 is performed, the connected AR devices (e.g., AR device A 112 and AR device B 114) do not need to be time-synchronized again. Therefore, the relative pose calculation is only performed when AR device A 112 and AR device B 114 are connected during a joint collaborative AR session (e.g., each AR device views the "same" virtual object in the real-world environment 102). In another example, the coordinate system alignment module 608 provides alignment data (e.g., the relative pose of AR device A 112 relative to AR device B 114) to the AR application 310 / graphics processing unit 318 for accurate placement / display of virtual objects.

[0074] Figure 77 is a block diagram illustrating a server 110 according to an example implementation. The server 110 includes an AR device interface 702, a point cloud alignment application 704, a relative pose module 706, and a coordinate system alignment module 708. The AR device interface 702 is configured to communicate with AR device A 112 and AR device B 114. In one example, the AR device interface 702 accesses point cloud data A 408 from AR device A 112 and point cloud data B 610 from AR device B 114.

[0075] The point cloud alignment application 704 accesses the point cloud data A 408 and the point cloud data B 610 from the AR device interface 702. The point cloud alignment application 704 performs a joint alignment of both the point cloud data A 408 and the point cloud data B 610. In one example, the point cloud alignment application 704 performs a joint multi-point set registration (JRMPC) algorithm on the point cloud data A 408 and the point cloud data B 610. In another example, the point cloud alignment application 704 performs an iterative closest point (ICP) algorithm on the point cloud data A 408 and the point cloud data B 610. One of ordinary skill in the art will recognize that the point cloud alignment application 704 may use other joint point cloud alignment algorithms.

[0076] The relative pose module 706 determines a relative pose based on the aligned point clouds performed at the point cloud alignment application 704. In one example, the relative pose module 706 determines a relative pose between the VIO reference frames of each AR device based on the aligned point clouds.

[0077] The coordinate system alignment module 708 uses the relative pose to align the VIO reference coordinate system of each AR device in 3D. In one example, the coordinate system alignment module 608 provides a coordinate system alignment 710 (e.g., the relative pose of AR device A 112 relative to AR device B 114) to AR device A 112 and AR device B 114 to accurately place / display virtual objects.

[0078] Figure 8 806) and provides coordinate system alignment 808 to both AR device A 112 and AR device B 114.

[0079] Fig. 9is an interaction diagram illustrating a collaborative AR experience according to an example implementation. AR device A 112 identifies a virtual object 902 in reference coordinate system B. AR device A 112 displays the virtual object based on the aligned reference system by (accessing a coordinate system alignment 904 between AR device A and AR device B, and displaying the virtual object 906 in reference coordinate system A based on the coordinate system alignment).

[0080] Fig.10 1 is a flowchart illustrating a method 1000 for aligning a reference coordinate system according to an example embodiment. The operations in the method 1000 may be performed by the AR device A 112 using the above description of Figure 6 Therefore, the method 1000 is described by way of example with reference to the AR device 112. However, it should be understood that at least some of the operations of the method 1000 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0081] In block 1002, the shared device application 328 receives point cloud data from the depth systems of the AR device A 112 and the AR device B 114. In block 1004, the point cloud alignment application 604 aligns the point cloud data from the AR device A 112 and the AR device B 114. In block 1006, the relative pose module 606 determines the relative pose of the AR device A 112 and the AR device B 114. In block 1008, the coordinate system alignment module 608 aligns the reference coordinate system based on the relative pose. In block 1010, the coordinate system alignment module 608 provides the aligned reference coordinate system to the AR application 310 of the AR device A 112 and / or the AR device B 114.

[0082] It should be noted that other embodiments can use different sorting, additional or less operations and different nomenclature or terminology to complete similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0083] Fig.11 1 is a flow chart showing a method 1100 for sharing an AR experience according to an example implementation. The operations in the method 1100 may be performed by the AR device A 112 using the method described above with respect to Figure 3 Therefore, the method 1100 is described by way of example with reference to the AR device A 112. However, it should be understood that at least some of the operations of the method 1100 may be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0084] In block 1102, the shared device application 328 receives virtual object data in reference frame b (e.g., the reference frame of AR device B 114). In block 1104, the shared device application 328 accesses the coordinate system alignment between AR device A 112 and AR device B 114. In block 1106, the AR application 310 displays the virtual object in reference frame a (e.g., the reference frame of AR device A 112).

[0085] It should be noted that other embodiments can use different sorting, additional or less operations and different nomenclature or terminology to complete similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0086] Fig.12 A routine 1200 according to an example implementation is shown. In block 1202, the routine 1200 generates a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera of a first augmented reality (AR) device and a first sparse 3D point generated by a first SLAM system at the first AR device. In block 1204, the routine 1200 generates a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of a second AR device and a second sparse 3D point generated by a second SLAM system at the second AR device. In block 1206, the routine 1200 determines a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud based on the corresponding predicted depths. In block 1208, the routine 1200 provides the relative pose to at least one of the first AR device or the second AR device.

[0087] Fig.13 is a diagram showing alignment of coordinate systems from two AR devices according to one embodiment. As shown, different images of a scene are captured by different devices, and sparse points are captured in each image. The depth of the sparse points is predicted using a pre-trained model, and a point cloud is generated for each image. The partial overlap of the point clouds is used to align the two point clouds, which are then used to determine the relative pose of each device.

[0088] The monocular camera of the first AR device generates a monocular image A 1302. The monocular camera of the second AR device generates a monocular image B 1304. The monocular image A 1302 and the sparse 3D points A 1310 are applied to the pre-trained model to predict the predicted point cloud A 1306. The monocular image B 1304 and the sparse 3D points A 1312 are applied to the same pre-trained model to predict the predicted point cloud A 1308. The predicted point cloud A 1306 is registered / aligned with the predicted point cloud A 1308. The relative pose of the AR devices is determined based on the registered point clouds 1314.

[0089] System with head wearable device

[0090] Fig.14 A network environment 1400 is shown in which a head wearable device 1402 may be implemented according to an example implementation. Fig.14 is a high-level functional block diagram of an example head wearable device 1402 that communicatively couples a mobile client device 1438 and a server system 1432 via various networks 1440.

[0091] The head wearable device 1402 includes a camera, such as at least one of a visible light camera 1412, an infrared emitter 1414, and an infrared camera 1416. The client device 1438 may be able to connect to the head wearable device 1402 using both communication 1434 and communication 1436. The client device 1438 is connected to the server system 1432 and the network 1440. The network 1440 may include any combination of wired and wireless connections.

[0092] The head wearable device 1402 also includes two image displays of the optical assembly image display 1404. The two image displays include one image display associated with a left side of the head wearable device 1402 and one image display associated with a right side of the head wearable device 1402. The head wearable device 1402 also includes an image display driver 1408, an image processor 1410, low power circuits 1426, and high speed circuits 1418. The image display 1404 of the optical assembly is used to present images and videos to a user of the head wearable device 1402, including images that may include a graphical user interface.

[0093] The image display driver 1408 commands and controls the image display in the image display 1404 of the optical assembly. The image display driver 1408 may deliver the image data directly to the image display in the image display 1404 of the optical assembly for presentation, or may have to convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data may be video data formatted according to a compression format (e.g., H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc.), and still image data may be formatted according to a compression format (e.g., Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (Exif), etc.).

[0094] As described above, the head wearable device 1402 includes a frame and handles (or temples) extending from the sides of the frame. The head wearable device 1402 also includes a user input device 1406 (e.g., a touch sensor or a push button), including an input surface on the head wearable device 1402. The user input device 1406 (e.g., a touch sensor or a push button) is used to receive input selections from the user to manipulate the graphical user interface of the presented image.

[0095] Fig.14 The components for the head wearable device 1402 shown in the figure are located on one or more circuit boards (e.g., PCB or flexible PCB) in the frame or temples. Alternatively or additionally, the depicted components can be located in a block, frame, hinge, or nose bridge of the head wearable device 1402. The left and right can include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0096] The head wearable device 1402 includes a memory 1422 that stores instructions for performing a subset or all of the functions described herein. The memory 1422 may also include a storage device.

[0097] like Fig.14As shown, the high-speed circuit 1418 includes a high-speed processor 1420, a memory 1422, and a high-speed wireless circuit 1424. In this example, the image display driver 1408 is coupled to the high-speed circuit 1418 and is operated by the high-speed processor 1420 to drive the left image display and the right image display in the image display 1404 of the optical assembly. The high-speed processor 1420 can be any processor capable of managing high-speed communications and the operation of any general computing system required by the head wearable device 1402. The high-speed processor 1420 includes the processing resources required to manage high-speed data transmission on the communication 1436 to the wireless local area network (WLAN) using the high-speed wireless circuit 1424. In some examples, the high-speed processor 1420 executes an operating system (e.g., a LINUX operating system or other such operating system of the head wearable device 1402), and the operating system is stored in the memory 1422 for execution. In addition to any other duties, the high-speed processor 1420 that executes the software architecture of the head wearable device 1402 is used to manage data transmission with the high-speed wireless circuit 1424. In some examples, high-speed wireless circuit 1424 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 1402.11 communication standard (also referred to herein as Wi-Fi). In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit 1424.

[0098] The low power wireless circuit 1430 and the high speed wireless circuit 1424 of the head wearable device 1402 may include a short range transceiver (Bluetooth TM ) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Client device 1438 including a transceiver for communicating via communication 1434 and communication 1436 can be implemented using details of the architecture of head wearable device 1402, as can other elements of network 1440.

[0099] The memory 1422 includes any storage device capable of storing various data and applications, including camera data generated by the left and right, infrared cameras 1416 and the image processor 1410, and images generated by the image display driver 1408 for display on the image display in the optical assembly 1404. Although the memory 1422 is shown as being integrated with the high-speed circuit 1418, in other examples, the memory 1422 can be a separate independent element of the head wearable device 1402. In some such examples, the circuit routing can provide a connection from the image processor 1410 or the low-power processor 1428 to the memory 1422 through a chip including the high-speed processor 1420. In other examples, the high-speed processor 1420 can manage the addressing of the memory 1422 so that the low-power processor 1428 will start the high-speed processor 1420 at any time when a read or write operation involving the memory 1422 is required.

[0100] like Fig.14 As shown in , the low-power processor 1428 or the high-speed processor 1420 of the head wearable device 1402 can be coupled to a camera (a visible light camera 1412; an infrared emitter 1414 or an infrared camera 1416), an image display driver 1408, a user input device 1406 (for example, a touch sensor or a push button), and a memory 1422.

[0101] The head wearable device 1402 is connected to a host computer. For example, the head wearable device 1402 is paired with a client device 1438 via communication 1436, or connected to a server system 1432 via a network 1440. For example, the server system 1432 can be one or more computing devices as part of a service or network computing system, which includes a processor, a memory, and a network communication interface to communicate with the client device 1438 and the head wearable device 1402 through the network 1440.

[0102] The client device 1438 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via the network 1440, communication 1434, or communication 1436. The client device 1438 may also store at least a portion of the instructions for generating binaural audio content in a memory of the client device 1438 to implement the functions described herein.

[0103] The output components of the head wearable device 1402 include visual components, such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by an image display driver 1408. The output components of the head wearable device 1402 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input devices 1406) of the head wearable device 1402, the client device 1438, and the server system 1432 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0104] The head wearable device 1402 may optionally include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 1402. For example, the peripheral elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0105] For example, the biometric component includes a component that detects expression (e.g., hand expression, facial expression, voice expression, body posture, or eye tracking), measures biological signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifies a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The positioning component includes a position sensor component that generates position coordinates (e.g., a global positioning system (GPS) receiver component), a WiFi or Bluetooth™ transceiver that generates positioning system coordinates, an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the client device 1438 via the low power wireless circuit 1430 or the high speed wireless circuit 1424 through the communication 1436.

[0106] When phrases similar to "at least one of A, B, or C," "at least one of A, B, and C," "one or more A, B, or C," or "one or more of A, B, and C" are used, it is intended that the phrase be interpreted to mean that A can exist alone in an embodiment, B can exist alone in an embodiment, C can exist alone in an embodiment, or any combination of elements A, B, and C can exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C.

[0107] Changes and modifications may be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0108] Fig.15 1500 is a block diagram illustrating a software architecture 1504 that can be installed on any one or more of the devices described herein. The software architecture 1504 is supported by hardware such as a machine 1502, which includes a processor 1520, a memory 1526, and an I / O component 1538. In this example, the software architecture 1504 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1504 includes layers such as an operating system 1512, a library 1510, a framework 1508, and an application 1506. In operation, the application 1506 calls an API call 1550 through the software stack and receives a message 1552 in response to the API call 1550.

[0109] The operating system 1512 manages hardware resources and provides public services. The operating system 1512 includes, for example, a kernel 1514, services 1516, and drivers 1522. The kernel 1514 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1514 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1516 can provide other public services to other software layers. Drivers 1522 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1522 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0110] The libraries 1510 provide a low-level common infrastructure used by the applications 1506. The libraries 1510 may include a system library 1518 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1510 may include an API library 1524, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1510 may also include various other libraries 1528 to provide many other APIs to the application 1506 .

[0111] The framework 1508 provides a high-level common infrastructure used by the applications 1506. For example, the framework 1508 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1508 can provide a wide range of other APIs that can be used by the applications 1506, some of which can be specific to a particular operating system or platform.

[0112] In an example implementation, applications 1506 may include a home application 1536, a contacts application 1530, a browser application 1532, a book reader application 1534, a location application 1542, a media application 1544, a messaging application 1546, a game application 1548, and a variety of other applications such as third-party applications 1540. Applications 1506 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1506 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1540 (e.g., applications developed by an entity other than the vendor of a particular platform using ANDROID ) may be used to create a third-party application 1540. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party application 1540 can call API call 1550 provided by operating system 1512 to facilitate the functions described in this article.

[0113] Fig.16 1600 within which instructions 1608 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1600 to perform any one or more of the methods discussed herein. For example, the instructions 1608 may cause the machine 1600 to perform any one or more of the methods described herein. The instructions 1608 transform a general purpose, unprogrammed machine 1600 into a specific machine 1600 that is programmed to perform the functions described and illustrated in the manner described. The machine 1600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1600 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1608, sequentially or otherwise, that specify actions to be taken by the machine 1600. Furthermore, while only a single machine 1600 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1608 to perform any one or more of the methodologies discussed herein.

[0114] The machine 1600 may include a processor 1602, a memory 1604, and an I / O component 1642 configured to communicate with each other via a bus 1644. In an example embodiment, the processor 1602 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1606 that executes instructions 1608 and a processor 1610. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig.16 Multiple processors 1602 are shown, but the machine 1600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0115] The memory 1604 includes a main memory 1612, a static memory 1614, and a storage unit 1616, which are all accessible by the processor 1602 via a bus 1644. The main memory 1604, the static memory 1614, and the storage unit 1616 store instructions 1608 that embody any one or more of the methodologies or functions described herein. The instructions 1608 may also reside, completely or partially, within the main memory 1612, within the static memory 1614, within the machine-readable medium 1618 within the storage unit 1616, within at least one of the processors 1602 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1600.

[0116] I / O components 1642 may include a variety of components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1642 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be understood that I / O components 1642 may be included in Fig.16 1642 may include an output component 1628 and an input component 1630. The output component 1628 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 1630 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and / or force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0117] In other example embodiments, the I / O component 1642 may include a biometric component 1632, a motion component 1634, an environmental component 1636, or a positioning component 1638, as well as various other components. For example, the biometric component 1632 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1634 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1636 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of dangerous gases to ensure safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1638 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0118] A variety of technologies may be used to achieve communication. I / O components 1642 also include communication components 1640, which are operable to couple machine 1600 to network 1620 or device 1622 via coupling 1624 and coupling 1626, respectively. For example, communication components 1640 may include a network interface component or another suitable device to interface with network 1620. In other examples, communication components 1640 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low power consumption), Components and other communication components for providing communication via other modalities. Device 1622 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0119] In addition, the communication component 1640 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1640 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1640, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, and the like.

[0120] Various memories (e.g., memory 1604, main memory 1612, static memory 1614, and / or memory of processor 1602) and / or storage unit 1616 may store one or more sets of instructions and data structures (e.g., software) that embody or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1608) when executed by processor 1602 cause various operations to implement the disclosed embodiments.

[0121] Instructions 1608 may be sent or received over network 1620 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 1640) and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1608 may be sent or received to device 1622 via coupling 1626 (e.g., a peer-to-peer coupling) using a transmission medium.

[0122] As used herein, the terms "machine storage medium", "device storage medium" and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGAs), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "computer storage medium" and "device storage medium" explicitly exclude carrier waves, modulated data signals and other such media, at least some of which are covered by the term "signal media" discussed below.

[0123] The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be understood to include any intangible medium that is capable of storing, encoding, or carrying instructions 1416 for execution by the machine 1400, and include digital or analog communication signals or other intangible media that facilitate the communication of such software. Thus, the terms "transmission medium" and "signal medium" should be taken to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0124] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Thus, these terms include both storage devices / medium and carrier / modulated data signals.

[0125] Although the embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes may be made to these embodiments without departing from the broader scope of the present disclosure. Therefore, the description and the drawings should be considered in an illustrative sense, rather than a restrictive sense. The drawings forming a part of the present invention show, by way of illustration and not limitation, specific embodiments in which the subject matter may be practiced. The embodiments shown are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, this specific embodiment should not be understood as a restrictive sense, and the scope of the various embodiments is limited only by the appended claims and the full scope of the equivalents given by these claims.

[0126] Such embodiments of the subject matter of the present invention may be referred to herein individually and / or collectively by the term "invention", which is merely for convenience and is not intended to voluntarily limit the scope of the present application to any single invention or inventive concept if, in fact, more than one invention or inventive concept is disclosed. Therefore, although specific embodiments have been shown and described herein, it should be understood that any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all changes or modifications of the various embodiments. Upon reviewing the above description, the combination of the above embodiments and other embodiments not specifically described herein will be obvious to those skilled in the art.

[0127] An abstract of the disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together in a single embodiment for the purpose of simplifying the disclosure. This method of the disclosure should not be interpreted as reflecting the intention that the claimed embodiment requires more features than the features explicitly recited in each claim. On the contrary, as reflected in the attached claims, the subject matter of the present invention lies in less than all the features of a single disclosed embodiment. Therefore, the claims are incorporated into the detailed description herein, wherein each claim exists independently as a separate embodiment.

[0128] Example

[0129] Example 1 is a method comprising: generating a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; generating a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera device of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determining a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud based on the corresponding predicted depths; and providing the relative pose to at least one of the first AR device or the second AR device.

[0130] Example 2 includes the method of Example 1, further including: rendering a first virtual object in a first display of a first AR device based on the relative posture; and rendering a second virtual object in a second display of a second AR device based on the relative posture, wherein the second virtual object corresponds to the first virtual object.

[0131] Example 3 includes the method of Example 1, further comprising: accessing first sparse 3D points from a six degree of freedom (6DOF) tracker of the first AR device.

[0132] Example 4 includes the method of Example 3, wherein the 6DOF tracker comprises a visual-inertial-simultaneous localization and mapping (VI-SLAM) system.

[0133] Example 5 includes the method of Example 1, wherein the pre-trained model is a machine learning model trained using images captured by multiple cameras.

[0134] Example 6 includes the method of Example 1, wherein the server is configured to: communicate with a first AR device and a second AR device; determine a relative pose by aligning overlapping areas of the first point cloud and the second point cloud; and provide the relative pose to at least one of the first AR device or the second AR device.

[0135] Example 7 includes the method of Example 1, wherein the first AR device communicates with the second AR device, wherein the first AR device is configured to receive a second point cloud from the second AR device and determine the relative pose.

[0136] Example 8 includes the method of Example 1, and further includes: generating a predicted depth of a third point cloud by applying a pre-trained model to a third single image generated by a third monocular camera device of a third AR device and a third sparse 3D point generated by a third SLAM system at the third AR device; determining a first relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud; determining a second relative pose between the first reference coordinate system of the first AR device and a third reference coordinate system of the third AR device by aligning the first point cloud with the third point cloud; providing the first relative pose to the second AR device; and providing the second relative pose to the third AR device.

[0137] Example 9 includes the method of Example 1, and further includes: generating a predicted depth of a third point cloud by applying a pre-trained model to a third single image generated by a third monocular camera device of a third AR device and third sparse 3D points generated by a third SLAM system at the third AR device; and using a new reference coordinate system common to the first AR device, the second AR device, and the third AR device, by aligning the overlapping areas of the first point cloud, the second point cloud, and the third point cloud with respect to the new reference coordinate system, to identify a first posture of the first AR device based on the depth of the first point cloud, to identify a second posture of the second AR device based on the depth of the second point cloud, and to identify a third posture of the third AR device based on the depth of the third point cloud.

[0138] Example 10 includes the method of Example 1, wherein the first AR device is configured to generate a first point cloud from the first single image and the first sparse 3D points, the first point cloud being denser than the first sparse 3D points.

[0139] Example 11 is a computing device comprising: a processor; and a memory storing instructions, which, when executed by the processor, configure the device to: generate a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; generate a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera device of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determine a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud based on the corresponding predicted depths; and provide the relative pose to at least one of the first AR device or the second AR device.

[0140] Example 12 is a computing device of Example 11, wherein the instructions further configure the device to: render a first virtual object in a first display of a first AR device based on a relative posture; and render a second virtual object in a second display of a second AR device based on the relative posture, wherein the second virtual object corresponds to the first virtual object.

[0141] Example 13 is the computing device of Example 11, wherein the instructions further configure the device to: access first sparse 3D points from a six degree of freedom (6DOF) tracker of the first AR device.

[0142] Example 14 is the computing device of Example 13, wherein the 6DOF tracker comprises a visual-inertial-simultaneous localization and mapping (VI-SLAM) system.

[0143] Example 15 is the computing device of Example 11, wherein the pre-trained model is a machine learning model trained using images captured by multiple cameras.

[0144] Example 16 is a computing device of Example 11, wherein the server is configured to: communicate with a first AR device and a second AR device; determine a relative pose by aligning overlapping areas of the first point cloud and the second point cloud; and provide the relative pose to at least one of the first AR device or the second AR device.

[0145] Example 17 is the computing device of Example 11, wherein the first AR device communicates with the second AR device, wherein the first AR device is configured to receive the second point cloud from the second AR device and determine the relative pose.

[0146] Example 18 is a computing device of Example 11, wherein the instructions further configure the device to: generate a predicted depth of a third point cloud by applying the pretrained model to a third single image generated by a third monocular camera device of a third AR device and a third sparse 3D point generated by a third SLAM system at the third AR device; determine a first relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud; determine a second relative pose between the first reference coordinate system of the first AR device and a third reference coordinate system of the third AR device by aligning the first point cloud with the third point cloud; provide the first relative pose to the second AR device; and provide the second relative pose to the third AR device.

[0147] Example 19 is a computing device of Example 11, which generates a predicted depth of a third point cloud by applying a pre-trained model to a third single image generated by a third monocular camera device of a third AR device and a third sparse 3D point generated by a third SLAM system at the third AR device; and uses a new reference coordinate system common to the first AR device, the second AR device, and the third AR device, to align the overlapping areas of the first point cloud, the second point cloud, and the third point cloud with respect to the new reference coordinate system to identify a first posture of the first AR device based on the depth of the first point cloud, a second posture of the second AR device based on the depth of the second point cloud, and a third posture of the third AR device based on the depth of the third point cloud.

[0148] Example 20 is a non-transitory computer-readable storage medium, which includes instructions that, when executed by a computer, cause the computer to: generate a predicted depth of a first point cloud by applying a pretrained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; generate a predicted depth of a second point cloud by applying the pretrained model to a second single image generated by a second monocular camera device of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determine a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by aligning the first point cloud with the second point cloud based on the corresponding predicted depths; and provide the relative pose to at least one of the first AR device or the second AR device.

[0149] As shown below by way of example, implementations of the described subject matter may include one or more features alone or in combination.

Claims

1. A method comprising: Generate a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; Generate a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determining a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud based on the corresponding predicted depths; as well as The relative pose is provided to at least one of the first AR device or the second AR device.

2. The method according to claim 1, further comprising: rendering a first virtual object in a first display of the first AR device based on the relative pose; as well as rendering a second virtual object in a second display of the second AR device based on the relative pose, The second virtual object corresponds to the first virtual object.

3. The method according to claim 1, further comprising: The first sparse 3D points from a six degree of freedom (6DOF) tracker of the first AR device are accessed.

4. The method according to claim 3, wherein: The 6DOF tracker includes a Visual-Inertial - Simultaneous Localization and Mapping (VI-SLAM) system.

5. The method according to claim 1, wherein: The pre-trained model is a machine learning model trained using images captured by multiple cameras.

6. The method according to claim 1, wherein: The server is configured as: Communicating with the first AR device and the second AR device; determining the relative pose by registering overlapping regions of the first point cloud and the second point cloud; as well as The relative pose is provided to at least one of the first AR device or the second AR device.

7. The method according to claim 1, wherein: The first AR device is in communication with the second AR device, wherein the first AR device is configured to receive the second point cloud from the second AR device and determine the relative pose.

8. The method according to claim 1, further comprising: generating a predicted depth of a third point cloud by applying the pre-trained model to a third single image generated by a third monocular camera of a third AR device and third sparse 3D points generated by a third SLAM system at the third AR device; Determining a first relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud; determining a second relative pose between a first reference coordinate system of the first AR device and a third reference coordinate system of the third AR device by registering the first point cloud with the third point cloud; providing the first relative posture to the second AR device; as well as The second relative posture is provided to the third AR device.

9. The method according to claim 1, further comprising: generating a predicted depth of a third point cloud by applying the pre-trained model to a third single image generated by a third monocular camera of a third AR device and third sparse 3D points generated by a third SLAM system at the third AR device; and Using a new reference coordinate system common to the first AR device, the second AR device and the third AR device, by aligning the overlapping areas of the first point cloud, the second point cloud and the third point cloud with respect to the new reference coordinate system, a first posture of the first AR device is recognized based on the depth of the first point cloud, a second posture of the second AR device is recognized based on the depth of the second point cloud, and a third posture of the third AR device is recognized based on the depth of the third point cloud.

10. The method according to claim 1, wherein: The first AR device is configured to generate the first point cloud from the first single image and the first sparse 3D points, the first point cloud being denser than the first sparse 3D points.

11. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to: Generate a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; Generate a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determining a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud based on the corresponding predicted depths; as well as The relative pose is provided to at least one of the first AR device or the second AR device.

12. The computing device according to claim 11, wherein: The instructions further configure the apparatus to: rendering a first virtual object in a first display of the first AR device based on the relative pose; and rendering a second virtual object in a second display of the second AR device based on the relative pose, The second virtual object corresponds to the first virtual object.

13. The computing device of claim 11, wherein: The instructions further configure the apparatus to: The first sparse 3D points from a six degree of freedom (6DOF) tracker of the first AR device are accessed.

14. The computing device of claim 13, wherein: The 6DOF tracker includes a Visual-Inertial - Simultaneous Localization and Mapping (VI-SLAM) system.

15. The computing device of claim 11, wherein: The pre-trained model is a machine learning model trained using images captured by multiple cameras.

16. The computing device of claim 11, wherein: The server is configured as: Communicating with the first AR device and the second AR device; determining the relative pose by registering overlapping regions of the first point cloud and the second point cloud; as well as The relative pose is provided to at least one of the first AR device or the second AR device.

17. The computing device of claim 11, wherein: The first AR device is in communication with the second AR device, wherein the first AR device is configured to receive the second point cloud from the second AR device and determine the relative pose.

18. The computing device of claim 11, wherein: The instructions further configure the apparatus to: generating a predicted depth of a third point cloud by applying the pre-trained model to a third single image generated by a third monocular camera of a third AR device and third sparse 3D points generated by a third SLAM system at the third AR device; Determining a first relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud; determining a second relative pose between a first reference coordinate system of the first AR device and a third reference coordinate system of the third AR device by registering the first point cloud with the third point cloud; providing the first relative posture to the second AR device; as well as The second relative posture is provided to the third AR device.

19. The computing device of claim 11, wherein: The instructions further configure the apparatus to: generating a predicted depth of a third point cloud by applying the pre-trained model to a third single image generated by a third monocular camera of a third AR device and third sparse 3D points generated by a third SLAM system at the third AR device; and Using a new reference coordinate system common to the first AR device, the second AR device and the third AR device, by aligning the overlapping areas of the first point cloud, the second point cloud and the third point cloud with respect to the new reference coordinate system, a first posture of the first AR device is recognized based on the depth of the first point cloud, a second posture of the second AR device is recognized based on the depth of the second point cloud, and a third posture of the third AR device is recognized based on the depth of the third point cloud.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: Generate a predicted depth of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera device of a first augmented reality (AR) device and first sparse 3D points generated by a first SLAM system at the first AR device; Generate a predicted depth of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of a second AR device and second sparse 3D points generated by a second SLAM system at the second AR device; determining a relative pose between a first reference coordinate system of the first AR device and a second reference coordinate system of the second AR device by registering the first point cloud with the second point cloud based on the corresponding predicted depths; as well as The relative pose is provided to at least one of the first AR device or the second AR device.

Images