Correlating device coordinate systems in multi-person augmented reality systems

Abstract

A method and system for mapping the coordinate system of a device of an AR system are disclosed. A third device may receive a first posture associated with a first coordinate system from a first device and a second posture associated with a second coordinate system from a second device. The third device may receive an image including a reference marker presented by each of the first device and the second device. The first posture and a portion of the image including the first reference marker may define a first set of 3D coordinates. The second posture and a portion of the image including the second reference marker may define a second set of 3D coordinates. A coordinate system transformation may be generated from the correspondence between the first set and the second set of 3D coordinates. The coordinate system transformation may be sent to the first device and / or the second device.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality (AR), and more particularly, and without limitation, to a coordinate system for associating multiple devices in an AR environment. Background Art

[0002] In the real-world environment where the device is located, AR presentations can project virtual objects in a virtual environment displayed on the device's display. The device's camera can capture the real-world environment. Virtual objects can be created to mimic real-world objects and presented on the device's display so that the virtual objects appear to be naturally located in the virtual environment. For example, a camera can capture a real-world video of a real-world environment that includes an empty picnic table. The device generates a virtual object of a picnic basket, and the virtual object is presented as if it were placed on the picnic table. The virtual object is presented in the display and looks essentially as if the corresponding real-world object is physically located on the picnic table.

[0003] In a multi-device AR system, each device can present a virtual environment that represents the real environment. In particular, each device presents the virtual environment from the perspective of that device. Therefore, in such a system, it is important that the virtual environment is presented consistently across devices. Summary of the invention

[0004] Aspects of the present disclosure include a method for mapping a coordinate system of a device in a multi-person AR system. The method includes: a third mobile device receives data indicating a first posture of a first mobile device, wherein the first posture is defined relative to a first coordinate system associated with the first mobile device; the third mobile device receives data indicating a second posture of a second mobile device, wherein the second posture is defined relative to a second coordinate system associated with the second mobile device; the third mobile device receives an image showing a first reference mark displayed by the first mobile device and a second reference mark displayed by the second mobile device; the third mobile device identifies a first set of three-dimensional coordinates associated with the first reference mark, the first set of three-dimensional coordinates being identified relative to the first coordinate system; the third mobile device identifies a second set of three-dimensional coordinates associated with the second reference mark, the second set of three-dimensional coordinates being identified relative to the second coordinate system; the third mobile device generates a coordinate system transformation based on the first set of three-dimensional coordinates and the second set of three-dimensional coordinates, the coordinate system transformation mapping coordinates between the first coordinate system and the second coordinate system; the third mobile device sends the coordinate system transformation to the second mobile device.

[0005] Another aspect of the present disclosure includes a system including one or more processors and a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform the above method.

[0006] Another aspect of the present disclosure includes a non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the above method.

[0007] Other areas of applicability of the present disclosure will become apparent from the detailed description provided below.It should be understood that the detailed description and specific examples, although indicating various embodiments, are intended only for illustrative purposes and are not intended to necessarily limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure is described in conjunction with the accompanying drawings.

[0009] Figure 1 is an illustration of an AR system according to at least one aspect of the present disclosure, in which coordinate systems of multiple devices can be aligned.

[0010] Figure 2 Examples of fiducial markers that may be presented on a first device for use in multi-device coordinate system alignment according to at least one aspect of the present disclosure are shown.

[0011] Figure 3A A process of generating a posture of a device relative to a coordinate system of another device according to at least one aspect of the present disclosure is shown.

[0012] Figure 3B A process of generating 3D coordinates of a second device relative to a coordinate system of the second device according to at least one aspect of the present disclosure is shown.

[0013] Figure 3C A process of generating a coordinate system transformation according to at least one aspect of the present disclosure is shown.

[0014] Figure 4 is a flow chart of a process for aligning coordinate systems of two mobile devices according to at least one aspect of the present disclosure.

[0015] Figure 5 An example of components of a computing system that executes an AR application according to at least one aspect of the present disclosure is shown.

[0016] In the drawings, similar components and / or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a dash and a second label that distinguishes the similar components. If only the first reference number is used in the specification, the specification applies to any similar components having the same first reference number, regardless of the second reference number. DETAILED DESCRIPTION

[0017] The following description only provides preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of preferred exemplary embodiments will provide effective instructions for implementing preferred exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope set forth in the appended claims.

[0018] An augmented reality application is executed on a device to present virtual objects in a video simultaneously captured of a real-world environment in which the device is located. The device may present the captured video on a display screen of the device, wherein the virtual objects are presented so that the virtual objects appear correctly in the virtual environment. For example, the virtual objects may be presented on the display screen in substantially the same position and orientation as the corresponding real-world objects appear physically in the real-world environment. In order to maintain the coherence of the virtual objects as the device moves, the device may track their position and orientation in the real-world environment to ensure that the virtual objects continue to appear correctly when the device's perspective changes. The position and orientation define the posture of the device. The device may define a coordinate system to map the virtual environment to the real-world environment and track its posture and virtual objects.

[0019] In a multi-device augmented reality application, the same virtual object can be presented on each display screen of multiple devices located in a real-world environment. Typically, each device performs a tracking process, such as a SLAM process, to track its posture (e.g., position and orientation) in the environment according to its own coordinate system. Since the coordinate systems of the devices are different, a transformation between the coordinate systems may be required in order to display various instances of the same virtual object on the devices in a coordinated and coherent manner. The transformation can be generated based on one or more of the devices that display known fiducial markers and the remaining devices that capture images of these fiducial markers. In particular, at least two devices present fiducial markers, and a third device generates an image showing the presented fiducial markers to derive the transformation between the coordinate systems of the two devices.

[0020] For example, during AR calibration, the first device and the second device may begin tracking their respective poses (e.g., position and orientation) in a real-world environment. The first device may determine a first pose (T1) relative to a first coordinate system of the first device using, for example, a SLAM process or other tracking technology. Similarly, the second device may determine a second pose (T2) relative to a second coordinate system of the second device. The first device may present a first reference marker on a display screen of the first device and send data about the first pose (T1) to a third device. The second device may also present a second reference marker on a display screen of the second device and send data about the second pose (T2) device to the third device. The third device may be directed to generate an image showing the reference markers displayed by the first device and the second device within the same image.

[0021] In some examples, the first device and the second device may present an instance of the same fiducial marker. Based on the image, the AR application of the third device detects the first feature point p from the first fiducial marker. 1i (i=0,1,2,3...n) and detect the second feature point p from the second reference mark 2i (i=0,1,2,3...n). The second feature point p 2i (i=0,1,2,3…n) and the first feature point p 1i The AR application derives the first feature point p in the first coordinate system from the first posture (T1) of the first device and the known information about the geometric shape of the first device. 1i In addition, the AR application derives a second feature point p in the second coordinate system from the second posture (T2) and the information about the geometry of the second device. 2i The AR application establishes a correspondence between the first set of 3D coordinates and the second set of 3D coordinates.

[0022] The third device uses the above correspondence to generate a coordinate system transformation. The coordinate system transformation can map points (e.g., coordinates) in the first coordinate system to corresponding points in the second coordinate system, and vice versa. In some cases, the coordinate system transformation may include estimates of one or more rigid transformations, such as rotation, translation, reflection, combinations of the above, etc.

[0023] The devices described herein can be any type of computing device, such as a mobile device (e.g., a smartphone, laptop, tablet computer, PDA, etc.), a desktop computing device, a dedicated hardware device such as a fixed or portable game console, etc. The devices described herein may include a built-in camera that can be used during the execution of an AR application. In some cases, the camera may not be built-in, but connected to the device. In these cases, the device and the camera can move independently of each other. The coordinate system of the device can be based on the position / orientation of the device or based on the position / orientation of the camera.

[0024] Figure 1 is an illustration of an AR system 100 that can map coordinate systems of multiple devices according to at least one aspect of the present disclosure. The AR system 100 can run a multi-device augmented reality application in which virtual objects can be presented within the display screen of each device. For example, a first device 104 can define a virtual object to be presented within an environment in which the first device is located. The virtual object can be presented on the display screen 108 of the first device 104 as if the virtual object were a physical object located in the environment. Since the position of the virtual object in the environment can be based on the coordinate system 102 of the first device, which may not be shared by the second device 116 of the multi-device AR application, the virtual object may not be correctly presented on the display screen 120 of the second device 116 unless the coordinate system of the first device 104 can be mapped to the coordinate system 118 of the second device 116.

[0025] During the initial calibration process of the AR application, the coordinate system 102 of the first device 104 can be mapped to the coordinate system 118 of the second device 116 without using fixed reference points in the environment. For example, each of the first device 104 and the second device can present an image of a fiducial marker on a respective display screen. The third device 128 can access the image of the first device 104 and the second device 116 displaying the fiducial marker and use the image (together with the pose of each of the first device and the second device and the device geometry information of each device) to generate a coordinate system transformation that maps the coordinates of the first coordinate system 102 associated with the first device to corresponding coordinates of the second coordinate system 118 associated with the second device.

[0026] In particular, during the calibration process, it can be determined that the current position of the device is the origin of its own coordinate system. The device can use internal sensors (e.g., accelerometers, global positioning system (GPS) sensors, compasses, combinations of the above, etc.), image processing (e.g., using machine learning, deep learning, etc.), or a combination of the above to track its own movement and update its position in its coordinate system. The device may not have data related to the environment in which it is located, but it can track its position relative to the device's initial calibration position (e.g., its origin). For example, if the internal sensors indicate that the device has moved one meter after calibration, the device's position can be determined as one meter from the origin (in a particular direction). Therefore, the device's coordinate system can be used to indicate the relative position of the device (e.g., relative to the device's position at calibration), and not necessarily the absolute position of the device, because the environment may be unknown.

[0027] In some cases, the device may perform a simultaneous localization and mapping (SLAM) process that may define the coordinate system of the device and the posture of the device. The SLAM process may also track devices and objects in the environment relative to the coordinate system of the device. The SLAM process may be used to track devices and objects in the environment even if the environment is unknown to the SLAM process (at least initially). The SLAM process may take as input variables such as, but not limited to, control data ct, sensor data st, and time intervals t, and generate an output that may include an approximate position xt of the device within a given time interval and a map mt of the environment.

[0028] SLAM can start with a calibration step, where an empty map of the environment can be initialized with the device at the origin. The SLAM process can update xt and mt when the device captures sensor data indicating movement in a particular direction (and optionally image data from the device's camera that can be used to indicate objects in the environment). SLAM can be an iterative process that updates xt and mt at set time intervals or when new sensor data or image data can be detected. For example, if no sensor changes occur between time intervals t and t+1, the SLAM process can delay updating the position and map to preserve processing resources. When a change in sensor data is detected that indicates a high probability that the device has moved from its previous position xt, the SLAM process can calculate the device's new position xt and update the map mt.

[0029] Once the first device and the second device detect the initial poses of the first device and the second device, respectively, the first device may present the fiducial marker 112 on the display screen 108 of the first device 104 and the second device may present the fiducial marker 124 on the display screen 120 of the second device 116. The fiducial markers 112 and 124 may be predetermined so that the devices of the multi-device AR system 100 may know the size, shape, color, pattern, shading, etc. The fiducial markers 112 and 124 may be the same or different. If the fiducial markers 112 and 124 are to be the same fiducial marker, the first device may send an identification of the fiducial marker to be displayed to the second device so that the second device displays the same fiducial marker. Alternatively, the second device may send an identification of the fiducial marker to be displayed to the first device so that the first device displays the same fiducial marker.

[0030] If the fiducial markers 112 and 124 displayed on the first device and the second device are different, the third device can identify the fiducial markers displayed by each device, for example, using imaging processing as described below to identify or by receiving the identification of the displayed fiducial markers from the first device and the second device. The third device can then determine that the feature points of one fiducial marker correspond to the feature points of another fiducial marker (for example, corresponding to the same 2D coordinates or the same physical position on the first device, etc.). For example, the third device can use a table including the correspondence between the feature points of two or more fiducial markers. The third device can use the identification of each fiducial marker and identify the corresponding feature points of each fiducial marker from the table. The fiducial marker 112 and the fiducial marker 128 are shown as different fiducial markers, wherein the fiducial marker 112 may include a group of black squares distributed over a predetermined area, and the predetermined area has a known size. The fiducial marker 124 may be a checkerboard pattern of black and white squares.

[0031] The third device 128 can be directed to obtain an image of the display screen 108 of the first device 104 and the display screen 120 of the second device 116 when the fiducial markers 112 and 124 are presented, so that both the fiducial markers 112 and 124 appear in the image. Since the size, color, pattern, etc. of the fiducial markers are known, the images of the fiducial markers 112 and 124 can be processed to define a first set of two-dimensional feature points corresponding to the fiducial marker 112 and a second set of two-dimensional feature points corresponding to the fiducial marker 124. The feature points can correspond to points within the fiducial marker, such as the center of a square, a vertex between squares, a corner of a fiducial marker, or any other point of the fiducial marker that can be easily identified based on the characteristics of the fiducial marker. The first set of two-dimensional feature points can be processed together with the first posture of the first device 104 and the known geometric information of the first device to identify a first set of 3D coordinates (relative to the first coordinate system 102), wherein each 3D coordinate can correspond to one of the first set of feature points. The second set of two-dimensional feature points can be processed together with the second posture of the second device 116 and the known geometric information of the second device to identify a second set of 3D coordinates (relative to the second coordinate system 118), where each 3D coordinate can correspond to a feature point of the second set of feature points. The correspondence between the first set of 3D coordinates and the second set of 3D coordinates can be used to generate a coordinate system transformation including one or more rigid transformations. A rigid transformation is a geometric transformation of Euclidean space that preserves the distance between pairs of points. A rigid transformation can include rotation (around one or more axes), translation, reflection, or a combination thereof. A rigid transformation can provide a mapping of coordinates of the first coordinate system 102 to corresponding coordinates of the second coordinate system 118.

[0032] The first coordinate system 102 may be mapped to the second coordinate system 118 during an initial calibration process. In some cases, the mapping process may be performed again after the initial calibration process, such as when the SLAM process is reset (as this will start a new coordinate system for the device), after calibration values ​​indicate that the mapping is no longer accurate, or after a predetermined time interval.

[0033] Figure 2 Examples of fiducial markers that can be presented on a device to map the coordinate system of each device according to at least one aspect of the present disclosure are shown. Fiducial markers 204-236 can be fiducial markers that can be used to map the coordinate system of a device. When placed in the real world, the fiducial markers can be used to detect the position, orientation, and / or scale of objects in the environment. The fiducial markers 204-236 can include predetermined sizes, shapes, and patterns that can be known to the device that captured the image of the fiducial marker. Using the known characteristics of the fiducial markers, the device can detect the fiducial markers in the image and calculate the position of the fiducial markers, define virtual objects to occupy space near or at the fiducial markers, detect the scale of objects within the environment, detect the orientation of the device relative to the fiducial markers, combinations of the above, and the like.

[0034] For example, the characteristics of each fiducial marker can be selected so that the fiducial marker and its feature points can be detected regardless of the specific rotation or transformation of the marker within the captured image. For example, the fiducial marker can include one or more shapes within the fiducial marker that appear different when rotated or transformed so as to indicate the degree of rotation and / or transformation when detected. The degree of rotation / transformation can be used to determine the orientation of the device that captured the image. For example, if the fiducial marker in the image is rotated 45 degrees, it can be determined that the device is also rotated 45 degrees.

[0035] In some cases, one or more rotations, affine transformations, Euclidean transformations, reflections, transpositions, and combinations thereof may be performed on an image of a fiducial marker and a processed fiducial marker that appears in a predetermined orientation may be output. For example, a device that photographs a fiducial marker may store characteristics of the fiducial marker (e.g., size, pattern, color, etc.). However, the fiducial marker may not appear as expected (e.g., rotated, blurred, stretched, etc.). The image may be processed to separate the fiducial marker and rotate and / or transform the fiducial marker so that the fiducial marker appears in the expected orientation. In other cases, the image of the fiducial marker may not be processed to change the orientation of the fiducial marker within the image.

[0036] The device detects the direction of the fiducial marker by detecting one or more feature points of the fiducial marker. The feature points can be detected using the characteristics of the detected fiducial marker and the known characteristics of the fiducial marker. For example, the feature points can be based on specific characteristics of the marker. For example, the fiducial marker 204 can be a checkerboard pattern. Feature points can be detected at the vertices between each group of four squares, the center of each white square or black square, the corners, the corners of each white square, the corners of each black square, and combinations thereof. Each fiducial marker may include one or more feature points that can be detected within the image. In some cases, each fiducial marker may include three or more feature points. Although any number of feature points can be detected, the more feature points that can be detected, the higher the accuracy of mapping one coordinate system to another coordinate system.

[0037] In some cases, each fiducial marker may include a different number of feature points than the other fiducial markers. For example, a first fiducial marker may include a first set of feature points, and a second fiducial marker may include a second set of feature points (where the number of feature points in the first set is not equal to the number of feature points in the second set). At least some of the feature points in the first set of feature points may correspond to at least some of the feature points in the second set of feature points. These corresponding feature points may be identified by a table of correspondences between different fiducial markers. For example, the table may indicate which feature points in the first set of feature points of the first fiducial marker correspond to which feature points in the second set of feature points of the second fiducial marker. In these cases, the first set of feature points may include three or more feature points corresponding to feature points in the second set of feature points.

[0038] The device may use image processing to detect the fiducial marker from other parts of the image and to detect feature points from the fiducial marker. One such image processing technique includes edge detection. Edge detection may include filtering techniques, wherein one or more filters may be applied to the image. The filter may modify the image by blurring, sharpening, transforming (for example, but not limited to, one or more affine transforms, Euclidean transforms, etc.), etc. The filter may reduce image noise by, for example, removing image artifacts and / or other parts of the image that do not correspond to the fiducial marker.

[0039] In some cases, some parts of the image may be processed more than other parts of the image. For example, one part of the image may appear blurry while another part of the image may appear sharp. Different filters may be applied to different parts of the image, and further, a different set of filters may be applied to different parts of the image. For example, a first part of the image may be filtered to sharpen the first part, and a second part of the image may be filtered using an affine transform filter and noise reduction. Any number of different filters may be applied to the image and / or each tile.

[0040] Once the filter is applied, edge detection can identify changes in pixel intensity gradients between adjacent pixels. Large changes in intensity between adjacent pixels can indicate the presence of an edge. For example, a first pixel adjacent to a pixel with a low intensity value has a high intensity value, which can provide an indication that the first pixel is part of an edge. In some cases, pixels that are not part of an edge can be suppressed (e.g., set to a predetermined red / green / blue value, such as black, where red=0, blue=0, and green=0, or any predetermined red / green / blue value). Edge detection operators such as Roberts crossover operator, Prewitt operator, Sobel operator, etc. can be used as part of the identification of pixel intensity gradients.

[0041] A non-maximum suppression process can be used to suppress pixels that do not strongly correspond to an edge. The non-maximum suppression process assigns an edge intensity value to each pixel identified as part of an edge using a pixel intensity gradient. For each pixel identified as part of an edge, the edge intensity value of the pixel can be compared to the edge intensity values ​​of the pixel's eight surrounding pixels. If the pixel has a higher edge intensity value (e.g., a local maximum) than the edge intensity values ​​of the surrounding pixels, the surrounding pixels are suppressed. Non-maximum suppression can be repeated for each pixel in the entire image.

[0042] Then, a double thresholding process can be performed to remove noise and / or false edge pixels caused by the previous image processing techniques applied herein. Two pixel intensity thresholds can be defined, one high and one low. The thresholds can be used to assign strong or weak intensity attributes to each pixel. Pixels including intensity values ​​above the high threshold can be assigned a strong intensity attribute, where pixels including intensity values ​​between the high threshold and the low threshold can be assigned a weak intensity attribute. Pixels including intensity values ​​below the low threshold can be suppressed (e.g., in the same manner as described above).

[0043] Next, a hysteresis process may be performed to remove pixels with weak intensity attributes (weak due to noise, color change, etc.). For example, a local statistical analysis (e.g., connected domain analysis, etc.) may be performed on each pixel with a weak intensity attribute. Pixels with weak intensity attributes that are not surrounded by pixels including strong intensity attributes may be suppressed. The remaining pixels (e.g., unsuppressed pixels) after the hysteresis process include only those pixels that belong to the edge. Although the above five processing processes are described in a specific order, each process may be performed any number of times (e.g., repeatedly) and / or in any order without departing from the spirit or scope of the present disclosure. In some cases, only a subset of the five processes need to be performed on the image. For example, the image processing may perform a recognition process of a pixel intensity gradient without first performing a filtering process. In some cases, a partially processed image may be received (e.g., one or more of the above processes have been performed). In this case, one or more additional processes may be performed to complete the image processing.

[0044] In some cases, signal processing (e.g., similar to a radio frequency signal) may be performed on the image. The image may be transformed into the frequency domain (e.g., using a Fourier transform, etc.) to represent the frequency at which specific pixel features exist in the image (e.g., pixel intensity, RGB value, etc.). In the frequency domain, one or more filters (e.g., but not limited to a Butterworth filter, a bandpass, etc.) may be applied to the image (e.g., during or after preprocessing, edge detection) to suppress or change specific frequencies. Suppressing specific frequencies may reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels of a specific color or color gradient, standardize color gradients, etc. A high-pass filter may bring out edges in an image (e.g., sharpen the color and / or intensity between adjacent pixels), while a low-pass filter may blend edges (e.g., blur). Image padding may be performed prior to signal processing to improve signal processing techniques. In some cases, different portions and / or blocks of an image may be processed differently, some of which are processed with a high-pass filter and others with a low-pass filter. In some cases, the threshold (e.g., the cutoff frequency of a high-pass or low-pass filter) can be modified for different portions of the image (e.g., based on image processing of one or more previous images, machine learning, and / or the like).

[0045] Signal processing can also determine other properties of the image, such as identifying the coherence of the relationships between pixels (e.g., for edge detection, segmentation, pattern analysis, etc.). The relationships between pixels can be used to further refine edge detection and / or identify structural features depicted within the image. For example, coherence can be used to identify related image portions (e.g., portions of the same object) and unrelated image portions.

[0046] Fiducial markers 204-236 are examples of fiducial markers that can be used to map one coordinate system of two or more devices to another coordinate system. For example, fiducial marker 204 can be a checkerboard pattern of alternating squares with two or more colors. In some cases, the colors can have high contrast, such as white and black. In other cases, one or more colors other than black and white can be used, such as red, green and / or blue (or alternatively, cyan, magenta and / or yellow). In some other cases, a contrasting pattern fill can be used, where one group of squares may not include a pattern, while another group of squares can use cross-hatching. Fiducial marker 204 may or may not include a border around the fiducial marker, because edge detection can be used to define the border of the fiducial marker.

[0047] The fiducial markers may have irregular shapes and may not follow a set pattern. For example, fiducial markers 208, 212, 216, 220, and 236 include a group of black squares scattered in a predetermined area. The square shape of the fiducial marker may be used in part to determine the specific outline of the fiducial marker. In addition, the scatter pattern of the group of squares (e.g., the distance between two or more specific squares, etc.) may be used to indicate the location of the device capturing the image. For example, the distance between two non-adjacent squares may be known to the device. The device may calculate the difference between the known distance and the distance detected in the captured image. The greater the difference between the known value and the distance calculated from the image, the farther the camera may be from the fiducial marker.

[0048] Similarly, the size of a particular set of squares can be calculated and compared to known sizes. Changes in the size of the squares can be used to determine the orientation of the device relative to the fiducial markers. For example, if one side of the square is larger than the other sides of the square, the device's camera may have captured the image of the fiducial marker from an angle that is offset from the fiducial marker rather than perpendicular to it.

[0049] In some cases, the fiducial markers may have non-square shapes, such as fiducial markers 224 and 228. Fiducial markers 224 and 228 may be circular shapes with inner circular shapes. In some cases, these fiducial markers may include one or more additional shapes within them, such as a line that bisects a circle. These additional shapes may indicate the orientation of the fiducial marker to indicate the orientation of the device.

[0050] Although Figure 2 The specific shapes shown in FIG. 1 have specific patterns, shapes, colors, orientations, etc., but the fiducial markers can have any specific shape, which can be geometric, such as the squares and circles shown, or amorphous.

[0051] Figure 3AThe process of generating 3D coordinates of the coordinate system of the first device relative to the first device according to at least one aspect of the present disclosure is shown. The third device can capture an image of the first device and the second device presenting the fiducial marker. The first device can present a fiducial marker, such as fiducial marker 304. The third device can detect one or more feature points of the fiducial marker 304. The feature point can be any visible unique feature of the fiducial marker. In some cases, the direction of the camera can be determined from the direction of the fiducial marker based on the characteristics of the fiducial marker. For example, the fiducial marker 304 can be oriented based on the color of the corner shape or the distance between non-adjacent shapes (e.g., 316, 320). The feature points can be detected based on specific characteristics of the fiducial marker, such as the size of a single part of the fiducial marker (e.g., 308 and 312), color, shape, pattern, and combination thereof. For example, the fiducial marker 304 can include a feature point at each vertex 324 formed by two adjacent shapes sharing the same color. In this case, the fiducial marker 304 can include 9 feature points. Other fiducial markers can include more or fewer feature points. In some cases, any fiducial marker including at least three feature points can be used as the fiducial marker 304. These feature points can be aggregated into a first group of feature points 328.

[0052] A pose (T1) 332 of the first device at the time the third device captured the image of the marker may be received from the first device. The pose may represent the position and orientation of the first device relative to the coordinate system of the first device. In some cases, the pose 332 may be represented by a rotation vector R 1 n and the translation vector t 1 n The rotation vector and the translation vector may be represented in a transformation matrix as shown. The posture (T1) may be determined using a SLAM process performed on the first device, image processing performed on an image captured by the first device, device geometry information (e.g., device size), camera information (e.g., zoom focal length, skew parameters, principal point, scale factor, etc.), internal sensors (e.g., accelerometer, gyroscope, compass, etc.), combinations of the above, and the like.

[0053] The posture of the first device (T1) and the information about the geometry of the first device can be used to identify the 3D coordinates of each feature point in the first set of feature points relative to the coordinate system of the first device 336. For example, since the posture of the first device can indicate the 3D coordinates of the first device in the first coordinate system, the geometry of the device can be used to determine the 3D coordinates of each feature point in the first set of feature points in the first coordinate system.

[0054] Figure 3BThe process of generating 3D coordinates of a coordinate system of a second device relative to a second device according to at least one aspect of the present disclosure is shown. The second device may display the same fiducial marker as the first device, or display a different fiducial marker, such as but not limited to fiducial marker 340. The fiducial marker 340 may include a set of black shapes distributed in a predefined area. These shapes may be distributed so that the direction of the fiducial marker can be used to determine the direction of the camera that obtains the image of the fiducial marker. One or more feature points may be detected from the fiducial marker 340 based on the characteristics of the fiducial marker 340, such as but not limited to the number of black shapes, the distance between specific black shapes (e.g., 344 or 356), a shape having one or more common vertices or a specific number of common vertices, one or more corners of the fiducial marker (e.g., 348), the size of the fiducial marker (e.g., 352), the pattern of the fiducial marker, the color and a combination thereof, etc. In some cases, any fiducial marker including at least three feature points may be used as the fiducial marker 340. These feature points may be aggregated into a second set of feature points 360.

[0055] A gesture (T2) 364 of the second device at the time the third device captured the image of the marker 340 may be received. In some cases, gesture 364 may be represented by a rotation vector R 2 n and the translation vector t 2 n The rotation vector and the translation vector may be represented within a transformation matrix as shown. The posture (T2) may be determined in a manner similar to that determined by the first device and as described. The posture (T2) of the second device and information about the geometry of the second device may be used to identify data indicating 3D coordinates 368 of each feature point in the second set of feature points relative to a coordinate system of the second device. The data indicating 3D coordinates 368 may be identified in a manner similar to that described above.

[0056] Figure 3C 3D coordinates 336 of the first set of feature points and 3D coordinates 368 of the second set of feature points can be used to define a first set of rigid transformations 372, the first set of rigid transformations 372 including one or more rigid transformations, and the first set of rigid transformations 372 can transform the coordinates in the first coordinate system to the coordinates in the second coordinate system. For example, each feature point n in the first set of feature points 1 There is a corresponding feature point n in the second set of feature points 2 This correspondence can be used to match each 3D coordinate in the first coordinate system with the corresponding coordinate in the second coordinate system. For example, for each feature point n 1 , we can identify the corresponding 3D coordinates [x 1 n ,y1 n ,z 1 n ] and the corresponding feature point n in the second feature point 2 Then, we can identify the feature point n in the second coordinate system 2 The associated 3D coordinates [x 2 n ,y 2 n ,z 2 n ]. Then the 3D coordinates [x 1 n ,y 1 n ,z 1 n ] and 3D coordinates [x 2 n ,y 2 n ,z 2 n ] between the 3D coordinates [x 1 n ,y 1 n ,z 1 n ] performs one or more rigid transformations, such as rotation, translation, reflection, etc., to convert the 3D coordinates [x 1 n ,y 1 n ,z 1 n ] is changed to 3D coordinates [x 2 n ,y 2 n ,z 2 n ]. The rigid transformation may include one or more rotations, one or more translations, one or more reflections, one or more combinations thereof, etc. In some cases, a second set of rigid transformations 372 may be defined, the second set of rigid transformations 372 including one or more rigid transformations, the second set of rigid transformations 372 may change the coordinates in the second coordinate system to the coordinates in the first coordinate system. The first set of rigid transformations 272 and the second set of rigid transformations 372 are collectively referred to as rigid transformations 372.

[0057] Depending on the selected coordinate system, the coordinate system transformation 376 can be generated using the first set of rigid transformations 372 or the second set of rigid transformations. For example, the third device (or alternatively, the first device, the second device, a user of any device, or a server) can select a reference coordinate system (e.g., the first coordinate system or the second coordinate system) on which the coordinates of the virtual object are based. After selection, a coordinate system transformation 376 can be generated, which transforms the coordinates in the non-selected coordinate system to the corresponding coordinates in the selected coordinate system. For example, the coordinate system transformation can apply each of one or more rigid transformations to identify the corresponding output coordinates (e.g., 376) in the second coordinate system from the input coordinates (e.g., 380) in the first coordinate system. In some cases, the coordinate system transformation can include a first set of rigid transformations and a second set of rigid transformations, so that the coordinate system transformation can transform the coordinates in the first coordinate system into the corresponding coordinates in the second coordinate system as needed, and transform the coordinates in the second coordinate system into the corresponding coordinates in the first coordinate system. The first set of rigid transformations includes one or more rigid transformations. The second set of rigid transformations includes one or more rigid transformations.

[0058] Figure 4 It is a flowchart of a process of mapping one coordinate system to another coordinate system according to at least one aspect of the present disclosure. In box 404, a third mobile device may receive a first posture (T1) of a first mobile device. The first mobile device may perform a position tracking process, such as a SLAM process, which continuously tracks the posture (T1) of the first mobile device associated with the coordinate system of the first mobile device. The first posture (T1) may be received at a specific moment, such as when an image of the first device is captured. The third mobile device may receive device geometry information associated with the first mobile device, such as device size, display screen size, etc. For example, the third mobile device may receive device geometry information and the first posture from another device (such as a server or a second mobile device, etc.), as well as an image of a fiducial marker (such as a fiducial marker as described below) displayed on the first mobile device. In some cases, the third mobile device may receive a model identifier of the first mobile device. The third mobile device may then use the model identifier to query (such as querying an internal storage table or querying a server) to obtain device geometry information. The third mobile device may also receive an identifier of the first fiducial marker, which indicates a specific fiducial marker displayed by the first mobile device.

[0059] At block 408, the third mobile device may receive a second posture (T2) of the second mobile device. Similar to the first device, the second mobile device may perform a position tracking process, such as a SLAM process, which continuously tracks the posture (T2) of the second mobile device associated with the coordinate system of the second mobile device. The second posture (T2) may be received at a specific time, such as when an image of the second device is captured. In some cases, the first posture and the second posture may be received almost simultaneously (e.g., received in parallel). In other cases, the first posture and the second posture may be received asynchronously, such as received successively, with one posture received followed by another posture. The third mobile device may receive device geometry information associated with the second mobile device, such as device size, display screen size, etc. For example, the third mobile device may receive device geometry information and the second posture from another device (e.g., a server or a second mobile device, etc.), as well as an image of a fiducial marker (e.g., a fiducial marker as described below) displayed on the second mobile device. In some cases, the third mobile device may receive a model identifier of the second mobile device. The third mobile device may then use the model identifier to query (e.g., query an internal storage table or query a server) to obtain device geometry information. The third mobile device may also receive an identifier of the second fiducial marker, which indicates a specific fiducial marker displayed by the second mobile device.

[0060] In box 412, the third mobile device can receive the image of the first fiducial marker presented on the display screen of the first mobile device and the second fiducial marker presented on the display screen of the second mobile device. The third device can receive instructions from each of the first device and the second device to present to the user, guiding the user to operate the camera of the third device to capture the image of the first fiducial marker and the second fiducial marker. For example, when each of the first mobile device and the second mobile device presents the first fiducial marker and the second fiducial marker, the third mobile device can use the camera to obtain the image of the display screen of the first mobile device and the second mobile device. In another example, the third mobile device can receive the image through the network.

[0061] A first set of feature points can be detected from a portion of an image including a first fiducial marker, and a second set of feature points can be detected from a portion of an image including a second fiducial marker. Since the fiducial marker has a known size and geometry, the feature points can be used to determine the position and / orientation of a third device relative to the fiducial marker. In conjunction with a posture of the device presenting the fiducial marker, the set of feature points can be used to define 3D coordinates of the set of feature points in the same coordinate system as the posture. In some cases, the set of feature points includes three feature points. In other cases, the set of feature points includes four or more feature points.

[0062] At block 416, the third device may define a first set of 3D coordinates. The first set of 3D coordinates may be defined using a first posture of the first mobile device, device geometry information of the first mobile device, and a first set of feature points. For example, by using the geometry of the mobile device, the physical location of each feature point displayed by the first mobile device may be estimated using the known size and geometry of the fiducial marker (e.g., associated with the first coordinate system). The 3D coordinates of each feature point in the first coordinate system may be determined using the posture of the first mobile device.

[0063] In box 420, the third device can define a second set of 3D coordinates. The second set of 3D coordinates can be defined using the second posture of the second mobile device, the device geometry information of the second mobile device, and the second set of feature points. For example, by using the geometry of the mobile device, the physical position of each feature point displayed by the second mobile device can be estimated using the known size and geometry of the fiducial marker (e.g., associated with the second coordinate system). The posture of the second mobile device can be used to determine the 3D coordinates of each feature point in the second coordinate system. In some cases, the first set of 3D coordinates and the second set of 3D coordinates can be defined in parallel. In other cases, the first set of 3D coordinates and the second set of 3D coordinates can be defined asynchronously (e.g., overlappingly or successively, one set of coordinates is defined before another set of coordinates).

[0064] At block 422, a correspondence between the first set of 3D coordinates and the second set of 3D coordinates may be generated. If the first mobile device and the second mobile device display the same reference image, each feature point in the first feature point will correspond to a feature point in the second feature point that forms a corresponding feature point pair. By matching the 3D coordinates in the first set of 3D coordinates and the second set of 3D coordinates associated with the feature point pair, a correspondence between the 3D coordinates may be generated. If the first mobile device and the second mobile device display different reference images, respectively, each of the first mobile device and the second mobile device may send an identification of the displayed reference marker to the third mobile device. The third mobile device may use a lookup table to determine a correspondence between the first feature point and the second feature point that form the corresponding feature point pair. Then, by matching the 3D coordinates in the first set of 3D coordinates and the second set of 3D coordinates associated with the feature point pair, a correspondence between the 3D coordinates may be generated. The correspondence may represent a mapping that associates coordinates of the first set of 3D coordinates (associated with the first coordinate system) and coordinates of the second set of 3D coordinates (associated with the second coordinate system).

[0065] At block 424, a coordinate system transformation may be generated using the first set of 3D coordinates and the second set of 3D coordinates. For example, since each feature point in the first feature point has a corresponding feature point in the second feature point, the 3D coordinates associated with the feature point in the first feature point may be matched to the 3D coordinates associated with the corresponding feature point in the second feature point. Each 3D coordinate associated with the first device may be matched to a 3D coordinate of the second device. After matching, one or more rigid transformations may be applied to the first 3D coordinate in the first coordinate system to change the first 3D coordinate to its matching 3D coordinate in the second coordinate system.

[0066] In some cases, a single matching pair of 3D coordinates can be used to define the rigid transformation required to map coordinates in one coordinate system to corresponding 3D coordinates in another coordinate system. The remaining 3D coordinate pairs can be used to verify the accuracy of one or more rigid transformations. In other cases, two or more matching pairs of 3D coordinates can be used to define the rigid transformation required to map coordinates in one coordinate system to corresponding 3D coordinates in another coordinate system. In other cases, each matching pair of 3D coordinates can be used to define the rigid transformation required to map coordinates in one coordinate system to corresponding 3D coordinates in another coordinate system.

[0067] The coordinate system transformation may include a first set of rigid transformations that apply one or more of rotations, translations, or reflections to coordinates in the first coordinate system to determine corresponding coordinates in the second coordinate system. The first set of rigid transformations includes one or more rigid transformations. In some cases, the coordinate system transformation may additionally (or alternatively) include a second set of rigid transformations that apply one or more of rotations, translations, or reflections to coordinates in the second coordinate system to determine corresponding coordinates in the first coordinate system. The second set of rigid transformations includes one or more rigid transformations.

[0068] The coordinate system transformation can map a point in the first coordinate system of the first mobile device to a corresponding point in the second coordinate system of the second mobile device. In some cases, the position calculated by the SLAM process of the first mobile device can be transformed into a corresponding position in the second coordinate system of the second mobile device. In other cases, the position calculated by the SLAM process of the second mobile device can be transformed into a corresponding position in the first coordinate system of the first mobile device.

[0069] In block 428, a coordinate system transformation may be sent to the second mobile device to enable the second mobile device to transform the received coordinates in the first coordinate system into the second coordinate system. In some cases, a coordinate system transformation may be sent to the first mobile device to enable the first mobile device to transform the received coordinates in the second coordinate system into the first coordinate system. In other cases, a coordinate system transformation may be sent to the first mobile device and the second mobile device to enable the coordinates to be transformed into the appropriate coordinate system. This may enable the respective device to transform the received coordinates into the local coordinate system of the mobile device, encode the coordinates of the respective mobile device into the coordinate system of the receiving device, and then send the encoded coordinates to the receiving device.

[0070] For example, in an AR application, a first mobile device can capture an image or video of an environment. The first mobile device can define a first instance of a virtual object to be presented on a display screen of the first mobile device such that the first instance of the virtual object appears to be physically and naturally located in the environment. The SLAM process can track the first mobile device as it moves in the environment, and the AR application can continue to present the first instance of the virtual object as if it were naturally located in the environment (regardless of changes in the position or orientation of the first mobile device).

[0071] The second mobile device can receive information associated with the virtual object from the first mobile device (or from the server), including characteristics that enable the second mobile device to render an instance of the virtual object on the display screen of the second mobile device and posture information indicating the position and orientation of the virtual object in the environment. The second device can use a coordinate system transformation to convert the coordinates of the posture information from the first coordinate system of the first mobile device to the coordinate system of the second mobile device. The second mobile device can capture an image or video of the environment. The second mobile device can then present a second instance of the virtual object within the captured image / video of the environment, so that the second instance of the virtual object appears to be physically (and naturally) located within the environment (regardless of changes in the position or orientation of the second mobile device).

[0072] Figure 4The process can be performed entirely by a third mobile device, partially by one or more of the first mobile device, the second mobile device, the third mobile device and / or the server, or entirely by the server. For example, the server can direct the mapping between the coordinate systems of the first mobile device and the second mobile device and execute all or part of the augmented reality application. The server can instruct the third mobile device to obtain an image of a reference marker presented on the first mobile device and the second mobile device. The first pose can be sent from the first mobile device to the server, the second pose can be sent from the second mobile device to the server, and the image can be sent from the third mobile device to the server. The server can then use the first pose, the second pose, and the image to generate a coordinate system transformation, and send the coordinate system transformation to the first mobile device and / or the second mobile device.

[0073] Although Figure 4 The boxes in are presented in a specific order, but they can be executed in any specific order. In some cases, before proceeding to the next box, Figure 4 Each box can be executed once or multiple times. Figure 4 describes the mapping between the coordinate systems associated with two devices, but Figure 4 The process can be extended to map the coordinate systems of any number of devices, for example, by causing each additional device to perform the process of 404-432. For example, a third mobile device can be used to obtain an image of an additional mobile device presenting a reference image having the reference image displayed by the first mobile device, the second mobile device, or any other previously mapped mobile device.

[0074] Can be modified Figure 4 The process of mapping the coordinate system of one device to the coordinate system of another device is performed while still mapping the coordinate system of one device to the coordinate system of another device. For example, the third device can receive images of each device as separate images, so that the first image can include the display screen of the first mobile device and the second image can include the display screen of the second mobile device. Between taking the first image and taking the second image, the first mobile device, the second mobile device, and the third mobile device can be stationary. In another case, the first fiducial marker can be the same fiducial marker as the second fiducial marker, or a different fiducial marker. If different fiducial markers are used, the third device can include a feature point table indicating common feature points in the fiducial markers (e.g., the same 2D coordinate positions). In some cases, the feature point table can be received from the first device and / or the second device along with the corresponding identifiers of the displayed fiducial markers.

[0075] Figure 5 5 shows an example of components of a computing system for executing an AR application according to at least one aspect of the present disclosure. The computing system 504 may be Figure 4Although these components are shown as part of computing system 504, computing system 504 may also be distributed such that some components may be located on a different hardware platform than other components.

[0076] The computing system 504 includes at least a processor 508, a memory 512, a storage device 516, an input / output peripheral (I / O) 520, a communication peripheral 524, one or more cameras 528, and an interface bus 532. The interface bus 532 can be used to communicate, send, and transfer data, control, and commands between the various components of the computing system 504. The memory 512 and the storage device 516 may include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage, such as Memory, and other tangible storage media. Any such computer-readable storage media can be used to store instructions or program codes that implement various aspects of the present disclosure. Memory 512 and storage device 516 can also include computer-readable signal media. Computer-readable signal media include propagated data signals containing computer-readable program code. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media include any computer-readable media that is not a computer-readable storage medium and can communicate, propagate, or transmit a program for use in conjunction with the computing system 504.

[0077] In addition, the memory 512 may include an operating system, programs, and applications. The processor 508 may be used to execute stored instructions and include, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. The memory 512 and / or the processor 508 may be virtualized and may be hosted in another computing system such as a cloud network or a data center. The I / O peripherals 520 may include a user interface such as a keyboard, a screen (e.g., a touch screen), a microphone, a speaker, other input / output devices, and computing components such as a graphics processing unit, a serial port, a parallel port, a universal serial bus, and other input / output peripherals. The I / O peripherals 520 are connected to the processor 508 by any port coupled to the interface bus 532. The communication peripherals 524 may be used to facilitate communication between the computing system 504 and other computing devices through a communication network, and include, for example, a network interface controller, a modem, a wireless and wired interface card, an antenna, and other communication peripherals.

[0078] Although the subject matter has been described in detail with respect to its specific embodiments, it should be understood that those skilled in the art, after obtaining an understanding of the foregoing, can easily generate changes, variations, and equivalents to these embodiments. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration rather than limitation, and does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter that are obvious to a person of ordinary skill in the art. In fact, the methods and systems described herein can be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes can be made to the form of the methods and systems described herein without departing from the spirit of the present disclosure. The attached claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of the present disclosure.

[0079] Unless expressly stated otherwise, it should be understood that terms such as "process," "compute," "determine," and "identify" are used throughout the discussion of this specification to refer to the actions or processes of a computing device (e.g., one or more computers or similar electronic computing devices) that manipulates or transforms data represented as physical electronic or magnetic quantities in a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0080] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0081] Embodiments of the methods disclosed herein may be performed in the operation of such a computing device. The order of the blocks presented in the above examples may be changed—for example, the blocks may be reordered, combined, and / or decomposed into sub-blocks. Certain blocks or processes may be performed in parallel.

[0082] Conditional language used herein, such as "may," "might," "for example," and the like, unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that certain examples include and other examples do not include certain features, elements, and / or steps. Thus, such conditional language generally does not imply that one or more examples require the features, elements, and / or steps in any way, or that one or more examples must include logic for deciding, with or without author input or prompting, whether such features, elements, and / or steps are included or will be performed in any particular example.

[0083] The terms "including", "having", "having", etc. are synonymous and are used inclusively in an open manner and do not exclude other elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusiveness (rather than exclusiveness), so that when used to connect a list of elements, for example, the term "or" represents one, some or all of the elements in the list. "Applicable to" or "for" used herein refers to open and inclusive language, and does not exclude devices that are applicable to or used to perform additional tasks or steps. In addition, the use of "based on" means openness and inclusiveness, because the process, step, calculation or other action "based on" one or more listed conditions or values ​​may actually be based on additional conditions or values ​​outside the list. Similarly, the use of "based at least in part" means openness and inclusiveness, because the process, step, calculation or other action "based at least in part on" one or more listed conditions or values ​​can be based on additional conditions or values ​​outside the list in practice. The titles, lists and numbers included herein are only for ease of explanation and are not meant to be limiting.

[0084] The various features and processes described above can be used independently of each other, or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, some method or process blocks can be omitted in some embodiments. The methods and processes described herein are also not limited to any particular order, and the blocks or states related thereto can be performed in other appropriate orders. For example, the described blocks or states can be performed in a different order than that specifically disclosed, or multiple blocks or states can be combined in a single block or state. The example blocks or states can be performed serially, in parallel, or in some other way. Blocks or states can be added to or deleted from the disclosed examples. Similarly, the example systems and components described herein can be configured to be different from those described. For example, compared to the disclosed examples, elements can be added, removed, or rearranged.

[0085] Although the principles of the present disclosure have been described above in conjunction with specific devices and methods, it should be clearly understood that this description is only by way of example and not limiting the scope of the present disclosure.

Claims

1. A method for mapping a device coordinate system, characterized in that: The method comprises: receiving, by a third mobile device, data indicating a first posture of the first mobile device, wherein the first posture is defined by a first coordinate system relative to the first mobile device; The third mobile device receives data indicating a second posture of the second mobile device, wherein the second posture is defined by a second coordinate system relative to the second mobile device; receiving, by the third mobile device, an image showing a first fiducial marker displayed by the first mobile device and a second fiducial marker displayed by the second mobile device; the third mobile device identifying a first set of three-dimensional coordinates associated with the first fiducial marker, the first set of three-dimensional coordinates being identified relative to the first coordinate system; the third mobile device identifying a second set of three-dimensional coordinates associated with the second fiducial marker, the second set of three-dimensional coordinates being identified relative to the second coordinate system; The third mobile device generates a coordinate system transformation based on the first set of three-dimensional coordinates and the second set of three-dimensional coordinates, the coordinate system transformation mapping coordinates in the first coordinate system to corresponding coordinates in the second coordinate system; and The third mobile device sends the coordinate system transformation to the second mobile device.

2. The method according to claim 1, characterized in that Identifying the first set of three-dimensional coordinates includes: A set of feature points of the first fiducial marker is detected, wherein each three-dimensional coordinate in the first set of three-dimensional coordinates corresponds to a feature point in the set of feature points.

3. The method according to claim 2, characterized in that The set of feature points includes three or more feature points.

4. The method according to claim 1, characterized in that The first pose is represented by a rotation vector and a translation vector, and wherein the first set of three-dimensional coordinates associated with the first fiducial marker is defined at least in part using the first pose and a geometry of the first mobile device.

5. The method according to claim 1, characterized in that: The coordinate system transformation is used to make a first virtual object to be presented by the first mobile device appear in the environment at substantially the same position and substantially the same orientation as a second virtual object presented by the second mobile device.

6. The method according to claim 1, characterized in that The coordinate system transformation includes a matrix for defining a mapping between the first coordinate system and the second coordinate system, and Sending the coordinate system transformation includes sending the matrix to the second mobile device via a data network between the third mobile device and the second mobile device.

7. The method according to claim 1, characterized in that Objects within the environment of the second mobile device are tracked based on the first coordinate system and the coordinate system transformation.

8. A mobile device, characterized in that: The mobile device comprises: one or more processors; A non-transitory computer readable medium storing a program, which, when executed by the one or more processors, causes the one or more processors to perform operations including: receiving data indicating a first posture of a first mobile device, wherein the first posture is defined by a first coordinate system relative to the first mobile device; receiving data indicating a second posture of a second mobile device, wherein the second posture is defined by a second coordinate system relative to the second mobile device; receiving an image, the image comprising a portion of a first display screen of the first mobile device and a portion of a second display screen of the second mobile device, wherein the first display screen of the first mobile device comprises a first fiducial marker and the second display screen of the second mobile device comprises a second fiducial marker; identifying a first set of three-dimensional coordinates associated with the first fiducial marker, the first set of three-dimensional coordinates being identified relative to the first coordinate system; identifying a second set of three-dimensional coordinates associated with the second fiducial marker, the second set of three-dimensional coordinates being identified relative to the second coordinate system; generating a coordinate system transformation using the first set of three-dimensional coordinates and the second set of three-dimensional coordinates, wherein the coordinate system transformation maps coordinates in the first coordinate system to corresponding coordinates in the second coordinate system; and The coordinate system transformation is sent to the second mobile device.

9. The mobile device according to claim 8, characterized in that Identifying the first set of three-dimensional coordinates includes: A set of feature points of the first fiducial marker is detected, wherein each three-dimensional coordinate in the first set of three-dimensional coordinates corresponds to a feature point in the set of feature points.

10. The mobile device according to claim 9, characterized in that The set of feature points includes three or more feature points.

11. The mobile device according to claim 8, characterized in that The first pose is represented by a rotation vector and a translation vector, wherein the first set of three-dimensional coordinates associated with the first fiducial marker is defined at least in part using the first pose and a geometry of the first mobile device.

12. The mobile device according to claim 8, characterized in that The coordinate system transformation is used to make a first virtual object to be presented by the first mobile device appear in the environment at substantially the same position and substantially the same orientation as a second virtual object presented by the second mobile device.

13. The mobile device according to claim 8, characterized in that The coordinate system transformation includes a matrix for defining a mapping between the first coordinate system and the second coordinate system, and Sending the coordinate system transformation includes sending the matrix to the second mobile device over a data network.

14. The mobile device according to claim 8, characterized in that Objects within the environment of the second mobile device are tracked based on the first coordinate system and the coordinate system transformation.

15. A non-transitory computer-readable medium, characterized in that The device stores instructions which, when executed by one or more processors, cause the one or more processors to perform operations including: receiving data indicating a first posture of a first mobile device, wherein the first posture is defined by a first coordinate system relative to the first mobile device; receiving data indicating a second posture of a second mobile device, wherein the second posture is defined by a second coordinate system relative to the second mobile device; receiving an image including a portion of a first display screen of the first mobile device and a portion of a second display screen of the second mobile device, wherein the first display screen of the first mobile device includes a first fiducial marker and the second display screen of the second mobile device includes a second fiducial marker; identifying a first set of three-dimensional coordinates associated with the first fiducial marker, the first set of three-dimensional coordinates being identified relative to the first coordinate system; identifying a second set of three-dimensional coordinates associated with the second fiducial marker, the second set of three-dimensional coordinates being identified relative to the second coordinate system; generating a coordinate system transformation using the first set of three-dimensional coordinates and the second set of three-dimensional coordinates, wherein the coordinate system transformation maps coordinates in the first coordinate system to corresponding coordinates in the second coordinate system; and The coordinate system transformation is sent to the second mobile device.

16. The non-transitory computer readable medium of claim 15, wherein: Identifying the first set of three-dimensional coordinates includes: A set of feature points of the first fiducial marker is detected, wherein each three-dimensional coordinate in the first set of three-dimensional coordinates corresponds to a feature point in the set of feature points.

17. The non-transitory computer readable medium of claim 16, wherein: The set of feature points includes three or more feature points.

18. The non-transitory computer readable medium of claim 15, wherein: The first pose is represented by a rotation vector and a translation vector, and wherein the first set of three-dimensional coordinates associated with the first fiducial marker is defined at least in part using the first pose and a geometry of the first mobile device.

19. The non-transitory computer readable medium of claim 15, wherein: The coordinate system transformation is used to make a first virtual object to be presented by the first mobile device appear in the environment at substantially the same position and substantially the same orientation as a second virtual object presented by the second mobile device.

20. The non-transitory computer readable medium of claim 15, wherein: The coordinate system transformation includes a matrix defining a mapping between the first coordinate system and the second coordinate system, wherein sending the coordinate system transformation includes sending the matrix to the second mobile device over a data network.

21. The non-transitory computer readable medium of claim 15, wherein: Objects within the environment of the second mobile device are tracked based on the first coordinate system and the coordinate system transformation.

Images