Method and system for associating device coordinate systems in a multi-person AR system

By receiving and processing equipment posture data, and using reference markers and image processing technology to generate coordinate system transformation, the problem of inconsistency in virtual objects in multi-device augmented reality systems is solved, and the consistency and coherence of virtual objects on multiple devices is achieved. .

CN114730212BActive Publication Date: 2024-11-26GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080079560.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-05
Filing Date
2020-12-02
Publication Date
2024-11-26
Estimated Expiration
2040-12-02

AI Technical Summary

Technical Problem

In multi-device augmented reality systems, maintaining consistency and coherence of virtual objects is a challenge because each device has its own unique coordinate system, resulting in inconsistent presentation of virtual objects on different devices.

Method used

By receiving and processing posture data from other devices, using reference markers and image processing techniques, coordinate system transformations are generated to map coordinate systems between different devices, ensuring the coherent presentation of virtual objects on multiple devices.

Benefits of technology

The consistency and coherence of virtual objects present on multiple devices is achieved, ensuring the consistent position and orientation of virtual objects between devices, and enhancing the coordination of augmented reality applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114730212B_ABST
    Figure CN114730212B_ABST
Patent Text Reader

Abstract

A method and system for mapping coordinate systems of devices for augmented reality applications are disclosed. A second mobile device may receive a first posture defined relative to a first coordinate system associated with a first mobile device. The second mobile device may receive an image of a display screen of the first mobile device presenting a reference marker. Data indicating three-dimensional coordinates of the first mobile device may be identified and used to define a second posture of the second mobile device associated with the coordinate system of the first device. A coordinate system transformation may be generated using the second posture, the coordinate system transformation mapping coordinates between a first coordinate system of the first mobile device and a second coordinate system of the second mobile device. Based on the coordinate system transformation, a first virtual object may be presented on the display screen of the second mobile device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality and, more particularly and without limitation, to mapping coordinate systems of multiple devices. Background Art

[0002] Virtual reality presentations (AR presentations) can project virtual objects within a real-world environment displayed on a device's display. The camera of the augmented reality device can capture the real-world environment. Virtual objects can be created and presented through the device's display so that the virtual objects appear to be naturally located in the environment. For example, a camera can capture a real-time video of a real-world environment that includes an empty picnic table. The virtual reality device can generate a virtual object that is presented as if it were placed on the picnic table. The virtual object is presented in the display device and appears essentially as if the object was physically located on the picnic table.

[0003] In a multi-device augmented reality system, each device can present a virtual representation of the real-world environment. In particular, each device presents a virtual representation of the environment from the perspective of that device. Therefore, in such a system, it is important that the virtual environment is presented consistently across devices. Summary of the invention

[0004] Aspects of the present disclosure include a method for mapping device coordinate systems in a multi-person augmented reality application. The method includes: a second mobile device receives data indicating a first posture of a first mobile device, wherein the first posture is defined relative to a first coordinate system associated with the first mobile device; the second mobile device determines a second posture of the second mobile device, wherein the second posture is defined relative to a second coordinate system associated with the second mobile device; the second mobile device receives an image of a display screen of the first mobile device, the image showing a marker; the second mobile device identifies three-dimensional coordinates associated with a reference marker based on the first posture and the above image, the three-dimensional coordinates are defined relative to the first coordinate system; based on the three-dimensional coordinates, a third posture of the second mobile device is determined, wherein the third posture is defined relative to the first coordinate system associated with the first mobile device; the second mobile device generates a coordinate system transformation based on the second posture and the third posture, the coordinate system transformation maps coordinates between the first coordinate system and the second coordinate system; the second mobile device presents a first virtual object on the display screen of the second mobile device based on the coordinate system transformation.

[0005] Another aspect of the present disclosure includes a system including one or more processors and a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform the above method.

[0006] Another aspect of the present disclosure includes a non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the above method.

[0007] Other areas of applicability of the present disclosure will become apparent from the detailed description provided below.It should be understood that the detailed description and specific examples, although indicating various embodiments, are intended only for illustrative purposes and are not intended to necessarily limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure is described in conjunction with the accompanying drawings.

[0009] Figure 1 is an illustration of an augmented reality system according to at least one aspect of the present disclosure, in which coordinate systems of multiple devices can be mapped.

[0010] Figure 2 Examples of fiducial markers that may be presented on a first device for multi-device coordinate system mapping in accordance with at least one aspect of the present disclosure are shown.

[0011] Figure 3A A process of generating a posture of a device relative to a coordinate system of another device according to at least one aspect of the present disclosure is shown.

[0012] Figure 3B A process of generating a coordinate system transformation according to at least one aspect of the present disclosure is shown.

[0013] Figure 4 is a flow chart of a process of mapping coordinate systems of two mobile devices according to at least one aspect of the present disclosure.

[0014] Figure 5 An example of components of a computing system that executes an augmented reality application according to at least one aspect of the present disclosure is shown.

[0015] In the drawings, similar components and / or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a dash and a second label that distinguishes the similar components. If only the first reference number is used in the specification, the specification applies to any similar components having the same first reference number, regardless of the second reference number. DETAILED DESCRIPTION

[0016] The following description only provides preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of preferred exemplary embodiments will provide effective instructions for implementing preferred exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope set forth in the appended claims.

[0017] An augmented reality application is executed on a device to present virtual objects in a video simultaneously captured of a real-world environment in which the device is located. The device may present the captured video on a display screen of the device, wherein the virtual objects are presented so that the virtual objects appear correctly in the real-world environment. For example, the virtual objects may be presented on the display in substantially the same position and orientation as the corresponding real-world objects appear physically in the real-world environment. In order to maintain the coherence of the virtual objects as the device moves, the device may track their position and orientation in the real-world environment to ensure that the virtual objects continue to appear correctly when the device's perspective changes. The position and orientation define the posture of the device. The device may define a coordinate system to map the real-world environment to the virtual environment and track its posture relative to the coordinate system.

[0018] In a multi-device augmented reality application, the same virtual object can be presented on each display of multiple devices located in a real-world environment. Typically, each device performs a tracking process, such as a SLAM process, to track its pose (e.g., position and orientation) in the real-world environment based on its own coordinate system. Because the coordinate systems of the devices are different, transformations between the coordinate systems may be required in order to display various instances of the same virtual object on the devices in a coordinated and coherent manner. The transformation can be generated based on one or more of the devices displaying known fiducial markers or natural images and the remaining devices that capture images of these fiducial markers or natural images.

[0019] In an example, during the AR calibration process, the first device may track its pose relative to the first coordinate system of the first device (T 11 ). Similarly, the second device can track its posture relative to the second coordinate system of the second device (T 22 The posture used here (T ij ) refers to the pose of device "i" in coordinate system "j". The first device may present a fiducial marker or marker image on the display screen of the first device and its pose (T 11 )(or a series of postures(T 11)) is sent to the second device along with a timestamp (or a series of timestamps, where each timestamp in the series corresponds to a gesture in the series of gestures. Each gesture can be tracked in the first coordinate system. The marker image can be any image with detectable features that can be used by the device to define the gesture of the device relative to the detectable features.

[0020] The second device can take a single or a series of images of the fiducial marker or marker image displayed by the first device. The best image in the series of images and the matching pose (T 11 ) of the timestamp of the posture of the second device (T 22 Based on the image, the second device detects feature points from the reference marker or marker image. After selection, the second device detects feature points from the posture (T 11 ), known information about the reference mark or the mark image, and known information about the geometry of the first device to derive the three-dimensional (3D) coordinates of the reference point to establish a correspondence between the 3D coordinates and the 2D feature points. By using the PNP (perspective-n-point) technology based on the correspondence, the second device can derive its posture (T 21 ). The second device has two postures (T 21 ) and (T 22 ) can be used to generate a coordinate system transformation. A coordinate system transformation can transform a position and orientation from a first coordinate system to a second coordinate system, and vice versa.

[0021] Figure 1 1 is a diagram of an augmented reality system 100 according to at least one aspect of the present disclosure, in which a transformation between coordinate systems of multiple devices can be determined. The augmented reality system 100 includes multiple devices, such as a first device 104 and a second device 116. The devices 104 and 116 are located in a real world environment. Each of the devices 104 and 116 executes an AR application to display a virtual representation of the real world environment, wherein the virtual environment includes virtual objects.

[0022] In the example, the first device 104 can present a virtual object on the display screen 108 of the first device 104 as if the virtual object were a physical object located in a real-world environment. The presentation of the virtual object in the virtual environment is based on the first coordinate system 102 of the first device 104. The second device 116 can also present another instance of the virtual object on its display screen 120. Here again, the presentation depends on the second coordinate system 118 of the second device 116. To present the instance of the virtual object in a coordinated and coherent manner on the display screens 108 and 120, a transformation 124 between the first coordinate system and the second coordinate system is required.

[0023] During calibration of an augmented reality application executed on a first device 104 and a second device 116, a second coordinate system 118 of the second device 116 can be mapped to a first coordinate system 102 of the first device 104 without using fixed reference points in the environment. Instead, as further described below, the first device 104 presents a reference to the second device 116, and the second device 116 generates and processes an image of the reference to define a transformation between the two coordinate systems. Other operations are performed during calibration. For example, the first device 104 can perform a process of tracking its posture relative to the first coordinate system 102. Similarly, the second device 116 can also track its posture relative to the second coordinate system 118. After calibration, each device can begin to present virtual objects in a coordinated and coherent manner based on the transformation between the coordinate systems and the tracked postures.

[0024] For example, during calibration, it may be determined that the current location of the first device 104 is at the origin of its own coordinate system 102. The first device 104 may use internal sensors (e.g., accelerometers, global positioning system (GPS) sensors, compasses, combinations thereof, etc.), image processing (e.g., using machine learning, deep learning, etc.), or a combination thereof to track its own movement and update its position within the coordinate system 102. The first device 104 may not be aware of the environment in which it is located, but it may track its position relative to the location where the first device 104 was originally calibrated (e.g., its origin). For example, if the internal sensors indicate that the first device 104 has moved one meter after calibration, the location of the first device 104 may be determined to be one meter from the origin (in a particular direction). Therefore, the coordinate system 102 of the first device 104 may be used to indicate the relative location of the first device 104 (e.g., relative to the location of the first device 104 at the time of calibration), but not necessarily the absolute location of the first device 104, because the environment may be unknown.

[0025] In some cases, the first device 104 can perform a simultaneous localization and mapping (SLAM) process that can define the coordinate system 102 and track the posture of the device within the environment. The SLAM process can be used to track the first device 104 in the real-world environment even if the real-world environment is unknown to the SLAM process (at least initially). The SLAM process can take variables such as, but not limited to, control data ct, sensor data st, and time interval t as inputs and generate an output that can include an approximate position xt of the device within a given time interval and a map mt of the environment.

[0026] SLAM can start with a calibration step, in which a coordinate system 102 representing a real-world environment can be initialized, with the first device 104 at the origin. When the first device 104 captures sensor data indicating movement in a particular direction (and optionally image data from the camera of the first device 104 that can be used to indicate objects in the environment), the SLAM process can update xt and mt. SLAM can be an iterative process that updates xt and mt at set time intervals or when new sensor data or image data can be detected. For example, if no sensor changes occur between time intervals t and t+1, the SLAM process can delay updating the position and map to preserve processing resources. When a change in sensor data is detected indicating that the device has moved from its previous position xt with a high probability, the SLAM process can calculate the new position xt of the device and update the map mt. The second device 116 can perform the same process to define the coordinate system of the second device 116 and the position and direction n of the second device 116 relative to the coordinate system 118.

[0027] Once the initial pose of the first device 104 and the second device 116 is detected, the first device 104 can present a fiducial marker 112 (or other type of reference) on the display screen 108 of the first device 104. The fiducial marker 112 can be predetermined so that the size, shape, color, pattern, shading, etc. can be known to the devices in the multi-device augmented reality system 100. For example, the fiducial marker 112 can be a checkerboard pattern of black and white squares of known size. When displaying the fiducial marker 112, the first device 104 can send to the second device 116 its pose defined in the first coordinate system 102, instructions for taking an image of the fiducial marker 112, and optionally, information about the geometry of the first device 104 (e.g., screen size) and a fiducial marker identifier indicating a specific fiducial marker presented by the first device 104. In some cases, the first device 104 can send a series of poses, wherein each pose in the series of poses includes a first timestamp corresponding to the moment when the pose was determined.

[0028] The second device 116 can be directed to capture an image of the fiducial marker displayed by the first device 104. If more than one gesture is received from the first device, the second device can select a gesture having a first timestamp that is closest to a second timestamp from the series of gestures, the second timestamp corresponding to the moment when the image was captured. In some cases, the second device 116 can capture a series of images to ensure that at least one image in the series of images captures specific features of the fiducial marker required to define a coordinate system transformation. Each image may include a second timestamp indicating the moment when the image was captured. The second device can select the best image from the series of images. If multiple gestures are received from the first device, the second device can select a gesture having a first timestamp that is closest to the second timestamp of the best image from the series of gestures.

[0029] In some cases, the second device 116 can use two (or more) gestures received from the first device 104. For example, the second device 116 identifies two or more gestures that are determined to be closest in time to the moment when the best image was captured (based on their respective timestamps). The second device can then interpolate between the two gestures received from the first device 104 using the corresponding timestamps of the gestures to generate a new gesture of the first device 104 that corresponds to the gesture of the first device at the moment when the best image was captured by the second device 116.

[0030] In some cases, the second device 116 may pre-store device geometry information for the first device 104 and / or possible fiducial markers that the first device 104 may present. For example, the second device 116 may load the device geometry information and known characteristics of the fiducial markers from local memory. If more than one type of device may be used in the AR application, the second device 116 may receive an identifier, such as a device model identifier, to determine the correct device geometry information. The device geometry information may be received from local memory or from a server. If the first device 104 does not send a fiducial marker identifier, the second device 116 may use a classifier in combination with one or more image processing techniques such as edge detection to identify the fiducial marker.

[0031] Since the size, color, pattern, etc. of the fiducial marker is known, the image of the fiducial marker 112 can be processed to define a set of two-dimensional (2D) feature points. Each 2D feature point can correspond to a point within the fiducial marker 112, such as the center of a square, a vertex between squares, a corner of a fiducial marker, or any other point of the fiducial marker that can be easily identified based on the features of the fiducial marker. The set of 2D feature points can be processed together with the posture of the first device 104 relative to the coordinate system 102 of the first device 104 in the environment and the known geometric information of the first device to identify the 3D coordinates of each feature point. The correspondence between the set of 2D feature points and each 3D coordinate can be used to calculate the posture of the second device 116 relative to the coordinate system 102 of the first device 104. The posture can be used to generate a coordinate system transformation, which can map the feature points of the coordinate system 102 of the first device 104 and the second coordinate system 118 of the second device 116.

[0032] The second coordinate system may be mapped to the first coordinate system during calibration. In some cases, the mapping process may be performed again after calibration, such as when the SLAM process is reset (as this will start a new coordinate system for the device), after the calibration values ​​indicate that the mapping is no longer accurate, or after a predetermined time interval.

[0033] Figure 2 Examples of fiducial markers that can be presented on a first device for multi-device coordinate system mapping according to at least one aspect of the present disclosure are shown. Markers 204-236 can be fiducial markers that can be used to map coordinates of a first coordinate system to corresponding coordinates of a second coordinate system. The fiducial markers 204-236 may include predetermined sizes, shapes, and patterns that can be known to the device that captured the image of the fiducial marker. Since the characteristics of the fiducial marker are known, the fiducial marker (once captured by the device's camera) can be detected using, for example, image processing. The features of the fiducial marker captured in the image can be used, for example, to define the position of the device relative to the fiducial marker, determine the scale of an object in a real-world environment, determine the orientation of the device, map the coordinate system of the device to another device, and so on.

[0034] For example, the characteristics of each fiducial marker can be selected so that the fiducial marker and its feature points can be detected regardless of the specific rotation or transformation of the marker within the captured image. For example, the fiducial marker can include one or more shapes within the fiducial marker that appear different when rotated or transformed so as to indicate the degree of rotation and / or transformation when detected. The degree of rotation / transformation can be used to determine the orientation of the device that captured the image. For example, if the fiducial marker in the image is rotated 45 degrees, it can be determined that the device is also rotated 45 degrees.

[0035] In some cases, one or more rotations, affine transformations, Euclidean transformations, reflections, transpositions, and combinations thereof may be performed on an image of a fiducial marker and a processed fiducial marker that appears in a predetermined orientation may be output. For example, a device that photographs a fiducial marker may store characteristics of the fiducial marker (e.g., size, pattern, color, etc.). However, the fiducial marker may not appear as expected (e.g., rotated, blurred, stretched, etc.). The image may be processed to separate the fiducial marker and rotate and / or transform the fiducial marker so that the fiducial marker appears in the expected orientation. In other cases, the image of the fiducial marker may not be processed to change the orientation of the fiducial marker within the image.

[0036] The device detects the direction of the fiducial marker by detecting one or more feature points of the fiducial marker. The feature points can be detected using the characteristics of the detected fiducial marker and the known characteristics of the fiducial marker. For example, the feature points can be based on specific characteristics of the marker. For example, the fiducial marker 204 can be a checkerboard pattern. Feature points can be detected at the vertices between each group of four squares, the center of each white square or black square, the corners of each white square, the corners of each black square, and combinations thereof. Each fiducial marker may include one or more feature points that can be detected within the image. In some cases, each fiducial marker may include three or more feature points. Although any number of feature points can be detected, the more feature points that can be detected, the higher the accuracy of the mapped coordinate system.

[0037] The device may use image processing to detect the fiducial marker from other parts of the image and to detect feature points from the fiducial marker. One such image processing technique includes edge detection. Edge detection may include filtering techniques, wherein one or more filters may be applied to the image. The filter may modify the image by blurring, sharpening, transforming (for example, but not limited to, one or more affine transforms, Euclidean transforms, etc.), etc. The filter may reduce image noise by, for example, removing image artifacts and / or other parts of the image that do not correspond to the fiducial marker.

[0038] In some cases, some parts of the image may be processed more than other parts of the image. For example, one part of the image may appear blurry while another part of the image may be sharp. Different filters may be applied to different parts of the image, and further, a different set of filters may be applied to different parts of the image. Different filters may be applied to different parts of the image. For example, a first part of the image may be filtered to sharpen the first part, and a second part of the image may be filtered using an affine transform filter and noise reduction. Any number of different filters may be applied to the image and / or each tile.

[0039] Once the filter is applied, edge detection can identify changes in pixel intensity gradients between adjacent pixels. Large changes in intensity between adjacent pixels can indicate the presence of an edge. For example, a first pixel adjacent to a pixel with a low intensity value has a high intensity value, which can provide an indication that the first pixel is part of an edge. In some cases, pixels that are not part of an edge can be suppressed (e.g., set to a predetermined red / green / blue value, such as black, where red=0, blue=0, and green=0, or any predetermined red / green / blue value). Edge detection operators such as Roberts crossover operator, Prewitt operator, Sobel operator, etc. can be used as part of the identification of pixel intensity gradients.

[0040] A non-maximum suppression process can be used to suppress pixels that do not strongly correspond to an edge. The non-maximum suppression process assigns an edge intensity value to each pixel identified as part of an edge using a pixel intensity gradient. For each pixel identified as part of an edge, the edge intensity value of the pixel can be compared to the edge intensity values ​​of the pixel's eight surrounding pixels. If the pixel has a higher edge intensity value (e.g., a local maximum) than the edge intensity values ​​of the surrounding pixels, the surrounding pixels are suppressed. Non-maximum suppression can be repeated for each pixel in the entire image.

[0041] Then, a double thresholding process can be performed to remove noise and / or false edge pixels caused by the previous image processing techniques applied herein. Two pixel intensity thresholds can be defined, one high and one low. The thresholds can be used to assign strong or weak intensity attributes to each pixel. Pixels including intensity values ​​above the high threshold can be assigned a strong intensity attribute, where pixels including intensity values ​​between the high threshold and the low threshold can be assigned a weak intensity attribute. Pixels including intensity values ​​below the low threshold can be suppressed (e.g., in the same manner as described above).

[0042] Next, a hysteresis process may be performed to remove pixels with weak intensity attributes (weak due to noise, color change, etc.). For example, a local statistical analysis (e.g., connected domain analysis, etc.) may be performed on each pixel with a weak intensity attribute. Pixels with weak intensity attributes that are not surrounded by pixels including strong intensity attributes may be suppressed. The remaining pixels after the hysteresis process (e.g., unsuppressed pixels) include only those pixels that belong to edge portions. Although the above five processing processes are described in a specific order, each process may be performed any number of times (e.g., repeatedly) and / or in any order without departing from the spirit or scope of the present disclosure. In some cases, only a subset of the five processes need to be performed on the image. For example, the image processing may perform a recognition process of a pixel intensity gradient without first performing a filtering process. In some cases, a partially processed image may be received (e.g., one or more of the above processes have been performed). In this case, one or more additional processes may be performed to complete the image processing.

[0043] In some cases, signal processing (e.g., similar to a radio frequency signal) may be performed on the image. The image may be transformed into the frequency domain (e.g., using a Fourier transform, etc.) to represent the frequency at which specific pixel features exist in the image (e.g., pixel intensity, RGB value, etc.). In the frequency domain, one or more filters (e.g., but not limited to a Butterworth filter, a bandpass, etc.) may be applied to the image (e.g., during or after preprocessing, edge detection) to suppress or change specific frequencies. Suppressing specific frequencies may reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels of a specific color or color gradient, standardize color gradients, etc. A high-pass filter may bring out edges in an image (e.g., sharpen the color and / or intensity between adjacent pixels), while a low-pass filter may blend edges (e.g., blur). Image padding may be performed prior to signal processing to improve signal processing techniques. In some cases, different portions and / or blocks of an image may be processed differently, some of which are processed with a high-pass filter and others with a low-pass filter. In some cases, the threshold (e.g., the cutoff frequency of a high-pass or low-pass filter) can be modified for different portions of the image (e.g., based on image processing of one or more previous images, machine learning, and / or the like).

[0044] Signal processing can also determine other properties of the image, such as identifying the coherence of the relationships between pixels (e.g., for edge detection, segmentation, pattern analysis, etc.). The relationships between pixels can be used to further refine edge detection and / or identify structural features depicted within the image. For example, coherence can be used to identify related image portions (e.g., portions of the same object) and unrelated image portions.

[0045] Marks 204-236 are examples of reference marks that can be used to map the coordinate systems of two or more devices. For example, reference mark 204 can be a checkerboard pattern of alternating squares with two or more colors. In some cases, the colors can have high contrast, such as white and black. In other cases, one or more colors other than black and white can be used, such as red, green and / or blue (or alternatively, cyan, magenta and / or yellow). In some other cases, a contrasting pattern fill can be used, where one group of squares can not include a pattern, and another group of squares can use cross hatching. Mark 204 may or may not include a border around the reference mark, because edge detection can be used to define the border of the reference mark.

[0046] The markers may have irregular shapes and may not follow a set pattern. For example, the fiducial markers 208, 212, 220, and 236 include a group of black squares scattered in a predetermined area. The square shape of the fiducial marker may be used in part to determine the specific outline of the fiducial marker. In addition, the scatter pattern of the group of squares (e.g., the distance between two or more specific squares, etc.) may be used to indicate the location of the device capturing the image. For example, the distance between two non-adjacent squares may be known to the device. The device may calculate the difference between the known distance and the distance detected in the captured image. The greater the difference between the known value and the distance calculated from the image, the farther the camera may be from the fiducial marker.

[0047] Similarly, the size of a particular set of squares can be calculated and compared to known sizes. Changes in the size of the squares can be used to determine the orientation of the device relative to the fiducial markers. For example, if one side of the square is larger than the other sides of the square, the device's camera may have captured the image of the fiducial marker from an angle offset from the fiducial marker rather than perpendicular to it.

[0048] In some cases, the fiducial markers may have non-square shapes, such as fiducial markers 224 and 228. Markers 224 and 228 may be circular shapes with inner circular shapes. In some cases, these fiducial markers may include one or more additional shapes within them, such as a line that bisects a circle. These additional shapes may indicate the orientation of the fiducial marker to indicate the orientation of the device.

[0049] Although Figure 2 The specific shapes shown in FIG. 1 have specific patterns, shapes, colors, orientations, etc., but the fiducial markers can have any specific shape, which can be geometric, such as the squares and circles shown, or amorphous.

[0050] Figure 3AAn example of generating a gesture of a coordinate system of one device relative to another device according to at least one aspect of the present disclosure is shown. A second device can capture an image of a first device presenting a fiducial marker 304, and one or more feature points of the fiducial marker 304 can be detected. A feature point can be any visible unique point of a fiducial marker. In some cases, detecting feature points within a fiducial marker can include detecting the size of a unique portion of the fiducial marker, such as size 308 and 312. In addition, distances between non-adjacent shapes, such as 316 and 320, can be detected to determine the direction of a camera capturing an image. In other cases, feature points detected within a fiducial marker can be used to determine the size of a unique shape or the distance between non-adjacent shapes. Feature points can be detected based on specific characteristics of the fiducial marker 304. For example, a fiducial marker 304 can include a feature point at each vertex 324 formed by two adjacent shapes sharing the same color. Mark 304 can include nine feature points. Other fiducial markers can include more or fewer feature points. Feature points can be aggregated into a group of feature points 328.

[0051] The second device may receive the gesture (T 11 )332. The posture can represent the position and orientation of the first device relative to the coordinate system of the first device. In some cases, the posture 332 can be represented by a rotation vector R[R1, R2, R3] and a translation vector t[t1, t1, t1]. In other cases, the posture 332 can be represented as a transformation matrix. The posture (T) can be determined using a SLAM process performed on the first device, image processing performed on images captured by the first device, device geometry information (e.g., the size of the device), camera information (e.g., zoom focal length, skew parameters, principal point, scale factor, etc.), internal sensors (e.g., accelerometer, gyroscope, compass, etc.), combinations of the above, etc. 11 ).

[0052] The posture of the first device (T 11 ) and a set of feature points can be used to identify data indicating three-dimensional coordinates 336 of each feature point in the set of feature points relative to a coordinate system of the first device. For example, each feature point can represent a 2D coordinate associated with a fiducial marker displayed on the first device. The posture of the first device represents the position and orientation of the first device. The combination of the device geometry information and the position and orientation of the first device can be used to generate a 3D coordinate (in the first coordinate system) for each feature point.

[0053] A correspondence between the feature points and the three-dimensional coordinates of the feature points may be established. The second device may use the correspondence between the three-dimensional coordinates and the set of feature points to perform a PNP process to generate a current posture of the second device relative to the coordinate system of the first device. For example, the set of feature points may be used to determine the position and orientation of the second device relative to the set of feature points. Since the correspondence links the three-dimensional coordinates (based on the coordinate system of the first device) and the set of feature points, the second device may identify its position and orientation or posture 344 relative to the coordinate system of the first device (T 21 ).

[0054] Figure 3B The process of generating a coordinate system transformation according to at least one aspect of the present disclosure is shown. The posture (T 21 )344 can be related to the posture of the second device relative to the coordinate system of the second device (T 22 ) are used together to generate a coordinate system transformation 352. The coordinate system transformation can be used to map points (eg, coordinates) of the second coordinate system to corresponding coordinates in the first coordinate system, and vice versa.

[0055] For example, point set 360 may be points corresponding to a virtual object in the second coordinate system. By converting point set 360 to a corresponding point set 356 in the first coordinate system, the virtual object may be presented in the display screen of the first device. The coordinate system transformation may convert points in the second coordinate system to corresponding points in the first coordinate system and / or convert points in the first coordinate system to corresponding points in the second coordinate system. Thus, the coordinate system transformation may be used to present a virtual object defined according to one coordinate system in substantially the same position and orientation in another coordinate system.

[0056] Figure 4 4 is a flowchart of a process for mapping coordinate systems of two mobile devices according to at least one aspect of the present disclosure. In block 404, a first posture (T 11 The first mobile device may perform a position tracking process, such as a SLAM process, that tracks a posture of the first mobile device associated with a coordinate system of the first mobile device (T 11 ). The first posture (T 11 ) may be a specific gesture at a specific moment. The first gesture may be associated with a first timestamp, the first timestamp corresponding to the moment when the first gesture is determined. In some cases, a series of gestures may be received, wherein each gesture in the series of gestures is associated with a corresponding first timestamp.

[0057] At block 406, the second device may determine a second posture (T 22 ). Second posture (T 22) can be defined relative to a coordinate system associated with the second mobile device. The second posture can be determined using a position tracking process (eg, a SLAM process).

[0058] In box 408, the second mobile device can receive an image of the fiducial marker presented on the display screen of the first mobile device. The image of the fiducial marker can be associated with a second timestamp, which indicates the moment when the image was taken. In some cases, a series of images of the fiducial marker can be taken, each of which is associated with a corresponding second timestamp. For example, the second mobile device can receive an instruction (from a server or a first mobile device) to use a camera of the second mobile device to take a picture of the display screen of the first mobile device while the first mobile device is presenting the fiducial marker. The instruction can be presented on the display screen of the second mobile device. In another example, the second mobile device can receive one or more pictures via a network connection, etc. The fiducial marker can be any marker with a predefined size or geometric shape.

[0059] Although fiducial markers are described, any type of marker can be used. Typically, while the mobile device is running (e.g., during an AR session), the mobile device generates an image. The image shows elements of the real-world environment. These elements can be used as Figure 4 An example process and markers in embodiments of the present disclosure. One example of an element is a fiducial marker. Other examples of elements include landmarks, such as a unique building or a portion thereof in a real-world environment, a recognizable feature in a real-world environment, a graphic in a real-world environment, a graphic overlaid on an image, etc.

[0060] In box 412, the second mobile device can receive alignment information. The alignment information can be sent from the first mobile device or from the server to the second mobile device. The alignment information may include a posture (or a series of postures) of the first mobile device, an identification of a fiducial marker displayed by the first mobile device, device geometry information of the first device, a device model identifier of the first device, and the like. In some cases, some alignment information may be received from the first mobile device, and other alignment information may be received from the server. For example, the device model identifier of the first mobile device may be used to look up device geometry information from a memory of the second mobile device or from a server. If the device model identifier of the first device matches the device model identifier of the second mobile device, it may not be necessary to obtain the device geometry information. In other cases, all alignment information may be received from the first mobile device. The second mobile device may receive the alignment information substantially simultaneously with, shortly before, or shortly after obtaining an image of the fiducial marker at box 410.

[0061] At block 416, a set of feature points can be detected from the image of the fiducial marker. If a series of images is captured, a best image can be selected from the series of images, and a set of feature points can be detected using the best image. The best image can be an image of a view captured of the entire fiducial marker, having the largest number of visible feature points, the clearest image, an image with the fewest image artifacts or defects, etc. Since the fiducial marker has a known size and geometry, the feature points can be used to determine the pose of the second mobile device relative to the fiducial marker. In some cases, the set of feature points includes three feature points. In other cases, the set of feature points includes four or more feature points.

[0062] In box 420, data indicating three-dimensional coordinates can be defined. The three-dimensional coordinates can be defined using a first posture, mobile device geometry information, and the above-mentioned set of feature points. If the second device receives a series of postures, the second mobile device can select a specific posture from the series of postures as the first posture. The specific posture can be a posture with a first timestamp that is closest to the second timestamp of the captured image (or if more than one image is captured, the best image selected). For example, if a series of postures p1, p2, p3 (respectively) are associated with first timestamps t1, t2, t3, and the captured image is associated with a second timestamp t2.1, posture p2 will be selected as the first posture because the first timestamp t2 associated with p2 is closest to the second timestamp t2.1 of the captured image. In some cases, if the timestamp of a posture is not approximately equal to the corresponding timestamp of the image, the second device can use the corresponding timestamp of each posture to interpolate between the two postures of the first device, thereby calculating the posture of the first device at the exact moment when the second device captures the best image. For example, postures p2 and p3 can be interpolated to calculate the image of the first device at the timestamp t2.1 of the captured image. By using the mobile device geometry to estimate the physical location of each feature point displayed on the first mobile device, the known size and geometry of the fiducial markers can be fully utilized. The pose of the first mobile device can be used to determine the three-dimensional coordinates of each feature point in the first coordinate system.

[0063] At block 424, a correspondence relationship between the three-dimensional coordinates and a set of feature points may be generated. The correspondence relationship may represent a mapping linking the three-dimensional coordinates (associated with the first coordinate system) and the feature points of the image of the fiducial marker.

[0064] At block 428, a third posture (T ) of the second mobile device relative to the first coordinate system may be defined using a correspondence between the three-dimensional coordinates and a set of feature points. 21). For example, the PNP process can use the above correspondence as input (along with camera calibration data such as zoom focal length, skew parameters, principal point, scale factor, etc.) to generate an estimated third pose of the second mobile device. The third pose of the second mobile device can be associated with the coordinate system of the first mobile device.

[0065] At block 432, a second pose (T) of the second mobile device relative to a coordinate system associated with the second mobile device may be used. 22 ) and a third posture (T 21 ) generates a coordinate system transformation. The coordinate system transformation can map a point in the first coordinate system of the first mobile device to a corresponding point in the second coordinate system of the second mobile device. In some cases, the position calculated by the SLAM process of the first mobile device can be converted into a position in the second coordinate system of the second mobile device. In other cases, the position calculated by the SLAM process of the second mobile device can be converted into a position in the first coordinate system of the first mobile device.

[0066] At block 436, an instance of the virtual object may be presented on the display screen of the second mobile device in a natural and coherent manner relative to a second instance of the virtual object presented on the display screen of the first mobile device based on the coordinate system transformation. For example, the second mobile device may receive data associated with the virtual object from the first mobile device, such as 3D coordinates of the virtual object in the first coordinate system. The second mobile device may convert the 3D coordinates of the virtual object associated with the first coordinate system into 3D coordinates in the second coordinate system using the coordinate system transformation. The second mobile device may present the instance of the virtual object using the 3D coordinates of the virtual object in the second coordinate system.

[0067] For example, in an augmented reality application, the second mobile device can take an image or video of the environment. The second mobile device can define instances of virtual objects to be presented on the display screen of the second mobile device as if they were physically located in the environment. The SLAM process can track the second mobile device as it moves in the environment so that the virtual objects continue to appear as if they are physically (and naturally) located in the environment (regardless of how the position or orientation of the second mobile device changes).

[0068] The first mobile device can then cause a second virtual object (e.g., an instance of a virtual object) to be presented within the display screen of the first mobile device at substantially the same position and orientation within the virtual environment as the virtual object. For example, the first device can capture an image / video of the environment and present the second virtual object within the captured image / video such that the second virtual object appears to be physically (and naturally) located within the environment (regardless of changes in the position or orientation of the second mobile device).

[0069] At block 440 , the first coordinate system and the second coordinate system may be synchronized by continuously tracking the first mobile device and the second mobile device according to the mapped coordinate system (eg, using respective SLAM processes executed on the respective mobile devices).

[0070] Figure 4 The process can be performed entirely by the first mobile device, partially by the first mobile device, entirely by the second mobile device, partially by the second mobile device, entirely by the server, or partially by the server. The server can guide the mapping of the coordinate systems of the first mobile device and the second mobile device and execute all or part of the augmented reality application. For example, the first mobile device can define a first posture (using the SLAM process performed by the first mobile device) and the second mobile device can define a second posture (using the SLAM process performed by the second mobile device). The second mobile device can be guided to obtain an image of a reference mark presented on the display screen of the first mobile device. The second mobile device can detect a set of feature points. In some cases, the first posture can be sent to the second mobile device and the second mobile device can identify the three-dimensional coordinates. In other cases, the set of feature points can be sent to the first mobile device and the first mobile device can identify the three-dimensional coordinates. In some other cases, the set of feature points can be sent to the server. The server can identify the three-dimensional coordinates.

[0071] Continuing with this example, the second posture can be defined by the second mobile device, or defined by the server, which sends the defined second posture back to the second mobile device. The second mobile device can use the second posture to generate a coordinate system transformation to map the position of the second coordinate system to the position of the first coordinate system. In some cases, the first mobile device can use the second posture to generate a coordinate system transformation to map the position of the first coordinate system to the position of the second coordinate system. The coordinate system transformation can be used with the SLAM process performed on the second mobile device to track virtual objects defined according to the coordinate system of the first device within the environment.

[0072] Although Figure 4 The boxes in are presented in a specific order, but they can be executed in any specific order. In some cases, before proceeding to the next box, Figure 4 Each box can be executed one or more times. Figure 4 describes the mapping of the coordinate systems of the two devices, but Figure 4 The process can be extended to map the coordinate systems of any number of devices, for example, by having each additional device perform the process of 404-432. For example, each additional device can use the first device as a reference object, or any other device having a coordinate system that has been previously mapped with the coordinate system of the first device.

[0073] Figure 5 5 shows an example of components of a computing system for executing an augmented reality application according to at least one aspect of the present disclosure. Figure 4 Although these components are shown as part of computing system 504, computing system 504 may also be distributed such that some components may be located on a different hardware platform than other components.

[0074] The computing system 504 includes at least a processor 508, a memory 512, a storage device 516, an input / output peripheral (I / O) 520, a communication peripheral 524, one or more cameras 528, and an interface bus 532. The interface bus 532 may be used to communicate, send, and transfer data, control, and commands between the various components of the computing system 504. The memory 512 and the storage device 516 may include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), a hard drive, a CD-ROM, an optical storage device, a magnetic storage device, an electronic non-volatile computer storage, such as a memory device. Memory, and other tangible storage media. Any such computer-readable storage media can be used to store instructions or program codes that implement various aspects of the present disclosure. Memory 512 and storage device 516 can also include computer-readable signal media. Computer-readable signal media include propagated data signals containing computer-readable program code. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media include any computer-readable media that is not a computer-readable storage medium and can communicate, propagate, or transmit a program for use in conjunction with the computing system 504.

[0075] In addition, the memory 512 may include an operating system, programs, and applications. The processor 508 may be used to execute stored instructions and include, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. The memory 512 and / or the processor 508 may be virtualized and may be hosted in another computing system such as a cloud network or a data center. The I / O peripherals 520 may include a user interface such as a keyboard, a screen (e.g., a touch screen), a microphone, a speaker, other input / output devices, and computing components such as a graphics processing unit, a serial port, a parallel port, a universal serial bus, and other input / output peripherals. The I / O peripherals 520 are connected to the processor 508 by any port coupled to the interface bus 532. The communication peripherals 524 may be used to facilitate communication between the computing system 504 and other computing devices through a communication network, and include, for example, a network interface controller, a modem, a wireless and wired interface card, an antenna, and other communication peripherals.

[0076] Although the subject matter has been described in detail with respect to its specific embodiments, it should be understood that those skilled in the art, after obtaining an understanding of the foregoing, can easily generate changes, variations, and equivalents to these embodiments. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration rather than limitation, and does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter that are obvious to a person of ordinary skill in the art. In fact, the methods and systems described herein can be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes can be made to the form of the methods and systems described herein without departing from the spirit of the present disclosure. The attached claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of the present disclosure.

[0077] Unless expressly stated otherwise, it should be understood that terms such as "process," "compute," "calculate," "determine," and "identify" are used throughout the discussion of this specification to refer to the actions or processes of a computing device (e.g., one or more computers or similar electronic computing devices) that manipulates or transforms data represented as physical electronic or magnetic quantities in a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0078] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0079] Embodiments of the methods disclosed herein may be performed in the operation of such a computing device. The order of the blocks presented in the above examples may be changed—for example, the blocks may be reordered, combined, and / or decomposed into sub-blocks. Certain blocks or processes may be performed in parallel.

[0080] Conditional language used herein, such as "may," "might," "for example," and the like, unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that certain examples include and other examples do not include certain features, elements, and / or steps. Thus, such conditional language generally does not imply that one or more examples require the features, elements, and / or steps in any way, or that one or more examples must include logic for deciding, with or without author input or prompting, whether such features, elements, and / or steps are included or will be performed in any particular example.

[0081] The terms "including", "having", "having", etc. are synonymous and are used inclusively in an open manner and do not exclude other elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusiveness (rather than exclusiveness), so that when used to connect a list of elements, for example, the term "or" represents one, some or all of the elements in the list. "Applicable to" or "for" used herein refers to open and inclusive language, and does not exclude devices that are applicable to or used to perform additional tasks or steps. In addition, the use of "based on" means openness and inclusiveness, because the process, step, calculation or other action "based on" one or more listed conditions or values ​​may actually be based on additional conditions or values ​​outside the list. Similarly, the use of "based at least in part" means openness and inclusiveness, because the process, step, calculation or other action "based at least in part on" one or more listed conditions or values ​​can be based on additional conditions or values ​​outside the list in practice. The titles, lists and numbers included herein are only for ease of explanation and are not meant to be limiting.

[0082] The various features and processes described above can be used independently of each other, or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, some method or process blocks can be omitted in some embodiments. The methods and processes described herein are also not limited to any particular order, and the blocks or states related thereto can be performed in other appropriate orders. For example, the described blocks or states can be performed in a different order than that specifically disclosed, or multiple blocks or states can be combined in a single block or state. The example blocks or states can be performed serially, in parallel, or in some other way. Blocks or states can be added to or deleted from the disclosed examples. Similarly, the example systems and components described herein can be configured to be different from those described. For example, compared to the disclosed examples, elements can be added, removed, or rearranged.

[0083] Although the principles of the present disclosure have been described above in conjunction with specific devices and methods, it should be clearly understood that this description is only by way of example and not limiting the scope of the present disclosure.

Claims

1. A method for associating device coordinate systems in a multi-person augmented reality system, characterized in that: The method comprises: receiving, by a second mobile device, data indicating a first posture of the first mobile device, wherein the first posture is defined by a first coordinate system relative to the first mobile device, wherein the first posture comprises a first timestamp corresponding to a time at which the first posture is determined; determining, by the second mobile device, a second posture of the second mobile device, wherein the second posture is defined by a second coordinate system relative to the second mobile device; The second mobile device receives an image of a display screen of the first mobile device, the image showing a mark; The second mobile device identifies three-dimensional coordinates associated with the marker based on the first gesture and the image, the three-dimensional coordinates being defined by the first coordinate system; determining a third posture of the second mobile device based on the three-dimensional coordinates, wherein the third posture is defined by the first coordinate system; and The second mobile device generates a coordinate system transformation based on the second gesture and the third gesture, the coordinate system transformation mapping coordinates in the first coordinate system to corresponding coordinates in the second coordinate system.

2. The method according to claim 1, characterized in that Determining the third posture of the second mobile device includes: identifying a set of feature points from the marker; and A correspondence between the set of feature points and the three-dimensional coordinates associated with the marker is generated.

3. The method according to claim 2, characterized in that The set of feature points includes three or more feature points.

4. The method according to claim 1, characterized in that: The first pose is represented by a rotation vector and a translation vector, and wherein the three-dimensional coordinates associated with the marker are defined at least in part using the first pose and a geometry of the first mobile device.

5. The method according to claim 1, characterized in that The first virtual object presented by the second mobile device corresponds to the second virtual object presented by the first mobile device, and wherein the first virtual object appears in the environment at substantially the same position and substantially the same orientation as the second virtual object.

6. The method according to claim 1, characterized in that The coordinate system transformation includes a transformation matrix defined according to a rotation and a translation between the second pose and the third pose.

7. The method according to claim 1, characterized in that The method further comprises: The second mobile device presents a first virtual object on a display screen of the second mobile device based on the coordinate system transformation.

8. A mobile device for associating a device coordinate system in a multi-person augmented reality system, characterized in that: The mobile device comprises: one or more processors; and A non-transitory computer-readable storage medium storing a program, which, when executed by the one or more processors, causes the one or more processors to perform operations including: receiving data indicating a first posture of another mobile device, wherein the first posture is defined by a first coordinate system relative to the other mobile device, wherein the first posture includes a first timestamp corresponding to a time when the first posture is determined; determining a second pose of the mobile device, wherein the second pose is defined by a second coordinate system relative to the mobile device; receiving an image of a display screen of the other mobile device, the image showing a marker; identifying three-dimensional coordinates associated with the marker based on the first posture and the image, the three-dimensional coordinates being defined by the first coordinate system; determining a third posture of the mobile device based on the three-dimensional coordinates, wherein the third posture is defined by the first coordinate system; and A coordinate system transformation is generated based on the second gesture and the third gesture, wherein the coordinate system transformation maps coordinates in the first coordinate system to corresponding coordinates in the second coordinate system.

9. The mobile device according to claim 8, characterized in that Determining the third posture of the mobile device includes: identifying a set of feature points from the marker; and A correspondence between the set of feature points and the three-dimensional coordinates associated with the marker is generated.

10. The mobile device according to claim 9, characterized in that The set of feature points includes three or more feature points.

11. The mobile device according to claim 8, characterized in that The first pose is represented by a rotation vector and a translation vector, and wherein the three-dimensional coordinates are defined at least in part using the first pose and a geometry of the other mobile device.

12. The mobile device according to claim 8, characterized in that The operations also include: A first virtual object is presented on the display screen, wherein the first virtual object corresponds to a second virtual object presented by the other mobile device, and wherein the first virtual object appears in the environment at substantially the same position and substantially the same orientation as the second virtual object.

13. The mobile device according to claim 8, characterized in that The coordinate system transformation includes a transformation matrix defined according to a rotation and a translation between the second pose and the third pose.

14. A non-transitory computer-readable medium, characterized in that The system stores instructions for associating device coordinate systems in a multi-person augmented reality system. When executed by one or more processors, the instructions cause the one or more processors to perform operations including: receiving data indicating a first posture of a first mobile device, wherein the first posture is defined by a first coordinate system relative to the first mobile device, wherein the first posture includes a first timestamp corresponding to a time when the first posture is determined; determining a second pose of a second mobile device, wherein the second pose is defined by a second coordinate system relative to the second mobile device; receiving an image of a display screen of the first mobile device, the image showing a marker; identifying three-dimensional coordinates associated with the marker based on the first posture and the image, the three-dimensional coordinates being defined by the first coordinate system; determining a third posture of the second mobile device based on the three-dimensional coordinates, wherein the third posture is defined by the first coordinate system; and A coordinate system transformation is generated based on the second gesture and the third gesture, wherein the coordinate system transformation maps coordinates in the first coordinate system to corresponding coordinates in the second coordinate system.

15. The non-transitory computer readable medium of claim 14, wherein: Determining the third posture of the second mobile device includes: identifying a set of feature points from the marker; and A correspondence between the set of feature points and the three-dimensional coordinates associated with the marker is generated.

16. The non-transitory computer readable medium of claim 14, wherein: The first posture is represented by a rotation vector and a translation vector, the rotation vector and the translation vector indicating a position of the first mobile device within the first coordinate system and an orientation of the first mobile device.

17. The non-transitory computer readable medium of claim 14, wherein: The three-dimensional coordinates are defined at least in part using the first pose and a geometry of the first mobile device.

18. The non-transitory computer readable medium of claim 14, wherein: The operations also include: A first virtual object is presented on a display screen of the second mobile device, wherein the first virtual object corresponds to a second virtual object presented by the first mobile device, and wherein the first virtual object appears in an environment at substantially the same position and substantially the same orientation as the second virtual object.