Relative spatial positioning of mobile devices

Abstract

In order to obtain relative positioning between multiple mobile devices, a first mobile device observes a second mobile device within the field of view of a camera of the first mobile device at time t1, determines a first position of the first mobile device at t1, and receives a second position of the second mobile device at t1 from the second mobile device. The first mobile device determines information about the orientation of the first mobile device relative to the second mobile device at t1 based at least in part on the first position and the observation of the second mobile device. The first mobile device identifies two constraints that relate the coordinate systems of the mobile devices based at least in part on the second position and the orientation information. Once at least six constraints are accumulated, the pose of the first mobile device relative to the second mobile device can be calculated.

Description

Background Art

[0001] Mixed reality, which may also be referred to as augmented reality, involves the merging of real-world objects and / or people with virtual objects to produce new environments and visualizations in which physical and digital objects coexist and interact in real time. Mixed reality devices enhance a user's view of the real world with virtual objects that are intended to appear as if they were actually placed in the real world. Mixed reality devices may allow a user to view their real-world surroundings through a translucent display. Virtual objects may then be rendered on the display. These virtual objects appear to be superimposed on the user's view of the real-world environment around them, thereby blending virtual reality with physical reality.

[0002] Mixed reality experiences can be shared among multiple mixed reality devices. This enables multiple users to have a common shared experience in a shared mixed reality environment. Shared mixed reality experiences can be useful and / or enjoyable in many different scenarios, such as gaming, where players can interact with each other and with virtual objects that are part of the game.

[0003] To facilitate a shared mixed reality experience, it is important that multiple devices can calculate their position and motion in the same coordinate system so that they can know each other's relative position. Most mixed reality devices are able to determine their own relative motion, but not necessarily their relative pose (i.e., position and orientation) with respect to other devices.

[0004] The common solution for aligning coordinate systems for multiple devices is to exchange 3D map information (or 2D image data) between devices so that the relative poses between these maps / images can be determined. These solutions can be implemented point-to-point or on a cloud service.

[0005] However, the exchange of map data can be cumbersome and presents privacy risks. Furthermore, traditional image feature matching can fail when the users' viewpoints are significantly different from each other, resulting in the devices being unable to determine a joint coordinate system and thus unable to share a mixed reality experience. For example, this can occur in a scenario where the users are looking at each other and the cameras are therefore viewing opposite sides of the same space, which is often a natural configuration for games and other types of shared mixed reality experiences. Summary of the invention

[0006] According to one aspect of the present disclosure, a method for obtaining relative positioning between multiple mobile devices is disclosed. The method can be performed by a first mobile device. The method includes observing a second mobile device within the field of view of a camera of the first mobile device at a first time point. The method also includes determining a first position of the first mobile device at the first time point. The first position is represented in a first coordinate system used by the first mobile device. The method also includes receiving a second position of the second mobile device at the first time point from the second mobile device. The second position is represented in a second coordinate system used by the second mobile device. The method also includes: determining orientation information about the orientation of the first mobile device relative to the second mobile device at the first time point based at least in part on the first position and the observation of the second mobile device at the first time point. The method also includes: identifying two constraints that relate the first coordinate system used by the first mobile device to the second coordinate system used by the second mobile device based at least in part on the second position and the orientation information. The method also includes: once at least six constraints have been accumulated, the pose of the first mobile device relative to the second mobile device is calculated.

[0007] In some embodiments, the method may additionally include: receiving user input when observing a second mobile device within a field of view of a camera of the first mobile device at a first point in time. In response to receiving the user input, a first position may be determined, a second position may be received from the second mobile device, orientation information may be determined, and two constraints may be identified. The method may be performed during a game involving multiple mobile devices, and the user input may be provided as part of the game.

[0008] In some embodiments, the method may further include: automatically detecting a second mobile device within a field of view of a camera of the first mobile device at a first point in time. In response to automatically detecting the second mobile device, a first position may be determined, a second position may be received from the second mobile device, orientation information may be determined, and two constraints may be identified. Automatically detecting the second mobile device may include automatically detecting an activated light emitter of the second mobile device.

[0009] In some embodiments, at the first point in time, both the second mobile device and the at least one additional mobile device may be visible within a field of view of a camera of the first mobile device.The method may also include distinguishing between the second mobile device and the at least one additional mobile device.

[0010] In some embodiments, determining the orientation information may include determining a geometric line in space corresponding to a direction in which the second mobile device was observed at the first point in time.

[0011] In some embodiments, calculating the pose can include calculating a six degree of freedom transform that relates a first coordinate system used by the first mobile device to a second coordinate system used by the second mobile device.

[0012] In some embodiments, the at least six constraints may include a plurality of constraints determined as a result of observing the event, and one or more additional constraints known independently of the observing the event.

[0013] In some embodiments, the method may additionally include: creating a first simultaneous localization and mapping (SLAM) map based on the field of view of the camera of the first mobile device. The first SLAM map may be merged with a second SLAM map created by the second mobile device.

[0014] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0015] Additional features and advantages will be set forth in the following description. The features and advantages of the present disclosure can be realized and obtained by the systems and methods particularly pointed out in the appended claims. The features of the present disclosure will become more apparent from the following description and the appended claims, or can be learned by the practice of the disclosed subject matter as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to describe the manner in which the above and other features of the present disclosure can be obtained, a more specific description will be given by referring to specific embodiments of the present disclosure illustrated in the accompanying drawings. For better understanding, the same elements are designated by the same reference numerals in the various drawings. It should be understood that the drawings depict some example embodiments, and the embodiments will be described and explained with additional specificity and detail through the use of the drawings, in which:

[0017] Figures 1A to 1C An example of a method for obtaining relative positioning between multiple mobile devices is illustrated.

[0018] Figure 2 The diagram illustrates the field of view of a mobile device at a point in time when multiple other mobile devices are visible within the field of view.

[0019] Figure 3 An example is illustrated in which the coordinate systems of two mobile devices are aligned with each other and are considered a single mobile device for the purposes of the positioning techniques disclosed herein.

[0020] Figure 4An example is illustrated in which simultaneous localization and mapping (SLAM) maps of two mobile devices are merged after the coordinate systems of the mobile devices are aligned with each other.

[0021] Figure 5 A method for obtaining relative positioning between multiple mobile devices according to the present disclosure is illustrated.

[0022] Figure 6 Illustrated are certain components that may be included within a mobile device configured to implement the techniques disclosed herein. DETAILED DESCRIPTION

[0023] The present disclosure generally relates to obtaining relative positioning between multiple mobile devices, each of which includes a camera and is capable of tracking its own position and motion in space using its own coordinate system. The techniques disclosed herein are applicable to any scenario in which it is desired that multiple mobile devices calculate their positions and motions in the same coordinate system so that they can know the relative positions of each other. As an example, the techniques disclosed herein can be used in the context of a shared mixed reality experience.

[0024] As used herein, the term "mobile device" refers to a portable computing device that includes a camera and is capable of implementing the spatial positioning technology disclosed herein. In some embodiments, the mobile device may be small enough for a user to hold and operate the mobile device in the user's hand. In some embodiments, the mobile device may be a wearable computing device. In some embodiments, the mobile device may be a mixed reality (or augmented reality) device that can provide a mixed reality (or augmented reality) experience to the user. Some examples of mobile devices include head-mounted displays, smart glasses, smart phones, tablet computers, and laptop computers. The mobile device may be able to connect to one or more computer networks, such as the Internet. The mobile device may also be able to establish peer-to-peer communications with other computing devices.

[0025] The technology disclosed herein utilizes direct observation of another mobile device within the field of view of a camera of a mobile device. Consider a simple example involving two mobile devices, a first mobile device and a second mobile device. Assume that the first mobile device observes the second mobile device within the field of view of the camera of the first mobile device. When this occurs, the first mobile device can use its own position and its observation of the second mobile device to constrain the orientation of the first mobile device relative to the second mobile device. In other words, the first mobile device can determine information about the orientation of the first mobile device relative to the second mobile device. This orientation information, together with the position of the second mobile device (as represented in the coordinate system used by the second mobile device), can be used to identify two constraints for relating the coordinate system used by the first mobile device to the coordinate system used by the second mobile device.

[0026] Once at least six constraints have been accumulated, the pose of the first mobile device relative to the second mobile device (and the pose of the second mobile device relative to the first mobile device) can be calculated. More specifically, the six (or more) constraints can be used to calculate a six-degree-of-freedom (6DoF) transformation that relates a first coordinate system used by the first mobile device to a second coordinate system used by the second mobile device.

[0027] As used herein, the term "observation event" refers to a situation in which one mobile device observes another mobile device within the field of view of its camera, and the corresponding position information and orientation information are used to determine two constraints that relate the coordinate systems of the two mobile devices. As will be discussed in more detail below, in some embodiments, an observation event can be triggered by user input. Alternatively, an observation event can be triggered when one mobile device automatically detects another mobile device in the field of view of its camera.

[0028] The disclosed techniques for obtaining relative positioning between multiple mobile devices may be particularly advantageous in situations where it is not possible or desirable for multiple mobile devices to share a three-dimensional (3D) environment map with each other. For example, this may occur when the viewpoints of the users are significantly different from each other.

[0029] In some embodiments, the disclosed technology can reduce processing requirements relative to known methods. As described above, the usual solution for aligning coordinate systems for multiple mobile devices is to exchange three-dimensional (3D) map information between mobile devices. However, the technology disclosed herein enables relative positioning between multiple mobile devices without exchanging 3D map information. This can reduce the amount of processing required to align the coordinate systems of multiple mobile devices. For example, mobile devices that align their coordinate systems according to the technology disclosed herein do not have to determine or exchange 3D map information. Therefore, by utilizing the technology disclosed herein, the amount of processing required to determine and / or exchange 3D map information can be saved.

[0030] The ability to obtain relative positioning between multiple mobile devices without exchanging 3D map information can also reduce the amount of map information that needs to be stored, thereby reducing the storage requirements of the mobile devices. In addition to reducing storage requirements, the technology disclosed herein can also reduce the amount of information communicated between multiple mobile devices (via a computer network or via peer-to-peer communication). Instead of exchanging 3D map information (which can be quite data-intensive), the mobile devices can simply exchange some location information and some orientation information associated with a particular point in time (as will be discussed in more detail below). This potentially reduces the amount of information communicated between multiple mobile devices, thereby potentially freeing up a large amount of communication bandwidth for other purposes.

[0031] However, despite the foregoing, in some cases, three-dimensional map information may still be exchanged according to the techniques disclosed herein. As will be discussed in more detail below, in some embodiments, the 3D maps being constructed by each mobile device may be merged into a larger, more complete map.

[0032] About Figures 1A to 1C An example of a method for obtaining relative positioning between multiple mobile devices is described. The example involves two mobile devices 102a, 102b, a first mobile device 102a and a second mobile device 102b. The coordinate system used by the first mobile device 102a will be referred to as the first coordinate system in this article, and the coordinate system used by the second mobile device 102b will be referred to as the second coordinate system in this article. The user of the first mobile device 102a will be referred to as the first user 104a, and the user of the second mobile device 102b will be referred to as the second user 104b.

[0033] Figure 1A It is illustrated that a first user 104a is aiming a first mobile device 102a at a second user 104b, and a second user 104b is aiming a second mobile device 102b at the first user 104a. For example, this may occur during a game in which users are supposed to shoot at each other using mobile devices. Figure 1B The diagram shows that when the first mobile device 102a is aimed at the second mobile device 102b (eg Figure 1A ), the field of view of the camera of the first mobile device 102a, as well as the second mobile device 102b can be displayed to the first user 104a on the display 106 of the first mobile device 102a. Figure 1C Illustrated are a trajectory 108a of a first mobile device 102a and a trajectory 108b of a second mobile device 102b as the first user 104a and the second user 104b move around over a period of time (eg, move around during a game).

[0034] like Figure 1C As shown in , at time t1, the first user 104a positions the first mobile device 102a so that the second mobile device 102b is within the field of view of the camera of the first mobile device 102a (and is therefore visible on the display 106). For example, in the context of a shooting game, the first user 104a may aim the first mobile device 102a at the second user 104b, who may be holding or wearing the second mobile device 102b and may aim the second mobile device 102b at the first user 104a. Figure 1BAs shown in , a crosshair 110 can be displayed on the display 106 of the first mobile device 102a. The crosshair 110 can help the first user 104a position the first mobile device 102a so that the second mobile device 102b is approximately located within the center of the field of view of the camera of the first mobile device 102a. The first user 104a then provides some input (e.g., clicks a button on the first mobile device 102a) so that the first mobile device 102a and the second mobile device 102b remember (e.g., store in memory) and communicate certain information associated with that specific point in time.

[0035] In particular, when the second mobile device 102b is observed within the field of view of the camera of the first mobile device 102a, the first mobile device 102a determines and remembers the position of the first mobile device 102a at time t1. This position expressed in the first coordinate system used by the first mobile device 102a will be referred to herein as p1 t1 (As used herein, the term px ty refers to the location of device x at time y.) The first mobile device 102a also sets p1 t1 Communicate to the second mobile device 102b. Assume that the first mobile device 102a and the second mobile device 102b are substantially time synchronized.

[0036] The second mobile device 102b also determines and remembers its position at time t1. This position, expressed in the second coordinate system used by the second mobile device 102b, will be referred to herein as p2 t1 The second mobile device 102b also sends p2 t1 Communicates to the first mobile device 102a.

[0037] The first mobile device 102a also determines and remembers information about its orientation relative to the second mobile device 102b at time t1. For example, the first mobile device 102a can determine and remember a geometric line in space that corresponds to the direction along which the second mobile device is viewed at the first time t1. This line will be referred to herein as line t1 If the second mobile device 102b is observed at the center of the field of view of the camera of the first mobile device 102a at time t1, this line in space will correspond to the optical axis of the camera.

[0038] Then, the position information (p1 t1 and p2 t1 ) and directional information (line t1 ) can be used to identify two constraints relating a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b. In particular, the position (p2 t1) should be along the line in the previously defined space (line t1 ) The geometric constraints of positioning provide two mathematical constraints for aligning the coordinate systems of the two mobile devices 102a, 102b.

[0039] This process of observing another mobile device and determining two constraints using the position and orientation information associated with the observation can then be repeated at least twice until at least six constraints are determined. However, it is not always necessary to make each observation with the same mobile device (although it can be). For example, in the depicted example, at time t2, the second mobile device 102b observes the first mobile device 102a within the field of view of the camera of the second mobile device 102b.

[0040] More specifically, at time t2, the second user 104b positions the second mobile device 102b so that the first mobile device 102a is within the field of view of the camera of the second mobile device 102b. The second user 104b then provides some input so that the second mobile device 102b and the first mobile device 102a remember and communicate certain information associated with time t2. In particular, the second mobile device 102b determines and remembers its location (p2 t2 ), which is represented in a second coordinate system used by the second mobile device 102b. The second mobile device 102b also sends p2 to the first mobile device 102a. t2 In addition, the first mobile device 102a determines and remembers its location (p1 t2 ), which is represented in a first coordinate system used by the first mobile device 102a. The first mobile device 102a also sends p1 to the second mobile device 102b. t2 The second mobile device 102b also determines and remembers information about its orientation relative to the first mobile device 102a at time t2. More precisely, the second mobile device 102b remembers a geometric line in space that corresponds to the direction along which the first mobile device 102a is viewed at time t2. This line will be referred to herein as line t2 .

[0041] This position and orientation information can then be used to identify two additional constraints relating a second coordinate system used by the second mobile device 102b to a first coordinate system used by the first mobile device 102a. Specifically, the position (p1) of the first mobile device 102a at time t2 t2 ) need to follow line t2 The geometric constraints of positioning provide two additional mathematical constraints for aligning the coordinate systems of the two mobile devices 102a, 102b.

[0042] Subsequently, at time t3, the first user 104a positions the first mobile device 102a so that the second mobile device 102b is within the field of view of the camera of the first mobile device 102a. The first user 104a then provides some input so that the first mobile device 102a and the second mobile device 102b remember and communicate certain information associated with time t3. In particular, the first mobile device 102a determines and remembers its position (p1) at time t3. t3 ), which is represented in a first coordinate system used by the first mobile device 102a. The first mobile device 102a also sends p1 to the second mobile device 102b. t3 The second mobile device 102b determines and remembers its location at time t3 (p2 t3 ), which is represented in a second coordinate system used by the second mobile device 102b. The second mobile device 102b also sends p2 to the first mobile device 102a. t3 The first mobile device 102a also determines and remembers information about its orientation relative to the second mobile device 102b at time t3. More precisely, the first mobile device 102a remembers a geometric line in space that corresponds to the direction along which the second mobile device 102b is viewed at time t3. This line will be referred to herein as line t3 .

[0043] This position and orientation information can then be used to identify two additional constraints relating a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b. In particular, the position (p2 t3 ) need to follow line t3 The geometric constraints of positioning provide two additional mathematical constraints for aligning the coordinate systems of the two mobile devices 102a, 102b.

[0044] In this example, six constraints have been identified after time t3. This is a sufficient number of constraints for enabling the calculation of a six-degree-of-freedom transformation, which relates the first coordinate system used by the first mobile device 102a to the second coordinate system used by the second mobile device 102b. As known to those skilled in the art, the term "six degrees of freedom" (6DoF) refers to the freedom of movement of a rigid body in three-dimensional space. In particular, a rigid body can freely change position on three vertical axes, such as forward / backward (surge), up / down (heave), left / right (sway) translation. This translation can be combined with a change in orientation by rotating around three vertical axes, which are generally referred to as the yaw axis (normal axis), the pitch axis (lateral axis) and the roll axis (longitudinal axis). In the context of the present disclosure, a "six-degree-of-freedom transformation" (6DoF transformation) specifies a mathematical operation that should be performed to convert the position and orientation information represented in one coordinate system into equivalent position and orientation information represented in another coordinate system.

[0045] In the example just described, six constraints are obtained from three observation events. Two observation events occur at the first mobile device 102a (at times t1 and t3), and one observation event occurs at the second mobile device 102b (at time t2). The specific details of this particular example should not be construed as limiting the scope of this disclosure. For example, the observation events do not have to occur in any particular order. In an alternative example, the first mobile device 102a may have two consecutive observation events (e.g., at t1 and t2) before the second mobile device 102b has an observation event (e.g., at time t3). In addition, both mobile devices 102a and 102b do not have to have observation events. In another alternative example, all three observation events may occur at the same mobile device.

[0046] In addition, more than three observation events may occur. As described above, a minimum of six constraints (from a minimum of three observation events) are required in order to compute a 6DoF transformation relating a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b. With minimal data (e.g., data from only three observation events), more than one solution associated with computing a 6DoF transformation may be obtained. However, with a set of redundant measurements, the solution generally becomes unique.

[0047] In some embodiments, both of the mobile devices 102a, 102b can calculate a 6DoF transformation. In the specific example just described, four constraints are determined based on observations made by the first mobile device 102a (at times t1 and t3), and two constraints are determined based on observations made by the second mobile device 102b (at time t2). The mobile devices 102a, 102b can share information about these constraints with each other so that both mobile devices 102a, 102b have enough information to calculate the 6DoF transformation. Alternatively, one of the mobile devices (e.g., the first mobile device 102a) can calculate the 6DoF transformation and share the 6DoF transformation with another mobile device (e.g., the second mobile device 102b). Alternatively again, another entity different from the mobile devices 102a-102b can calculate the 6DoF transformation and share the 6DoF transformation with the mobile devices 102a-102b. The other entity can be a server that communicates electronically with the mobile devices 102a-102b via one or more computer networks. In some embodiments, the other entity can be a cloud-based entity.

[0048] For the purpose of computing the 6DoF transform, constraints determined as a result of an observation event can be combined with additional constraints that are known to be independent of the observation event. Figure 1A-Figure 1C In the example shown in , the six constraints determined as a result of three observation events (at times t1, t2, and t3) can be combined with additional constraints such as a known common vertical orientation (two constraints), height above the ground (one constraint), major wall orientation (one constraint), etc. The common vertical orientation can be determined from inertial measurements or vertical vertical vanishing points / lines. The height above the ground can be determined when observing the ground plane. The orientation relative to the room can be determined when observing the walls. Redundant sets of constraints can be used to obtain more accurate and / or robust results.

[0049] exist Figure 1A-Figure 1C In the example shown, the user input triggers an observation event. In other words, the user input causes the mobile devices 102a, 102b to remember and communicate certain information associated with a particular point in time and use that information to determine constraints for the coordinate systems that relate the mobile devices 102a, 102b. The user input can be provided when the user sees another mobile device in the field of view of the camera of the user's mobile device (e.g., as seen on the display 106).

[0050] In some embodiments, user input triggering observation events can be integrated into the user experience as part of the gameplay. This can be applicable to some mixed reality experiences, such as games where users are supposed to shoot each other. In this type of scenario, when a user shoots, it can generally be assumed that he or she is aiming at the other user. After the players shoot each other several times, the disclosed technology can be used to determine the relative pose between the coordinate systems of the two mobile devices. This information can then be used to enrich the user's experience.

[0051] Integrating observation events into the user experience as part of the gameplay can provide certain benefits over at least some known methods. As described above, a common solution for aligning coordinate systems for multiple devices is to exchange three-dimensional map information (or two-dimensional image data) between the devices. Using at least some known methods, user input may be required to exchange such information / data. In addition, the required user input can be in addition to any user input that would otherwise occur naturally as part of the user experience (e.g., not just shooting as part of the gameplay). As disclosed herein, integrating observation events into the user experience as part of the gameplay eliminates the need for such additional user input, making the overall experience for the user more natural and more enjoyable.

[0052] However, despite the foregoing, user input need not be integrated into the user experience. In fact, it is not necessary to rely on user input to trigger an observation event. Alternatively, instead of relying on such user input, the mobile device can be configured to automatically detect the presence of another mobile device in the field of view of its camera, and in response to detecting another mobile device in this manner, automatically remember and communicate with the location and orientation information. In other words, in some embodiments, an observation event can be triggered when a mobile device automatically detects another mobile device in the field of view of its camera.

[0053] The mobile device may use an object recognition algorithm to detect the presence of another mobile device within the field of view of its camera. This process may be simplified when the detected mobile device is equipped with a light emitter that can be observed by the observing mobile device. For example, referring again to Figure 1A-Figure 1CIn the example shown in , it is assumed that the second mobile device 102b is equipped with a light emitter and the light emitter is activated at time t1. Instead of relying on input from the first user 104a to trigger an observation event, the first mobile device 102a can instead automatically detect the second mobile device 102b within the field of view of its camera. The activated light emitter on the second mobile device 102b can help facilitate this automatic detection. Some examples of light emitters that can be used in conjunction with the techniques disclosed herein include flashlights on mobile phones, privacy lights associated with cameras, and infrared emitters used by some three-dimensional cameras on some head-mounted displays (e.g., in mixed reality headsets). Infrared emitters can be observed by infrared / depth cameras of similarly equipped devices.

[0054] In some embodiments, the identity of a particular mobile device may be transmitted via a unique pattern transmitted by a light transmitter on the mobile device. In other words, the mobile device may turn its light transmitter on and off according to a specific pattern, and the pattern used by the particular mobile device may be unique (at least for a set of mobile devices used simultaneously in the same location). Another mobile device that is observing the pattern may be able to determine the identity of the light emitting mobile device based on the unique pattern transmitted.

[0055] although Figures 1A to 1C The example shown in includes two mobile devices 102a, 102b, but in some cases, it may be necessary to determine the relative pose between more than two mobile devices. The technology disclosed herein can also be extended to more than two mobile devices, as long as each mobile device has sufficient constraints relative to other mobile devices to allow relative positioning.

[0056] When more than two mobile devices are involved, there are additional difficulties because the field of view of a particular mobile device may include more than one mobile device. When a mobile device attempts to determine its pose relative to a particular mobile device and there are multiple mobile devices visible within the field of view of its camera, it may be difficult to determine which of the multiple mobile devices is the particular mobile device that is the subject of the pose calculation.

[0057] Figure 2 The field of view of a mobile device is illustrated, as seen, for example, on a display 206. A plurality of other mobile devices 202a, 202b are visible in the field of view. Figure 2 The mobile device shown in FIG. 2 will be referred to as an observing mobile device, and the mobile devices 202a - 202b visible within the field of view of the observing mobile device will be referred to as a first observed mobile device 202a and a second observed mobile device 202b.

[0058] Assume that an observing mobile device is trying to calculate its pose relative to a first observed mobile device 202a. It may be difficult for the observing mobile device to determine which of the mobile devices 202a, 202b visible in the field of view of its camera is the first observed mobile device 202a. In other words, it may be difficult for the observing mobile device to distinguish the first observed mobile device 202a from the second observed mobile device 202b. Therefore, when an observation event is triggered (e.g., in response to user input, or alternatively, when the observing mobile device automatically detects the mobile devices 202a, 202b), it may be difficult for the observing mobile device to decide whether it should determine and remember information about its orientation relative to the first observed mobile device 202a, or whether it should determine and remember information about its orientation relative to the second observed mobile device 202b.

[0059] This problem can be solved in several different ways. In some embodiments, a robust algorithm such as Random Sample Consensus (RANSAC) can be used. RANSAC is an outlier detection method, and the correspondence with other mobile devices can be considered as outliers. This can be particularly effective in the case where a relatively small number of mobile devices are visible in the camera's field of view at any given time.

[0060] As another example, in some embodiments, an image recognition algorithm may be used to associate observation events corresponding to the same device. Figure 2 In the example shown in FIG. 1 , it is assumed that the observing mobile device is at a previous point in time (i.e., before the two mobile devices 202a-202b are observed simultaneously, as shown in FIG. 1 ). Figure 2 In response to performing the observation, the observing mobile device may store an image of the first observed mobile device 202a. Then, when the observing mobile device simultaneously observes the first observed mobile device 202a and the second observed mobile device 202b (as shown in FIG. Figure 2 ), previously stored images may be used to detect a first observed mobile device 202a and distinguish it from a second observed mobile device 202b.

[0061] For the purposes of the techniques disclosed herein for obtaining relative positioning between multiple mobile devices, once the coordinate systems of two mobile devices are aligned relative to each other, they can be treated as a single mobile device. This in turn will facilitate aligning one or more additional mobile devices with the two previously aligned mobile devices.

[0062] An example of this feature is Figure 3. Assume that the first mobile device 302a and the second mobile device 302b align their coordinate systems with each other according to the technology disclosed herein. Then, the first mobile device 302a and the second mobile device 302b can be regarded as a "single" mobile device 312 (for the purpose of aligning their shared coordinate system with another mobile device). Therefore, subsequently, the third mobile device 302c can align itself with the coordinate system used by the "single" mobile device 312 (i.e., used by both the first mobile device 302a and the second mobile device 302b).

[0063] As described above, the coordinate systems of the two mobile devices can be aligned with each other after a minimum of three observation events (each observation event generates two of the six constraints required to calculate the 6DoF transformation). However, in this example, these observation events can occur as a result of an interaction between the third mobile device 302c and the first mobile device 302a and / or as a result of an interaction between the third mobile device 302c and the second mobile device 302b.

[0064] Figure 3 The diagram shows the trajectory 308a of the first mobile device 302a, the trajectory 308b of the second mobile device 302b, and the trajectory 308c of the third mobile device 302c when the users of these devices move around over a period of time. Assume that the first observation event occurs at time t1, wherein the third mobile device 302c observes the first mobile device 302a, and both devices 302a and 302c determine and remember the corresponding position and orientation information to determine the two constraints for calculating the relative posture between these devices 302a and 302c (in the above manner). Then, the second observation event occurs at time t2, wherein the second mobile device 302b observes the third mobile device 302c, and both devices 302b and 302c determine and remember the corresponding position and orientation information for determining two additional constraints. Then, the third observation event occurs at time t3, wherein the first mobile device 302a observes the third mobile device 302c, and both devices 302a and 302c determine and remember the corresponding position and orientation information for determining two additional constraints. This will create sufficient constraints to compute the 6DoF transform for aligning the respective coordinate systems, since the first mobile device 302a and the second mobile device 302b are treated as a "single" mobile device 312 for the purpose of aligning their shared coordinate systems with the third mobile device 302c.

[0065] As mentioned above, the common scheme for registering coordinate systems for multiple mobile devices is to exchange three-dimensional map information between mobile devices. The three-dimensional map information is usually determined according to a synchronous positioning and mapping (SLAM) algorithm. The 3D map created using the SLAM algorithm may be referred to as a SLAM map in this article. According to the present disclosure, once the coordinate systems of more than two mobile devices are aligned as described herein, the SLAM map being constructed by each mobile device can be merged into a larger, more complete SLAM map.

[0066] Figure 4 The diagram shows a first mobile device 402a creating a first SLAM map 414a and a second mobile device 402b creating a second SLAM map 414b. If the first mobile device 402a and the second mobile device 402b align their coordinate systems with each other based on the observed events according to the techniques disclosed herein, the first SLAM map 414a and the second SLAM map 414b can be combined into a larger, more complete SLAM map. Figure 4 4. The merged SLAM map 416 is illustrated in FIG. 4 as a merged SLAM map 416. The merged SLAM map 416 may include information from the first SLAM map 414a and the second SLAM map 414b. The merged SLAM map 416 may be used by both the first mobile device 402a and the second mobile device 402b.

[0067] Being able to merge SLAM maps in this way can enable easier positioning of additional mobile devices. In addition, it can also enable persistence (cross-time shared coordinate system) based on a map of a more complete surrounding environment. In other words, the technology disclosed herein can also improve traditional sharing and persistence methods based on 3D maps. The disclosed technology can provide specific benefits related to merging SLAM maps without direct visual overlap, which will allow traditional alignment based on images. For example, it is assumed that the first SLAM map 414a and the second SLAM map 414b are generated from relative viewpoints. For example, if the first mobile device 402a and the second mobile device 402b are positioned relative to each other (for example, separated by a table from each other), so that there is no overlap between the field of view of the camera of the first mobile device 402a and the field of view of the camera of the second mobile device 402b, then this may be the case. In this example, the merged SLAM map 416 will include map information about a much larger area than the first SLAM map 414a or the second SLAM map 414b alone.

[0068] Figure 5A method 500 for obtaining relative positioning between multiple mobile devices according to the present disclosure is illustrated. For clarity, the method 500 will be described in relation to some of the previously described systems, devices, components, and data. The method 500 will be described from the perspective of a first mobile device 102a interacting with a second mobile device 102b. As previously described, the user of the first mobile device 102a will be referred to as the first user 104a, and the user of the second mobile device 102b will be referred to as the second user 104b.

[0069] In step 502, at a first point in time (t1), a first mobile device 102a observes a second mobile device 102b within the field of view of a camera of the first mobile device 102a. This may occur as a result of a first user 104a aiming the first mobile device 102a at a second user 104b. The second user 104b may be holding or wearing the second mobile device 102b. The second user 104b may aim the second mobile device 102b at the first user 104a.

[0070] In step 504, the first mobile device 102a determines the location (p1 t1 ).p1 t1 The value of is expressed in a first coordinate system used by the first mobile device 102a.

[0071] In step 506, the first mobile device 102a receives the position (p2 t1 ).p2 t1 The value of is expressed in a second coordinate system used by the second mobile device 102b.

[0072] In step 508, the first mobile device 102a determines the location of the mobile device 102a based at least in part on the first location (p1 t1 ) and observation of the second mobile device 102b at time t1 to determine information about the orientation of the first mobile device 102a relative to the second mobile device 102b at time t1. For example, the first mobile device 102a may determine and remember geometric lines in space. t1 ), geometric line t1 ) corresponds to the direction along which the second mobile device 102b is observed at time t1.

[0073] In step 510, the first mobile device 102a determines the location of the mobile device 102a based at least in part on the second location (p2 t1 ) and directional information (e.g., by line t1denoted by ), to identify two constraints relating a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b. For example, the position (p2 t1 ) should be along the previously defined space line (line t1 ) The geometric constraints of positioning provide two mathematical constraints to align a first coordinate system used by the first mobile device 102a with a second coordinate system used by the second mobile device 102b.

[0074] Steps 502 to 510 of method 500 correspond to a single observation event. In some embodiments, an observation event similar to that represented by steps 502 to 510 may be triggered by a user input. More specifically, when in step 502, the first mobile device 102a observes the second mobile device 102b within the field of view of its camera at time t1, the first user 104a may provide some type of user input (e.g., click a button on the first mobile device 102a). Steps 504 to 510 may be performed in response to receiving the user input.

[0075] In some embodiments, method 500 may be performed during a game involving a first mobile device 102a and a second mobile device 102b. User input that causes steps 504 to 510 to be performed may be provided as part of the game. For example, the game may be a shooting game in which users 104a-104b should shoot each other using mobile devices 102a-102b. Step 502 of method 500 (in which the first mobile device 102a observes the second mobile device 102b within the field of view of its camera) may occur as a result of the first user 104a aiming the first mobile device 102a at the second user 104b, who is holding or wearing the second mobile device 102b. As part of the game, when the first user 104a takes some action (e.g., activates a trigger button on the first mobile device 102a) that causes the first mobile device 102a to shoot at the second user 104b, user input that causes steps 504 to 510 to occur may be provided.

[0076] Alternatively, an observation event such as represented by steps 502 to 510 may be triggered when one mobile device automatically detects another mobile device in the field of view of its camera. More specifically, when in step 502, the first mobile device 102a observes the second mobile device 102b within the field of view of its camera at time t1, the first mobile device 102a may automatically detect the second mobile device 102b (e.g., via an object recognition algorithm). Steps 504 to 510 of method 500 may be performed in response to the first mobile device 102a automatically detecting the second mobile device 102b. In other words, the first mobile device 102a may perform steps 504 to 510 of method 500 in response to the first mobile device 102a detecting the second mobile device 102b without any additional user input. In some embodiments, the second mobile device 102b may include a light emitter. If the light emitter is activated when the first mobile device 102a observes the second mobile device 102b within the field of view of its camera at time t1, the activated light emitter may make it easier for the first mobile device 102a to automatically detect the second mobile device 102b.

[0077] In step 502, when the first mobile device 102a observes the second mobile device 102b within the field of view of its camera, there may be one or more additional mobile devices also visible within the field of view. (e.g., Figure 2 , multiple mobile devices 202a-202b may be visible in the field of view of the camera at the same time.) Therefore, in some embodiments, the method 500 may additionally involve distinguishing the second mobile device 102b from one or more additional mobile devices that are also visible in the field of view of the camera. This may be achieved in various ways, such as by using an image recognition algorithm or a random sample consensus (RANSAC) algorithm, as described above.

[0078] The observation event represented by steps 502 to 510 provides two constraints that relate a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b. Once at least six constraints have been accumulated, the first mobile device 102a can calculate a pose of the first mobile device 102a relative to the second mobile device 102b in step 512. For example, the first mobile device 102a can use six (or more) constraints to calculate a 6DoF transformation that relates a first coordinate system used by the first mobile device 102a to a second coordinate system used by the second mobile device 102b.

[0079] In some embodiments, the six (or more) constraints used to calculate the pose in step 512 may include (a) a plurality of constraints determined as a result of the observation event, and (b) one or more additional constraints known independently of the observation event. Some examples of these additional constraints include a known common vertical orientation, height above the ground, orientation of a major wall, etc. These additional constraints may be used to obtain more accurate and / or robust results.

[0080] Figure 6 Illustrated are certain components that may be included within a mobile device 602 configured to implement the techniques disclosed herein. Figure 6 The mobile device 602 shown in FIG. 6 represents one possible implementation of the mobile devices 102a - 102b , 202a - 202b , 302a - 302c , 402a - 402b described previously.

[0081] The mobile device 602 includes a processor 601. The processor 601 may be a general-purpose single-chip or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a dedicated microprocessor (e.g., a Digital Signal Processor (DSP)), a microcontroller, a programmable gate array, etc. The processor 601 may be referred to as a central processing unit (CPU). Figure 6 Just a single processor 601 is shown in the mobile device 602 of FIG. 4 , but in an alternative configuration, a combination of processors (eg, an ARM and DSP) could be used.

[0082] The mobile device 602 also includes a memory 603 in electronic communication with the processor 601. The memory 603 can be any electronic component capable of storing electronic information. For example, the memory 603 can be implemented as a random access memory (RAM), a read-only memory (ROM), a disk storage medium, an optical storage medium, a flash memory device in a RAM, an on-board memory included in the processor 601, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) memory, a register, etc., including combinations thereof.

[0083] Instructions 605 and data 607 may be stored in memory 603. Instructions 605 may be executed by processor 601 to implement some or all of the steps, operations, actions, or other functions disclosed herein. Executing instructions 605 may involve using data 607 stored in memory 603. Unless otherwise specified, any of the various examples of modules and components described herein may be implemented in part or in whole as instructions 605, which are stored in memory 603 and executed by processor 601. Any of the various examples of data described herein may be in data 607, which is stored in memory 603 and used during execution of instructions 605 by processor 601.

[0084] Instructions 605 may include a device tracking module 621. The device tracking module 621 may be configured to track the position and movement of the mobile device 602 in space. The mobile device 602 may include various components that facilitate such tracking, including (one or more) cameras 631, inertial measurement units (IMUs) 633, and (one or more) other sensors 635. The device tracking module 621 may perform optical-based tracking based on input received from the camera 631. In some embodiments, the device tracking module 621 may also utilize input received from the IMU 633 and the sensor 635 to enhance tracking. Tracking may include, for example, six degrees of freedom (6DoF) device tracking. This may be implemented using simultaneous localization and mapping (SLAM) and / or visual inertial odometers (VIO). The device tracking module 621 may be configured to build and / or update a SLAM map 614 of an unknown environment while keeping track of the position of the mobile device 602 within the environment.

[0085] Instructions 605 may also include a relative positioning module 623. Relative positioning module 623 may be configured to determine the relative positioning between mobile device 602 and one or more other mobile devices according to the techniques disclosed herein. For example, relative positioning module 623 may be configured to implement such as Figure 5 The method 500 is shown in FIG.

[0086] The mobile device 602 may include a display 637. In some embodiments (e.g., in embodiments where the mobile device 602 is a mixed reality device), the display 637 may include one or more semi-transparent lenses on which images of virtual objects may be displayed. Different stereo images may be displayed on the lenses to create the appearance of depth, while the semi-transparent nature of the lenses allows the user to see both the real world and the virtual objects rendered on the lenses. The mobile device 602 may also include a graphics processing unit (GPU) 639. The processor (s) 601 may direct the GPU 639 to render virtual objects and cause the virtual objects to appear on the display 637.

[0087] Mobile device 602 may also include a light emitter 641. Light emitter 641 may be activated to make mobile device 602 more easily detected by another mobile device. In some embodiments (e.g., if mobile device 602 is a mobile phone), light emitter 641 may be a flash or a privacy light. Alternatively, light emitter 641 may be an infrared emitter used by some three-dimensional cameras on some head-mounted displays (e.g., in a mixed reality headset). In some embodiments, one or more cameras 631 of mobile device 602 may be an infrared / depth camera configured to detect infrared light emitted by another mobile device.

[0088] The mobile device 602 may also include (one or more) input devices 611. The input device 611 may be used to provide user input for triggering an observation event. In some embodiments, the (one or more) input device 611 may include one or more buttons. Some other examples of input devices 711 include a keyboard, a mouse, a microphone, a remote control device, a button, a joystick, a trackball, a touchpad, and a light pen.

[0089] The mobile device 602 may also include one or more communication interfaces 613 for communicating with other electronic devices such as other mobile devices. For example, the mobile device 602 may use the communication interface 613 to send location information and / or orientation information to another mobile device in conjunction with determining its position relative to the other mobile device. The (one or more) communication interface 613 may be based on wired communication technology, wireless communication technology, or both. Some examples of the communication interface 613 include a universal serial bus (USB), an Ethernet adapter, a wireless adapter operating according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, Wireless communication adapter and infrared (IR) communication port.

[0090] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules, components, etc. may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a non-transitory computer-readable medium having computer executable instructions stored thereon, which, when executed by at least one processor, perform some or all of the steps, operations, actions, or other functions disclosed herein. Instructions may be organized into routines, programs, objects, components, data structures, etc., which may perform specific tasks and / or implement specific data types, and may be combined or distributed as desired in various embodiments.

[0091] The steps, operations and / or actions of the methods described herein may be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps, operations and / or actions is required for the normal operation of the described method, the order and / or use of specific steps, operations and / or actions may be modified without departing from the scope of the claims.

[0092] The term "determining" (and its grammatical variations) encompasses a variety of actions, and thus, "determining" may include mathematically calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), ascertaining, etc. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0093] The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, it should be understood that reference to "one embodiment" or "an embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the described features. For example, any element or feature described with respect to an embodiment herein may be combined with any element or feature of any other embodiment described herein, where compatible.

[0094] The present disclosure may be implemented in other specific forms without departing from its spirit or features. The described embodiments should be considered illustrative rather than restrictive. Therefore, the scope of the present disclosure is indicated by the appended claims rather than by the foregoing description. Changes within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A method for obtaining relative positioning between a plurality of mobile devices, the method being performed by a first mobile device, the method comprising: observing a second mobile device within a field of view of a camera of the first mobile device at a first point in time; determining a first position of the first mobile device at the first point in time, the first position being expressed in a first coordinate system used by the first mobile device; receiving, from the second mobile device, a second position of the second mobile device at the first point in time, the second position being expressed in a second coordinate system used by the second mobile device; determining, based at least in part on the first location and the observation of the second mobile device at the first point in time, orientation information regarding an orientation of the first mobile device relative to the second mobile device at the first point in time; identifying two constraints relating the first coordinate system to be used by the first mobile device to the second coordinate system to be used by the second mobile device based at least in part on the second position and the orientation information; as well as Once at least six constraints are accumulated, a pose of the first mobile device relative to the second mobile device is calculated.

2. The method according to claim 1, further comprising: When the second mobile device is observed within the field of view of the camera of the first mobile device at the first point in time, user input is received, wherein in response to receiving the user input: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

3. The method according to claim 2, wherein: The method is performed during a game involving the plurality of mobile devices; and The user input is provided as part of the game.

4. The method according to claim 1, further comprising: Automatically detecting the second mobile device within the field of view of the camera of the first mobile device at the first point in time, wherein in response to automatically detecting the second mobile device: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

5. The method of claim 4, wherein automatically detecting the second mobile device comprises: An activated light transmitter of the second mobile device is automatically detected.

6. The method according to claim 1, wherein: At the first point in time, both the second mobile device and at least one additional mobile device are visible within the field of view of the camera of the first mobile device; and The method also includes distinguishing the second mobile device from the at least one additional mobile device.

7. The method of claim 1 , wherein determining the orientation information comprises: A geometric line in space is determined, the geometric line corresponding to the direction in which the second mobile device was observed at the first point in time.

8. The method of claim 1, wherein calculating the pose comprises calculating a six degree of freedom transformation that relates the first coordinate system used by the first mobile device to the second coordinate system used by the second mobile device.

9. The method of claim 1, wherein the at least six constraints include: multiple constraints determined as a result of observing events; as well as One or more additional constraints that are known independently of the observed event.

10. The method according to claim 1, further comprising: creating a first simultaneous positioning and mapping map based on the field of view of the camera of the first mobile device; as well as The first simultaneous positioning and mapping map is merged with a second simultaneous positioning and mapping map created by the second mobile device.

11. A first mobile device, configured to obtain a relative position between the first mobile device and a second mobile device, comprising: camera; one or more processors; a memory in electronic communication with the one or more processors; as well as Instructions stored in the memory, the instructions executable by the one or more processors to: At a first point in time, observing the second mobile device within a field of view of the camera of the first mobile device; determining a first position of the first mobile device at the first point in time, the first position being expressed in a first coordinate system used by the first mobile device; receiving, from the second mobile device, a second position of the second mobile device at the first point in time, the second position being expressed in a second coordinate system used by the second mobile device; determining, based at least in part on the first location and the observation of the second mobile device at the first point in time, orientation information regarding an orientation of the first mobile device relative to the second mobile device at the first point in time; identifying two constraints relating the first coordinate system to be used by the first mobile device to the second coordinate system to be used by the second mobile device based at least in part on the second position and the orientation information; as well as Once at least six constraints are accumulated, a pose of the first mobile device relative to the second mobile device is calculated.

12. The first mobile device according to claim 11 further includes an input device, wherein the input device is configured to: receive user input when the second mobile device is observed within the field of view of the camera of the first mobile device at the first time point, wherein in response to receiving the user input: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

13. The first mobile device according to claim 11 further includes additional instructions, which can be executed by the one or more processors to: automatically detect the second mobile device within the field of view of the camera of the first mobile device at the first time point, wherein in response to automatically detecting the second mobile device: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

14. The first mobile device of claim 11, wherein the instructions executable by the one or more processors to determine the orientation information comprise: Instructions executable by the one or more processors to determine a geometric line in space, the geometric line corresponding to a direction in which the second mobile device was observed at the first point in time.

15. The first mobile device of claim 11, wherein the instructions executable by the one or more processors to calculate the pose comprise: Instructions executable by the one or more processors to compute a six degree of freedom transform relating the first coordinate system used by the first mobile device to the second coordinate system used by the second mobile device.

16. A computer-readable medium comprising instructions executable by one or more processors to cause a first mobile device to: observing a second mobile device within a field of view of a camera of the first mobile device at a first point in time; determining a first position of the first mobile device at the first point in time, the first position being expressed in a first coordinate system used by the first mobile device; receiving, from the second mobile device, a second position of the second mobile device at the first point in time, the second position being expressed in a second coordinate system used by the second mobile device; determining, based at least in part on the first location and the observation of the second mobile device at the first point in time, orientation information regarding an orientation of the first mobile device relative to the second mobile device at the first point in time; identifying two constraints relating the first coordinate system to be used by the first mobile device to the second coordinate system to be used by the second mobile device based at least in part on the second position and the orientation information; as well as Once at least six constraints are accumulated, a pose of the first mobile device relative to the second mobile device is calculated.

17. The computer-readable medium of claim 16, further comprising additional instructions executable by the one or more processors to cause the first mobile device to: receive user input when the second mobile device is observed within the field of view of the camera of the first mobile device at the first point in time, wherein in response to receiving the user input: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

18. The computer-readable medium of claim 16, further comprising additional instructions executable by the one or more processors to: automatically detect the second mobile device within the field of view of the camera of the first mobile device at the first point in time, wherein in response to automatically detecting the second mobile device: the first position is determined, the second position is received from the second mobile device, the orientation information is determined, and the two constraints are identified.

19. The computer-readable medium of claim 16, wherein the instructions executable by the one or more processors to determine the orientation information comprise: Instructions executable by the one or more processors to determine a geometric line in space, the geometric line corresponding to a direction in which the second mobile device was observed at the first point in time.

20. The computer-readable medium of claim 16, wherein the instructions executable by the one or more processors to calculate the pose comprise: Instructions executable by the one or more processors to compute a six degree of freedom transform relating the first coordinate system used by the first mobile device to the second coordinate system used by the second mobile device.

Images