Relative spatial localization of mobile devices
Patent Information
- Application Number
- CA3132049
- Authority / Receiving Office
- CA · CA
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-01-30
- Filing Date
- 2020-01-30
- Publication Date
- 2026-10-20
- Estimated Expiration
- 2040-01-30
AI Technical Summary
Existing mixed-reality devices face challenges in determining relative pose with other devices, especially when users' viewpoints differ significantly, leading to failures in shared mixed-reality experiences due to cumbersome data exchange and privacy risks associated with sharing three-dimensional map information.
A method for mobile devices to calculate relative localization by observing each other, determining position and orientation constraints, and accumulating at least six constraints to calculate a six degrees of freedom transformation, allowing alignment of coordinate systems without exchanging 3D map information, thereby reducing processing and storage requirements.
Enables efficient and secure relative localization between multiple mobile devices, facilitating shared mixed-reality experiences without the need for extensive data exchange, improving processing efficiency and user experience by integrating observation events into game play or automatic detection.
Abstract
Description
<DP=1>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753 ²RELATIVE SPATIAL LOCALIZATION OF MOBILE DEVICES²BACKGROUND²
[0001] Mixed reality, which can also be known as augmented reality, ²involves the²merging of real-world objects and / or people with virtual objects to produce ²new²environments and visualizations where physical and digital objects co-exist ²and interact in ²real time. Mixed-reality devices augment a user's view of the real world with ²virtual objects ²that aim to look as if they are actually placed within the real world. A mixed-²reality device ²may allow the user to view their real-world surroundings through a semi-²transparent display.²Virtual objects can then be presented on the display. These virtual objects ²appear to be²superimposed on the user's view of their real-world surroundings, thereby ²merging virtual ²reality with physical reality.²
[0002] A mixed-reality experience can be shared among multiple mixed-²reality devices. ²This enables multiple users to have common shared experiences within a shared ²mixed-²reality environment. There are many different scenarios where a shared mixed-²reality²experience could be useful and / or enjoyable, such as a game where players are ²able to ²interact with one another and with virtual objects as part of the game.²
[0003] To facilitate shared mixed-reality experiences, it is important ²for multiple ²devices to be able to compute their position and motion in the same coordinate ²system so²that they can be aware of each other's relative position. Most mixed-reality ²devices are able²to determine their own relative motion, but not necessarily their relative ²pose (i.e., position ²and orientation) with respect to other devices.²
[0004] The typical solution to register the coordinate system for ²multiple devices is to ²exchange three-dimensional map information (or two-dimensional image data) ²between²devices so that the relative pose between these maps / images can be determined. ²These²solutions can either be implemented peer-to-peer or over a cloud service.²
[0005] The exchange of map data, however, can be cumbersome and present ²privacy ²risks. Moreover, when users' viewpoints differ significantly from each other ²the traditional ²image feature matching can fail, resulting in the inability of devices to ²determine a joint²coordinate system and thus to share a mixed-reality experience. This may ²happen, for²example, in scenarios where users look at each other and the camera and ²therefore view ²opposite sides of the same space, which is often the natural configuration for ²games and ²other types of shared mixed-reality experiences.²1²<DP=2>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²SUMMARY²
[0006] In accordance with one aspect of the present disclosure, a method ²for obtaining ²a relative localization between a plurality of mobile devices is disclosed. ²The method may ²be performed by a first mobile device. The method includes observing a second ²mobile²device within a field of view of a camera of the first mobile device at a ²first point in time.²The method also includes determining a first position of the first mobile ²device at the first ²point in time. The first position is represented in a first coordinate system ²used by the first ²mobile device. The method also includes receiving from the second mobile ²device a second ²position of the second mobile device at the first point in time. The second ²position is²represented in a second coordinate system used by the second mobile device. ²The method²also includes determining orientation information about an orientation of the ²first mobile ²device with respect to the second mobile device at the first point in time ²based at least in ²part on the first position and the observation of the second mobile device at ²the first point ²in time. The method also includes identifying two constraints that relate the ²first coordinate²system used by the first mobile device to the second coordinate system used by ²the second²mobile device based at least in part on the second position and the ²orientation information. ²The method also includes calculating a pose of the first mobile device ²relative to the second ²mobile device once at least six constraints have been accumulated.²
[0007] In some embodiments, the method may additionally include ²receiving user input²when the second mobile device is observed within the field of view of the ²camera of the²first mobile device at the first point in time. The first position may be ²determined, the second ²position may be received from the second mobile device, the orientation ²information may ²be determined, and the two constraints may be identified in response to ²receiving the user ²input. The method may be performed during a game that involves the plurality ²of mobile² devices, and the user input may be provided as part of the game.²
[0008] In some embodiments, the method may additionally include ²automatically ²detecting the second mobile device within the field of view of the camera of ²the first mobile ²device at the first point in time. The first position may be determined, the ²second position ²may be received from the second mobile device, the orientation information may ²be²determined, and the two constraints may be identified in response to ²automatically detecting²the second mobile device. Automatically detecting the second mobile device may ²include ²automatically detecting an activated light emitter of the second mobile ²device.²
[0009] In some embodiments, the second mobile device and at least one ²additional ²mobile device may both be visible within the field of view of the camera of ²the first mobile²2²<DP=3>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²device at the first point in time. The method may additionally include ²distinguishing ²between the second mobile device and the at least one additional mobile ²device.²
[0010] In some embodiments, determining the orientation information may ²include ²determining a geometric line in space corresponding to a direction along which ²the second² mobile device was observed at the first point in time.²
[0011] In some embodiments, calculating the pose may include calculating ²a six degrees ²of freedom transformation that relates the first coordinate system used by the ²first mobile ²device to the second coordinate system used by the second mobile device.²
[0012] In some embodiments, the at least six constraints may include a ²plurality of²constraints that are determined as a result of observation events and one or ²more additional²constraints that are known independently of the observation events.²
[0013] In some embodiments, the method may additionally include creating ²a first ²simultaneous localization and mapping (SLAM) map based on the field of view of ²the ²camera of the first mobile device. The first SLAM map may be merged with a ²second SLAM²map that is created by the second mobile device.²
[0014] This Summary is provided to introduce a selection of concepts in ²a simplified ²form that are further described below in the Detailed Description. This ²Summary is not ²intended to identify key features or essential features of the claimed subject ²matter, nor is it ²intended to be used as an aid in determining the scope of the claimed subject ²matter.²
[0015] Additional features and advantages will be set forth in the ²description that²follows. Features and advantages of the disclosure may be realized and ²obtained by means ²of the systems and methods that are particularly pointed out in the appended ²claims. Features ²of the present disclosure will become more fully apparent from the following ²description ²and appended claims, or may be learned by the practice of the disclosed ²subject matter as²set forth hereinafter.²BRIEF DESCRIPTION OF THE DRAWINGS²
[0016] In order to describe the manner in which the above-recited and ²other features of ²the disclosure can be obtained, a more particular description will be rendered ²by reference ²to specific embodiments thereof which are illustrated in the appended ²drawings. For better²understanding, the like elements have been designated by like reference ²numbers throughout²the various accompanying figures. Understanding that the drawings depict some ²example ²embodiments, the embodiments will be described and explained with additional ²specificity ²and detail through the use of the accompanying drawings in which:²
[0017] Figures 1A¨C illustrate an example of a method for obtaining a ²relative²3²<DP=4>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²localization between a plurality of mobile devices.²
[0018] Figure 2 illustrates a field of view of a mobile device at a ²point in time when a ²plurality of other mobile devices are visible within the field of view.²
[0019] Figure 3 illustrates an example in which two mobile devices' ²coordinate systems²are aligned with respect to each other and are considered to be a single ²mobile device for²the purpose of the localization techniques disclosed herein.²
[0020] Figure 4 illustrates an example in which two mobile devices' ²simultaneous ²localization and mapping (SLAM) maps are merged after the mobile devices' ²coordinate ²systems are aligned with respect to each other.²
[0021] Figure 5 illustrates a method for obtaining a relative localization ²between a²plurality of mobile devices in accordance with the present disclosure.²
[0022] Figure 6 illustrates certain components that may be included ²within a mobile²device that is configured to implement the techniques disclosed herein.²DETAILED DESCRIPTION²
[0023] The present disclosure is generally related to obtaining a relative ²localization²between a plurality of mobile devices, each of which includes a camera and is ²able to keep ²track of its own position and motion in space using its own coordinate system. ²The ²techniques disclosed herein are applicable to any scenario in which it is ²desirable for a ²plurality of mobile devices to compute their position and motion in the same ²coordinate²system so that they can be aware of each other's relative position. As one ²example, the²techniques disclosed herein may be utilized in the context of shared mixed-²reality ²experiences.²
[0024] As used herein, the term "mobile device" refers to a portable ²computing device ²that includes a camera and is capable of implementing the spatial localization ²techniques²disclosed herein. In some embodiments, a mobile device may be small enough for ²a user to²hold and operate the mobile device in the user's hand. In some embodiments, a ²mobile ²device may be a wearable computing device. In some embodiments, a mobile ²device may ²be a mixed reality (or augmented reality) device that is capable of providing ²a mixed reality ²(or augmented reality) experience for users. Some examples of mobile devices ²include head-².. mounted displays, smartglasses, smartphones, tablet computers, and laptop ²computers.²Mobile devices may be capable of connecting to one or more computer networks, ²such as ²the Internet. Mobile devices may also be capable of establishing peer-to-peer ²communication with other computing devices.²
[0025] The techniques disclosed herein utilize direct observations of a ²mobile device²4²<DP=5>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²within the field of view of another mobile device's camera. Consider a simple ²example ²involving two mobile devices, a first mobile device and a second mobile ²device. Suppose ²that the first mobile device observes the second mobile device within the ²field of view of ²the first mobile device's camera. When this occurs, the first mobile device is ²able to use its²own position and its observation of the second mobile device to constrain the ²orientation of²the first mobile device with respect to the second mobile device. In other ²words, the first ²mobile device is able to determine information about the orientation of the ²first mobile ²device with respect to the second mobile device. This orientation information, ²along with ²the position of the second mobile device (as represented in the coordinate ²system used by²the second mobile device), may be used to identify two constraints for ²relating the²coordinate system used by the first mobile device to the coordinate system ²used by the ²second mobile device.²
[0026] Once at least six constraints have been accumulated, the pose of ²the first mobile ²device relative to the second mobile device (and vice versa) may be ²calculated. More²specifically, the six (or more) constraints may be used to calculate the six ²degrees of freedom²(6DoF) transformation that relates the first coordinate system used by the ²first mobile device ²to the second coordinate system used by the second mobile device.²
[0027] As used herein, the term "observation event" refers to a ²situation in which one ²mobile device observes another mobile device within its camera's field of ²view, and the²corresponding position information and orientation information is used to ²determine two²constraints that relate the two mobile devices' coordinate systems. As will be ²discussed in ²greater detail below, in some embodiments an observation event may be ²triggered by user ²input. Alternatively, an observation event may be triggered when one mobile ²device ²automatically detects another mobile device in its camera's field of view.²
[0028] The disclosed techniques for obtaining the relative localization ²between a²plurality of mobile devices may be particularly advantageous in situations ²where it is not ²possible or desirable for the plurality of mobile devices to share a three-²dimensional (3D) ²environment map with one another. This may occur, for example, when users' ²viewpoints ²differ significantly from each other.²
[0029] In some embodiments, the disclosed techniques may reduce processing²requirements relative to known approaches. As indicated above, the typical ²solution to ²register the coordinate system for a plurality of mobile devices is to ²exchange three-²dimensional (3D) map information between the mobile devices. The techniques ²disclosed ²herein, however, enable the relative localization between a plurality of ²mobile devices to be²5²<DP=6>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²obtained without exchanging 3D map information. This may reduce the amount of ²processing that is required to align the coordinate systems of a plurality of ²mobile devices. ²For example, mobile devices that align their coordinate systems in accordance ²with the ²techniques disclosed herein do not have to determine or exchange 3D map ²information.²Therefore, the amount of processing that is required to determine and / or ²exchange 3D map²information may be saved by utilizing the techniques disclosed herein.²
[0030] The ability to obtain the relative localization between a ²plurality of mobile ²devices without exchanging 3D map information may also reduce the amount of ²map ²information needs to be stored, and consequently reduce storage requirements ²for mobile²devices. In addition to reducing storage requirements, the techniques ²disclosed herein may²also reduce the amount of information that is communicated between a plurality ²of mobile ²devices (either via computer network(s) or via peer-to-peer communications). ²Instead of ²exchanging 3D map information, which can be quite data intensive, the mobile ²devices may ²simply exchange some position information and some orientation information ²associated²with specific points in time (as will be discussed in greater detail below). ²This potentially²reduces the amount of information that is communicated between the plurality ²of mobile ²devices, thereby potentially freeing a significant amount of communication ²bandwidth for ²other purposes.²
[0031] Notwithstanding the foregoing, however, three-dimensional map ²information²may still be exchanged under some circumstances in accordance with the ²techniques ²disclosed herein. As will be discussed in greater detail below, in some ²embodiments, the 3D ²maps that are being constructed by each mobile device can be merged into a ²larger, more ²complete map.²
[0032] An example of a method for obtaining a relative localization ²between a plurality²of mobile devices will be described in relation to Figures 1A¨C. This example ²involves two²mobile devices 102a¨b, a first mobile device 102a and a second mobile device ²102b. The ²coordinate system used by the first mobile device 102a will be referred to ²herein as a first ²coordinate system, and the coordinate system used by the second mobile device ²102b will ²be referred to herein as a second coordinate system. The user of the first ²mobile device 102a²will be referred to as the first user 104a, and the user of the second mobile ²device 102b will²be referred to as the second user 104b.²
[0033] Figure 1A illustrates the first user 104a aiming the first mobile ²device 102a at ²the second user 104b, and the second user 104b aiming the second mobile device ²102b at ²the first user 104a. This may occur, for example, during a game in which the ²users are²6²<DP=7>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²supposed to shoot at each other using the mobile devices. Figure 1B ²illustrates the field of ²view of the camera of the first mobile device 102a when the first mobile ²device 102a is ²aimed at the second mobile device 102b (as shown in Figure 1A), as it may be ²displayed to ²the first user 104a on a display 106 of the first mobile device 102a. Figure ²1C illustrates the²trajectory 108a of the first mobile device 102a and the trajectory 108b of the ²second mobile²device 102b as the first user 104a and the second user 104b move around over a ²period of ²time (e.g., while moving around during a game).²
[0034] As shown in Figure 1C, at time tithe first user 104a positions ²the first mobile ²device 102a so that the second mobile device 102b is located within the field ²of view of the²camera of the first mobile device 102a (and is therefore visible on the ²display 106). For²example, in the context of a shooting game, the first user 104a may aim the ²first mobile ²device 102a at the second user 104b, who may be holding or wearing the second ²mobile ²device 102b and possibly aiming the second mobile device 102b at the first ²user 104a. As ²shown in Figure 1B, crosshairs 110 may be displayed on the display 106 of the ²first mobile²device 102a. The crosshairs 110 may help the first user 104a to position the ²first mobile²device 102a so that the second mobile device 102b is located approximately ²within the ²center of the field of view of the camera of the first mobile device 102a. The ²first user 104a ²then provides some input (e.g., clicking a button on the first mobile device ²102a) that causes ²the first mobile device 102a and the second mobile device 102b to remember ²(e.g., store in²memory) and communicate certain information associated with that specific ²point in time.²
[0035] In particular, the first mobile device 102a determines and ²remembers its position ²at time ti when the second mobile device 102b is observed within the field of ²view of the ²camera of the first mobile device 102a. This position, which is represented in ²the first ²coordinate system used by the first mobile device 102a, will be referred to ²herein as phi.²(As used herein, the term pxty refers to the position of device x at time y.) ²The first mobile²device 102a also communicates phi to the second mobile device 102b. It is ²assumed that ²the first mobile device 102a and the second mobile device 102b are ²substantially time ²synchronized.²
[0036] The second mobile device 102b also determines and remembers its ²position at²time ti. This position, which is represented in the second coordinate system ²used by the²second mobile device 102b, will be referred to herein as p2a. The second ²mobile device ²102b also communicates p2ti to the first mobile device 102a.²
[0037] The first mobile device 102a also determines and remembers ²information about ²its orientation at time ti with respect to the second mobile device 102b. For ²example, the²7²<DP=8>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²first mobile device 102a may determine and remember a geometric line in space ²corresponding to the direction along which the second mobile device 102b was ²observed at ²time ti. This line will be referred to herein as linen_ If the second mobile ²device 102b was ²observed in the center of the field of view of the camera of the first mobile ²device 102a at²time ti, then this line in space would correspond to the optical axis of the ²camera.²
[0038] This position information (phi and p2t1) and orientation ²information (linen) may ²then be used to identify two constraints that relate the first coordinate ²system used by the ²first mobile device 102a to the second coordinate system used by the second ²mobile device ²102b. In particular, the geometric constraint that the position of the second ²mobile device²102b at time ti (p2t1) should be located along the previously defined line in ²space (lineti)²provides two mathematical constraints to align the coordinate systems of both ²mobile ²devices 102a¨b.²
[0039] This process of observing another mobile device and using ²position and ²orientation information associated with that observation to determine two ²constraints may²then be repeated at least two more times until at least six constraints are ²determined.²However, it does not always have to be the same mobile device that does the ²observing each ²time (although it could be). For instance, in the depicted example, the second ²mobile device ²102b observes the first mobile device 102a within the field of view of the ²camera of the ²second mobile device 102b at time t2.²
[0040] More specifically, at time t2 the second user 104b positions the ²second mobile²device 102b so that the first mobile device 102a is located within the field ²of view of the ²camera of the second mobile device 102b. The second user 104b then provides ²some input ²that causes the second mobile device 102b and the first mobile device 102a to ²remember ²and communicate certain information associated with time t2. In particular, ²the second²mobile device 102b determines and remembers its position at time t2 (p212), ²which is²represented in the second coordinate system used by the second mobile device ²102b. The ²second mobile device 102b also sends p2t2 to the first mobile device 102a. In ²addition, the ²first mobile device 102a determines and remembers its position at time t2 ²(plc), which is ²represented in the first coordinate system used by the first mobile device ²102a. The first²mobile device 102a also sends p112 to the second mobile device 102b. The ²second mobile²device 102b also determines and remembers information about its orientation at ²time t2 with ²respect to the first mobile device 102a. More precisely, the second mobile ²device 102b ²remembers the geometric line in space corresponding to the direction along ²which the first ²mobile device 102a was observed at time t2. This line will be referred to ²herein as linetz.²8²<DP=9>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²
[0041] This position and orientation information may then be used to ²identify two ²additional constraints that relate the second coordinate system used by the ²second mobile ²device 102b to the first coordinate system used by the first mobile device ²102a. In particular, ²the geometric constraint that the position of the first mobile device 102a at ²time t2 (plc)²needs to be located along 1inet2 provides an additional two mathematical ²constraints to align²the coordinate systems of both mobile devices 102a¨b.²
[0042] Subsequently, at time t3 the first user 104a positions the first ²mobile device 102a ²so that the second mobile device 102b is located within the field of view of ²the camera of ²the first mobile device 102a. The first user 104a then provides some input ²that causes the²first mobile device 102a and the second mobile device 102b to remember and ²communicate²certain information associated with time t3. In particular, the first mobile ²device 102a ²determines and remembers its position at time t3 (pie), which is represented ²in the first ²coordinate system used by the first mobile device 102a. The first mobile ²device 102a also ²sends plt3 to the second mobile device 102b. The second mobile device 102b ²determines²and remembers its position at time t3 (p2t3), which is represented in the ²second coordinate²system used by the second mobile device 102b. The second mobile device 102b ²also sends ²p2t3 to the first mobile device 102a. The first mobile device 102a also ²determines and ²remembers information about its orientation at time t3 with respect to the ²second mobile ²device 102b. More precisely, the first mobile device 102a remembers the ²geometric line in²space corresponding to the direction along which the second mobile device 102b ²was²observed at time t3. This line will be referred to herein as 1inet3.²
[0043] This position and orientation information may then be used to ²identify two ²additional constraints that relate the first coordinate system used by the ²first mobile device ²102a to the second coordinate system used by the second mobile device 102b. In ²particular,²the geometric constraint that the position of the second mobile device 102b at ²time t3 (p2t3)²needs to be located along 1inet3 provides two additional mathematical ²constraints to align ²the coordinate systems of both mobile devices 102a¨b.²
[0044] In the present example, six constraints have been identified ²after time t3. This is ²a sufficient number of constraints to enable calculation of a six degrees of ²freedom²transformation that relates the first coordinate system used by the first ²mobile device 102a²to the second coordinate system used by the second mobile device 102b. As is ²known by ²those skilled in the art, the term "six degrees of freedom" (6DoF) refers to ²the freedom of ²movement of a rigid body in three-dimensional space. In particular, a rigid ²body is free to ²change position as forward / backward (surge), up / down (heave), left / right ²(sway) translation²9²<DP=10>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²in three perpendicular axes. Such translation may be combined with changes in ²orientation ²through rotation about three perpendicular axes, which are often termed yaw ²(normal axis), ²pitch (transverse axis), and roll (longitudinal axis). In the context of the ²present disclosure, ²a "six degrees of freedom transformation" (6DoF transformation) specifies the²mathematical operations that should be performed to convert position and ²orientation²information that is represented in one coordinate system to equivalent ²position and ²orientation information that is represented in another coordinate system.²
[0045] In the example just described, six constraints are obtained from ²three observation ²events. Two observation events occur at the first mobile device 102a (at times ²ti and t3),²and one observation event occurs at the second mobile device 102b (at time ²t2). The specific²details of this particular example should not be interpreted as limiting the ²scope of the ²present disclosure. For instance, it is not necessary for the observation ²events to occur in ²any particular order. In an alternative example, the first mobile device 102a ²could have two ²consecutive observation events (e.g., at ti and t2) before the second mobile ²device 102b has²an observation event (e.g., at time t3). Also, it is not necessary for both ²mobile devices²102a¨b to have observation events. In another alternative example, all three ²observation ²events may occur at the same mobile device.²
[0046] In addition, more than three observation events may occur. As ²indicated above, ²a minimum of six constraints (from a minimum of three observation events) are ²needed in²order to calculate a 6DoF transformation that relates the first coordinate ²system used by the²first mobile device 102a to the second coordinate system used by the second ²mobile device ²102b. With minimal data (e.g., data from only three observation events), it is ²possible that ²more than one solution may be obtained in connection with calculating the 6DoF ²²transformation. With a redundant set of measurements, however, the solution ²typically² becomes unique.²
[0047] In some embodiments, both of the mobile devices 102a¨b may ²calculate the ²6DoF transformation. In the particular example that was just described, four ²constraints are ²determined based on observations made by the first mobile device 102a (at ²times ti and t3), ²and two constraints are determined based on observations made by the second ²mobile device²102b (at time t2). The mobile devices 102a¨b may share information about these ²constraints²with one another so that both mobile devices 102a¨b have sufficient ²information to calculate ²the 6DoF transformation. Alternatively, one of the mobile devices (e.g., the ²first mobile ²device 102a) may calculate the 6DoF transformation and share the 6DoF ²transformation ²with the other mobile device (e.g., the second mobile device 102b). ²Alternatively still,²<DP=11>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²another entity that is distinct from the mobile devices 102a¨b may calculate ²the 6DoF ²transformation and share the 6DoF transformation with the mobile devices ²102a¨b. This ²other entity may be a server that is in electronic communication with the ²mobile devices ²102a¨b via one or more computer networks. In some embodiments, this other ²entity may be² a cloud-based entity.²
[0048] The constraints that are determined as a result of observation ²events may be ²combined with additional constraints that are known independently of ²observation events ²for purposes of calculating the 6DoF transformation. For instance, with ²respect to the ²example shown in Figures 1A¨C, the six constraints that are determined as a ²result of the².. three observation events (at times ti, t2, and t3) may be combined with ²additional constraints²such as a known common vertical orientation (two constraints), a height above ²ground (one ²constraint), a dominant wall orientation (one constraint), and so forth. A ²common vertical ²orientation may be determined from inertial measurements or vertical vanishing ²²points / lines. The height above ground may be determined when a ground plane is ²observed.²The orientation with respect to a room may be determined when walls are ²observed. A²redundant set of constraints can be used to obtain more accurate and / or robust ²results.²
[0049] In the example shown in Figures 1A¨C, user input triggers the ²observation ²events. In other words, user input causes the mobile devices 102a¨b to ²remember and ²communicate certain information associated with specific points in time, and ²to use this²information to determine constraints for relating the coordinate systems of ²the mobile²devices 102a¨b. This user input may be provided when a user sees another ²mobile device in ²the field of view of the camera of the user's mobile device (as seen, for ²example, on a display ²106).²
[0050] In some embodiments, having user input trigger the observation ²events can be².. integrated into the user experience as part of game play. This may be ²applicable to some²mixed-reality experiences, such as a game where users are supposed to shoot at ²each ²other. In this type of scenario, when the user shoots, one can in general ²assume that he or ²she is aiming at the other user. After players have shot a few times at each ²other, the ²disclosed techniques can be used to determine the relative pose between the ²coordinate².. systems of both mobile devices. This information can then be used to enrich ²the users'²experience.²
[0051] Integrating the observation events into the user experience as ²part of game play ²may provide certain benefits relative to at least some known approaches. As ²indicated ²above, the typical solution to register the coordinate system for multiple ²devices is to²11²<DP=12>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²exchange three-dimensional map information (or two-dimensional image data) ²between ²devices. With at least some known approaches, user input may be required to ²exchange such ²information / data. Moreover, the required user input may be in addition to ²whatever user ²input would otherwise occur naturally as part of the user experience (e.g., ²something more²than shooting as part of game play). Integrating the observation events into ²the user²experience as part of game play, as disclosed herein, eliminates the need for ²this extra user ²input, thereby making the overall experience more natural and more enjoyable ²for the user.²
[0052] Notwithstanding the foregoing, however, it is not necessary for ²user input to be ²integrated into the user experience. In fact, it is not necessary to rely on ²user input to trigger²observation events. Alternatively, instead of relying on such user input, a ²mobile device may²be configured to automatically detect the presence of another mobile device in ²the field of ²view of its camera, and to automatically remember and communicate the relevant ²position ²and orientation information in response to detecting another mobile device in ²this way. In ²other words, in some embodiments, observation events may be triggered when one ²mobile²device automatically detects another mobile device in its camera's field of ²view.²
[0053] A mobile device may use an object recognition algorithm to detect ²the presence ²of another mobile device within its camera's field of view. This procedure can ²be simplified ²when the mobile device being detected is equipped with a light emitter that ²can be observed ²by the mobile device that is making the observation. For example, referring ²again to the²example shown in Figures 1A¨C, suppose that the second mobile device 102b is ²equipped²with a light emitter and that the light emitter is activated at time ti. ²Instead of relying on ²input from the first user 104a to trigger the observation event, the first ²mobile device 102a ²may instead automatically detect the second mobile device 102b within its ²camera's field of ²view. The activated light emitter on the second mobile device 102b may help to ²facilitate²this automatic detection. Some examples of light emitters that may be used in ²connection²with the techniques disclosed herein include a flashlight on a mobile phone, a ²privacy light ²associated with a camera, and an infrared emitter that is used by some three-²dimensional ²cameras on some head-mounted displays (e.g., in mixed-reality headsets). An ²infrared ²emitter may be observed by an infrared / depth camera of a similarly equipped ²device.²
[0054] In some embodiments, the identity of a particular mobile device may ²be²communicated via a unique pattern emitted by the light emitter on that mobile ²device. In ²other words, a mobile device may turn its light emitter on and off in ²accordance with a ²particular pattern, and the pattern used by a particular mobile device may be ²unique (at least ²with respect to a set of mobile devices that are being used at the same time ²in the same²12²<DP=13>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²place). Another mobile device that is observing the pattern may be able to ²determine the ²identity of the light-emitting mobile device based on the unique pattern that ²is emitted.²
[0055] Although the example shown in Figures 1A¨C included two mobile ²devices ²102a¨b, under some circumstances it may be desirable to determine the relative ²pose²between more than two mobile devices. The techniques disclosed herein can also ²be²extended to more than two mobile devices, as long as each mobile device has ²sufficient²constraints with respect to the other mobile devices to allow for relative ²localization.²
[0056] When more than two mobile devices are involved, there is an ²additional ²difficulty because the field of view of a particular mobile device may include ²more than one²mobile device. When a mobile device is attempting to determine its pose with ²respect to a²particular mobile device and there are a plurality of mobile devices visible ²within the field ²of view of its camera, it may be difficult to determine which of the plurality ²of mobile ²devices is the particular mobile device that is the subject of the pose ²calculations.²
[0057] Figure 2 illustrates a field of view of a mobile device as seen, ²for example, on a²display 206. A plurality of other mobile devices 202a¨b are visible in the ²field of view. The²mobile device whose field of view is shown in Figure 2 will be referred to as ²the observing ²mobile device, and the mobile devices 202a¨b that are visible within the field ²of view of the ²observing mobile device will be referred to as a first observed mobile device ²202a and a ²second observed mobile device 202b.²
[0058] Suppose that the observing mobile device is attempting to calculate ²its pose with²respect to the first observed mobile device 202a. It may be difficult for the ²observing mobile ²device to determine which of the mobile devices 202a¨b that are visible in its ²camera's field ²of view is the first observed mobile device 202a. In other words, it may be ²difficult for the ²observing mobile device to distinguish between the first observed mobile ²device 202a and²the second observed mobile device 202b. Therefore, when an observation event ²is triggered²(e.g., in response to user input or, alternatively, when the observing mobile ²device ²automatically detects the mobile devices 202a¨b), it may be difficult for the ²observing ²mobile device to decide whether it should determine and remember information ²about its ²orientation with respect to the first observed mobile device 202a or the ²second observed² mobile device 202b.²
[0059] This problem can be addressed in several different ways. In some ²embodiments, ²robust algorithms such as random sample consensus (RANSAC) may be used. RANSAC ²is ²an outlier detection method, and correspondences to other mobile devices may ²be ²considered as outliers. This may be particularly effective where a relatively ²small number²13²<DP=14>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²of mobile devices are visible in the camera's field of view at any given time.²
[0060] As another example, in some embodiments it may be possible to use ²image ²recognition algorithms to associate observation events that correspond to the ²same device. ²For instance, referring again to the example shown in Figure 2, suppose that ²the observing²mobile device detects the first observed mobile device 202a by itself at a ²previous point in²time (i.e., prior to observing the two mobile devices 202a¨b at the same time ²as shown in ²Figure 2). The observing mobile device may store an image of the first ²observed mobile ²device 202a in response to making that observation. Subsequently, when the ²observing ²mobile device observes both the first observed mobile device 202a and the ²second observed²mobile device 202b at the same time (as shown in Figure 2), the previously ²stored image²may be used to detect the first observed mobile device 202a and distinguish it ²from the ²second observed mobile device 202b.²
[0061] Once two mobile devices' coordinate systems are aligned with ²respect to each ²other, they can be considered to be a single mobile device for the purpose of ²the techniques²disclosed herein for obtaining a relative localization between a plurality of ²mobile devices.²This would, in turn, facilitate aligning one or more additional mobile devices ²to both ²previously aligned mobile devices.²
[0062] An example of this feature is shown in Figure 3. Suppose that a ²first mobile ²device 302a and a second mobile device 302b align their coordinate systems ²with one²another in accordance with the techniques disclosed herein. The first mobile ²device 302a²and the second mobile device 302b may then be considered to be a "single" ²mobile device ²312 for purposes of aligning their shared coordinate system with another ²mobile device. ²Thus, a third mobile device 302c may subsequently align itself with the ²coordinate system ²used by this "single" mobile device 312 (i.e., used by both the first mobile ²device 302a and² the second mobile device 302b).²
[0063] As discussed above, the coordinate systems of two mobile devices ²may be ²aligned with one another after a minimum of three observation events (each ²observation ²event producing two of the six constraints that are needed to compute a 6DoF ²transformation). In the present example, however, these observation events may ²occur as a²result of interactions between the third mobile device 302c and the first ²mobile device 302a²and / or as a result of interactions between the third mobile device 302c and ²the second mobile ²device 302b.²
[0064] Figure 3 illustrates the trajectory 308a of the first mobile ²device 302a, the ²trajectory 308b of the second mobile device 302b, and the trajectory 308c of ²the third mobile²14²<DP=15>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²device 302c as the users of these devices move around over a period of time. ²Suppose that ²a first observation event occurs at time ti in which the third mobile device ²302c observes ²the first mobile device 302a and both devices 302a, 302c determine and ²remember the ²corresponding position and orientation information to determine two ²constraints for²calculating the relative pose between these devices 302a, 302c (in the manner ²described²above). Then, a second observation event occurs at time t2 in which the second ²mobile ²device 302b observes the third mobile device 302c and both devices 302b, 302c ²determine ²and remember the corresponding position and orientation information for ²determining two ²additional constraints. Subsequently, a third observation event occurs at time ²t3 in which²the first mobile device 302a observes the third mobile device 302c and both ²devices 302a,²302c determine and remember the corresponding position and orientation ²information for ²determining two additional constraints. This would produce enough constraints ²to compute ²the 6DoF transformation for aligning the respective coordinate systems because ²the first ²mobile device 302a and the second mobile device 302b are considered to be a ²"single"²mobile device 312 for purposes of aligning their shared coordinate system with ²the third²mobile device 302c.²
[0065] As indicated above, the typical solution to register the ²coordinate system for a ²plurality of mobile devices is to exchange three-dimensional map information ²between the ²mobile devices. This three-dimensional map information is typically determined ²in²accordance with a simultaneous localization and mapping (SLAM) algorithm. A 3D ²map²that is created using a SLAM algorithm may be referred to herein as a SLAM ²map. In ²accordance with the present disclosure, once the coordinate systems of two or ²more mobile ²devices are aligned as described herein, the SLAM maps that are being ²constructed by each ²mobile device can be merged into a larger, more complete SLAM map.²
[0066] Figure 4 illustrates a first mobile device 402a that creates a first ²SLAM map 414a²and a second mobile device 402b that creates a second SLAM map 414b. If the ²first mobile ²device 402a and the second mobile device 402b align their coordinate systems ²with one ²another based on observation events in accordance with the techniques ²disclosed herein, ²then the first SLAM map 414a and the second SLAM map 414b may be merged into a²larger, more complete SLAM map, which is illustrated in Figure 4 as a merged ²SLAM map²416. The merged SLAM map 416 may include information from both the first SLAM ²map ²414a and the second SLAM map 414b. The merged SLAM map 416 may be used by both ²²the first mobile device 402a and the second mobile device 402b.²
[0067] Being able to merge SLAM maps in this way may enable easier ²localization of²<DP=16>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²additional mobile devices. In addition, it may enable persistence (sharing ²coordinate ²systems across time) based on more complete maps of the surrounding ²environment. In ²other words, the techniques disclosed herein may also improve traditional 3D ²map-based ²sharing and persistence approaches. The disclosed techniques may provide ²particular²benefits in connection with merging SLAM maps that have no direct visual ²overlap that²would allow for traditional image based alignment. For example, suppose that ²the first ²SLAM map 414a and the second SLAM map 414b are generated from opposing ²viewpoints. ²This may be the case, for instance, if the first mobile device 402a and the ²second mobile ²device 402b are positioned opposite each other (e.g., across the table from ²one another) such²that there is no overlap between the field of view of the camera of the first ²mobile device²402a and the field of view of the camera of the second mobile device 402b. In ²this example, ²the merged SLAM map 416 would include map information about a much larger area ²than ²either the first SLAM map 414a or the second SLAM map 414b individually.²
[0068] Figure 5 illustrates a method 500 for obtaining a relative ²localization between a²plurality of mobile devices in accordance with the present disclosure. For the ²sake of clarity,²the method 500 will be described in relation to some of the systems, devices, ²components, ²and data described previously. The method 500 will be described from the ²perspective of a ²first mobile device 102a that is interacting with a second mobile device 102b. ²As before, the ²user of the first mobile device 102a will be referred to as the first user ²104a, and the user of² the second mobile device 102b will be referred to as the second user 104b.²
[0069] In step 502, the first mobile device 102a observes the second ²mobile device 102b ²within a field of view of the camera of the first mobile device 102a at a ²first point in time ²(ti). This may occur as a result of the first user 104a aiming the first ²mobile device 102a at ²the second user 104b. The second user 104b may be holding or wearing the ²second mobile²device 102b. The second user 104b may be aiming the second mobile device 102b ²at the²first user 104a.²
[0070] In step 504, the first mobile device 102a determines the position ²of the first ²mobile device 102a at time ti (phi). The value of plu is represented in a ²first coordinate ²system that is used by the first mobile device 102a.²
[0071] In step 506, the first mobile device 102a receives from the second ²mobile device²102b the position of the second mobile device 102b at time ti (p2ti). The ²value of p2ti is ²represented in a second coordinate system that is used by the second mobile ²device 102b.²
[0072] In step 508, the first mobile device 102a determines information ²about the ²orientation of the first mobile device 102a with respect to the second mobile ²device 102b at²16²<DP=17>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²time ti based at least in part on the first position (phi) and the observation ²of the second ²mobile device 102b at time ti. For example, the first mobile device 102a may ²determine ²and remember a geometric line in space (linett) corresponding to the direction ²along which ²the second mobile device 102b was observed at time ti.²
[0073] In step 510, the first mobile device 102a identifies two constraints ²that relate the²first coordinate system used by the first mobile device 102a to the second ²coordinate system ²used by the second mobile device 102b based at least in part on the second ²position (p2t1) ²and the orientation information (as represented, for example, by linett). For ²example, the ²geometric constraint that the position of the second mobile device 102b at ²time ti (p2t1)²should be located along the previously defined line in space (linett) provides ²two²mathematical constraints to align the first coordinate system used by the ²first mobile device ²102a with the second coordinate system used by the second mobile device 102b.²
[0074] Steps 502 through 510 of the method 500 correspond to a single ²observation ²event. In some embodiments, an observation event like the one represented by ²steps 502²through 510 may be triggered by user input. More specifically, when in step ²502 the first²mobile device 102a observes the second mobile device 102b within its camera's ²field of ²view at time ti, the first user 104a may provide some type of user input ²(e.g., clicking a ²button on the first mobile device 102a). Steps 504 through 510 may be ²performed in ²response to receiving this user input.²
[0075] In some embodiments, the method 500 may be performed during a game ²that²involves the first mobile device 102a and the second mobile device 102b. The ²user input ²that causes steps 504 through 510 to be performed may be provided as part of ²the game. For ²example, the game may be a shooting game in which the users 104a¨b are ²supposed to shoot ²at each other using the mobile devices 102a¨b. Step 502 of the method 500, in ²which the²first mobile device 102a observes the second mobile device 102b within its ²camera's field²of view, may occur as a result of the first user 104a aiming the first mobile ²device 102a at ²the second user 104b, who may be holding or wearing the second mobile device ²102b. The ²user input that causes steps 504 through 510 to occur may be provided when the ²first user ²104a takes some action (e.g., activating a trigger button on the first mobile ²device 102a) that²causes the first mobile device 102a to shoot at the second user 104b as part ²of the game.²
[0076] Alternatively, an observation event like the one represented by ²steps 502 through ²510 may be triggered by when one mobile device automatically detects another ²mobile ²device in its camera's field of view. More specifically, when in step 502 the ²first mobile ²device 102a observes the second mobile device 102b within its camera's field ²of view at²17²<DP=18>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²time ti, the first mobile device 102a may automatically detect the second ²mobile device ²102b (e.g., via an object recognition algorithm). Steps 504 through 510 of the ²method 500 ²may be performed in response to the first mobile device 102a automatically ²detecting the ²second mobile device 102b. In other words, the first mobile device 102a may ²perform steps²504 through 510 of the method 500 in response to the first mobile device 102a ²detecting the²second mobile device 102b, without the need for any additional user input. In ²some ²embodiments, the second mobile device 102b may include a light emitter. If ²this light ²emitter is activated when the first mobile device 102a observes the second ²mobile device ²102b within its camera's field of view at time ti, then this activated light ²emitter may make²it easier for the first mobile device 102a to automatically detect the second ²mobile device²102b.²
[0077] In step 502, when the first mobile device 102a observes the ²second mobile device ²102b within its camera's field of view, there may be one or more additional ²mobile devices ²that are also visible within the field of view. (For example, as shown in ²Figure 2, a plurality²of mobile devices 202a¨b may be visible in a camera's field of view at the ²same time.) Thus,²in some embodiments, the method 500 may additionally involve distinguishing ²between the ²second mobile device 102b and one or more additional mobile devices that are ²also visible ²within the camera's field of view. This may be accomplished in various ways, ²such as ²through the use of an image recognition algorithm or a random sample consensus² (RANSAC) algorithm, as discussed above.²
[0078] The observation event represented by steps 502 through 510 ²provides two ²constraints that relate the first coordinate system used by the first mobile ²device 102a to the ²second coordinate system used by the second mobile device 102b. Once at least ²six ²constraints have been accumulated, then in step 512 the first mobile device ²102a may²calculate the pose of the first mobile device 102a relative to the second ²mobile device 102b.²For example, the first mobile device 102a may use the six (or more) ²constraints to calculate ²a 6DoF transformation that relates the first coordinate system used by the ²first mobile device ²102a to the second coordinate system used by the second mobile device 102b.²
[0079] In some embodiments, the six (or more) constraints that are used ²to calculate the².. pose in step 512 may include (a) a plurality of constraints that are ²determined as a result of²observation events, and (b) one or more additional constraints that are known ²independently ²of observation events. Some examples of these additional constraints include a ²known ²common vertical orientation, a height above ground, a dominant wall ²orientation, and so ²forth. These additional constraints can be used to obtain more accurate and / or ²robust results.²18²<DP=19>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²
[0080] Figure 6 illustrates certain components that may be included ²within a mobile ²device 602 that is configured to implement the techniques disclosed herein. ²The mobile ²device 602 shown in Figure 6 represents one possible implementation of the ²mobile devices ²102a¨b, 202a¨b, 302a¨c, 402a¨b that were described previously.²
[0081] The mobile device 602 includes a processor 601. The processor 601 ²may be a²general purpose single- or multi-chip microprocessor (e.g., an Advanced RISC ²(Reduced ²Instruction Set Computer) Machine (ARM)), a special purpose microprocessor ²(e.g., a ²digital signal processor (DSP)), a microcontroller, a programmable gate array, ²etc. The ²processor 601 may be referred to as a central processing unit (CPU). Although ²just a single²processor 601 is shown in the mobile device 602 of Figure 6, in an alternative ²configuration,²a combination of processors (e.g., an ARM and DSP) could be used.²
[0082] The mobile device 602 also includes memory 603 in electronic ²communication ²with the processor 601. The memory 603 may be any electronic component capable ²of ²storing electronic information. For example, the memory 603 may be embodied as ²random²access memory (RAM), read-only memory (ROM), magnetic disk storage media, ²optical²storage media, flash memory devices in RAM, on-board memory included with the ²processor 601, erasable programmable read-only memory (EPROM), electrically ²erasable ²programmable read-only memory (EEPROM) memory, registers, and so forth, ²including ²combinations thereof²
[0083] Instructions 605 and data 607 may be stored in the memory 603. The ²instructions²605 may be executable by the processor 601 to implement some or all of the ²steps, ²operations, actions, or other functionality disclosed herein. Executing the ²instructions 605 ²may involve the use of the data 607 that is stored in the memory 603. Unless ²otherwise ²specified, any of the various examples of modules and components described ²herein may be²implemented, partially or wholly, as instructions 605 stored in memory 603 and ²executed²by the processor 601. Any of the various examples of data described herein may ²be among ²the data 607 that is stored in memory 603 and used during execution of the ²instructions 605 ²by the processor 601.²
[0084] The instructions 605 may include a device tracking module 621. ²The device²tracking module 621 may be configured to track the position and the motion of ²the mobile²device 602 in space. The mobile device 602 may include various components that ²facilitate ²such tracking, including one or more cameras 631, an inertial measurement unit ²(IMU) 633, ²and one or more other sensors 635. The device tracking module 621 may perform ²optical ²based tracking based on input received from the camera(s) 631. In some ²embodiments, the²19²<DP=20>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²device tracking module 621 may also utilize input received from the IMU 633 ²and the ²sensor(s) 635 to enhance tracking. The tracking can comprise, for example, six ²degrees of ²freedom (6DoF) device tracking. This can be implemented using simultaneous ²localization ²and mapping (SLAM) and / or visual-inertial odometry (VI). The device tracking ²module² 621 may be configured to construct and / or update a SLAM map 614 of an unknown²environment while simultaneously keeping track of the location of the mobile ²device 602 ²within the environment.²
[0085] The instructions 605 may also include a relative localization ²module 623. The ²relative localization module 623 may be configured to determine the relative ²localization²between the mobile device 602 and one or more other mobile devices in ²accordance with²the techniques disclosed herein. For example, the relative localization module ²623 may be ²configured to implement methods such as the method 500 shown in Figure 5.²
[0086] The mobile device 602 may include a display 637. In some ²embodiments (e.g., ²in embodiments where the mobile device 602 is a mixed-reality device), the ²display 637²may include one or more semitransparent lenses on which images of virtual ²objects may be²displayed. Different stereoscopic images may be displayed on the lenses to ²create an ²appearance of depth, while the semitransparent nature of the lenses allows the ²user to see ²both the real world as well as the virtual objects rendered on the lenses. The ²mobile device ²602 may also include a graphics processing unit (GPU) 639. The processor(s) ²601 may²direct the GPU 639 to render the virtual objects and cause the virtual objects ²to appear on²the display 637.²
[0087] The mobile device 602 may also include a light emitter 641. The ²light emitter ²641 may be activated in order to make it easier for the mobile device 602 to ²be detected by ²another mobile device. In some embodiments (e.g., if the mobile device 602 is ²a mobile²phone), the light emitter 641 may be a flashlight or a privacy light. ²Alternatively, the light²emitter 641 may be an infrared emitter that is used by some three-dimensional ²cameras on ²some head-mounted displays (e.g., in mixed-reality headsets). In some ²embodiments, one ²or more of the cameras 631 of the mobile device 602 may be an infrared / depth ²camera that ²is configured to detect infrared light emitted by another mobile device.²
[0088] The mobile device 602 may also include one or more input devices ²611. The²input device(s) 611 may be used to provide user input for triggering ²observation events. In ²some embodiments, the input device(s) 611 may include one or more buttons. ²Some other ²examples of input devices 711 include a keyboard, mouse, microphone, remote ²control ²device, button, joystick, trackball, touchpad, and lightpen.²<DP=21>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²
[0089] The mobile device 602 may also include one or more communication ²interfaces ²613 for communicating with other electronic devices, such as other mobile ²devices. For ²example, the mobile device 602 may use a communication interface 613 to send ²position ²information and / or orientation information to another mobile device in ²connection with²determining its pose with respect to the other mobile device. The ²communication²interface(s) 613 may be based on wired communication technology, wireless ²communication technology, or both. Some examples of communication interfaces ²613 ²include a Universal Serial Bus (USB), an Ethernet adapter, a wireless adapter ²that operates ²in accordance with an Institute of Electrical and Electronics Engineers (IEEE) ²802.11²wireless communication protocol, a Bluetooth wireless communication adapter, ²and an²infrared (IR) communication port.²
[0090] The techniques described herein may be implemented in hardware, ²software, ²firmware, or any combination thereof, unless specifically described as being ²implemented ²in a specific manner. Any features described as modules, components, or the ²like may also²be implemented together in an integrated logic device or separately as ²discrete but²interoperable logic devices. If implemented in software, the techniques may be ²realized at ²least in part by a non-transitory computer-readable medium having computer-²executable ²instructions stored thereon that, when executed by at least one processor, ²perform some or ²all of the steps, operations, actions, or other functionality disclosed ²herein. The instructions²may be organized into routines, programs, objects, components, data ²structures, etc., which²may perform particular tasks and / or implement particular data types, and which ²may be ²combined or distributed as desired in various embodiments.²
[0091] The steps, operations, and / or actions of the methods described ²herein may be ²interchanged with one another without departing from the scope of the claims. ²In other²words, unless a specific order of steps, operations, and / or actions is ²required for proper²functioning of the method that is being described, the order and / or use of ²specific steps, ²operations, and / or actions may be modified without departing from the scope of ²the claims.²
[0092] The term "determining" (and grammatical variants thereof) ²encompasses a wide ²variety of actions and, therefore, "determining" can include calculating, ²computing,²processing, deriving, investigating, looking up (e.g., looking up in a table, ²a database or²another data structure), ascertaining and the like. Also, "determining" can ²include receiving ²(e.g., receiving information), accessing (e.g., accessing data in a memory) ²and the like. Also, ²"determining" can include resolving, selecting, choosing, establishing and the ²like.²
[0093] The terms "comprising," "including," and "having" are intended to ²be inclusive²21²<DP=22>²CA 03132049 2021-08-30²WO 2020 / 190387 PCT / US2020 / 015753²and mean that there may be additional elements other than the listed elements. ²Additionally, ²it should be understood that references to "one embodiment" or "an embodiment" ²of the ²present disclosure are not intended to be interpreted as excluding the ²existence of additional ²embodiments that also incorporate the recited features. For example, any ²element or feature²described in relation to an embodiment herein may be combinable with any ²element or²feature of any other embodiment described herein, where compatible.²
[0094] The present disclosure may be embodied in other specific forms ²without ²departing from its spirit or characteristics. The described embodiments are to ²be considered ²as illustrative and not restrictive. The scope of the disclosure is, ²therefore, indicated by the²appended claims rather than by the foregoing description. Changes that come ²within the²meaning and range of equivalency of the claims are to be embraced within their ²scope.²22²
Claims
<DP=1>²CLAIMS²1. A method for obtaining a relative localization between a plurality of ²mobile devices, ²the method being performed by a first mobile device, the method comprising:²observing a second mobile device within a field of view of a camera of the ²first ²mobile device at a first point in time;²determining a first position of the first mobile device at the first point in ²time, the ²first position being represented in a first coordinate system used by the ²first mobile device;²receiving from the second mobile device a second position of the second mobile ²²device at the first point in time, the second position being represented in a ²second coordinate ²system used by the second mobile device;²determining orientation information about an orientation of the first mobile ²device ²with respect to the second mobile device at the first point in time based at ²least in part on ²the first position and the observation of the second mobile device at the ²first point in time;²identifying two constraints that relate the first coordinate system used by ²the first ²mobile device to the second coordinate system used by the second mobile device ²based at ²least in part on the second position and the orientation information; and²calculating a pose of the first mobile device relative to the second mobile ²device ²once at least six constraints have been accumulated.²2. The method of claim 1, further comprising receiving user input when the ²second ²mobile device is observed within the field of view of the camera of the first ²mobile device ²at the first point in time, wherein the first position is determined, the ²second position is ²received from the second mobile device, the orientation information is ²determined, and the ²two constraints are identified in response to receiving the user input.²3. The method of claim 2, wherein:²the method is performed during a game that involves the plurality of mobile²devices; and²the user input is provided as part of the game.²4. The method of claim 1, further comprising automatically detecting the ²second ²mobile device within the field of view of the camera of the first mobile ²device at the first ²point in time, wherein the first position is determined, the second position ²is received from ²the second mobile device, the orientation information is determined, and the ²two ²constraints are identified in response to automatically detecting the second ²mobile device.²5. The method of claim 4, wherein automatically detecting the second mobile ²device ²comprises automatically detecting an activated light emitter of the second ²mobile device.²23²<DP=2>²6. The method of claim 1, wherein:²the second mobile device and at least one additional mobile device are both ²visible ²within the field of view of the camera of the first mobile device at the first ²point in time; ²and²the method further comprises distinguishing between the second mobile device ²and ²the at least one additional mobile device.²7. The method of claim 1, wherein determining the orientation information ²comprises ²determining a geometric line in space corresponding to a direction along which ²the second ²mobile device was observed at the first point in time.²8. The method of claim 1, wherein calculating the pose comprises ²calculating a six ²degrees of freedom transformation that relates the first coordinate system ²used by the first ²mobile device to the second coordinate system used by the second mobile ²device.²9. The method of claim 1, wherein the at least six constraints comprise:²a plurality of constraints that are determined as a result of observation ²events; and ²one or more additional constraints that are known independently of the ²observation²events.²10. The method of claim 1, further comprising:²creating a first simultaneous localization and mapping (SLAM) map based on the²field of view of the camera of the first mobile device; and²merging the first SLAM map with a second SLAM map that is created by the²second mobile device.²11. A first mobile device that is configured to obtain a relative ²localization between the ²first mobile device and a second mobile device, comprising:²a camera;²one or more processors;²memory in electronic communication with the one or more processors; and²instructions stored in the memory, the instructions being executable by the ²one or²more processors to:²observe the second mobile device within a field of view of the camera of the ²first mobile device at a first point in time;²determine a first position of the first mobile device at the first point in ²time, ²the first position being represented in a first coordinate system used by the ²first ²mobile device;²receive from the second mobile device a second position of the second²24²<DP=3>²mobile device at the first point in time, the second position being ²represented in a ²second coordinate system used by the second mobile device;²determine orientation information about an orientation of the first mobile ²device with respect to the second mobile device at the first point in time ²based at ²least in part on the first position and the observation of the second mobile ²device at ²the first point in time;²identify two constraints that relate the first coordinate system used by the ²first mobile device to the second coordinate system used by the second mobile ²device based at least in part on the second position and the orientation ²information; ²and²calculate a pose of the first mobile device relative to the second mobile ²device once at least six constraints have been accumulated.²12. The first mobile device of claim 11, further comprising an input device ²that is ²configured to receive user input when the second mobile device is observed ²within the ²field of view of the camera of the first mobile device at the first point in ²time, wherein the ²first position is determined, the second position is received from the second ²mobile device, ²the orientation information is determined, and the two constraints are ²identified in²response to receiving the user input.²13. The first mobile device of claim 11, further comprising additional ²instructions that ²are executable by the one or more processors to automatically detect the ²second mobile ²device within the field of view of the camera of the first mobile device at ²the first point in ²time, wherein the first position is determined, the second position is ²received from the ²second mobile device, the orientation information is determined, and the two ²constraints ²are identified in response to automatically detecting the second mobile ²device.²14. The first mobile device of claim 11, wherein the instructions that are ²executable by ²the one or more processors to determine the orientation information comprise ²instructions ²that are executable by the one or more processors to determine a geometric ²line in space ²corresponding to a direction along which the second mobile device was observed ²at the ²first point in time.²15. The first mobile device of claim 11, wherein the instructions that are ²executable by ²the one or more processors to calculate the pose comprise instructions that ²are executable ²by the one or more processors to calculate a six degrees of freedom ²transformation that ²relates the first coordinate system used by the first mobile device to the ²second coordinate ²system used by the second mobile device.²
