Multiple integrated models for device positioning

Abstract

A system and method determine a location of a device. The device collects sensor data using one or more sensors. Based on the sensor data, one or more positioning models are selected from a plurality of positioning models. The selected models are applied to generate one or more potential locations. A current location of the device is determined based on the one or more potential locations.

Description

[0001] Divisional

[0002] This application is a continuation of International Application No. PCT / US2019 / 039606, International Filing Date June 27, 2019, Chinese National Application No. 201980043077.8, which claims priority to U.S. Provisional Application No. 62 / 858, 1 10, filed June 12, 2019, the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to location determination, and more specifically to determining device location using sensor data. BACKGROUND

[0004] Computer-mediated reality technology allows users with handheld or wearable devices to change their perception of the visual or audible aspects of their environment as experienced through the device. Augmented reality (AR) is a type of computer-mediated reality that specifically changes the perception of a physical, real-world environment using sensory data generated at a computing device. AR technology can provide computer-mediated reality that includes a visualization of a user's real-world environment that includes one or more augmentations, such as a virtual object at a particular location within the visualization of the real-world environment, such that the virtual object appears to be an immersive aspect of the real world. SUMMARY

[0005] According to particular embodiments, a method includes generating computer-mediated reality data, the method comprising: generating three-dimensional (3-D) map data and camera location data at a client device; transmitting the 3-D map data and the client data to a mobile server; receiving world map data at the client device from the remote server, wherein the world map data is generated using the 3-D map data; and generating a computer-mediated reality image at the client device.

[0006] According to another particular embodiment, the augmented reality engine comprises a locally stored animation engine running on a portable computer, the animation engine comprising: a first input receiving a stream of digital images produced by a camera integrated in the portable computer, the digital images representing a near real-time view of the environment as seen by the camera; a second input receiving a geo-positioning from a geo-positioning system integrated in the portable computer; a 3D mapping engine receiving the first and second inputs and estimating a distance between the camera positioning and one or more mapping points at a particular point in time; an output comprising the stream of digital images produced by the camera overlaid with a computer-generated image, wherein the computer-generated image is located at a particular positioning in the 3D map and remains located at this particular positioning as the user moves the camera to different positionings in the space; and a non-locally stored object detection engine in network communication with the locally stored animation engine; a first input is received from the locally stored animation engine, the first input comprising a digital image from the stream of digital images produced by the camera; a second input is received from the locally stored animation engine, the second input comprising a geo-positioning associated with the digital image received from the locally stored animation engine.

[0007] In various embodiments, a device collects sensor data (e.g., images, audio), global positioning system (GPS) data, Wi-Fi availability data, orientation and motion data (e.g., via a gyroscope, accelerometer, inertial measurement unit (IMU), etc.), light levels, sound levels, pressure data, etc. Based on the sensor data, the device selects one or more localization models for use in determining a location of the device. The selected one or more models are applied to the sensor data to determine one or more potential locations for the device, and a current location of the device is determined from the one or more potential locations. In the event there is more than one potential location, the current location can be determined as an average or weighted average of the potential locations, by selecting one of the potential locations, or by other suitable techniques. In some embodiments, the selection of one or more models and / or one or more potential locations can be based on which one or more models were previously used in similar locations. Two or more locations can be considered similar based on general geographic proximity and / or similarity in one or more sensor data, or based on determinations of the sensor data (e.g., indoors vs. outdoors, low lighting vs. high lighting, etc.).

[0008] Other advantages of the present disclosure are described herein. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 FIGURE illustrates an embodiment of a networked computing environment for generating and displaying augmented reality data.

[0010] Figure 2 FIGURE illustrates a process executable by a computing system of Figure 1 for generating and displaying augmented reality data.

[0011] Figure 3 FIGURE illustrates a block diagram of an example computer system suitable for use in Figure 1 a networked computing environment of

[0012] Figure 4 is a flowchart illustrating a method for determining a location of a device according to one embodiment. DETAILED DESCRIPTION

[0013] A system and method create a three-dimensional (3-D) map and use the 3-D map to enable augmented reality (AR). In one embodiment, the 3-D map has a resolution of one centimeter, although maps having other resolutions can be used, including maps having variable resolutions. In various embodiments, map creation is achieved on a client (e.g., by a mobile phone or headset) that is paired with a backend server that provides previously compiled imaging and map building information to the client device.

[0014] In one embodiment, the system selects images and global positioning system (GPS) coordinates on a client (e.g., on a handheld or worn electronic computing device) and pairs the selected data with a 3-D map. The 3-D map is built from a camera recording module and an inertial measurement unit (IMU), which can include an accelerometer and / or a gyroscope. The selected data is sent to a server. The server and the client computing device cooperatively process the selected data to establish objects and geometry for AR, and to determine potential interactions between virtual or non-virtual objects in AR. For example, the augmented reality can be an augmented reality of a real-world room, and can include one or more animated virtual objects that can interact with each other (e.g., to play a game).

[0015] By using the selected data and 3-D maps, the system can use neural networks to achieve object detection and geometry estimation, thereby providing AR. An example of a neural network is a computational model for machine learning that uses a large collection of connected units (artificial neurons). The units are connected in software, and if the combined input signal to a unit is large enough, the unit fires its own output signal. In this way, artificial neural networks mimic biological neural networks learning to perform tasks. The system uses deep learning (e.g., multi-layer neural networks) to understand AR data by context (e.g., for the aforementioned object detection and geometry estimation tasks).

[0016] In some embodiments, the system aggregates local 3D maps to create one or more global 3-D maps (e.g., by linking multiple local 3D maps together). The aggregated 3-D maps are combined into a global 3-D map on the server to generate a single digital 3D map of the environment, or "world." For example, multiple local 3-D maps generated by one or more devices for similar GPS coordinates, including portions that match within a predetermined threshold, can be determined to overlap. In this way, the overlapping portions can be used to stitch the two local 3-D maps together. The global 3-D map is used to store the location of virtual objects at particular GPS coordinates, which are further indexed down to particular 3D locations (e.g., within a foot of error) by 3-D points and viewable images. For example, if the "world" is a house, various local 3-D maps of various rooms can be combined into a single global 3-D map that represents the interior of the house as a whole.

[0017] Illustrative processes send map data to and from a server. As described herein, a 3-D map is a collection of 3-D points in space that represent a world or portion of a world in a manner similar to 3-D pixels (or "voxels"). When image data is available and useful, e.g., when it is determined to be image data for a location that has not yet been incorporated into a 3-D map, the image data can be sent along with the 3-D map. Certain embodiments send 3-D maps without image data. Figure 1

[0018] ​In various embodiments, the client device generates a 3-D map using a 3-D algorithm executed by the processor. The client device sends images, 3-D maps, and GPS data in an efficient manner. For example, images can be selectively sent so as not to stall transmission or processing. In one example, images can be selectively sent when the camera of the client device has a novel viewpoint (e.g., the space captured by the image has not been included in one or more images at the server, or such images at the server are older than a threshold amount of time), but not when one or more images for the current viewpoint have already been provided. For example, images are designated for sending by the algorithm when the field of view of the camera has minimal (e.g., less than a threshold) overlap with previous images in past or recent camera poses, or when the viewpoint has not been observed for an amount of time dependent on the expected movement of the object. As another example, an image can be provided if more than a threshold amount of time has lapsed since a previous image in the current (or approximately overlapping) viewpoint was provided. This can enable stored images associated with the map to be updated to reflect the current (or at least recent) state of the real-world location.

[0019] In various embodiments, the server device includes a real-time detection system that detects objects based on 3-D data and images, and estimates the geometry of the real-world environment. For example, a 3-D map of a non-realistic room (e.g., a semi-dense and / or dense 3-D reconstruction) can be determinable with images captured by client devices such that the room is fully represented within the 3-D map at a particular resolution or level of detail.

[0020] The server fuses images, 3-D data, and / or GPS data using the real-time detection system to construct a consistent and easily indexed 3-D map of the world, a synthesized real-world map. Once stored, the real-world map can be searched to locate previously stored real and / or virtual objects at particular locations in the 3-D map of the world.

[0021] In various embodiments, map construction (creation of new 3-D map data) and tracking (location of objects, including client devices) are done on the client. A sparse digital reconstruction of the real world is captured along with the location of the camera relative to the real world. Map construction includes creating a point cloud, or collection of 3-D points. By serializing and transmitting the point cloud information, the system communicates the sparse representation back to the server along with GPS data. Cloud processing enables multiplayer capabilities (sharing of map data between independent client devices in real-time or near real-time) and object detection with working physical memory (storing maps and virtual object data on the device for future AR experiences) between two or more client devices. For example,

[0022] The server includes a database of 3-D maps and images. The server uses GPS data to determine whether a 3-D map for that coordinate has been previously stored. If located, the stored 3-D data is transmitted back to the client device. For example, a user at a home location can accept previously stored data associated with the home location, in addition, 3-D map and image data can be added to the stored composite world 3-D map.

[0023] Figure 1 is a block diagram of an AR computing system according to one embodiment, which includes a client device 102 that cooperates with a cloud server element accessed via a network 104. For example, the cloud server element can be a component of a server device configured to produce AR data. In the illustrated embodiment, the client device 102 includes a game engine 106 (e.g., a Unity game engine or another physics / rendering engine) and an AR platform 108. The AR platform 108 can perform segmentation and object recognition. Figure 1 The AR platform 108 illustrated in the middle includes a complex computer vision module 110 that performs client image processing, including image segmentation and local 3-D estimation, among other things.

[0024] The complex computer vision module 110 ingests pixel data from a camera of the client device and sensor data from other sensors of the client device, such as an inertial measurement unit. The complex computer vision module 110 generates a feature vector by performing feature extraction on the pixel data, triangulating the position and viewpoint of the camera based on the sensor data and / or the pixel data, and performing feature matching on the extracted features, thereby providing an N-point perspective (PnP) estimate of the camera viewpoint and what objects are in the viewpoint. As described below, different models can be employed to perform simultaneous localization and mapping, depending on the context.

[0025] The AR platform 108 also includes a localization and map building module 112. In one embodiment, the localization and map building module 112 uses functionality from a simultaneous localization and mapping (SLAM) software development kit (SDK). The SLAM SDK functionality includes a map building system that builds a cloud of points, a cloud of lines, a cloud of planes, or any other cloud of geometric shapes for approximating real world geometry, and provides a tracking function to discover the current location of a camera in three-dimensional space. The SLAM process further projects animations or augmented values (e.g., virtual objects) into AR so that they appear to be in the real world. In other embodiments, the localization and map building module 112 can use different or additional methods to map an environment around the client device 102 and / or determine the location of the client device in that environment.

[0026] The localization and map building module 112 can have a plurality of models that can be used to determine the current location of the client device 102. In one embodiment, the models include point cloud based models (e.g., as provided by SLAM), plane matching models, line matching models, geographic information system (GIS) models, building recognition models, landscape recognition models, cube matching models, cylinder matching models, horizon matching models, light source matching models, and object and / or semantic matching models such as exit matching models, street light matching models, tree matching models, text matching models (e.g., for billboards or street signs), or other models for matching real world objects to virtual approximations. In other embodiments, different or additional models can be available to the localization and map building module 112.

[0027] A point cloud model represents a physical space as a collection of points located in a virtual world that correspond to locations of object surfaces in the real world. For example, a wall can be represented by a collection of points distributed over the wall. A given point cloud model can include a fixed number of points that are dynamically assigned to represent the physical space, such as by including a relatively small number of points on large flat surfaces (e.g., walls) while including denser clusters of points around more complex objects (e.g., plants, furniture, etc.). In some embodiments, the localization and map building module 112 has a plurality of point cloud models available to it that have varying resolutions and scales. Thus, for example, objects near the user's location can be represented by higher resolution point clouds than those of objects farther away. The point cloud models determine the location of a sensor by matching sensor data (e.g., camera images) generated by the device to expected sensor readings for different locations that are generated based on the point cloud.

[0028] A plane matching model represents the physical space as a set of planes in the virtual that are at locations corresponding to large, relatively flat surfaces in the real world. For example, each of the floor, the walls, and the tabletops in a room can be represented by a plane in the plane matching model. The plane matching model determines the position of a device by matching planes identified in sensor data (e.g., camera images) generated by the device to planes in the model.

[0029] A line matching model is similar to a plane matching model except that it defines lines in the virtual world at locations corresponding to lines in the real world. For example, a line can be the edge of a piece of furniture, the joint between two walls, etc. The line matching model determines the position of a device by matching lines identified in sensor data (e.g., camera images) generated by the device to lines in the model.

[0030] A geographic information system (GIS) model matches pixel data to satellite imaging data. The GIS estimates a viewpoint based on objects detected at the horizon of the viewpoint. The size and scale of the objects at the horizon are estimated by the GIS model based on the pixel data (e.g., a number of pixels wide and / or high). The GIS model then matches the estimated objects to objects included in the satellite imaging data that the GIS model projects to the horizon of the viewpoint. This can be performed by applying one or more mathematical transformations and / or machine-learned models (e.g., deep neural networks) to the satellite imaging data and / or the pixel data. Based on the matched objects, the GIS model identifies a likely viewpoint of the camera (and, by extension, the client device 102) that generated the pixel data.

[0031] An architectural recognition model, and other object and / or semantic matching models (e.g., a tree recognition model), match pixel data to an object or semantic that it is trained to recognize. For example, an architectural recognition model can be a machine-learned model trained to recognize buildings based on a training data set of images of buildings labeled as positive cases and images of other objects labeled as negative cases. After training, the architectural recognition model can predict whether pixel data (e.g., an image) includes a building. Similarly, models can be trained to recognize various objects that can be present in an image, such as street signs, trees, benches, etc.

[0032] Regardless of the set of available models, the client device 102 selects one or more of the models to use for localization based on the collected raw data. For example, a plane and line matching model can be most accurate indoors, while a landscape and building model can be most accurate outdoors. Thus, the collected data (e.g., lighting levels and images) can be analyzed by the localization and map building module 112 to determine whether the client device 102 is currently indoors or outdoors, and then select the more effective model.

[0033] In one embodiment, each model is assigned a score that indicates the likelihood that the model will generate an accurate position for the client device based on the current circumstances (e.g., as indicated by the collected data). The score can be based on a decision tree, where the parameters of the decision tree are set manually. Alternatively, the score can be based on a machine learning model trained with reinforcement learning that uses inputs and parameters from different models of different AR map types, where the reinforcement “reward” is proportional to the model performance (e.g., position estimation accuracy).

[0034] The model or models are selected based on the scores. For example, the client device 102 can select the model with the highest score, the three highest scoring models, or all models with scores above a threshold. Additionally or alternatively, the selection can be based on which models were previously selected at similar locations. For example, if the GPS coordinates for the client device 102 are within a threshold of GPS coordinates for which a plane matching model was previously successfully used, the score for the plane matching model can be boosted, or the plane matching model can be selected unless the score for another model exceeds the score for the plane matching model by a threshold amount. Similarly, if a particular model was previously selected but failed to provide an accurate position, the score for that model can be reduced. Thus, successes and failures of models at similar GPS coordinates can be used to inform the client device 102 which model or models to select to determine a more precise position (e.g., to within a centimeter) than provided by the GPS coordinates.

[0035] In a similar manner, the client device 102 can factor in the success and failure of models at locations similar to the current location as indicated by the collected data. For example, locations can be classified as being indoors or outdoors, and based on the collected data indicating whether the client device 102 is indoors or outdoors, certain models can be excluded or selected (or the corresponding scores increased or decreased). By employing models for objects that are typically indoors or outdoors, the collected data can be analyzed by one or more models (e.g., object recognition models) to determine whether it was collected indoors or outdoors. For example, using models for trees, sky, and grass to determine that the data was collected outside, and models for walls, sofas, and tables to determine that the data was collected inside.

[0036] Similarly, the selected model or models can be based on lighting levels. For example, one model (e.g., line cloud) can be particularly effective in low light conditions relative to other models, and thus is preferentially selected if the lighting level detected by the client device 102 is below a threshold. Similarly, other models (e.g., point cloud) can be less effective in low light conditions, and thus are avoided (assuming another model is available). Depending on the embodiment, other techniques for the selected data can be employed that work to influence which model or models are selected to favor selection of those models that are most likely to produce accurate results in the current conditions.

[0037] The localization and mapping module 112 can apply the selected model or models to generate a potential location for the client device 102. In the case where only a single model is selected, the generated location can be used as the actual location of the client device 102. However, in the case where GPS coordinates are available, if the location determined using the model differs from the GPS coordinates by more than a threshold amount, the client device 102 can apply additional models, or seek to verify the current location, such as by displaying a message requesting the user to confirm whether the location estimate is accurate, and updating the location estimate based on the received response.

[0038] When more than one model is selected, each selected model can generate a potential location for client device 102. The positioning and mapping module 112 determines the current location for client device 102 based on the cumulative potential locations of the selected models. In one embodiment, the current location is calculated as the mean of the potential locations generated by the selected models. In another embodiment, a weighted average of the potential locations is used. For example, the contribution of each model can be weighted by the score of that model from the model selection phase. Alternatively, the weighting can be pre-set to favor a particular model over others, for example, based on historical accuracy. In a further embodiment, potential locations can be combined in other ways, such as taking the potential location that most closely matches the device's GPS coordinates.

[0039] exist Figure 2 In the illustrated embodiment, the client computing device 102 further includes a map retrieval module 114 and a deep learning module 116. The map retrieval module 114 retrieves previously generated 3D maps (e.g., via network 104). In some embodiments, the map retrieval module 114 may locally store some 3D maps (e.g., a 3D map of a user's home location). The deep learning module 116 uses machine learning algorithms for object recognition. The deep learning module 116 may acquire trained machine learning algorithms via network 104. In some embodiments, the deep learning module 116 may also provide object recognition results and / or user feedback for further model training.

[0040] In the illustrated embodiment, the cloud server element accessed via network 104 (e.g., at a server computing device) includes an AR backend engine 118 communicating with a world map building module 120, an object recognition and semantic segmentation module 122, a map database 124, an object database 126, and a deep learning training module 128. In other embodiments, additional or different components may be included. Furthermore, functionality may be distributed differently from the functionality described herein. For example, some or all of the object recognition functionality may be executed at client device 102.

[0041] The AR backend engine receives data from the client device 102 and sends 3D maps and / or other data to the client device 102. Based on the type of data received, the AR backend engine sends the received data to a world map building module 120 and / or an object recognition and semantic segmentation module 122 for further processing.

[0042] A world map building module 120 fuses different local 3-D maps together to create a composite real-world map. As noted previously, GPS positioning data from client devices 102 that initially generate maps can be used to identify local maps that can abut or overlap. Pattern matching can then be used to identify overlapping portions of maps or two local maps that abut each other (e.g., because they include opposite sides of the same object). If two are determined to be overlapping or abutting, map building that indicates how the two maps relate to each other can be stored, or they can be combined into a single 3-D map.

[0043] An object recognition and semantic segmentation module 122 uses image and / or object information and collected 3-D data to identify features in the real world. In this way, for example, the server 104 determines that a chair is at a 3-D location and accesses an object database 126 associated with that location. A deep learning training module 128 can be used to fuse map information with object information, for example, to add a representation of an object to a 3-D map at a location represented in the 3-D map. In this way, the system 100 can connect 3-D information for object recognition and fusion back into 3-D maps. Alternatively or additionally, the deep learning module 128 can be used to train or update a model of machine learning for object recognition. For example, results of applying the object recognition and semantic segmentation module 122 to 3-D data can be verified (e.g., by a human) so that the deep learning training module 128 can expand a training data set or update a model based on the verification.

[0044] A map database 124 includes one or more computer-readable media configured to store map data generated by client devices 102. Map data can include local maps of 3-D point clouds stored in association with image and other sensor data collected by client devices 102 at a location. Map data can also include map building information that indicates geographic relationships between different local maps. Similarly, an object database 126 includes one or more computer-readable media configured to store information about recognized objects. For example, the object database 126 can include a list of known objects (e.g., chairs, desks, trees, buildings, etc.) with corresponding locations and attributes of those objects. These attributes can be generic to an object type or defined specifically for each instance of an object (e.g., all chairs can be considered furniture, but the location of each chair is defined individually). Although the map database 124 and the object database 126 are shown as single entities, they can be distributed across multiple storage media at multiple devices (e.g., as a distributed database), or can be a combined database.

[0045] Figure 1 is a flowchart according to one embodiment that illustrates a process performed by the client device 102 and the server device to generate and display AR data. The client 102 and server computing devices can be similar to those shown in Figure 3 FIG. 1. Dotted lines represent communication of data between the client 102 and the server, while solid lines indicate communication of data within one of the devices. In other embodiments, functionality can be distributed differently between the devices and / or different devices can be used.

[0046] At 202, raw data is collected at the client device by one or more sensors. In one embodiment, the raw data includes images and inertial measurement information. The images are captured by one or more cameras. The inertial measurement information can be collected using GPS and gyroscope and / or accelerometer components on the client device or on another device communicatively connected to the client device. In alternative embodiments, additional raw data can be collected, such as a stress level, a lighting level, a sound level, audio data, etc.

[0047] The client device 102 can maintain a local map store at 204. The local map store includes local point cloud data. The point cloud data includes localizations in space that form a mesh surface that can be constructed such that the point cloud data represents one or more objects in a location.

[0048] If a 3-D map is initialized at 206, the client device 102 can initiate SLAM functionality at 208. The SLAM functionality includes a map building system that builds a point cloud and tracking functionality to find the location of the camera in the space and / or orient the camera. The SLAM process projects virtual objects so that they appear to be in the real world in the AR experience. If no 3-D map is initialized at 206, the client device 102 can identify a stored 3-D map in the local map store 204 or by querying the cloud map store 220 and load the stored 3-D map for use. If no map is located at 210, the system creates a map using an initialization module at 212.

[0049] At 214, in the event that a novel viewpoint is detected (e.g., if a corner is turned to an area that has not yet been mapped / imaged, or in the event that there is overlap and not all of the real-world currently visible portion has been mapped / imaged), the system can record data gathered or inferred (e.g., using object recognition) about the local environment at 216. For example, in response to determining that the client device 102 currently has a novel viewpoint, images captured by the camera at that viewpoint can be sent by the client device 102 to the server. The client device 102 can use the novel viewpoint detector to determine when and how to transmit images with 3-D data. The local environment inference can include updated keyframes for the local map construction system, as well as serialized image and / or map data.

[0050] At the server side, the novel viewpoint data (e.g., including point cloud information with mesh data) can be stored in a cloud map storage at 218. The server can add different portions of the real-world map from the stored cloud map storage 220 as well as an object database 222 to the novel viewpoint data, such as labels for recognized objects using machine learning as described above. The cloud environment inference 224, including the added data such as semantic labels for recognized objects, can be sent back to the client device 102. The added data can include points and meshes as well as object data, and can be stored in the local map storage 204.

[0051] Figure 3 FIG. 3 is a high-level block diagram illustrating an example computer 300 suitable for use as a client device 102 or a server. The example computer 300 includes at least one processor 302 coupled to a chipset 304. The chipset 304 includes a memory controller hub 320 and an input / output (I / O) controller hub 322. A memory 306 and a graphics adapter 312 are coupled to the memory controller hub 320, and a display 318 is coupled to the graphics adapter 312. A storage device 308, keyboard 310, pointing device 314, and network adapter 316 are coupled to the I / O controller hub 322. Other embodiments of the computer 300 have different architectures.

[0052] In Figure 1In the illustrated embodiment, storage device 308 is a non-transitory computer-readable storage medium, such as a hard disk drive, CD-ROM, DVD, or solid-state storage device. Memory 306 stores instructions and data used by processor 302. Pointing device 314 is a mouse, trackball, touchscreen, or other type of pointing device, and it is used in conjunction with keyboard 310 (which may be an on-screen keyboard) to input data into computer system 300. Graphics adapter 312 displays images and other information on display 318. Network adapter 316 couples computer system 300 to one or more computer networks.

[0053] Depend on Figure 4 The type of computer used by the entity may vary depending on the embodiment and the processing power required by the entity. For example, the server may include a distributed database system comprising multiple blade servers working together to provide the described functionality. Furthermore, the computer may lack some of the components described above, such as the keyboard 310, graphics adapter 312, and display 318.

[0054] Figure 4 An embodiment of a method 400 for determining the location of a device is illustrated. Figure 4 The steps are illustrated from the perspective of client device 102 executing method 400. However, some or all of these steps may be performed by other entities or components. Additionally, some embodiments may perform the steps in parallel, in a different order, or perform different steps.

[0055] exist ​ In the illustrated embodiment, method 400 begins with client device 102 collecting (410) sensor data using one or more sensors. Sensor data may include images, video, audio, light level, sound level, inertial data, orientation, pressure, and / or any other available data from the sensors. Based on the sensor data, client device 102 selects (420) one or more positioning models. As previously described, the selected model or models may be based on scores assigned to the models, where the scores are based on sensor data and / or other factors (such as historical performance at a particular location or environment).

[0056] The client device 102 applies 430 the selected positioning model to generate a potential location for the client device. In one embodiment, each model outputs a potential location. The potential location can be defined based on a coordinate system within the local map (e.g., selected based on the GPS coordinates of the client device 102). For example, the GPS coordinates can indicate that the client device 102 is likely somewhere within a building, so a local map for that building is used. The selected model is then applied to determine a more accurate location of the client device 102 within the building (e.g., targeting a precision of one centimeter).

[0057] The client device 102 determines 440 the current location based on the potential location. In the case where only one model is selected, it can be assumed that the potential location it generated is correct. Alternatively, the potential location can be checked for plausibility based on available sensor data (e.g., within a threshold distance from the location indicated by the GPS coordinates, the detected lighting levels are consistent with expectations for the potential location, the images captured by the client device 102 are consistent with other images captured at similar locations, etc.). When more than one model is selected, the potential locations can be combined to yield the current location. As previously described, averaging, weighted combination, or other suitable techniques can be used to generate the current location from the potential locations.

[0058] Those of skill in the art can make many uses of the apparatus and techniques disclosed herein, and many modifications and departures can be made therefrom without departing from the concepts described. For example, components or features illustrated or described as part of the disclosure can be missing, combined, or arranged differently than illustrated or described. Examples of apparatuses according to the disclosure can include fewer or different components than those described with reference to one or more of the preceding figures. Thus, the disclosure is not limited to the specific implementations described herein, but rather the possible scope of the disclosure is to be given that full breadth of equivalents and equivalents thereof as set forth in the appended claims and functions to be given the full breadth of equivalents to which it is entitled.

Claims

1. A method for determining a location of a mobile device, the method comprising: collecting image data captured by a camera on the mobile device; identifying one or more features from the image data; determining that the mobile device is indoors or outdoors based on the identified features; selecting a positioning model from a plurality of positioning models based on the mobile device being indoors or outdoors, including assigning a score to each positioning model in the plurality of positioning models based on the mobile device being indoors or outdoors; providing the image data as input to the selected positioning model, the selected positioning model outputting a candidate location; and determining the location of the mobile device based on the candidate location.

2. The method of claim 1, wherein identifying the one or more features from the image data comprises: applying an object recognition model to the image data to identify a physical object in an environment captured by the image data as a feature of the one or more features.

3. The method of claim 2, wherein determining that the mobile device is indoors or outdoors based on the identified features comprises: characterizing the physical object as an indoor object or an outdoor object, wherein the determining is based on the characterizing of the physical object.

4. The method of claim 3, wherein the object recognition model is configured to identify objects including trees, sky, grass, chairs, tables, street lights, billboards, and road signs.

5. The method of claim 2, wherein selecting the positioning model is further based on the physical object.

6. The method of claim 1, wherein the plurality of positioning models includes a combination of: a point cloud based model, a plane matching model, a line matching model, a geographic information system (GIS) model, a building recognition model, an object recognition model, a semantic matching model, a cube matching model, a cylinder matching model, a horizon matching model, a light source matching model, or a landscape recognition model.

7. The method of claim 1, further comprising: selecting one or more additional positioning models from the plurality of positioning models; and providing the image data as input to the one or more additional positioning models, the one or more additional positioning models outputting one or more additional candidate locations, wherein determining the location of the mobile device is further based on the one or more additional candidate locations. determining an average location of the candidate location and the one or more additional candidate locations.

8. The method of claim 7, wherein determining the location of the mobile device comprises:

9. The method of claim 1, wherein selecting the positioning model is further based on GPS data indicating a current location of the mobile device.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations comprising: collecting image data captured by a camera on the mobile device; identifying one or more features from the image data; determining that the mobile device is indoors or outdoors based on the identified features; selecting a positioning model from a plurality of positioning models based on the mobile device being indoors or outdoors, including assigning a score to each positioning model in the plurality of positioning models based on the mobile device being indoors or outdoors; providing the image data as input to the selected positioning model, the selected positioning model outputting a candidate location; and determining the location of the mobile device based on the candidate location. determine a position of the mobile device based on the candidate position.

11. The storage medium of claim 10, wherein identifying the one or more features from the image data comprises: apply an object recognition model to the image data to identify one or more physical objects in an environment captured by the image data as the one or more features.

12. The storage medium of claim 11, wherein determining that the mobile device is indoors or outdoors based on the identified feature comprises: characterize the physical object as an indoor object or an outdoor object, wherein the determining is based on the characterization of the physical object.

13. The storage medium of claim 12, wherein the object recognition model is configured to identify objects including trees, sky, grass, chairs, tables, street lights, billboards, and road signs.

14. The storage medium of claim 11, wherein selecting the localization model is further based on the physical object.

15. The storage medium of claim 10, wherein the plurality of localization models includes a combination of: a point cloud based model, a plane matching model, a line matching model, a geographic information system (GIS) model, a building recognition model, an object recognition model, a semantic matching model, a cuboid matching model, a cylinder matching model, a horizon matching model, a light source matching model, and a landscape recognition model.

16. The storage medium of claim 10, wherein the operations further comprise: selecting one or more additional localization models from the plurality of localization models; and providing the image data as input to the one or more additional localization models, the one or more additional localization models outputting one or more additional candidate positions, wherein determining the position of the mobile device is further based on the one or more additional candidate positions.

17. The storage medium of claim 16, wherein determining the location of the mobile device comprises: determining an average position of the candidate position and the one or more additional candidate positions.

18. The storage medium of claim 10, wherein selecting the localization model is further based on GPS data indicating a current position of the mobile device.

19. The storage medium of claim 10, wherein the computing device is the mobile device.

20. A device comprising: a processor; and a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising: collecting image data captured by a camera on a mobile device; identifying one or more features from the image data; determining whether the mobile device is indoors or outdoors based on the identified features; selecting a localization model from a plurality of localization models based on whether the mobile device is indoors or outdoors, including assigning a score to each localization model in the plurality of localization models based on whether the mobile device is indoors or outdoors; providing the image data as input to the selected localization model, the selected localization model outputting a candidate position; and determining a position of the mobile device based on the candidate position. ​

Images