Privacy filtering of area description files prior to upload

Abstract

A mobile device includes at least one imaging sensor for acquiring images of the mobile device's environment, a privacy filter module, a spatial feature detection module, a collection module, and a network interface. The privacy filter module performs at least one image-based privacy filtering process using the acquired images to generate a filtered image. The spatial feature detection module determines a set of spatial features in the filtered image. The collection module generates a region description file representing the set of spatial features. The network interface transmits the region description file to a remote computing system. The collection module can select only a subset of the set of spatial features to include in the region description file.

Description

Technical Field

[0001] This disclosure generally relates to visual mapping systems, and more particularly to the positioning of mobile devices that use visual information. Background Technology

[0002] Visual mapping systems rely on spatial features (also known as “visual features”) detected in images captured by the mobile device, along with inertial information, to determine the mobile device’s current position and orientation in three-dimensional (3D) space. Typically, position and orientation are determined within a defined coordinate system to facilitate various functionalities requiring synchronization with a known, fixed coordinate system (such as virtual reality (VR) functionality, augmented reality (AR) functionality, or enabling interaction between multiple mobile devices for games or other devices). Simultaneous localization and mapping (SLAM) techniques enable mobile devices to map previously unmapped areas while concurrently learning their position and orientation within those areas. Thus, when the mobile device returns to the same area, its current position and orientation within that area can be easily determined through the detection of previously observed spatial features in a process known as “localization.” However, when the mobile device first enters an area, it lacks these previously detected localization cues. In conventional visual mapping systems, the mobile device must “learn” the area through the implementation of a visual mapping process—a process that is time-consuming and resource-intensive. To avoid the delays involved in performing visual mapping on previously unmapped areas, conventional visual mapping systems can instead recover the orientation or location of a mobile device based on non-visual orientation inputs, such as Global Positioning System (GPS) information or location mapping fed back via inertial sensors. However, these non-visual mapping schemes can be unreliable (e.g., poor GPS reception indoors or in areas surrounded by high obstacles), inaccurate, and error-prone due to sensor and measurement drift. Attached Figure Description

[0003] By referring to the accompanying drawings, those skilled in the art can better understand this disclosure, and many of its features and advantages will become apparent to them. The use of the same reference numerals in different drawings indicates similar or identical items.

[0004] Figure 1 This is an illustration of a cloud-based visual mapping system for crowdsourcing the creation and updating of region description files for use in a mobile device for positioning, according to at least one embodiment of the present disclosure.

[0005] Figure 2 The illustration shows at least one embodiment according to the present disclosure. Figure 1 A block diagram of the region description file server for the visual mapping system.

[0006] Figure 3 These are illustrations based on at least one embodiment of the present disclosure. Figure 1 A block diagram of a mobile device's visual mapping system.

[0007] Figure 4 This is a flowchart illustrating a method for selecting one of a previously mapped positioning process or a previously unmapped positioning process according to at least one embodiment of the present disclosure.

[0008] Figure 5 This is a flowchart illustrating a method for crowdsourcing the generation of location region description files according to at least one embodiment of the present disclosure.

[0009] Figure 6 This is a flowchart illustrating a method for locating a mobile device using a location region description file obtained from a remote region description file server, according to at least one embodiment of the present disclosure.

[0010] Figure 7 This is a flowchart illustrating a method for crowdsourcing the updating of a location area description file according to at least one embodiment of the present disclosure.

[0011] Figure 8 This is a flowchart illustrating a method for evaluating candidate spatial features included in a location region description file, according to at least one embodiment of the present disclosure.

[0012] Figure 9 This is a flowchart illustrating a method for evaluating spatial features to be removed from a location region description file according to at least one embodiment of the present disclosure.

[0013] Figure 10 This is a diagram illustrating a secondary query interface of an LADF database of a region description file server for performing location region description file queries according to at least one embodiment of the present disclosure.

[0014] Figure 11 This is a flowchart illustrating a method for locating a secondary query of a region description file according to at least one embodiment of the present disclosure.

[0015] Figure 12 This is a diagram illustrating an alternative secondary query interface of an LADF database for performing location region description file queries according to at least one embodiment of the present disclosure. Detailed Implementation

[0016] The following description is intended to convey a thorough understanding of this disclosure by providing numerous specific embodiments and details involving a vision mapping system. However, it should be understood that this disclosure is not limited to these specific embodiments and details, which are merely examples, and the scope of this disclosure is therefore intended to be limited only by the following claims and their equivalents. It should also be understood that, given known systems and methods, those skilled in the art will also appreciate that the use of this disclosure for its intended purpose and benefit in any number of alternative embodiments depends on specific design and other needs.

[0017] Figure 1-12 An example system and technique for the cloud-based creation, modification, and provision of location area data files for mobile devices are illustrated. In at least one embodiment, the visual mapping system includes an Area Description File (ADF) server communicatively coupled to multiple mobile devices via one or more wired or wireless networks. When a mobile device enters a previously unmapped area, it initiates a area learning process in which it acquires imagery of the area, detects spatial features using the acquired imagery, and transmits a representation of the spatial features, their relative geometry, associated statistics, and concurrently acquired sensor data in the ADF to the ADF server. The ADF server uses the ADF to generate a Location ADF (LADF), which represents a sparse point cloud of the spatial features of the area. Note that the term "file" (as used in "area description file" or otherwise herein) refers to any data structure and any combination of such data structures used to associate data with other information domains. The LADF can then be downloaded to another mobile device in the area covered by the LADF. Mobile devices receiving LADF (Local Area Data) can acquire images of a region, detect spatial features within them, and compare the detected spatial features and their relative geometry with the spatial features and relative geometry represented by LADF to position the mobile device within the region using the reference coordinate system presented by LADF. In this way, the discovery and mapping of previously unmapped regions can be efficiently crowdsourced through the passive acquisition and uploading of data by mobile devices.

[0018] In some embodiments, the ADF server receives ADF files from multiple mobile devices for the same region or for adjacent regions, and operates to merge these multiple ADF files into a merged ADF file (a cluster of merged ADF files), which can then be used to generate one or more LADFs for a region or its sub-regions. Furthermore, because changes may occur within a region, the ADF server can utilize crowdsourced updates of the LADF generated for a region based on feedback from mobile devices using the LADF for positioning within the region. For example, in response to feedback indicating that some spatial features in the LADF will not be easily observed by mobile devices, the ADF server can remove these spatial features from the LADF. Conversely, in response to feedback indicating that spatial features not included in the LADF will be easily observed by mobile devices in the region, the ADF server can add these spatial features to the LADF.

[0019] Because point clouds and associated data represented by ADFs and LADFs provide a visual representation of a region to some extent, acquiring ADFs from mobile devices and distributing ADFs to mobile devices can have privacy implications. Therefore, in some embodiments, the visual mapping system implements certain privacy controls. One such privacy control may include configuring the mobile device to implement one or more privacy filtering processes on the ADF data before the ADF is uploaded to the ADF server, and thus rendering information represented by the uploaded version of the ADF that cannot be used for the purpose of recreating useful visual content of the region. Another such privacy control may include a two-level LADF query for accessing LADFs distributed to mobile devices. When a mobile device enters an area where it does not have an available LADF, the mobile device acquires imagery, detects spatial features from the acquired imagery, and submits an LADF request to the ADF server, wherein the LADF request includes a set of detected spatial features and an indicator of the mobile device's location (e.g., GPS coordinates or one or more wireless base station identifiers). The ADF server may then perform a first query phase to identify a set of candidate LADFs that match one of the selected set of spatial features or location indicators, and then select an LADF from the set of candidate LADFs based on the other of the detected spatial features or location identifiers. In this way, the set of spatial identifiers submitted by the mobile device can be used as evidence that the mobile device is in the identified area or has access to it, and thus significantly reduces the risk of a breach of confidentiality in supplying LADF to the requesting mobile device.

[0020] Figure 1A visual mapping system 100 according to at least one embodiment of the present disclosure is illustrated. In the depicted example, the visual mapping system 100 includes an ADF server 102 communicatively coupled to one or more mobile devices 104. As represented by cloud 106, the ADF server 102 includes a computing system remotely located at the mobile device 104 and coupled to the mobile device 104 via one or more wired or wireless networks (such as via wireless local area network (WALN), cellular data network, the Internet, or a combination thereof). Although described herein in the context of a single server example, in other embodiments, the ADF server 102 may be implemented as a computing system including a cluster of servers. References below Figure 2 An example implementation of the ADF server 102 is described in more detail.

[0021] Mobile device 104 can include any of a variety of portable electronic devices operated by one or more users 110, such as head-mounted displays (HMDs), tablet computers, computing-enabled cellular phones (e.g., "smartphones"), laptop computers, personal digital assistants (PDAs), game console systems, drones, etc. In the depicted example, mobile device 104 includes a housing 112 having a surface 114 opposite another surface 116, whereby housing 112 is typically oriented with respect to user 110, such that the user faces surface 114 of housing 112. Further, in the depicted embodiment, mobile device 104 includes a display 118 deployed on surface 116 for presenting visual information to user 110. Thus, for ease of reference, surface 116 is referred to herein as a "forward" surface and surface 114 as a "user-facing" surface as a reflection oriented in this example, but these surfaces are not limited by these relationships.

[0022] The mobile device 104 also includes multiple sensors that acquire information about the region 122 in which the mobile device 104 is currently positioned. The mobile device 104 acquires visual information (images) for region 122 via one or more imaging sensors, such as imaging sensors 124, 126 deployed, for example, on the forward surface 116. Imaging sensors 124, 126 can be positioned and oriented on the forward surface 116 such that their respective fields of view overlap begin at a specified distance from the mobile device 104, thereby enabling depth sensing of objects in region 122 located within the overlapping fields of view via multi-view analysis. Alternatively, a depth sensor 130 deployed on surface 116 can be used to provide depth information for objects in the region.

[0023] In one embodiment, depth sensor 130 projects a modulated light pattern from forward surface 116 onto region 122 using a modulated light projector, and when it is reflected back from an object in region 122, the reflection of the modulated light pattern is acquired using one or both of imaging sensors 124, 126. These modulated light patterns can be either spatially modulated or temporally modulated light patterns. The acquired reflection of the modulated light is referred to herein as a “depth image” or “depth picture.” Depth sensor 120 can then calculate the depth of the object (i.e., the distance of the object from mobile device 104) based on the analysis of the depth picture. The depth data obtained from depth sensor 130 can be used to calibrate or otherwise augment depth information obtained from multi-view analysis (e.g., stereo analysis) of image data acquired by imaging sensors 124, 126. Alternatively, depth data from depth sensor 130 can be used instead of depth information obtained from multi-view analysis. See below for reference. Figure 3 A more detailed example of the electronic system of the mobile device 104 is described.

[0024] In operation, mobile device 104 acquires images of region 122 via one or both of imaging sensors 124, 126, modifies or otherwise processes the acquired images, and provides the processed acquired images for display on display device 108. Processing of the acquired images can include, for example, spatial or color filtering, adding augmented reality (AR) overlays, converting the real content of the images into corresponding virtual reality (VR) content, etc. To provide this AR or VR functionality, mobile device 104 relies on accurate determination of its current six degrees of freedom (6DOF) orientation and position (collectively referred to herein as “pose”) relative to a specified coordinate system. Therefore, in some embodiments, mobile device 104 also uses images acquired by imaging sensors 124, 126 and non-image sensor data (e.g., inertial sensor data) to determine the relative position / orientation (i.e., position / orientation relative to region 122) of mobile device 104 using one or more of the following: Simultaneous Localization and Mapping (SLAM) processes, visual rangefinding processes, or other visual mapping processes.

[0025] Conventionally, when a mobile device enters a new area (i.e., an area that a particular mobile device has not previously entered), the mobile device must either complete a region learning process (e.g., a SLAM process) to learn the new area, or rely on non-visual cues (e.g., GPS coordinates) to provide some indication of the mobile device's current pose relative to the coordinate system. In contrast, the visual mapping system 100 utilizes crowdsourcing provided by prior exploration caused by other mobile devices in the area to allow a new mobile device to quickly locate itself in the coordinate system without extensive visual mapping of the area. For this purpose, when mobile device 104 moves about area 122, mobile device 104 connected to ADF server 102 operates to acquire and upload ADF 132 to ADF server 102. As described in more detail below, ADF 132 includes a set of spatial features detected by mobile device 104, the relative geometry of the spatial features, various statistical metadata about the spatial features in a manner substantially independent of the mobile device's posture, lighting conditions or other temporary environmental conditions or device-specific conditions, and sensor data acquired from an inertial management unit (IMU) or other non-image sensors during the acquisition of images associated with the spatial features. ADF server 102, based on the information contained in ADF 132, filters and merges ADFs from multiple mobile devices 104 in the same (or adjacent) region, and generates one or more LADFs 134 from the resulting merged ADFs, each representing a sparse point cloud of spatial features for a region or its corresponding sub-region.

[0026] Therefore, when another mobile device 104 subsequently enters the new area, mobile device 104 can query ADF server 102 for the LADF associated with the area. In response to the query, ADF server 102 can provide LADF 134 to the requesting mobile device 104. Mobile device 104 can acquire an image of area 122, detect certain spatial features contained therein, and compare the detected spatial features and their relative geometry with the spatial features represented by LADF 134 and their associated relative geometry, and locate its current pose based on the comparison of spatial features (i.e., either determine its current pose or correct a previously determined pose that has drifted) (such as through the application of a loop closure algorithm). Mobile device 104 can also provide feedback to ADF server 102 based on the use of LADF 134 by the mobile device (such as feedback data indicating that the spatial features in LADF 134 were not observed by mobile device 104, or feedback data indicating that no spatial features in LADF 134 were observed by mobile device 104).

[0027] The crowdsourcing and cloud-based creation, refinement, and distribution of LADF 134 allow mobile devices 104 to quickly and efficiently locate themselves when entering new areas, thus avoiding the need for time-consuming area learning processes or the use of less accurate and often unreliable measurements from GPS or inertial sensors. Furthermore, because the spatial features of LADF 134 can be referenced to a specific coordinate system, multiple mobile devices 104 using LADF 134 can locate their current pose to a common coordinate system, thereby facilitating more efficient and accurate interaction between mobile devices 104 that rely on device pose information, such as multiplayer gaming, shared AR or VR functionality, etc.

[0028] Figure 2 The illustrations depict at least one embodiment according to the present disclosure. Figure 1 An example implementation of the ADF server 102 is shown below. In the depicted example, the ADF server 102 includes a computing system having a network interface 202, an ADF data storage 204, an LADF data storage 206, a feature scoring database 208, a georeferenced data storage 210, a merging module 210, a georeferenced module 214, a spatial feature filtering module 216, a location generation module 218, and a query module 220. Although illustrated as separate data storages, one or more of the following can be implemented together as a single data storage: data storages 204, 206, 208, and 210.

[0029] Modules 212, 214, 216, 218, and 220 can be implemented as hard-coded logic (e.g., application-specific integrated circuits or programmable logic), one or more processors 222, or combinations thereof, executing software instructions 224 stored in memory 226 or other storage devices. Furthermore, although described as a single server for ease of illustration, the ADF server 102 can instead be implemented as a computing system comprising multiple servers. For example, the functionality of modules 212, 214, 216, and 218 can be implemented on one server, and the functionality of querying module 220 and the LADF data storage 206 can be implemented on another server.

[0030] As a general overview, the merger module 212 operates via network interface 202 from one or more mobile devices 232 that have entered a region of LADF that has not yet been compiled. Figure 1In one embodiment of mobile device 104, ADF 132 is received. Merger module 212 merges one or more ADFs 132 from mobile device 232 to generate merged ADF data for storage in ADF data store 204. Georeferencing module 214 may provide georeferencing to the merged ADF using georeferencing information from georeferencing data store 210 (which may include, for example, imagery from Google Street View and associated georeferencing locations). Location generation module 218 generates one or more LADFs from the resulting merged ADF data and stores and indexes the one or more LADFs in LADF data store 206 for subsequent retrieval. Query module 220 receives LADF request 236 from mobile device (one embodiment of mobile device 104) via network interface 202, searches LADF data store 206 for LADF 134 corresponding to LADF request 236, and provides LADF 134 to mobile device 234. Mobile device 234 uses LADF 134 for location of mobile device 234. During this positioning process, mobile device 234 can provide LADF feedback 238 regarding LADF 134 to ADF server 102 via network interface 202. ADF server 102 can use LADF feedback 238 to adjust the feature scores of spatial features represented in feature score data storage 208, and improve LADF 134 by adding or removing spatial features based on these feature scores. See below for reference. Figure 4-12 These operations are described in more detail.

[0031] As described, mobile devices 232 and 234 represent embodiments of mobile device 104. Due to their respective operational requirements, mobile device 232, which performs the localization process using LADF, may not require the same capabilities as mobile device 234, which performs the ADF upload process. For illustration purposes, mobile device 234 may utilize depth sensor 120 or a stereo camera configuration to facilitate SLAM operations associated with the process of generating and uploading the ADF; however, mobile device 232 will typically not require depth sensor 120 and will only require a monocular camera configuration to facilitate the localization process.

[0032] Figure 3An example processing system 300 implemented by a mobile device 104 according to at least one embodiment of the present disclosure is illustrated. The processing system 300 includes a display 118, imaging sensors 124, 126, and a depth sensor 130. The processing system 300 also includes a graphics processing unit (GPU 302), frame buffers 303 and 305, an application processor 304, a display controller 306, system memory 308, a collection of non-image sensors 310, and a user interface 312. The user interface 312 includes one or more components (such as a touchscreen 314, a mouse, a keyboard, a microphone 316, various buttons or switches, and various haptic actuators 318) manipulated by a user to provide user input to the mobile device 104. The collection of non-image sensors 310 can include any of a variety of sensors used to provide a non-image background or state of the mobile device 104. Examples of such sensors include an inertial management unit (IMU) 320, which includes one or more of the following: a gyroscope 321, a magnetometer 322, and an accelerometer 323. Non-image sensors may also include, for example, an ambient light sensor 326 and various wireless receiving or transmitting sensors (such as a GPS sensor 328), a wireless local area network (WLAN) interface 330, a cellular interface 332, a peer-to-peer (P2P) wireless interface 334, and a near field communication (NFC) interface 336. Non-image sensors may also include user input components (such as a touchscreen 314 or a microphone 316) for the user interface 312.

[0033] Mobile device 104 also has access to various data storage 338 storing information or metadata used in conjunction with its image processing, location mapping, and location utilization processes. Data storage 338 can include: spatial feature data storage, which stores metadata of 2D or 3D spatial features identified from images acquired by the imaging sensors of mobile device 104; SLAM data storage, which stores SLAM-based information (such as region 122 already explored by mobile device 104). Figure 1 The data storage 338 includes mapping information of sub-regions of the mobile device 104; and AR data storage, which stores AR overlap information or VR information (such as a CAD-based representation of the relative position of the object of interest in region 122). The data storage 338 may be implemented on one or more storage components of the mobile device 104 (such as on a hard disk drive, solid-state memory, or removable storage medium (not shown)).

[0034] In operation, imaging sensors 124, 126 acquire images of the region and buffer the acquired images in frame buffers 303, 305. For images to be displayed in their original or modified form, GPU 302 processes the acquired images for display (e.g., by rendering AR overlay), and display controller 306 controls display 118 to display the processed images. Further, as described herein, mobile device 104 operates by uploading an ADF 134 for a previously unmapped location and downloading an LADF for a previously mapped location, and using the downloaded LADF to facilitate positioning of mobile device 104. For this purpose, one or more software programs may be stored in system memory 308 or other non-transitory computer-readable medium and executed by one or both of application processor 304 and GPU 302 to provide the ADF generation and LADF utilization functionality. For ease of illustration, one or more software programs in Figure 3 The programs are described as ADF generator 342 and LADF handler 344. These programs can be implemented as threads or other processes in an operating system (OS) executed by the processing system 300, threads, processes or subroutines of the same software application, or separately executed software applications. Furthermore, in some embodiments, some or all of the functionality of the programs 342, 344 described herein can be implemented via an ASIC, programmable logic, or other hard-coded logic.

[0035] The ADF generation program 342 may include, for example, a privacy filter module 346, a spatial feature detection module 348, and an ADF collection module 350. (As in...) Figure 3 As depicted, the privacy filter module 346 may include one or more image content filters (such as a text filter module 352 and a face filter module) and a downsampling module 356. The LADF processor 344 may include, for example, a spatial feature detection module 358 (which may be a spatial feature detection module 348), a request module 360, a positioning module 362, and a feedback module 364. The operation of the processing system 300, including the operation of the ADF generation program 342 and the LADF processor 344, is described in detail below.

[0036] As described above, when mobile device 104 is in an unmapped region (i.e., a region where ADF server 102 does not have an available LADF), mobile device 104 can operate in ADF generation mode, where mobile device 104 generates an ADF representing the spatial features detected in the unmapped region and uploads the ADF to ADF server 102. However, if the region has been mapped and ADF server 102 has an LADF for the region, mobile device 104 can instead operate in LADF positioning mode, where mobile device 104 obtains the LADF for the region from ADF server 102 and performs a positioning process using the sparse point cloud represented by the LADF to locate mobile device 104 to the region.

[0037] Figure 4 An example method 400 for selecting between these operating modes according to at least one embodiment is illustrated. For ease of description, in Figure 3 Method 400 is described in the example context of processing system 300. Method 400 is initiated at block 402, where mobile device 104 determines that it has moved to an area that has not yet been previously mapped by mobile device 104. In response, mobile device 104 initiates a positioning process or motion tracking process.

[0038] At block 404, mobile device 104 queries ADF server 102 to determine if LADF is available for the region. See below for reference. Figure 10-12 As described, the ADF server 102 can implement a two-level query process, wherein the mobile device 104 provides a set of spatial features of images acquired by the mobile device in the area and one or more location indicators (e.g., GPS coordinates or base station identifiers of WALN or cell tower base stations detected by the mobile device 104) and the ADF server 102 queries the LADF data storage 206 to identify the corresponding LADF.

[0039] If LADF is not available from ADF server 192, the region is considered an unmapped location, and therefore at block 406, mobile device 104 and ADF server 102 coordinate to perform the ADF / LADF generation process for the unmapped location. (See below for more information.) Figure 5 The process is described in more detail. When LADF is available from ADF server 102, the area is considered the mapped location, and therefore at block 408, mobile device 104 and ADF server 102 coordinate to perform the LADF location and update process, which is referenced below. Figure 6 To describe in more detail.

[0040] Figure 5An example method 500 for implementing an ADF / LADF generation process according to at least one embodiment is illustrated. As described above, the ADF / LADF generation process is performed in response to the determination that the LADF is unavailable for a newly encountered area by the mobile device 104. Therefore, after receiving user consent for the passive acquisition of area description data, method 500 is initiated at block 502, wherein an area learning process is initiated using the sensors of the mobile device 104. To perform the area learning process, as the mobile device 104 moves through the area, the mobile device 104 acquires image 372 via imaging sensors 124, 126. Figure 3 And the corresponding depth information 373 is acquired via depth sensor 130. When the image 372 and depth information 373 are acquired by mobile device 104, mobile device 104 may use, for example, an interactive game to induce the user to explore the area of ​​mobile device 104.

[0041] Concurrently, the ADF collection module 350 acquires non-image sensor data 374 from one or more of the non-image sensors in the set 310. Figure 3 For illustration, the ADF collection module 350 can acquire sensor data from the accelerometer 323 during image 372 acquisition, and thus indicate the orientation of the mobile device 104 with respect to gravity at the time of image acquisition. Similarly, sensor data acquired from the gyroscope 321 can be used to determine the direction of travel of the mobile device 104 with respect to visual features represented in the acquired images. Further, the non-image sensor data 374 can include sensor data that can operate as a location indicator for the mobile device 104. These location indicators can be georeferenced location identifiers (such as latitude / longitude coordinates represented in the sensor data provided by the GPS sensor 328). Alternatively, these location indicators can be inferred location indicators. For illustration, when WLAN base stations and cell tower base stations are assumed to be fixed, the detection of a WLAN base station or cell tower base station serves as an indication that the mobile device is close to the detected base station, and therefore the base station identifier (BSID), media access control (MAC) address, or other identifier of the base station can be used as an inferred location identifier for the mobile device.

[0042] As described below, the acquired image 372 and depth information 373 are used to determine a point cloud of spatial features, which is ultimately uploaded to the ADF server 102. Although this point cloud is not the image data itself, depending on the density and other conditions of the point cloud, it is possible to use the unmodified point cloud to reconstruct certain visual content originally present in the acquired image 372. For example, if the mobile device 104 is placed particularly close to a document, it is possible to use the point cloud determined from an image of the document to reproduce the text of the document. Therefore, to prevent the unintentional disclosure of visual content, the privacy filter module 346 can implement at least two privacy controls.

[0043] A first privacy control is implemented at block 504, wherein the privacy filter module 346 performs one or more content filtering processes on the acquired image 372 to remove image content from areas that may have privacy implications. For example, at block 504, the text filter module 352 may perform a text filtering process, wherein each image acquired at block 502 is scanned using one or more well-known text recognition algorithms to determine whether any region exists in the image potentially representing text content. For each region detected as a potential text region, the text filter module 352 may blur or delete the region, for example, by performing a blending operation using pixels in the region or adjacent regions, by replacing pixel values ​​in the region with the same default pixel region, and by otherwise deleting pixel values ​​in the region. Similarly, at block 504, the face filter module 354 may implement a face filtering process, wherein each acquired image is scanned using one or more well-known face recognition algorithms to determine whether any region exists in the image potentially representing a face, and the image is filtered to remove image content from each such identified region. In this way, the image is pre-filtered to remove potentially sensitive visual content before spatial feature detection, and therefore the point cloud resulting from the spatial features cannot be used to reconstruct potentially sensitive visual content.

[0044] At block 506, the spatial feature detection module 348 analyzes the filtered image to detect the spatial features contained within it. Any of a variety of spatial feature extraction algorithms can be used (such as, for example, Scale Invariant Feature Transform (SIFT), Fast Robust Feature Extraction (SURF), Gray-Level Patch, Gradient Position and Orientation Histogram (GLOH), Zernike Moments, Binary Robust Independent Fundamental Features (BREIF), Oriented BRISK (ORB), Binary Robust Invariant Scale-Variable Keypoint (BRISK), Difference of Gaussians (DOG), Fast Retinal Keypoint (FREAK), etc.). The spatial feature detection module 348 provides these detected spatial features as spatial feature data 376. For illustration, the FREAK algorithm provides comparisons of pixel pairs within the image patch, where the output of each comparison is either "0" or "1" based on whether the pixel is brighter or darker in the pair. In the FREAK algorithm, 512 such comparisons are computed on the image patch, and the result is a spatial feature descriptor (i.e., a 512-bit binary string representing the corresponding spatial feature of the image patch) and a (x,y,z) vector identifying the position of the spatial feature in the 3D reference frame.

[0045] Furthermore, the spatial feature detection module 348 determines statistical metadata 378 for each detected point. The statistical metadata 378 describes the corresponding spatial features in a manner that is substantially independent of the specific viewpoint of the mobile device 104 or ambient lighting when the corresponding image is acquired. For example, the statistical metadata 378 may include values ​​representing the average and standard deviation of the brightness of pixels representing spatial features, brightness gradients (or other visual characteristic gradients) in one or more directions, etc.

[0046] Although the privacy filter module 346 has purified the image of detected text and facial content, and therefore the original point cloud of spatial features represented by spatial feature data 376 is essentially devoid of text and facial content, the original point cloud can still possess sufficient spatial feature density to allow for some level of reconstruction of the visual appearance of the region from the original point cloud. Therefore, as a second privacy control, at block 508, the downsampling module 356 of the privacy filter module 346 downsamples the original point cloud to generate a filtered point cloud 380 of spatial features. Figure 3The filtered point cloud contains only a selected subset of the spatial features of the original point cloud. This has the advantages of reducing the likelihood of reconstructing the visual appearance of a region from the resulting point cloud and reducing the amount of data required to represent the resulting filtered point cloud 380. The downsampling process can be controlled by one or more downsampling criteria. For example, in some embodiments, the downsampling process may include random selection of spatial features to exclude from the filtered point cloud, removal of every Xth spatial feature (X is an integer greater than 2), or selection of no more than a maximum number of spatial features to include in the filtered point cloud. As another example, the downsampling process can be controlled by a maximum spatial feature density criterion specifying a maximum number of spatial features per cubic unit (i.e., unit volume). For example, the maximum spatial feature density criterion may specify that the filtered point cloud 380 contains at least one spatial feature per cubic foot, and thus downsample the original point cloud such that no more than one spatial feature per cubic foot is represented in the resulting filtered point cloud.

[0047] At block 510, the ADF collection module 350 uses the filtered point cloud 380, statistical metadata 378, and non-image sensor data 374 to generate an ADF 132. In one embodiment, the ADF 132 includes a file or other data structure representing the filtered point cloud 380 as a list or other set of multidimensional coordinates, along with a field for each multidimensional coordinate storing the statistical metadata 378 representing the spatial features. For illustration, each multidimensional coordinate may include a (x,y,z) floating-point vector representing the 3D position of the corresponding spatial feature in the (X,Y,Z) coordinate system of the mobile device 104. The data structure may also include one or more fields storing the non-image sensor data 374, and a field storing a unique identifier (UID) assigned to the ADF 132 by the ADF server 102, as described below.

[0048] At block 512, the ADF collection module 350 provides ADF 132 to the appropriate network interface (e.g., cellular interface 332 or WLAN interface 330) for transmission to the ADF server 102. In some embodiments, the ADF collection module 350 signals a request to upload the ADF to the ADF server 102, and the ADF server 102 responds using the UID assigned to the ADF, which the ADF collection module 350 inserts into the appropriate field in the ADF 132 before it is uploaded. Further, while the process of blocks 502-512 has been described as a sequential process, wherein the ADF 132 is completed before it is uploaded, in some embodiments, the ADF 132 is generated and uploaded in an iterative process, wherein the ADF 132 is generated as a sequence of ADF blocks, each ADF block containing a subset of point clouds generated from a portion of the acquired image, and each ADF block is tagged with the UID assigned to the ADF and uploaded to the ADF server when it is generated. ADF server 102 can therefore store these ADF blocks individually, or combine ADF blocks represented by the same UID into a single ADF 132.

[0049] The ADF 132 uploaded by mobile device 104 represents certain visual features present in the area where mobile device 104 is located. However, due to privacy controls implemented by mobile device 104, as described above with references 504 and 508, the information contained in ADF 132 provided to ADF server 102 is essentially devoid of anything that could have privacy implications. Instead, while the resulting ADF 132 contains a sparse point cloud that can describe the edges, corners, and other visual features of the area, the sparse point cloud contains insufficient information to support a reproduction of the visual appearance of the area in a way that is meaningful to human perception, to the extent that it would be sufficient to allow another mobile device to subsequently locate the area based on these described visual features. Thus, ADF server 102 should not possess any information from mobile device 104 that has the potential to display measurable sensitive information about the mapped area.

[0050] The role of the ADF server 102 in the ADF / LADF generation process is initiated at block 514, where the ADF 132 is received from the mobile device 104 via network interface 202 by the merging module 212. Once received, the merging module 212 temporarily indexes and stores the ADF 132 in the ADF data storage 204. The ADF 132 can be indexed for the temporary storage device based on the UID assigned to it, the spatial features represented in the point cloud represented by the ADF 132, and the location identifier(s) included in the ADF 132. In at least one embodiment, each ADF 132 uses a reference... Figure 10-12 The two-level query method described below is stored and indexed in ADF data store 204.

[0051] In some instances, multiple mobile devices may have uploaded corresponding ADFs for a region or for adjacent regions. Therefore, ADF server 102 operates at block 516 to merge these co-located ADFs before the resulting merged ADFs are processed into one or more LADFs. The merging process at block 516 can be initiated in response to any of a variety of triggers. For example, the merging process can be triggered by a specified amount of time failure, by the receipt of a specified number of ADFs from a mobile device, by a request for LADFs from a mobile device, etc. In response to such a trigger, at block 518, merging module 212 signals query module 220 to query ADF data store 204 to identify the existence of any “nearby” ADFs—that is, ADFs covering the same or adjacent regions. In some embodiments, nearby ADFs can be identified based on a comparison of location indicators associated with ADFs in ADF data store 240. For example, assuming the upload of ADF 132 from mobile device 104 has triggered a merging process and the uploaded ADF 132 includes one or more WALN MAC addresses as location identifiers, then those ADFs in the ADF data store 240 that are identified by query module 220 as having the same WALN MAC address as the uploaded ADF or a WALN MAC address known to be near an area covered by a WALN base station identified by the uploaded ADF are identified as nearby ADFs. As another example, the uploaded ADF may be provided with GPS coordinates as a location indicator, and based on its corresponding GPS coordinates, query module 220 can identify the ADF in the ADF data store 240 as nearby. In other embodiments, nearby ADFs may be identified based on a comparison of the spatial features represented in the uploaded ADF with the spatial features represented by the ADF stored in the ADF data store 240. Those ADFs with sufficiently overlapping sets of spatial features can therefore be identified as representing the same area or a nearby area, and are thus identified as nearby ADFs. Furthermore, refer to Figure 10-12 As described below, ADF server 102 can implement two-level queries against one or both of ADF data storage 204 and LADF data storage 206, such that each stored ADF / LADF is indexed for storage based on both its spatial feature set and one or more location identifiers, and nearby ADFs are identified by performing a two-level indexing process using both the spatial feature set of the uploaded ADF and one or more location indicators of the uploaded ADF.

[0052] A collection of one or more nearby or co-located ADFs identified by query module 220 is referred to herein as an "ADF cluster". When an ADF cluster is identified, at block 520, merging module 212 operates to perform one or more deduplication processes to remove duplicate spatial features represented in the ADFs of the ADF cluster. Further, if the ADF cluster represents an excessively large region or contains an excessively large number of spatial features, merging module 212 may divide the ADF cluster into one or more smaller ADF clusters, each of which may be processed as described above. At block 522, merging module 212 employs one or more well-known ring closure algorithms to determine the relative alignment between the ADFs of the ADF cluster. At block 524, merging module 212 analyzes the ADF cluster to selectively remove spatial features that are not reliably observed in most or all of the ADFs of the ADF cluster identified using one or more stability criteria. This serves the purpose of eliminating spatial features that may not represent permanent visual features of the region (and therefore unreliable sources for localization) and reducing the total number of spatial features in the combined point cloud represented by the ADF cluster. When selecting spatial features to remove, the merging module 212 may employ a scoring system based on stability criteria (such as assigning a spatial feature score based on its frequency of observation within the ADF of the ADF cluster). Furthermore, since the updated ADF is more likely to represent the current state of the region, the merging module 212 may assign inclusion preferences to spatial features in the updated ADF.

[0053] In some instances, the non-image sensor data 374 supplied with the ADF can allow the ADF to be georeferenced. For example, the non-image sensor data 374 may include indicators of the GPS coordinates and georeferenced orientation of the mobile device 104 when spatial features in the ADF are detected. Utilizing the relative alignment of the ADFs in the ADF cluster determined at block 522, if one ADF in the ADF cluster is georeferenced, then at block 526, the other ADFs in the ADF cluster can be georeferenced based on the application of coordinate system transformation using the relative alignment of the georeferenced ADFs and geographic location information. Furthermore, if georeferenced visual reference data is available (such as via Street View tools provided by Google), the ADFs in the ADF cluster can be georeferenced using that georeferenced visual reference data.

[0054] In the case where ADFs in an ADF cluster are merged, at block 528, the location generation module 218 uses the merged ADFs to generate one or more LADFs representing the region by the merged ADFs. If the merged ADFs are small enough or cover a sufficiently small area, the merged ADFs can be stored and indexed as a single LADF. However, if the merged ADFs exceed a threshold in size, cover too large an area, or contain multiple separate sub-regions (e.g., identified by walls, partitions, doors, and windows in the point cloud of the ADFs), the location generation module 218 can spatially partition the merged ADFs to generate multiple LADFs, each covering a different sub-region. In such instances, the location generation module 218 can attempt to identify logical partition lines (e.g., by identifying multiple rooms within an area based on detection of walls or other room partitions within the point cloud represented by the merged ADFs) and create separate LADFs for each identified room, offering the additional benefit of limiting the scope of the LADF to a single room.

[0055] For each LADF generated, at block 530, the positioning generation module 218 provides the LADF to the LADF data storage 206 for indexing and storage. As described above and as detailed below, in some embodiments, the LADF data storage 206 employs a two-level indexing for each LADF, wherein each LADF is indexed by one or more location markers and by the spatial features represented by the LADF. Thus, when a LADF is stored in the LADF data storage 206, the LADF data storage 206 stores the LADF in a data storage entry indexed by one of its location indicators or its set of spatial features (and the relative geometry therein), and another of the location indicators or spatial feature sets is used to select among a plurality of similarly indexed LADFs.

[0056] At the end of the process at block 530 of method 500, ADF server 102 has generated one or more LADFs for previously unseen areas using one or more ADFs uploaded from one or more mobile devices 104 that have already had the opportunity to explore the area. Therefore, ADF server 102 is prepared to supply the LADFs to any other mobile devices that are encountering the area for the first time, as described below. Figure 6 As described.

[0057] Figure 6 The illustration depicts an execution method according to at least one embodiment. Figure 4Block 408 represents an example method 600 for the LADF localization and update process. Method 600 is initiated at block 602 when the mobile device 104 enters an area that it has not previously encountered and mapped. In response to this determination, the mobile device initiates a spatial feature detection process for the area. For this purpose, the mobile device 104 triggers imaging sensors 124, 126 to begin imaging 382 of the area. Figure 3 The spatial feature detection module 358 acquires the image 382 and triggers the depth sensor 130 to begin acquiring depth information for the area. From the image 382 and the depth information, the spatial feature detection module 358 detects an initial set of spatial features representing the area. At block 604, the request module 360 ​​determines one or more location indicators of the location of the mobile device 104 at the time of spatial feature detection in the area. As described above, these location indicators may be specific geographic location indicators (such as GPS coordinates obtained from the GPS sensor 328), inferred location indicators (such as WLAN MAC address, WLAN BSID, or cell tower BSID, or combinations thereof). At block 606, the request module 360 ​​uses the initial spatial feature set and one or more location indicators to generate an LADF request 236 and transmits the LADF request 236 to initiate an investigation into the availability of LADF in the area.

[0058] In response to the receipt of LADF request 236 at query module 220 via ADF server 102 on network interface 202, at block 608 query LADF data storage 206 to identify suitable LADFs for use by mobile device 104 in the region. In at least one embodiment, query module 220 and LADF data storage 206 employ a two-level indexing scheme to identify suitable LADFs based on both an initial spatial feature set and a location indicator, as referenced. Figure 10-12 The following is a detailed description. Assuming a suitable LADF is identified, at block 610, query module 220 transmits the selected LADF (such as LADF 134) to the requesting mobile device 104.

[0059] Upon receiving LADF 134 in response to LADF request 236, request module 360 ​​provides LADF 134 to both positioning module 362 and feedback module 364. At block 612, positioning module 362 employs one or more well-known loop closure algorithms to compare a sparse point cloud based on spatial features from the region detected by spatial feature detection module 358 with the spatial features represented in LADF 134 (the result being a determined pose 384 of the mobile device 104 in the identified coordinate system). Figure 3The mobile device 104 is positioned in the coordinate system represented in LADF 134. Furthermore, since LADF is georeferenced, the determined pose 384 can also be georeferenced through appropriate translation. With the mobile device 104 positioned and pose 384 determined, one or more components of the mobile device 104 at block 614 can provide functionality dependent on accurate pose information (such as the implementation of AR or VR content, multiplayer games, navigation tools, etc.).

[0060] The generation of LADF 134 depends on the detection and selection of spatial features observed in the region by one or more mobile devices 104. However, because the region may have transient objects included in the observed spatial features or because the configuration of the region may change over time, the LADF may become more useless or "stale". Therefore, in some embodiments, with user permission, mobile device 104 can provide feedback on the LADF, which can be used by ADF server 102 to improve or "refresh" the LADF for subsequent use by other mobile devices. For this purpose, at block 616, feedback module 364 can compare the spatial features identified by spatial feature detection module 358 in the region with the spatial features of the sparse point cloud represented by LADF 134 and generate LADF feedback 238 representing one or more of the following: spatial features of LADF 134 also observed by spatial feature detection module 358; spatial features of LADF 134 not observed by spatial feature detection module 358; and spatial features not present in LADF 134 observed by spatial feature detection module 358. LADF feedback 238 is uploaded to ADF server 102. At block 618, spatial feature filter module 216 receives LADF feedback 238, thus updating the feature scores of spatial features in future score data storage 208, and can then update LADF to include new spatial features or remove previously included spatial features based on the updated feature scores.

[0061] Figure 7-9An example of a process for improving LADF 134 based on LADF feedback 238 according to at least one embodiment is illustrated. Initially, all spatial features represented in LADF 134 are provided with initial feature scores, which may be the same for all spatial features in LADF 134, or may be based on one or more properties of the spatial features. For example, a spatial feature may be assigned an initial feature score based on the number of times it appears in the ADF from which the ADF cluster from which LADF 134 was generated. For the purposes of the following description, it is assumed that a higher feature score reflects a more reliable spatial feature, and conversely, a lower feature score indicates a less reliable spatial feature. The LADF feedback 238 supplied by mobile device 104 can include one or more entries, wherein each entry either confirms the observation by mobile device 104 of a spatial feature present in LADF 134, rejects the observation by mobile device of a spatial feature present in LADF 134, or suggests a new spatial feature observed by mobile device 104 that does not exist in LADF 134. Thus, the feature score of a spatial feature can be adjusted based on the entry and the type of feedback it represents.

[0062] Figure 7 Method 700 represents the processing implemented by the ADF server 102 for each entry of the LADF feedback 238. At block 702, the spatial feature filter module 216 accesses the selected entry of the LADF feedback 238 to determine the type of feedback represented by the entry. If the feedback entry confirms that the mobile device 104 has observed the corresponding spatial feature of the LADF 134, at block 704, the spatial feature filter module 216 extracts one or more parameters (such as a timestamp indicating the time of observation of the spatial feature by the mobile device 104) belonging to the spatial feature as represented in the feedback entry, and records the extracted parameters in the feature score data storage 208. Further, at block 706, the spatial feature filter module 216 increments the feature score of the corresponding spatial feature to reflect its most recent observation.

[0063] Returning to block 702, if the feedback entry indicates that the mobile device 104 did not observe the spatial features represented in LADF 134, at block 708, the spatial feature filter module 216 extracts one or more parameters belonging to the identifier of the spatial feature as represented in the feedback entry (such as a timestamp indicating the time when the mobile device 104 was acquiring an image of an area in which the spatial feature should have been observed but was not), and records the extracted parameters in the feature score data storage 208. Further, at block 710, the spatial feature filter module 216 reduces the feature score of the corresponding spatial feature to reflect its missed observation.

[0064] Returning to block 702, if the feedback entry indicates that the mobile device 104 has reliably observed a spatial feature not represented in LADF 134, at block 712, the spatial feature filter module 216 extracts one or more parameters (such as a timestamp indicating the time when the mobile device 104 first or last observed the spatial feature, the frequency of observation of the spatial feature (e.g., the percentage of time the spatial feature is observed when the mobile device 104 is oriented in the direction containing the spatial feature)) belonging to the previously unobserved spatial feature as indicated in the feedback entry, and records the extracted parameters in the feature score data storage 208. Further, at block 714, the spatial feature filter module 216 also creates a feature score for the corresponding spatial feature in the feature score data storage 208, and assigns an initial value to the feature score if the spatial feature has not yet been represented in the feature score data storage 208. Otherwise, if a feature score already exists for the newly observed spatial feature (i.e., the spatial feature was previously observed by another mobile device 104), the spatial feature filter module 216 increments the feature score for the spatial feature. Method 700 then returns to block 702 to repeat the process for the next entry in LADF feedback 238.

[0065] Figure 8An example method 800 for evaluating candidate spatial features to be included in LADF 134, according to at least one embodiment, is illustrated. As described above, mobile device 104 can provide LADF feedback 238 identifying newly observed spatial features in a region that may be candidates for inclusion in LADF 134, and spatial feature filter module 216 can create a feature score the first time a spatial feature is detected, increasing the feature score each time the same spatial feature is observed by another mobile device 104. Thus, the feature score of each of these spatial features serves as an indicator of the feasibility of a spatial feature for inclusion in LADF 134 (i.e., how “reliable” the spatial feature is for positioning purposes). Therefore, in response to a triggering condition (such as aging of LADF 134 exceeding a certain threshold), or in response to sufficient feedback from the mobile device 104 indicating that a correction to LADF 134 will be guaranteed, at block 802, the spatial feature filter module 216 selects one of the candidate spatial features under consideration from the feature score data storage 208 for inclusion in LADF 134, and at block 804, the spatial feature filter module 216 compares the feature score of the selected candidate spatial feature with a specified threshold (specified as “THRESH_H”). This threshold may be a fixed threshold or may be based on one or more current conditions. For example, in one embodiment, the threshold THRESH_H may be set to the median current feature score of the spatial features currently included in LADF 134.

[0066] If the feature score of a candidate spatial feature does not exceed the threshold THRESH_H, the spatial feature filter module 216 stops any further consideration of the candidate spatial feature (in this round of evaluation), and method 800 returns to block 802 for the selection of the next candidate spatial feature. Otherwise, if the feature score exceeds the threshold THRESH_H, at block 806, the spatial feature filter module 216 signals the positioning generation module 218 to include the selected candidate spatial feature in LADF 134. In some embodiments, including a new spatial feature in LADF 134 may require the elimination of another spatial feature from LADF 134. In such a case, the spatial feature filter module 216 may select, for example, the spatial feature with the lowest feature score among all spatial features currently included in LADF 134. After including the candidate spatial feature in LADF 134, method 800 returns to block 802 for the selection and evaluation of the next candidate spatial feature.

[0067] Figure 9An example method 900 for evaluating spatial features to be removed from LADF 134, according to at least one embodiment, is illustrated. As described above, mobile device 104 can provide LADF feedback 238 confirming whether spatial features of LADF 134 have been observed by mobile device 104, and therefore spatial feature filter module 216 can adjust the feature scores of the spatial features. Thus, the feature score of each of these spatial features serves as an indicator of the current reliability of the spatial feature for positioning purposes. Therefore, in response to a triggering condition, at block 902, spatial feature filter module 216 selects spatial features currently included in LADF 134, and at block 904, spatial feature filter module 216 compares the feature score of the selected spatial feature with a specified threshold (specified as “THRESH_L”). This threshold can be a fixed threshold or can be based on one or more current conditions. For example, in one embodiment, the threshold THRESH_L can be set to the median current feature score of candidate spatial features not currently included in LADF 134.

[0068] If the feature score of a candidate spatial feature exceeds the threshold THRESH_L, the spatial feature filter module 216 stops any further consideration of the selected spatial feature (in this round of evaluation), and method 900 returns to block 902 for the selection of the next spatial feature in LADF 134. Otherwise, if the feature score threshold is below THRESH_L, at block 906, the spatial feature filter module 216 signals the positioning generation module 218 to remove the selected spatial feature from LADF 134. In some embodiments, removing a spatial feature from LADF 134 may require the selection of another spatial feature to replace the removed spatial feature and thus serve as the initiator. Figure 8 The triggering condition for the candidate spatial feature evaluation process in Method 800. After removing the selected spatial feature from LADF 134, Method 900 returns to Block 902 for the selection and evaluation of the next spatial feature in LADF 134.

[0069] The LADF maintained by ADF server 102 contains representations of the visual features and geometric information of regions of the world. This information is potentially sensitive, and therefore ADF server 102 takes precautions to prevent unauthorized or unintentional access to the LADF content stored in LADF data store 206 and the ADF content stored in ADF data store 204 (which, in some embodiments, may include the same data store). For this purpose, ADF server 102 employs privacy protection in the form of a two-level indexing scheme for the LADF content in LADF data store 206. Figure 10-12The illustration shows an example configuration of the query module 220 and LADF data store 206 based on this two-level indexing scheme, and their operation. The ADF data store 204 can be configured similarly as described below.

[0070] Figure 10 The illustration shows an example two-layer query interface 1000 implemented by query module 220 and LADF data storage 206. As depicted, each LADF (e.g., LADF 134) stored by LADF data storage 206 is stored in a corresponding LADF entry 1002 in the LADF database 1004 implemented in LADF data storage 206. Each LADF entry 1002 includes multiple fields, including a UID field 1006 storing the UID associated with the LADF stored therein, a spatial feature field 1008 storing a set of spatial features currently represented by the LADF, and a location field 1010 storing a location indicator or other location data associated with the LADF.

[0071] Without sufficient protection, an unauthorized party could potentially obtain LADF134 from LADF data storage 206, thus risking exposure of sensitive information. The two-layer query interface 1000 can more adequately protect the confidentiality of LADF data in LADF data storage 206 by requiring the requesting mobile device 104 or other requesters to prove they are in (or already are in) the area associated with the requested LADF. Some location indicators (such as GPS coordinates) can be easily spoofed or otherwise forged. Furthermore, other types of location indicators (such as WLAN MAC addresses or cell tower BSIDs) can cover areas much farther than the area of ​​interest and may therefore lack sufficient granularity. Thus, a location indicator alone may not be sufficient proof that the requesting mobile device is actually in the specified area. Therefore, the query interface 1000 can instead require the mobile device 104 to submit a set of spatial features observed by the mobile device 104 in the specified area to prove its presence there. The query interface 1000 can then identify a matching LADF by comparing this set of spatial features with the spatial features of the LADF entry. However, many areas have similar structural configurations (e.g., similar rooms in an office building), and therefore there is a risk that multiple LADFs can adequately match the set of submitted spatial features, and thus an incorrect LADF can be supplied to the requesting mobile device, which is doubly problematic, representing both a potential breach of confidentiality and providing the mobile device 104 with an incorrect reference for positioning.

[0072] Therefore, to ensure that the mobile device is indeed in the area it represents and to ensure that the correct LADF is supplied to the mobile device, the query interface 1000 uses both a set of spatial features and one or more location indicators submitted by the mobile device to identify the correct LADF from the LADF database 1004. It should be understood that, under this method, the mobile device or other requester proves that the requester is in the identified area by submitting spatial features observed in the identified area, and therefore the release of the LADF for the identified area carries a low risk of unintentionally disclosing potentially sensitive information, since the requester can simply look around the indicated area to obtain visual information many orders of magnitude more detailed than that represented by the supplied LADF.

[0073] For this purpose, query module 220 includes a spatial feature query interface 1012 and a location selection module 1014. Spatial feature query interface 1012 performs an initial query for candidate LADFs using the spatial feature set 1016 from the LADF request 236 submitted by mobile device 104, based on a search of the spatial feature field 1008 of the LADF entries 1002 in LADF database 1004. This search considers the relative geometry of the spatial features, as they are projected onto the image represented by LADF request 236, to check for consistency with those in LADF database 1004. Location selection module 1014 selects from identified candidate LADFs based on a comparison of the location information of each LADF in the location field 1010 of the candidate LADFs using one or more location indicators 1018 submitted by LADF request 236.

[0074] Furthermore, to prevent the search or other access to LADF entry 1002 using a location indicator or other key other than the set of spatial features, in some embodiments, LADF data storage 206 is configured to index LADF entry 1002 via spatial feature field 1008 and is also configured to avoid indexing LADF entry 1002 via location field 1010. In this configuration, LADF entry 1002 of LADF database 1004 is searchable via the spatial feature set of LADF, but cannot be initially searched by location. In this way, the submission of a location indicator alone will not result in the identification or generation of any LADF 134 from LADF database 1004.

[0075] Figure 11 The illustrations depict at least one embodiment according to the present disclosure. Figure 10Example method 1100 for operating the two-layer query interface 1000. Method 1100 is initiated at block 1102, wherein an LADF request 236 for an indicated area from a mobile device or other requester is submitted. In at least one embodiment, in order to accept the LADF request 236 for processing, the ADF server 102 requires the LADF request 236 to include a set of spatial features 1016 observed at least in the indicated area and one or more location indicators 1018 obtained or determined by a mobile device in the indicated area.

[0076] At block 1104, the spatial feature query interface 1012 searches the LADF database 1004 for LADF entries 1002 to find LADF entries 1002 that have a spatial feature set in a spatial feature field 1008 that sufficiently overlaps with the spatial feature set 1016 of LADF request 236, and that have a relative geometry consistent with the geometry of the spatial features projected from the spatial feature set 1016. In one embodiment, for each LADF entry 1002 analyzed, the spatial feature query interface 1012 can determine a count of matching spatial features between the spatial feature set 1016 and the spatial feature field 1008 of the LADF entry 1002. The spatial feature query interface 1012 can then select N candidate LADFs (e.g., based on the count values ​​determined for the LADF entry 1002) Figure 10 Candidate LADFs 1021, 1022, and 1023). For example, in some instances, each LADF with a count greater than a specified threshold can be selected as a candidate LADF. In other instances, the spatial feature query interface 1012 can select the N LADFs with the highest count values ​​as N candidate LADFs.

[0077] At block 1106, the spatial feature query interface 1012 verifies that at least one candidate LADF has been identified from the LADF database 1004. If not, at block 1108, the query module 220 signals to the requesting mobile device that the LADF is unavailable for the indicated area. Otherwise, one or more identified candidate LADFs are supplied to the location selection module 1014 (e.g., by identifying the UID of the candidate LADF), and at block 1110, the location selection module 1014 scores the match between each candidate LADF and one or more location indicators 1018 compared to the location indicated by the location field 1010 of the candidate LADF. At block 1112, the location selection module 1014 verifies whether there is a sufficient match between the location information of one or more location indicators 1018 and one of the candidate LADFs. If not, at block 1114, the query module 220 signals to the requesting mobile device that the LADF is unavailable for the indicated area. Otherwise, at block 1116, the location selection module 1014 selects the candidate LADF with the best location matching score as the LADF co-located with the identified area, and in response to LADF request 236, provides the selected LADF as LADF 134 for the area to the mobile device.

[0078] In addition to providing protection against unauthorized or unintentional access to LADFs in the LADF database 1004, performing LADF queries using spatial features and, for example, the geometry detected by the mobile device, ensures that the LADF ultimately selected through the two-stage query process will allow the mobile device to successfully locate itself using the matching LADF. This is because the first stage of the LADF query process performs a “localization” process that is the same as or similar to the localization process performed at the mobile device to identify candidate LADFs to find the mobile device’s pose relative to the LADF using the observed spatial features and the image geometry of the LADF.

[0079] Figure 12 The illustration shows an alternative configuration of a two-layer query interface 1000 according to at least one embodiment. (The last sentence appears to be incomplete and possibly refers to a different configuration.) Figure 10 As described in the implementation, the LADF data storage 206 implements an LADF database 1204 comprising multiple LADF entries 1202, each LADF entry 1202 having a UID field 1006, a spatial feature field 1008, and a location field 1010. However, with Figure 10In contrast to the previous implementation, the LADF database 1204 indexes LADF entries 1202 based on the location field 1010. Further, in this embodiment, the query module 220 includes a location query interface 1212 for the LADF database 1204 and a spatial feature selection module 1214.

[0080] In the depicted implementation, the two-level query of the LADF is implemented as a first-level search of the LADF entries 1202 by the LADF location query interface 1212, wherein the LADF has location information of a set of one or more location indicators 1018 that sufficiently match the LADF request 236 to identify the location of one or more candidate LADFs. The spatial feature selection module 1214 then compares the spatial feature set 1016 of the LADF request 236 to select the candidate LADF that best matches the spatial feature set 1016 as the LADF 134 to be supplied to the requesting mobile device in response to the LADF request 236.

[0081] Many of the inventive functionalities and principles described above are well-suited for implementations of integrated circuits (ICs) such as application-specific integrated circuits (ASICs) or therein. It should be anticipated that, regardless of the considerable effort and numerous design choices motivated by considerations such as availability, current technology, and economics, those skilled in the art, guided by the concepts and principles disclosed herein, will be readily able to generate such ICs with minimal experimentation. Therefore, to keep the discussion of such software and ICs brief and minimize any risk based on the principles and concepts of this disclosure, further discussion, if any, will be limited to the essence of the principles and concepts within preferred embodiments.

[0082] In this document, relational terms (such as first and second, etc.) may be used only to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” or any other variations thereof are intended to cover non-proprietary inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Without further constraints, an element preceding “comprises…a” does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element. As used herein, the term “another” is defined as at least a second or more. As used herein, the terms “including” and / or “having” are defined as “comprising.” Referring to electro-optical technology, as used herein, the term “coupled” is defined as connected, but not necessarily directly and mechanically. As used herein, the term "program" is defined as a sequence of instructions designed for execution on a computer system. A "program" or "computer program" can include subroutines, functions, flows, object methods, object implementations, executable applications, applets, service programs, source code, object code, shared libraries / dynamically loaded libraries, and / or any sequence of instructions designed for execution on a computer system.

[0083] The specification and drawings should be considered illustrative only, and therefore the scope of this disclosure is intended to be limited only by the following claims and their equivalents. Note that not all activities or elements described above in the general description are required; a particular activity or part of a device may not be required, and one or more other activities or included elements may be performed in addition to those described. Furthermore, the order in which the activities are listed need not be the order in which they are performed. Unless otherwise specified, the steps of the flowchart depicted above may be in any order, and depending on the implementation, steps may be eliminated, repeated, and / or added. Moreover, concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and changes may be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure.

[0084] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. Any benefit, advantage, solution to problem, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more significant shall not be construed as essential, required, or fundamental features of any or all of the claims.

Claims

1. At a mobile device communicatively coupled to a remote computing system, a method comprising: capturing imagery of an area in which the mobile device is positioned; performing at least one image-based privacy filtering process using the captured imagery to generate filtered imagery; determining a set of spatial features in the filtered imagery; selecting, for an area description file, a subset of the set of spatial features having fewer spatial features than the set of spatial features based on a maximum spatial feature density criterion; generating the area description file representing the subset of spatial features; and communicating the area description file representing the subset of spatial features to the remote computing system. Performing the at least one image-based privacy filtering process includes performing a text filtering process:

2. The method of claim 1, wherein, performing a text detection process to detect one or more first regions of the imagery that potentially represent text content; and modifying the imagery to remove image content from the detected one or more first regions to obtain text-filtered imagery. Performing the at least one image-based privacy filtering process further includes:

3. The method of claim 2, wherein, performing a face detection process to detect one or more second regions of the text-filtered imagery that potentially represent a human face; and modifying the text-filtered imagery to remove image content from the detected one or more second regions. Performing the at least one image-based privacy filtering process includes:

4. The method of claim 1, wherein, performing a face detection process to detect one or more regions of the imagery that potentially represent a human face; and modifying the imagery to remove image content from the detected one or more regions. The maximum spatial feature density criterion includes a specified maximum number of spatial features per cubic unit.

5. The method of claim 1, wherein, 6. A mobile device comprising: at least one imaging sensor that captures imagery of an environment of the mobile device; at least one processor; a storage component storing a set of executable instructions configured to manipulate the at least one processor to: perform at least one image-based privacy filtering process using the captured imagery to generate filtered imagery; determine a set of spatial features in the filtered imagery; select, for an area description file, only a subset of the set of spatial features having fewer spatial features than the set of spatial features by selecting spatial features of the subset based on a maximum spatial feature density criterion; and generate the area description file representing the subset of spatial features; and a network interface coupled to the at least one processor that communicates the area description file to a remote computing system. The set of executable instructions is further configured to manipulate the at least one processor to perform a text filtering process: detect one or more first regions of the imagery that potentially represent text content and modify the imagery to remove image content from the detected one or more first regions to obtain text-filtered imagery. The set of executable instructions is configured to manipulate the at least one processor to:

7. The mobile device of claim 6, wherein, ​ ​ 8. The mobile device of claim 7, wherein, ​ detecting one or more second regions of the text filtered imagery that potentially represent a human face and modifying the text filtered imagery to remove image content from the detected one or more second regions.

9. The mobile device of claim 6, wherein, The set of executable instructions is configured to manipulate the at least one processor to: detect one or more regions of the imagery that potentially represent a human face and modify the imagery to remove image content from the detected one or more regions.

10. The mobile device of claim 6, wherein, The maximum spatial feature density criterion includes a specified maximum number of spatial features per cubic unit. The maximum spatial feature density criterion includes a specified maximum number of spatial features per cubic unit.

Images