LiDAR-enabled wayfinding systems

Abstract

The LiDAR-enabled wayfinding system includes a LiDAR-equipped mapper mobile device adapted to perform LiDAR scans within a building, thereby creating a building universal scene description (USD); a management server operable to receive the building USD and generate a building bundle file including a plurality of waypoints associated with a corresponding plurality of waypoint fingerprints; and a LiDAR-equipped user mobile device adapted to perform LiDAR scans of a user position within the building and generate a user USD and a user position fingerprint, the user position fingerprint being compared to the plurality of waypoint fingerprints to identify at least one waypoint proximate to the user.

Description

[Background technology]

[0001] Various technologies have been developed to track a user's indoor location and display it on a mobile device's map. These technologies typically involve stationary hardware to which the mobile device connects to triangulate based on the user's distance from each hardware device. Such hardware devices can include Bluetooth Low Energy beacons (BLE), WiFi access points, or Bluetooth devices such as lights and badge readers. Hardware-based wayfinding is preferred because Global Positioning System (GPS) signals are often inaccurate indoors and cannot provide floor-specific signals.

[0002] The drawbacks of hardware-based indoor wayfinding are that these systems are expensive, navigation routes for the wayfinding system must be generated before fingerprinting, and they are typically battery-powered and require regular maintenance.

[0003] These and other limitations in the prior art will become apparent to those skilled in the art upon reading the following detailed description and upon viewing the drawings. Summary of the Invention

[0004] An example of a LiDAR-enabled wayfinding system includes a LiDAR-equipped mapper mobile device adapted to perform LiDAR scans within a building, thereby creating a building universal scene description (USD); a management server operable to receive the building USD and generate a building bundle file including a plurality of waypoints associated with a corresponding plurality of waypoint fingerprints; and a LiDAR-equipped user mobile device adapted to perform LiDAR scans of a user position within the building and generate a user position USD and a user position fingerprint, the user position fingerprint being compared to the plurality of waypoint fingerprints to identify waypoints proximate to the user position.

[0005] An example method for LiDAR-enabled wayfinding includes LiDAR scanning a plurality of areas within a building with a mapper mobile device LiDAR device, creating a building USD from the LiDAR scans within the building, and generating a building bundle file including a plurality of waypoints within the building and a plurality of segments connecting the plurality of waypoints.

[0006] An example of a non-transitory computer-readable medium includes a code segment for LiDAR scanning an area within a building with a user LiDAR device, thereby generating a user position universal scene description (USD); a code segment for creating a position fingerprint from the position USD; a code segment for comparing the position fingerprint with a plurality of waypoint fingerprints associated with a plurality of waypoints of the building, thereby predicting a waypoint position for the user within the building; and a code segment for providing directions for navigating from the user position to a desired destination within the building.

[0007] These and other embodiments, features and advantages will become apparent to those skilled in the art upon reading the following detailed description and reviewing the several figures in the drawings. [Brief explanation of the drawings]

[0008] Some embodiments will now be described with reference to the drawings, in which like reference numerals are provided to like components. These example embodiments are intended to illustrate, but not to limit, the present invention. The drawings include the following figures:

[0009] [Figure 1] FIG. 1 shows a LiDAR-enabled wayfinding system.

[0010] [Figure 2A] FIG. 2A is a front view of an example of a LiDAR-equipped mobile device.

[0011] [Figure 2B] FIG. 2B is a rear view of the example LiDAR-equipped mobile device of FIG. 2A.

[0012] [Figure 3] FIG. 3 is a block diagram of an example LiDAR-equipped mobile device.

[0013] [Figure 4] FIG. 4 is a block diagram illustrating an example of a server device.

[0014] [Figure 5] FIG. 5 is a diagram illustrating the use of a LiDAR-equipped mobile device.

[0015] [Figure 6] 6A-6D are diagrams illustrating several different types of LiDAR-equipped user mobile devices.

[0016] [Figure 7] FIG. 7 is a flow diagram illustrating an example process performed by the LiDAR-equipped user mobile device of FIG.

[0017] [Figure 8] FIG. 8 is a flow diagram illustrating an example process performed by the LiDAR-equipped mapper mobile device of FIG.

[0018] [Figure 9] FIG. 9 is a flow diagram illustrating an exemplary process performed by the administrator station of FIG.

[0019] [Figure 10] FIG. 10 is a flow diagram illustrating an exemplary process performed by the management server of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0020] 1 includes a LiDAR-enabled wayfinding system 10 including a LiDAR-equipped mapper mobile device 12, an administration server 14, a plurality of LiDAR-equipped mobile user devices 16, and an administrator station 18. The administration server 14, in this example, communicates with the mapper mobile device 12, the plurality of user devices 16, and the administrator station 18 over a network, such as the Internet 20, thereby providing access to an administration server database 22.

[0021] 2A and 2B are front and rear views of an example LiDAR-equipped mobile device 24 that, with appropriate software, can be used with the hardware / software platform of LiDAR-equipped mapper mobile device 12 and / or LiDAR-equipped mobile user device 16. For example, mobile device 24 can be an iPhone™ Pro 13 manufactured by Apple Inc. of Cupertino, California.

[0022] 2A, mobile device 24 includes housing 26 and touchscreen 28, which displays a plurality of home screen application (“app”) icons 30 and a plurality of fixed screen app icons 32. Tapping an app icon on touchscreen 28 launches the associated app. In FIG. 2B, the back of LiDAR-equipped mobile device 24 includes housing 26 and area 34 housing three camera lenses 36, a flash 38, and a LiDAR module 40.

[0023] FIG. 3 shows an electronic block diagram of the LiDAR-equipped mobile device 24. Note that this diagram is for illustrative purposes only and is not limiting. Here, the LiDAR-equipped mobile device 24 includes a main circuit 42 and input / output (I / O) components such as a touchscreen 28, a camera / flash 36 / 38, a LiDAR module 40, a speaker 44, and a microphone 46. The main circuit 42 is powered by a battery 48 and is turned on and off by a switch 50. In this embodiment, the main circuit 42 is provided with a universal serial bus (USB) 52. A transmit / receive (Tx / Rx) switch 54 and a Bluetooth / GPS (BT / GPS) module 56 couple an antenna 58 to the main circuit 42.

[0024] The main circuitry 42 of the LiDAR-equipped mobile device 24 includes a processor (CPU) 60 capable of executing applications (apps) and a read-only memory (ROM) 62 coupled to the CPU 60. The ROM 62 may be, for example, an electrically erasable programmable read-only memory (EEPROM) or flash memory and may store data, code segments, and objects such as app "A." Other memory includes a random access memory (RAM) 64 and a removable subscriber identity module (SIM) 66 that identifies the subscriber and the device. The exemplary main circuitry 42 also includes a CODEC 68, a baseband and audio / speech processing digital signal processor (DSP) 70, a digital-to-analog converter (DAC) and analog-to-digital converter (ADC) 72, and a radio frequency (RF) module 74 for frequency conversion, power amplification, etc.

[0025] The LiDAR module 40 of the LiDAR-equipped mobile device 24 operates to scan the field of view (FOV) of one or more camera lenses 36 with infrared (IR) laser pulses and detect the time it takes for each pulse to reflect back. The distance "d" between the LiDAR module 40 and the point on the surface from which the pulse reflects is simply d = (c * t) / 2, where c is the speed of light and t is the elapsed time between transmission and reception of the pulse. In this example, the LiDAR-equipped mobile device 24 is used to scan the interior of a building by sequentially scanning rooms and other areas within the building by moving the mobile device to point it at various locations, such as within rooms, hallways, and open areas of the building.

[0026] 4, exemplary computer 76 includes a microprocessor (PC) 78, read-only memory (ROM) 80, random access memory (RAM) 82, mass storage device 84, network interface 86, and input / output (I / O) 88. Computer 76 is suitable for use as administrator station 18 if I / O 88 includes a computer monitor, keyboard, and mouse, and is suitable for use as management server 14 if mass storage device 88 is separate from or includes management server database 22. Computer 76 can also be used as an integrated computer / server, integrating the functionality of management server 14 and management station 18.

[0027] FIG. 5 illustrates the use of a LiDAR-equipped mobile device 24 as an environmental scanner, such as the mapper mobile device 12 or the user mobile device 16, to generate a 3D model of one or more interior areas of a building, where the 3D model may be stored in Universal Scene Description (USD) format. In this example, the LiDAR-equipped mobile device 24 is held in the user's hand with the touchscreen 28 facing the user, thereby facilitating LiDAR scanning of the environment in multiple directions "d" and orientations "o." An app controlling the LiDAR scanning process (e.g., app "A" in FIG. 3 ), in this example, may ask the user to pick up the LiDAR-equipped mobile device 24 and slowly scan the environment (e.g., a room) by walking around and changing the direction and orientation of the device until the entire room has been scanned. Preferably, app A attempts to provide visual indicators that walls, windows, and other objects are being successfully scanned. If the user is moving too quickly, app A may recommend slowing down. Once scanning one room is complete, the process may be repeated for other rooms, hallways, open spaces, etc. Other data such as compass, altimeter, BLE Bluetooth, WiFi and / or GPS data (if available) may also be included as detailed information.

[0028] 6A-6D show several example wayfinding displays by a LiDAR-equipped user mobile device 16. Here, "display," as used herein, means a visual display, such as on a touchscreen 28, an audible display, e.g., via a speaker 44, or other user feedback, such as a tactile display. FIG. 6A shows an example touchscreen display 28A that includes a cross-sectional 3D rendering 90 of a building and instructions 92 on how to navigate within the building from a user location 94 to a destination 96. FIG. 6B shows an example touchscreen display 20B that includes a 2D map 98 and instructions 100 on how to navigate from a user location 102 to a destination 104 along a route 106. FIG. 6C shows an augmented reality (AR) display 108 on touchscreen display 28C that uses one or more cameras 36 and LiDAR module 40 to display the current environment with an overlay of instructions 110, 112 and a route line 114 to a desired destination (in this example, the "check-in" counter). FIG. 6D shows a textual directions display 116 on a touchscreen display 28D with the option to have the directions spoken by selecting a "sound" icon 118.

[0029] 7 is a flow diagram of an example process (a.k.a., a "method") 120 executed by a LiDAR-equipped user mobile device 16, such as by app "A" of FIG. 3. Process 120 begins at 122, where a user performs a LiDAR scan of their local environment (e.g., a room) at operation 124, thereby generating a location 3D model as a user location USD. Next, at operation 126, a user location fingerprint is created from the location USD, and at operation 128, the user location fingerprint is compared to the building USD waypoint fingerprints to predict a building waypoint for the user location. Note that a user location fingerprint may be associated with multiple waypoint fingerprints (e.g., a building may have multiple rooms with similar layouts). In that case, a ranked list of matches may be provided, as in the example shown below. A ranked list of possible waypoints (1) 82% Waypoint 47 (2) 13% Waypoint 52 (3) 3% Waypoint 98 Process 120 may then repeat operations 124-128 until the most likely waypoint is selected or a sufficiently high degree of confidence is achieved in the location of the waypoint. Finally, in operation 130, the user is provided with directions from the user's location to the desired destination within the building. After arriving at the destination, process 120 ends at 132.

[0030] 8 is a flow diagram of an example process 134 executed on a LiDAR-equipped mapper mobile device 12, for example, by app "A" of FIG. 3. Process 134 begins at 136, where a LiDAR scan is performed by the mobile device 12 in operation 138 of multiple areas within a building (e.g., rooms, hallways, open spaces, etc.). This LiDAR scan is then used to create building USD, optionally along with a building bundle file containing metadata associated with the building, in operation 140. The building USD (and bundle file, if present) are then transferred to the management server 14 in operation 142. Process 134 then ends at 144.

[0031] FIG. 9 is a flow diagram of an example process 146 implemented on the administrator station 18 of FIG. 1 . Process 146 begins at 148, where a building USD is accessed in operation 150, e.g., from the management server 14. Next, in operation 152, building waypoints and connecting segments are designated for the building. For example, a room may have a waypoint and a hallway outside the room may be another waypoint, with a connecting segment between the two. It should be understood that the designation of waypoints and connecting segments may be determined manually by the administrator or may be generated automatically. Next, in operation 154, a path between the waypoints is designated automatically or manually. Alternatively, the path may be determined later, e.g., on the LiDAR-equipped mapper mobile device 12. Optionally, in operation 156, the administrator may designate points of interest (POIs) for the building. Operation 158 manages the building bundle file by creating or updating a file containing metadata, e.g., waypoints, segments, paths, POIs, etc. Process 146 ends at 160.

[0032] FIG. 10 is a flow diagram of an example process 162 implemented on the management server 14 of FIG. 1. Process 162 begins at 164, where operation 166 determines whether there has been a server request. If there is no server request, operation 166 remains idle. If there is a server request from a user, e.g., a user of a LiDAR-equipped user mobile device 16, operation 168 determines whether the user needs new or updated building USD and / or building bundle files. If not, control returns to operation 166. If new or updated building USD and / or building bundle files are needed, operation 170 provides the new or updated building USD and / or building bundle files to the user, e.g., via a download operation over the Internet 20. If there is a server request from a mapper, e.g., a server request from a LiDAR-equipped mapper mobile device 12, operation 172 stores the building USD along with the building bundle files, e.g., in database 22 of FIG. 1. If operation 166 receives a server request from an administrator, e.g., from administrator station 18, operation 174 provides the administrator with access to the specified building USD. This server request may be sent, for example, in response to operation 150 accessing a building USD of FIG. 9. In operation 176, waypoints and connecting segments are created, for example, in response to operation 152 specifying waypoints and connecting segments of FIG. 9. In operation 178, a route is created automatically, in response to operation 154 of FIG. 9, or by a combination of the two. In operation 180, points of interest (POIs) are created in response to operation 156 of FIG. 9. Finally, operation 182 updates (or creates) a building bundle file using metadata derived from operations 176-180 and any operation 158 managing bundle files of FIG. 9. Process control then returns to operation 166 to await further service requests.

[0033] The Universal Scene Description (USD) is a framework for the exchange of 3D computer graphics data developed by Pixar Animation Studios ("Pixar") in Emeryville, California (now a subsidiary of Walt Disney Studios in Burbank, California). The USD framework was first released as open source software in 2016.

[0034] In Apple's recent release of iOS 16, a new application program interface (API) for a technology known as "RoomPlan" uses the iPhone's built-in LiDAR and camera to build 3D floor plans. A description of RoomPlan, along with sample code, can be found at https: / / developer.apple.com / documentation / roomplan. RoomPlan can be invoked by apps to create 3D models of rooms. The RoomPlan framework uses iPhone sensors, trained ML models, and RealityKit's rendering capabilities to capture the physical surroundings of a room. For example, the framework examines the iPhone's camera feed and LiDAR readings to identify walls, windows, openings, and doors. RoomPlan also recognizes room features, furniture, and appliances, such as fireplaces, beds, and refrigerators, and provides that information to apps.

[0035] To start capturing, the app displays a view (RoomCapture View) where the user can see their room in augmented reality (AR), which displays virtual cues as they move around the room: A real-time graphic overlay is displayed on the physical structure of the room to communicate the progress of the scan. If the framework requires a specific type of device movement or viewpoint to complete the capture, the UI will display instructions explaining how to position the device.

[0036] When the app determines that the current scan is complete, a view displays a small-scale version of the scanned room for user confirmation. Alternatively, apps can display custom graphics during the scanning process by directly creating and using a scanning session object (RoomCaptureSession). The framework outputs scan data as parametric data, making it easy for apps to modify individual components of the scanned room. RoomPlan also provides these results in Universal Scene Description (USD) format.

[0037] In computer science, a fingerprinting algorithm is a procedure that maps an arbitrarily large data item, software, or other digital file (a "digital object") to a much shorter string of bits known as its "fingerprint," which, for all practical purposes, uniquely identifies the original digital object. Typically, fingerprinting algorithms use high-performance hash functions to uniquely identify the digital object.

[0038] In the current example, a fingerprint generated from user location USD for a room will differ somewhat from a fingerprint for the room generated from building USD. The building bundle may aid the comparison process by providing metadata about the room, including the total volume of the room, the fractal dimension of the color patterns of the floor, walls, and ceiling, and the fractal dimension of large-scale objects in the scan. See, for example, Fractal Dimension (FD): An Image as a Single Real Number, MAST Research Project, University of Plymouth, URL accessed November 17, 2022: https: / / www.plymouth.ac.uk / research / materials-and-structures-research-group / fractal-dimension-fd-image-as-a-single-real-number.

[0039] In this example, the set of numbers derived from the above process can now be considered to be the coordinates of a point in feature space. This "feature vector" represents the associated scans, and this association is recorded in a database. In an exemplary embodiment, recursive subdivision of the feature space is used to organize the feature vector, thereby reducing subsequent search time using an index.

[0040] As the user navigates, LiDAR scans are taken and a feature vector is generated for the scan. This feature vector is used to identify the closest scan stored in the index. The Euclidean distance between the search feature vector and the feature vectors of potential matches is used as a score of potential matches. These potential matches may be ranked by probability ("score") as described above.

[0041] While various embodiments have been described using specific terms and devices, such description is for purposes of illustration only. The terms used are terms of description and not of limitation. Those skilled in the art will understand that changes and modifications may be made without departing from the spirit or scope of the various inventions supported by the written disclosure and drawings. Furthermore, it will be understood that aspects of various other embodiments may be interchanged in whole or in part.

Claims

1. a LiDAR-equipped mapper mobile device adapted to perform LiDAR scanning within a building and thereby create a building universal scene description (USD); a management server operable to receive the building USD and to maintain a building bundle file including a plurality of waypoints associated with a corresponding plurality of waypoint fingerprints; a LiDAR-equipped user mobile device adapted to LiDAR scan a user location within the building and generate a user location USD and a user location fingerprint, the user location fingerprint being compared to the plurality of waypoint fingerprints to identify waypoints proximate to the user location; 1. A LiDAR-enabled wayfinding system comprising:

2. The LiDAR-enabled wayfinding system of claim 1 , further comprising a management station connected to said management server thereby managing said building bundle file.

3. The LiDAR-enabled wayfinding system of claim 2 , wherein the building bundle file further includes a plurality of segments connecting the plurality of waypoints.

4. The LiDAR-enabled wayfinding system of claim 3 , wherein the building bundle file further includes a plurality of routes that include the plurality of waypoints and at least some of the plurality of segments.

5. The LiDAR-enabled wayfinding system of claim 4 , wherein the building bundle file further includes one or more points of interest (POIs).

6. the building includes a plurality of rooms; 3. The LiDAR-enabled wayfinding system of claim 2, wherein the LiDAR-equipped mapper mobile device is adapted to sequentially LiDAR scan the plurality of rooms, thereby at least partially creating the building USD.

7. 7. The LiDAR-enabled wayfinding system of claim 6, wherein the LiDAR-equipped mapper mobile device comprises at least one position sensor selected from the group consisting essentially of a compass, an altimeter, a BLE Bluetooth receiver, a WiFi receiver, and a GPS device.

8. The LiDAR-enabled wayfinding system of claim 2 , wherein both the plurality of waypoint fingerprints and the user position fingerprint are created by a hashing process.

9. 10. The LiDAR-enabled wayfinding system of claim 8, wherein the LiDAR-equipped user mobile device performs a ranking of correlations between the location fingerprint and the plurality of waypoint area fingerprints, thereby generating a ranked list of waypoints proximate to the user.

10. a user performing a LiDAR scan within a building, thereby generating a user-located universal scene description (USD); creating a user location fingerprint from the user location USD; comparing the user location fingerprint to a plurality of waypoint fingerprints associated with a plurality of waypoints of the building, thereby predicting a waypoint location of the user within the building; providing the user with directions for navigating from the user location to a desired destination within the building; A method for LiDAR-enabled wayfinding, comprising:

11. 11. The method for LiDAR-enabled wayfinding of claim 10, wherein LiDAR scanning within a building includes LiDAR scanning around the user in multiple directions and orientations.

12. LiDAR scanning a plurality of areas within a building with a mapper mobile device LiDAR device; creating a building USD from a LiDAR scan of the building; generating a building bundle file including a plurality of waypoints within the building and a plurality of segments connecting the plurality of waypoints; 12. The method for LiDAR enabled wayfinding of claim 11, comprising:

13. 13. The method for LiDAR-enabled wayfinding of claim 12, wherein the building USD includes at least one location parameter selected from the group consisting of compass heading, altitude, beacon identifier, and GPS location information.

14. 12. The method for LiDAR-enabled wayfinding of claim 11, further comprising the step of performing a ranking of correlations between the location fingerprint and the plurality of waypoint fingerprints, thereby providing a ranked list of waypoints proximate to the user.

15. A non-transitory computer-readable medium containing code segments executable on a user LiDAR device, a code segment for LiDAR scanning an area within a building with a user LiDAR device, thereby generating a user location universal scene description (USD); a code segment for creating a location fingerprint from the location USD; a code segment for comparing the location fingerprint to a plurality of waypoint fingerprints associated with a plurality of waypoints of the building, thereby predicting a waypoint location of the user within the building; and a code segment for providing directions for navigating from the user location to a desired destination within the building; A non-transitory computer-readable medium comprising code segments executable on a user LiDAR device, including:

16. 16. A non-transitory computer-readable medium comprising code segments executable on a user LiDAR device as described in claim 15, wherein the LiDAR scan of an area within a building includes LiDAR scanning the area in multiple directions and orientations.

17. a code segment for LiDAR scanning, by a mapper mobile device, of multiple areas within a building; a code segment for creating a building USD from a LiDAR scan of the building; a code segment for generating a building bundle file including a plurality of waypoints within the building and a plurality of segments connecting the plurality of waypoints; 20. A non-transitory computer-readable medium comprising a code segment executable on a user LiDAR device as described in claim 16, comprising:

18. 20. A non-transitory computer-readable medium comprising code segments executable on a user LiDAR device as described in claim 17, wherein the building USD includes at least one location parameter selected from the group consisting of compass heading, altitude, beacon, and GPS location information.

19. 17. A non-transitory computer-readable medium comprising a code segment executable on a user LiDAR device as described in claim 16, further comprising a code segment for performing a ranking of correlations between the location fingerprint and the plurality of waypoint fingerprints, thereby providing a ranked list of waypoints proximate to the user.

Images