Systems and methods for hands-free pedestrian navigation

Abstract

To provide hands-free pedestrian navigation system and method.SOLUTION: A hands-free pedestrian navigation method includes the steps of: (i) mounting on a user's head a display for projecting a visual image in front of the user's gaze and an IMU; and (ii) obtaining from a GPS unit carried by the user an approximate user location for locating the user in a computerized map. Confirmation is obtained from the user that the user's gaze is directed to a specified landmark in sight of the user and azimuth is computed between the user location and the landmark location extracted from the computerized map. Vocal prompts are provided and ancillary visual prompts are projected on the display to navigate the pedestrian.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a pedestrian navigation system. In some embodiments, this relates to the calibration of the magnetometer of an inertial magnetic unit (IMU) to ensure that the direction of movement of a pedestrian coincides with the intended destination.

Background Art

[0002] The following are listed prior art references that are considered relevant as background to the present invention, the contents of which are incorporated herein by reference. It should not be assumed that the references herein are meant to imply that they are relevant in any way to the patentability of the invention disclosed herein. Each reference is identified by a number in square brackets, so the prior art is referred to by numbers in square brackets throughout the specification. [1]Nehla Ghouaiel, Jean-Marc Cieutat, Jean Pierre Jessel. “Haptic System for Eyes Free and Hands Free Pedestrian Navigation.” ACHI 2013: The Sixth International Conference on Advances in Computer-Human Interactions, Feb 2013, France pp.330-335 <hal-00908028> [2]McGookin, D., Brewster, S., & Priego, P. (2009).「Audio Bubbles:Employing Non-speech Audio to Support Tourist Wayfinding」.In M. Ercan Al Audio Bubbles: tinsoy, U. Jekosch, & S. Brewster (Eds.), Haptic and Audio Interaction Design: 4th International Conference, HAID 2009 Dresden, Germany, September 10-11, 2009 Proceedings (Vol. 5763, pp. 41-50). Springer. https: / / doi.org / 10.1007 / 978-3-642-04076-4_5 [3]Ioannis Giannopoulos et al. GazeNav: Gaze-Based Pedestrian Navigation, Mob ileHCI ’15, August 25 - 29, 2015, Copenhagen, Denmark [4]Michael Minock et al. A Test-Bed for Text-to-Speech-Based Pedestrian Navigation Systems [5]Johan Boye et al. Walk This Way: Spatial Grounding for City Exploration J. Mariani et al. (eds.), Natural Interaction with Robots, Knowbots and Smartphones: Putting Spoken Dialog Systems into Practice, DOI 10.1007 / 978-1-4614-8280-2 6, Springer Science+Business Media New York 2014 [6]DWF van Krevelen and R. Poelman A Survey of Augmented Reality Technologies, Applications and Limitations in The International Journal of Virtual Reality, 2010, 9(2):1-20 [7]Bartie P, Mackaness W, Lemon O, Dalmas T, Janarthanam S, Hill R, Dickinson A & Liu X (2018) A dialogue based mobile virtual assistant for tourists: The SpaceBook Project, Computers, Environment and Urban Systems, 67, pp. 110-123. [8]Gabriel Skantze Error Handling in Spoken Dialogue Systems PhD Thesis, Stockholm, Sweden 2007 [9]Ricky Jacob, PhD Thesis "Integrating Haptic Feedback into Mobile Location Based Services" Department of Computer Science National University of Ireland Maynooth, July 2013

[0003] The proliferation of smartphones has led to the emergence of various personal or pedestrian navigation systems that leverage the widespread use of smartphones and their standard built-in GPS. The most common systems are handheld; that is, users must manually hold the system and, in many cases, read a map of their immediate vicinity downloaded based on the GPS coordinates of their smartphone.

[0004] This method, which requires holding a smartphone and looking down at the screen or raising it to bring the screen to eye level, is disruptive and imposes undesirable limitations. For example, tourists walking down the streets of a foreign city are engrossed in looking at tourist attractions. It is disruptive for tourists to be forced to turn their gaze to their smartphone screen to find directions to a particular attraction. Furthermore, tourists need to hold their smartphones, which can also be undesirable.

[0005] This recognition of inconvenience led to the development of so-called hands-free navigation systems.

[0006] Ghouaiel et al. [1] disclose a hands-free navigation system that uses vibration elements and augmented reality in a haptic modality to guide pedestrians in an urban environment via hands-free and eyes-free navigation.

[0007] McGookin et al. [2] have disclosed the concept of an audio bubble, which is a virtual sphere filled with sound, centered around a physical landmark, and provides navigation homing information to help the user find the landmark more easily.

[0008] Giannopoulos et al. [3] disclose using a head-mounted eye tracker to provide gaze-based interaction for pedestrian navigation, providing navigation information when the street the user is looking at is the street being tracked.

[0009] Minock et al. [4] have disclosed an Android system that supports eyes-free and hands-free navigation in urban areas. In one mode, a human operator sends text messages, implemented via TTS, to the user's earpiece. The operator checks the subject's GPS location on a map, listens to the subject's speech, and reviews 1fps video acquired from the subject's phone, which is worn in the form of a necklace.

[0010] Boye et al. [5] describe a voice-interactive prototype for pedestrian navigation in Stockholm that addresses the issue of GPS reading noise using various grounding strategies, which involves first establishing the user's current location and direction using nearby reference landmarks before giving instructions to the navigation system.

[0011] The discussion of grounding in this paper is relevant to the present invention as it addresses a similar problem to which the present invention pertains. However, the proposed solution is entirely different. In particular, Boye et al. [5] use landmarks to guide the user to the first node in the map and monitor the user's progress toward the designated node in real time. As long as the distance to the next node is decreasing, there is no problem. When the distance to the next node begins to increase, the system also checks the distance to the subsequent node B to determine whether the user has passed the expected node A and is moving in the right direction. However, if it is determined that the distance to both the expected next node A and the expected consecutive node B is increasing, and this pattern continues for 10 seconds, the system assumes the user is walking in the wrong direction and issues a replan.

[0012] What this means is that Boye et al.[5] are taking note of the problem of correctly guiding mobile users to designated destinations and the shortcomings of GPS in accurately determining a user's location, while using a trial-and-error approach to correct directional errors. In other words, their approach takes into account the fact that landmark-based guidance can result in pedestrians being led away from their intended destinations, but it is about detecting errors and taking appropriate corrective actions.

[0013] Krevelen et al. [6] have disclosed a detailed overview of AR techniques, including a description of head tracking. They note that inertial navigation systems employ accelerometers and gyroscopes. These instruments, when combined with accurate azimuth information, can provide a practical dead reckoning for estimating position by time measurement. To minimize drift-related errors, the estimations need to be updated regularly with accurate measurements. They also examine tracking in unprepared environments and acknowledge problems with the calibration of these devices.

[0014] Bartie et al. [7] similarly provide a complete overview of pedestrian navigation systems, including augmented reality (AR) applications, which typically use a head-mounted camera to image the landscape in the user's line of sight and transmit the images to a navigation server via the user's smartphone. The problem with such approaches is that they require relatively high bandwidth to transmit digital images, which drains the smartphone's battery. They also explain that because Global Navigation Satellite System (GNSS) can only calculate direction from movement history, it is necessary to clarify which direction to start walking from when GNSS direction values ​​are unreliable. They state that because the smartphone was not held flat and was in the user's pocket or bag, the smartphone's magnetometer could not be used, resulting in a very noisy output.

[0015] A solution proposed by [7] to guide the user is to refer to a well-known nearby landmark and ask the user to keep them to the right or left when departing (for example, “Keep Blackwell’s Bookstore to your right”). This would mean the user would not have to backtrack, if the Global Navigation System (GNSS) can derive the correct direction from the orbital history.

[0016] Bartie[7] et al. also mention the need to customize the naming selection based on the user's approach angle so that the system's reference to buildings matches the user's view. This is done by linking the address information of specific residents to the road network so that a direction is assigned to each resident's point.

[0017] Skantze[8] further provides background on many of the issues related to pedestrian navigation and grounding. Section 6.1 describes the Higgins Spoken Dialogue System, used in a simulated 3D model of a virtual city to guide pedestrians using verbal dialogue. This system cannot access the user's location by positioning devices. Instead, the user must determine their location by engaging in dialogue that describes their surroundings. However, the authors acknowledge that this is not feasible in real cities because it is impossible to fully model a city and real-world applications rely on GPS.

[0018] US2003 / 018430 discloses a method and apparatus for pedestrian navigation operating in dead reckoning mode, using a standard small inertial navigation system (INS) module as the source of acceleration signals. Figure 12K is a flowchart of an algorithm for automatic detection and correction of magnetic disturbances in a digital magnetic compass based on a comparison of displacement bearings provided by the digital magnetic compass and / or compass output and one or more gyroscopes that are initially aligned.

[0019] US2015 / 0354951 discloses a method and apparatus for determining a mismatch between a pedestrian holding or wearing the apparatus in a different orientation, in a constrained or unconstrained manner, and the apparatus itself. The apparatus may be head-mounted, such as smart glasses, smart goggles, ear-mounted systems, or helmet-mounted systems. One embodiment provides a pedestrian dead reckoning method for correcting a mismatch between the orientation of the apparatus and the orientation of the pedestrian.

[0020] US2013 / 0322667 discloses a personal navigation system including a head-mounted hearing aid with a speaker for emitting sound toward the user's ears when the user wears the hearing aid in a intended operating position on the user's head, and a hearing aid housing an inertial magnetic unit for determining the user's head movement. The system includes a GPS unit for determining the user's geographic location, a sound generator connected to output an audio signal to a speaker, and a pair of filters with a head-related transfer function connected in parallel between the sound generator and the speaker for generating a binaural acoustic signal, where the speaker is configured to emit sound toward the user's ears so that the user perceives it as sound from a sound source positioned in a direction corresponding to the head-related transfer function.

[0021] US20120050144 discloses a wearable head-mounted augmented reality computing device that may include a magnetometer or gyroscope for determining direction. A set of sensors constituting the INS would need to be mounted on the user's head so that they could be used to accurately determine the direction of the field of view. The device could be used for pedestrian routing / navigation. For this purpose, a display (displayed as part of a smartphone) and reflective devices are mounted on a harness worn on the head and configured so that the displayed information is actually superimposed. A microphone and earphones may be mounted on the harness to provide verbal commands and voice output, but integrating such functions to assist navigation is not proposed. In fact, the proposed use of the device is for navigation, and no explanation is provided as to how the device could be used for such purposes; the explanation is limited to providing a head-mounted see-through display that can use a smartphone with sensors that can determine the direction of the user's gaze. Such a device would be cumbersome, require a custom harness to support the smartphone, and perhaps most importantly, would make the smartphone inaccessible for normal use.

[0022] US20200195333 discloses a system comprising two wearable devices associated with the right and left sides of the user's body, each containing an RF beacon. Head posture or gaze detection is estimated by comparing the signal intensity (e.g., RSSI) or RF signal phase from the wearable devices to a digital camera device. An angular deviation (e.g., gaze) between the gaze detection and the camera is estimated, and the camera is activated or controlled based on comparing the angular deviation to a set threshold.

[0023] The use of IMUs for human movement and posture is described in Martin Veskrna's 2013 master's thesis, "Positioning system for small devices using principles of inertial navigation system," from the Faculty of Informatics, Masaryk University, and in "Self-Contained Position Tracking of Human Movement Using Small Inertial / Magnetic Sensor Module," presented by Xiaoping Yun et al. at the 2007 IEEE International Conference on Robotics and Automation (Rome, Italy, 10-14 Apr. 2007).

[0024] Both of these papers relate to pedestrian navigation and the need to calibrate inertial navigation systems to correct for magnetic sensor drift. Different techniques are used. For example, gyroscopes are commonly provided to deliver directional signals.

[0025] Jacobs [9] provides a detailed investigation of the then (2013) approaches for pedestrian navigation and states that, in relation to mobile-based location information services (MBLS), the most common interaction method used in MBLS is the use of an overlay of information on a map for navigation assistance and spatial query response. There are also systems that provide text feedback including turn-by-turn route guidance, and systems that integrate landmark information and photos. Researchers have also integrated panoramic images of locations or other geotagged images with information overlays to provide feedback to users. He notes that due to the availability of low-cost on-board digital compasses, there is a growing popularity of bearing-based mobile spatial interaction, shifting from purely location-based systems to direction recognition systems.

[0026] He also mentions the use of augmented reality in mobile spatial interaction systems where the position and orientation of a mobile device are used with the device's camera and overlaid on real-time images to provide spatial information to the user.

[0027] US20160224855 uses an optical sensor to estimate the orientation deviation between a portable device such as a mobile phone / smartphone and the person carrying the device. In a conventional inertial navigation system, usually, alignment of the inertial sensors within the platform (e.g., alignment of the portable electronic device containing the inertial sensors with the forward axis, lateral axis, and vertical axis of the platform) is required. If the inertial sensors are not properly aligned, the position and orientation calculated using measurements from the inertial sensors do not represent the state of the platform. Therefore, to achieve a high-precision navigation solution, it is necessary to connect the inertial sensors within the platform.

[0028] The above commentary merely superficially addresses a variety of different efforts for providing a hands - free system that provides or, in some cases, avoids the need to initialize or calibrate a pedestrian navigation system and an inertial guidance system. Thus, Bartie et al. [7] avoid the need to do so by providing a completely voice - based navigation system that does not rely on the smartphone's magnetometer.

[0029] Such an approach can be highly distracting. For a single tourist in an unfamiliar environment, the constant verbal interaction might provide reassurance, but the user will effectively have to keep paying attention to the navigation system. If a vacationing couple uses such a system, it will be difficult for them to converse without constant interruption.

[0030] Boye et al. [5] also employ a voice - based system that corrects alignment errors by trial and error.

[0031] WO2019 / 215739 in the name of the applicant of this application discloses an augmented reality (AR) device and a clip - on unit attached to removable glasses. The device has a housing that removably supports the clip - on unit, an exit window and an entrance window within the housing for the user to observe the scenery, a communication interface for coupling to a handheld device, a line - of - sight guidance unit operably coupled to an IMU that displays at least one marker in the user's field of view to direct the user's line of sight to a designated feature in the scenery, and an optical system within the housing for projecting the marker at a distance for overlaying on the scenery the user has viewed.

[0032] U.S. Patent No. 10,012,506 discloses a navigation guidance system that uses an electronic compass to determine a desired straight-line travel path between two waypoints and monitors the user's direction of travel. When the system determines that the direction of travel is within an established acceptable range, the system sends an alert to the user via one or more alert indicator components.

[0033] US2016 / 189426 discloses a method for generating virtual proxy objects and describes a method for controlling the location of virtual proxy objects in an augmented reality environment. A head-mounted display device (HMD) can identify real-world objects for generating virtual proxy objects, generate virtual proxy objects corresponding to the real-world objects, and use the HMD to display the virtual proxy objects so that they are perceived as existing in an augmented reality environment displayed to the end user of the HMD. [Overview of the Initiative] [Problems that the invention aims to solve]

[0034] The present invention aims to provide a system and method for pedestrian navigation that avoids some of the above-mentioned drawbacks. [Means for solving the problem]

[0035] Specifically, the present invention provides a solution to the need for a compact and lightweight accessory that can be clipped onto the user's eyeglass frame and includes an IMU, a microdisplay, optical instruments, and an audio interface for receiving both auxiliary guidance in the form of visual markings projected into the user's far field of view via a see-through display, as well as verbal guidance to the desired destination, by orienting the user towards a designated landmark.

[0036] The IMU includes a magnetometer and / or a gyroscope that provides directional signals in a hands-free navigation system that uses a smartphone with built-in GPS to orient the user to a map and provide voice instructions to guide the user in the requested direction.

[0037] The present invention also provides a method for correctly orienting a user toward a map so that correct navigation commands can be issued to direct the user toward a desired destination along the correct route.

[0038] The present invention can be used in navigation systems that do not require a camera to provide initial alignment, and therefore can avoid the increased bandwidth requirements of such systems. It also avoids trial-and-error corrections, as it is frustrating for pedestrians to realize they have walked hundreds of meters in the wrong direction and then have to turn back.

[0039] The present invention provides a method for hands-free pedestrian navigation to help a user reach a destination, and this method includes: (a)(i) a see-through display for projecting a visual image in front of the user's line of sight, and (ii) a process for attaching the IMU to the user's head; (b) The process of obtaining the user's approximate location from a GPS unit carried by the user; (c) The process of identifying the user on a computerized map of the area containing the user, using the user's approximate location; (d) A step of determining a landmark in the area that has a known landmark location within the user's field of view; (e) A step to confirm that the user's gaze is directed towards a landmark; (f) The process of determining the calculated bearing between the user's location and the location of a landmark extracted from a computerized map; And, (g) A process of guiding pedestrians by providing audio prompts and projecting supplementary visual prompts through a see-through display.

[0040] This approach offers a very lightweight solution to a problem inherent in all navigation systems, establishing not only the user's location but, more importantly, the user's direction of travel. Once this is established, the computerized navigation system can direct the user on a map and guide them to the requested destination in a similar manner to vehicle navigation systems, which give clear voice instructions only when the user needs to change direction. For example, when a pedestrian reaches an intersection where they need to turn right, they can be instructed by voice to turn right on the next street and proceed 500 meters along Lincoln Avenue. The system can then identify that the user has followed the instructions based on the direction measured by the IMU and therefore can cease giving further instructions until another change of direction is requested.

[0041] Conversely, if a user turns in the wrong direction, the system can instantly identify this and warn the user in real time, "No, this is not the road. Return to the main road, go another 100 meters, and turn right." Alternatively, the system can readjust the route according to pre-programmed settings, depending on whether the readjusted route is significantly less optimal than the original route. If there is no real difference whether the user turns now or another 100 meters later, the system can decide to leave the user in the correct position without issuing a potentially confusing verbal warning.

[0042] In some embodiments, the navigation system can initiate calibration by instructing the user to look at a landmark selected by the system. Alternatively, the user can select a landmark in their field of view and verbally inform the navigation system that their gaze is directed towards the designated landmark. In some embodiments, the IMU is part of a device that includes a see-through display that allows the user to project markers onto distant objects in their field of view. In such cases, the user can verbally inform the navigation system that their gaze is directed towards the designated landmark, and the IMU can be calibrated as described above.

[0043] The device is preferably head-mounted and includes an IMU and a communication interface to the user's smartphone (usually a wireless connection such as Bluetooth®), enabling hands-free navigation, although a wired connection is also possible. The user hears instructions via the smartphone, but for privacy and courtesy, voice instructions are relayed to earphones. The earphones may also connect to the smartphone via a short-range wireless interface or wire. The user transmits voice commands via a miniature microphone, which may be part of a device that can clip onto clothing or similarly connects to the smartphone wired or wirelessly, the latter preferably via a Bluetooth® connection.

[0044] In some embodiments, the navigation system is based on voice-controlled virtual assistant AI technology, such as Amazon Alexa, used in combination with a compact, visual-based accessory, as disclosed in WO2020 / 215739 above. This can be clipped to eyewear frames such as Echo Frame or Bose Frame, or to eyewear frames supporting a microphone and earphones, as needed. This enables friendly voice-based interaction combined with visual instructions superimposed on reality. [Brief explanation of the drawing]

[0045] To understand the present invention and to see how it can actually be implemented, embodiments are described here only as non-limiting examples with reference to the accompanying drawings.

[0046] [Figure 1a] This is an illustration of a user wearing the device according to the present invention. [Figure 1b] This is a diagram illustrating a device with typical dimensions. [Figure 2] This is a block diagram of the system according to the present invention. [Figure 3a] Figure 3a is a diagram illustrating the removable device as viewed from the rear. [Figure 3b] Figure 3b is an illustration of the removable device as seen from the front. [Figure 3c] Figure 3c is a partially cropped view of the removable device as seen from the rear, showing partial details of the optical instrument housed inside. [Figure 3d] Figure 3d is a schematic diagram of the optical instrument. [Figure 4a] Figure 4a is a block diagram of the devices in the wireless configuration. [Figure 4b] Figure 4b is a block diagram of the devices in a wired configuration. [Figure 5] Figure 5 shows the main operations for calibrating the IMU magnetometer and providing navigation instructions to the user. [Figure 6] Figure 6 shows the main steps involved in verifying that the user is following the correct route. [Figure 7] Figure 7 shows an alternative embodiment for calibrating the magnetometer when identifiable landmarks cannot be recognized. [Modes for carrying out the invention]

[0047] Figure 1a shows a device (10) detachably clipped to the front of an eyeglass frame 11 worn by a user (12). By such means, the device (10) is mounted directly in front of the user's eyes, allowing the user to observe the landscape through an exit window (13). The user can also see graphical annotations projected onto their eyes by the detachable device (10) and superimposed on the actual landscape. Preferably, the eyeglass frame (11) is a so-called “smart frame” supporting a microphone and earphone, which can be retrofitted to a conventional eyeglass frame with additional separate microphone and earphone, or mounted individually as a separate accessory. Preferably, the device (10) is an integrated accessory configured to be mounted to the front of an eyeglass frame, and since none of its components are attached to the temple pieces (side arms) of the user's eyeglasses, the device can be mounted to a conventional eyeglass frame. However, the present invention does not preclude the possibility of mounting or constructing some of the components to the side arms of a custom frame.

[0048] Figure 1b shows typical dimensions of a device (10) according to a preferred embodiment, the minimum components of which are an IMU, a microdisplay, a see-through display including a translucent mirror and optical equipment as described below with reference to Figures 2 and 3, and communication means. To enable such miniaturization, the field of view of the see-through display is small, and the optical exit window can be moved to align with the user's eye by lateral adjustment of the device. When using wireless communication, the device includes a small battery. To reduce power consumption, only the most important components are used or included. Most importantly, the camera is not essential and can be omitted, partly to reduce size, but mainly to avoid excessive power consumption, the addition of electronic equipment for image compression, and high-bandwidth communication (such as WiFi) for transmitting video to a smartphone. All of these make the device bulky and drain the battery.

[0049] Figure 2 is a block diagram of a system 20 that integrates a smartphone (21) having a GPS unit (22) and a host smartphone application (23) into a voice-controlled navigation system, shown as a host cloud application (24) and a clip-on device (10). The clip-on device (10) includes an IMU (25) and a microdisplay (26) and is connected to the smartphone control application (27) via BLE (Bluetooth® Low Energy). A smart frame (11) is connected to the host smartphone application (23) via Bluetooth®. The smart frame could be an Amazon Echo Frame with a microphone and earphone (28) as integrated components, and the host cloud application (24) could be Amazon Alexa and Skills. Such an implementation eliminates the need to integrate audio components into the device (10), and while this option is clearly possible, it comes at the cost of some extra bulk and increased battery consumption. For simplicity and consistency, we will refer to software applications loaded onto a smartphone as smartphone applications, and software applications operated over the internet as cloud applications.

[0050] The smartphone control application (27) connects to the navigation application cloud (29), where all navigation tasks are performed and communicated to the control application via the internet. The host smartphone application (23) connects to the host cloud application (24) via the internet. In this diagram, short-range Bluetooth® communication is indicated by black arrows, and internet communication is indicated by white arrows. The host smartphone application 23 performs speech / text conversion and interfaces to the navigation application cloud (29) via the internet. Amazon Sharing Alexa Skills are used to transfer data between the host cloud application (24) and the navigation application cloud (29), and can also be used to activate necessary functions such as navigation in the navigation application cloud (29). More specifically, Amazon enables device manufacturers to integrate Alexa voice functionality into their connected products using Alexa Voice Service (AVS), a cloud-based service that provides APIs for interfacing with Alexa. This requires that communication between the host smartphone application (23) and the smartphone control application (27) takes place via the clouds (24) and (29). However, other platforms may also support direct internal communication between two smartphone applications (23) and (27).

[0051] Figure 3a shows a rear view of the device (10). The user (12) sees the actual landscape through the exit window (30) and also sees graphical annotations superimposed on this landscape. These annotations may include marks, text, graphic shapes, etc. The detachable device has a housing (31) to which a mounting bracket (32) is fixed for supporting a clip-on unit (not shown) as described in WO 2019 / 215739, thereby clipping the device (10) to the front of the user's glasses. Alternatively, the device (10) may be magnetically attached to the user's glasses or may be another type of head-mounted device. Optionally, a USB connector (34) is provided to connect the device (10) to a smartphone (19). A system will be established.

[0052] Figure 3b shows a front view of the detachable device (10) as seen from the front, i.e., facing the user (12). In some embodiments, an optional window (36) is provided through which an optional built-in camera (37), positioned inside the device and schematically shown by a dotted line in Figure 3c, images the external landscape. Light from the landscape also passes through an exit window (30) and an entrance window (30') for the user to observe is also shown. When provided, the camera can be used as an auxiliary navigation aid, as described in WO 2019 / 215739 above, and can also be used to calibrate the IMU in the same way as Google Live View, but with the advantage that the user does not need to hold the smartphone.

[0053] Figure 3c shows the internal structure of the device (10) in more detail. The printed circuit board (PCB) (38) supports the electronic circuitry for the electronic operation of the removable device. The PCB (38) also has an IMU (25), a USB connector (34) (required only when using wired communication), a microdisplay (26), and an optional built-in camera (37) connected to it.

[0054] A microdisplay (26) is a high-brightness monochrome display having the following main characteristics: Pixel count: 304x256 Pixel size: 12μm x 12μm Active area: 3.648 mm x 3.972 mm

[0055] Device 10 houses an optical instrument (40) shown in Figure 3d, which includes two doublets (41), (41') and a combined prism (42). The doublets (41), (41') form a combined objective lens with a focal length equivalent to 21 mm. The light-emitting surface of the microdisplay (26) is positioned at the focal plane of this combined objective lens, thereby creating an image on the display at infinity, so that the user (12) can see the image on the display projected onto the actual landscape. The general image on the screen includes a cross-shaped marker or reticle that the user uses to point out objects in the field of view. As indicated by the arrows, the image is first reflected by the mirror surface (45) of the combined prism (42), and then directed towards the user's (12) eye by a partial mirror surface (46), thereby allowing light from the display (26) to reach the user's eye, while simultaneously allowing the user to see the actual scenery transmitted through the partial mirror surface (46) via the exit window (30) in Figure 3b. Typically, the partial mirror surface (46) has less than 50% reflection and less than 50% transmission. Since the focal length of the combined doublet is 21 mm, the display screen captures a field of view of H: 10 × V: 8.3 degrees (diagonal 13 degrees). The optical design allows the user to use an eye motion box with a diameter of 4.5 mm.

[0056] This configuration allows for the attachment of a very compact device to the user's eyeglass frame in a way that the exit window (30) is aligned with the user's eye, thereby enabling lateral adjustment to eliminate the need for a large eye-motion box that would necessitate the use of bulkier optical equipment. Many conventional see-through AR systems require a wide field of view to display graphical annotations in reality. Such systems are heavy, cumbersome, and expensive. Furthermore, they often use a built-in camera to image landmarks, allowing navigation systems to determine the user's location based on the landmark images. This further increases the bulk of the AR device and adds considerable overhead to the communication bandwidth.

[0057] In the device according to the present invention, a compact device is possible due to minimal AR annotations on markers and directions only, and a small field of view that does not rely on a camera. By combining this with voice, complex annotations (i.e., landmark names, complex directions) can be given verbally, enabling the use of a small field of view. This realizes hands-free, compact AR pedestrian navigation while on the go.

[0058] Figures 4a and 4b are block diagrams of the device (10) in wireless and wired configurations, respectively. Thus, as shown in Figure 4a, the optical device (40) projects a microdisplay 26 onto the user's (12) eyes, on which a screen image is overlaid onto the external scenery. The device (10) houses a magnetic inertia unit (IMU) 25 for tracking the device's movement and thereby tracking the user's head movement. The microdisplay (26) and IMU (25) are connected to a microcontroller (72). The microcontroller (72) also processes data for bidirectional communication with a Low Energy (LE) Bluetooth® unit (74) via an RF antenna (75). The Bluetooth® unit (74) communicates with a Bluetooth® unit (76) on a smartphone (21). An internal battery (77) powers the device components.

[0059] Figure 4b shows an alternative embodiment using USB wired communication. The microcontroller (72) includes an SPI (Serial Peripheral Interface) driver and USB-to-SPI converters (78), (78') that connect the display (26) and IMU (25) to a USB hub (79), respectively. The USB hub (79) connects to a smartphone (21) via a USB cable.

[0060] Figure 5 shows the main operations related to calibrating the IMU magnetometer and providing navigation instructions to the user. For clarity, the magnetometer is a component of the IMU (25) and is not shown separately in the drawing. The user (12) first attaches the device (10) to the spectacle frame (11) and turns on the power. In the first step, the magnetometer is calibrated to establish the orientation of the device (10) relative to Earth coordinates when the user is looking at a known landmark.

[0061] The calibration of this magnetometer (80) is performed as follows. The smartphone control application (27) acquires GPS data from the smartphone GPS unit (22). The smartphone control application determines the "line of sight" and retrieves relevant landmarks from the GPS / map database, i.e., Google Maps, based on the GPS location and estimated observation area. The map is stored in the navigation application cloud (29), from which the relevant portion of the map corresponding to the "line of sight" is downloaded to the smartphone. The smartphone control application identifies a visible landmark near the user (such as a known department store), draws a marker on the device's (10) display (26), and instructs the user to direct their gaze toward this landmark until the marker is overlaid on it. The user verbally confirms that their gaze is directed toward the landmark. The smartphone control application reads the azimuth angle measured by the magnetometer, calculates the azimuth angle between the user's location and the landmark's location based on the map, calibrates the magnetometer, and marks a "V" on the display, allowing the user to begin navigation.

[0062] Landmarks may be suggested by a smartphone application (27), but they may also be suggested by the user. In this case, the user might say, "I can see a Starbucks Coffee on the right," and the system will recognize it, and the process will continue as before. This corresponds to the traditional use of Alexa where the user voices the request. Alexa converts the voice to text and sends it to the Alexa cloud, where it is parsed and processed to derive an appropriate response, which is then transmitted to the user as a text string. This response is then converted to speech locally and spoken.

[0063] Once the magnetometer is calibrated, the smartphone application (27) can graphically indicate the calculated direction of travel by displaying auxiliary visual prompts, such as an arrow pointing magnetic north and / or text such as north-northwest, on the microdisplay (26). The displayed image is superimposed on the external landscape by an optical instrument (40). However, in a simplified device without a see-through display, the IMU can be mounted on the head without projecting a marker, or by projecting a marker using a non-see-through display such as Google Glass. Alignment with named landmarks is easily confirmed verbally, and the user's orientation is determined based on IMU measurements transmitted to the smartphone application (27). While this is likely to be less accurate, it still allows the user to focus on landmarks that the smartphone application (27) can direct the user towards and provide comprehensive navigation instructions. While verbal commands are preferred, the invention also envisions the use of a push-button microswitch on a clip-on device (10), which can be pressed by the user to confirm to the smartphone application when the user's gaze is directed towards a landmark. Of course, this isn't completely hands-free operation, but it still eliminates the need for manual operation on the smartphone. Furthermore, since calibration is performed only during the initialization and verification of the navigation application, such a variant can be considered a voice navigation system in all intents and purposes.

[0064] Once calibration is performed, navigation (81) may be initiated. • The user tells the system where they want to go. The smartphone application calculates a route, compiles landmarks along the route, acquires IMU and GPS data, and, when ready, displays graphical navigation assistance features on the display (26), such as arrows indicating the direction the user should go, and / or provides verbal instructions to assist with navigation. During navigation, the smartphone application provides the user with verbal instructions and graphic annotations. During navigation, either the user or the smartphone application can initiate verification mode to confirm that the user is still on orbit.

[0065] Figure 6 shows the main operations related to how the user is following the correct route. As shown, verification can be initiated by the user, for example, if there is doubt, as shown in (82), or by the smartphone application, as shown in (83). If initiated by the user, the user verbally confirms the landmarks they will pass, for example, "I see a Starbucks on the right." The smartphone application checks if this is correct and responds, if correct, "Good, keep walking." Otherwise, the smartphone application guides the user to a nearby landmark and instructs the user to perform a calibration routine similar to that shown as (80) in Figure 5, updating the navigation instructions as needed.

[0066] The smartphone verification (83) operates similarly to the initial calibration, but the smartphone verbally prompts the user to confirm that the landmarks that should be in the field of view are correctly visible. Regardless of whether the verification was initiated by the user of the smartphone application, the landmarks can be selected by either the user or the smartphone application. For example, the user could simply say: "Help," at which point the smartphone application would execute the verification protocol (83).

[0067] As described above, one embodiment of the present invention can be implemented without a display, but it can also be implemented only with a smart glass frame or earphone including an IMU, microphone, and earphone.

[0068] In this case, the user turns their head towards the approximate direction of the landmark when calibration or verification is required. The direction determined in this way is less precise than the direction achieved using projected markers, but more precise than the direction available with the current magnetometer. However, it is clear that graphical annotation is not possible in this case, and only verbal instructions are available.

[0069] Figure 7 illustrates an alternative embodiment for calibrating the IMU when identifiable landmarks cannot be identified. This can occur in situations where there are no special attributes to the elements surrounding the user, such as when a building has no signs and neither the user nor the system can verbally describe the objects around the user. In such cases, the smartphone application provides the user's mobile phone with a panoramic image of the user's field of view based on the user's GPS location. Such images can be obtained from a cloud database such as Street View. The user finds and clicks on the object they are looking at in the displayed panoramic image. Based on the selected area of ​​the panoramic image, the smartphone application can associate the selected image with the actual object the user is looking at, identify it, and determine the direction of the user's gaze.

[0070] It will be understood that modifications can be made without departing from the claimed scope of the present invention. For example, it is not necessary to integrate the microphone and earphone into the eyeglass frame. The microphone can be clipped to the user's jacket and connected to a smartphone in the same way as the earphone. In such a case, either or both can be connected wirelessly to the smartphone, typically using the smartphone's Bluetooth® interface, or via a suitable cable.

[0071] While embodiments have been described with particular reference to the calibration of an IMU using a magnetometer, it should also be noted that, as described above, the IMU may also use a gyroscope that provides relative angular displacement. In such a case, the calibration of the IMU determines the initial azimuth angle of the pedestrian with respect to a known landmark, and the gyroscope shows the rotation of the IMU, and therefore the user, with respect to the initial azimuth angle.

[0072] The present invention can be implemented using a display that provides visual navigation signals, even if it is not see-through. For example, LEDs having different colors or geometric shapes may be mounted spatially in relation to a head-mountable device so that they are visible when the user is looking in that direction from a distance. The illuminated LEDs may be visible to the user's eye even when out of focus and may indicate direction. The user may direct their gaze by a particular one of these LEDs, or, if calibration or verification is required as described above, turn their head roughly in the direction of a landmark. The LEDs may be incorporated into eyeglass lenses or mounted in front of eyeglass lenses.

[0073] It should be noted that features described with reference to one or more embodiments are described as examples, not as limitations to those embodiments. Therefore, unless otherwise specified, or unless a particular combination is clearly unacceptable, optional features described with reference to only some embodiments are assumed to be equally applicable to all other embodiments. It will also be understood that the software according to the present invention may be implemented by a computer program readable by a computer processor for performing the methods of the present invention. The present invention further envisions a machine-readable memory that concretely embodies a program of machine-executable instructions for performing the methods of the present invention.

Claims

1. A hands-free pedestrian navigation method to help a user reach their destination, (a) (i) a display for projecting a visual image in front of the user's line of sight, and (ii) a process for attaching the IMU to the user's head. (b) A step of obtaining the user's approximate location from a GPS unit carried by the user, (c) The process of using the user's approximate location to locate the user on a computerized map of the area containing the user. (d) A step of determining a landmark in the area that has a known landmark location within the user's field of view, (e) A process to obtain confirmation from the user that the user's gaze is directed towards the landmark, (f) A step of determining the calculated azimuth angle between the user's location and the location of a landmark extracted from the computerized map, (g) Calibrating the magnetometer or gyroscope element of the IMU using the calculated azimuth angle, and (h) The process of navigating the user by providing voice prompts and projecting supplementary visual prompts via a display. Methods that include...

2. The aforementioned method, (a) When the user's gaze is directed toward the landmark, the process of obtaining the azimuth angle measured from a magnetometer attached to the head of the IMU, and (b) The process of orienting the user by calculating the offset between the measured azimuth angle and the calculated azimuth angle. The method according to claim 1, further comprising:

3. The method according to claim 1, wherein at least one of (i) confirmation from the user that the user's gaze is directed toward the landmark, (ii) identification of the landmark, and (iii) identification of the destination is communicated by voice.

4. The method according to any one of claims 1 to 3, wherein the landmark is identified by displaying a panoramic photograph of the user's field of view on a smartphone carried by the user and identifying an object in the panoramic photograph selected by the user.

5. The method according to any one of claims 1 to 4, comprising the steps of: projecting a marker into the user's field of view via the display and obtaining confirmation from the user that the user's gaze is directed toward the landmark when the marker is aligned with the landmark; optionally, confirming that the user is following the correct route by displaying a visual representation in the user's field of view via the display; and optionally, warning the user that they are deviating from the correct route by displaying a visual representation in the user's field of view via the display.

6. The method according to any one of claims 1 to 5, wherein the user communicates information by voice via a software application having an interface to a third-party virtual assistant coupled to a navigation server.

7. The method according to claim 6, wherein the third-party virtual assistant is a cloud application that supports Amazon® Alexa and Skills and is connected to the navigation server via the Internet.

8. A computer program product comprising a computer-readable memory for storing program code instructions that, when executed by a computer processing unit, perform the method according to any one of claims 1 to 7.

9. A pedestrian navigation system for guiding users to their destination, A GPS unit carried by the user, A head-mountable device for a user to wear, including an IMU and a display for projecting visual images in front of the user's line of sight, A portable computing device carried by a user, operably coupled to the GPS unit and the head-mountable device, having memory and a processing unit, wherein the memory and processing unit are (a) Obtaining the user's approximate location from a GPS unit carried by the user, (b) Using the user's approximate location to pinpoint the user's location on a computerized map of the area containing the user. (c) In the area, determine a landmark whose location is known to the user within their field of view. (d) Confirmation that the user's gaze is directed towards the landmark is obtained through the user's voice, (e) Determining the calculated azimuth angle between the user's location and the location of the landmark extracted from the computerized map, (f) Calibrating the magnetometer or gyroscope element of the IMU using the calculated azimuth angle, and (g) A portable computing device programmed to navigate the user by providing voice prompts and projecting auxiliary visual prompts through the display, A system that includes this.

10. (g) When the user's gaze is directed toward the landmark, obtain the azimuth angle measured from the magnetometer of the head-mountable device, (h) The system according to claim 9, further comprising orienting a pedestrian by calculating an offset between a measured azimuth angle and a calculated azimuth angle.

11. The system according to claim 9 or 10, further comprising a remote navigation server storing map data, which communicates with the portable computing device in order to transmit the computerized map to the portable computing device.

12. The system according to any one of claims 9 to 11, wherein the portable computing device is a smartphone with a built-in GPS unit.

13. The system according to any one of claims 9 to 12, wherein the head-mountable device is detachably mounted to an eyeglass frame, the display includes a microdisplay for displaying an image and an optical device for projecting the image onto a view seen by the user, and optionally a microphone and earphones are incorporated into the eyeglass frame and configured to be coupled to the portable computing device to transmit voice information and to transmit voice commands to guide the user to a desired destination.

14. The system according to claim 13, wherein the image is a marker projected onto the landmark when the user's line of sight is aligned with the landmark.

15. The system according to claim 13, wherein the processing unit is programmed to transmit navigation assistance to the microdisplay.

16. The system according to any one of claims 9 to 15, further comprising a microphone and earphones coupled to the portable computing device for transmitting voice information and voice commands to guide the user to a desired destination.

17. The system according to claim 16, wherein the portable computing device runs a software application that has an interface to a third-party virtual assistant connected to a navigation server, and optionally, the microphone and earphones are configured to connect to a host application on the portable computing device via Bluetooth®, and the host application is configured to communicate over the Internet with a cloud application that supports Amazon® Alexa and Skills.

18. The head-mountable device is detachably attached to the eyeglass frame, The aforementioned eyeglass frame is an Amazon® Echo Frame, The system according to claim 17, wherein the portable computing device runs a software application that has an interface to a third-party virtual assistant coupled to a navigation server, the eyeglass frame connects to a host application of the portable computing device via Bluetooth®, and the host application is configured to communicate over the Internet with a cloud application that supports Amazon® Alexa and Skills.

19. The system according to claim 17 or 18, wherein the host application is configured to perform speech-to-text conversion and connect to a navigation application cloud over the Internet, and Amazon® Sharing Alexa Skills is used to transfer data between the host application and the navigation application cloud, to activate the navigation functions of the navigation application cloud, and to return navigation commands to the host application over the Internet.

20. The system according to any one of claims 9 to 19, wherein the display is a see-through display.

Images