An augmented reality based navigation method

Abstract

The application provides an AR-based navigation method, which acquires initial position information and destination position information of a user wearing a head-mounted display device; calculates a navigation path according to the initial position information and the destination position information; displays the navigation path on the head-mounted display device; acquires real-time position information of the head-mounted display device; acquires predicted position information of the head-mounted display device; calculates an error between the real-time position information and the predicted position information, feeds back to the head-mounted display device, and updates the predicted position information with the real-time position information. The application adopts a navigation mode combining a cloud and a head-mounted display device, reduces navigation delay and device navigation power consumption, and significantly improves user navigation experience; adopts a phased positioning and tracking method, and performs coordinate correction and IMU error elimination, further reduces the navigation power consumption of the head-mounted display device, and increases the endurance of the head-mounted display device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of navigation technology, and in particular to a navigation method based on augmented reality. BACKGROUND

[0002] Smart wearable devices have become a trend in the technology industry. Smart wearable devices are a general term for devices that are developed by applying wearable technology to the intelligent design of daily wear. The advent of wearable smart devices means the intelligent extension of humans. Through these devices, humans can better perceive external and self information, can more efficiently process information with the aid of computers, networks, or even other people, and can achieve more seamless communication.

[0003] The existing navigation method based on head-mounted display devices (such as AR glasses) needs to process a large amount of information: video information, depth information, IMU information, positioning information, and the data sources are complex. Almost all of them need to be processed at the same time, and each type of information acquisition and processing will have certain errors, and the intersection of multiple data will reduce the accuracy of navigation positioning and tracking. In addition, the information processed is large, and in addition, the user's motion state needs to be tracked in a timely manner, and real-time adjustment and planning of the navigation path are also needed, which will bring great pressure to the head-mounted display device, increase power consumption, and reduce its battery life. Moreover, most of the existing navigation algorithms are arranged in the cloud or in the terminal. The former arrangement method will cause a large delay, reducing the effect of navigation, and the latter requires a certain device computing power. These two ways have reduced the navigation experience to some extent.

[0004] Most of the existing navigation methods based on head-mounted display devices identify the real environment based on the head-mounted display device to determine the spatial distance of the physical scene and generate corresponding navigation markers, such as turn signs and AR navigation routes. This requires the head-mounted display device to have a certain computing power, but the algorithms carried by the head-mounted display device are not accurate enough for the physical spatial distance recognition of the actual environment navigation, especially for devices such as monocular cameras. In addition, carrying a strong algorithm will increase the power consumption of the head-mounted display device and reduce the battery life. These problems are not conducive to long-term outdoor navigation.

[0005] In addition, although the existing smart glasses combined with AR technology can complete route navigation, the power consumption of the existing navigation method based on head-mounted display devices is high, and it relies on accurate positioning information and information from other sensors, such as depth sensors. These technologies have limitations, and in addition, the user's motion needs to be tracked in real time, which will cause tracking delays or instability in the superposition of AR information and reality, thereby affecting the navigation experience. SUMMARY

[0006] In order to overcome the above technical defects, the present application aims to provide an augmented reality-based navigation method for reducing power consumption of a navigation system and improving navigation accuracy.

[0007] The present application discloses an augmented reality-based navigation method, comprising the following steps: obtaining initial position information and destination position information of a user wearing a head-mounted display device; calculating a navigation path according to the initial position information and the destination position information, wherein the navigation path comprises at least one path and a plurality of key point position information; displaying the navigation path on the head-mounted display device; obtaining real-time position information of the head-mounted display device at present, and transmitting the real-time position information to a cloud; obtaining current predicted position information calculated by the head-mounted display device; calculating an error between the real-time position information and the predicted position information, feeding back the error to the head-mounted display device, and updating the predicted position information with the real-time position information.

[0008] Preferably, the cloud stores a city three-dimensional model, the city three-dimensional model comprising key points of the city and position information of each key point, the position information being three-dimensional coordinates in a city three-dimensional model coordinate system; the obtaining of the real-time position information of the head-mounted display device at present and the transmitting of the real-time position information to the cloud; the obtaining of the current predicted position information of the user calculated by the head-mounted display device comprise: obtaining the real-time position information of the head-mounted display device at present, transmitting the real-time position information to the cloud, and calculating, by the cloud, real-time coordinates corresponding to the real-time position information in the city three-dimensional model coordinate system as the real-time position information; and obtaining the current predicted position information of the user in the city three-dimensional model coordinate system calculated by the head-mounted display device.

[0009] Preferably, the obtaining of the real-time position information of the head-mounted display device at present comprises: dividing the navigation path into a plurality of segments, obtaining real-time positioning information of the head-mounted display device by a positioning device when the head-mounted display device reaches a preset distance from an end point of a current segment, and transmitting the real-time positioning information to the cloud, and calculating, by the cloud, positioning coordinates corresponding to the real-time positioning information in the city three-dimensional model coordinate system as the real-time position information.

[0010] Preferably, the dividing of the navigation path into a plurality of segments comprises any one of the following: taking a turning point of the navigation path as a division point to divide the navigation path into a plurality of segments, so that each segment is a straight line; and taking a preset distance length as a division point to divide the navigation path into a plurality of segments.

[0011] Preferably, the step of dividing the navigation path into several segments using the turning points as dividing points, so that each segment is a straight line, includes: if the distance of the segment is greater than a first preset distance, then the segment is further divided into two or more sub-segments with the first preset distance as the interval.

[0012] Preferably, obtaining the predicted position information of the head-mounted display device in the cloud calculated by the head-mounted display device includes: obtaining the user's motion speed through the inertial sensor of the head-mounted display device, thereby calculating the user's motion displacement; and calculating the predicted position information of the head-mounted display device in the coordinate system of the city's three-dimensional model based on the starting position of the current segment and the user's motion displacement.

[0013] Preferably, feeding the error back to the head-mounted display device includes: feeding the error back to the head-mounted display device and correcting the calculations of the inertial sensor in the head-mounted display device based on an integrator algorithm.

[0014] Preferably, the step of feeding back the error to the head-mounted display device and correcting the calculations of the inertial sensor in the head-mounted display device based on the integrator algorithm includes: determining the road type of the road segment where the partial location information is located based on partial location information in the key point location information on the navigation path, wherein the road type includes one or more of straight roads, uphill roads, downhill roads, and turning roads; feeding back the error to the integrator algorithm corresponding to different road types; and the integrator algorithm corresponding to different road types is different.

[0015] Preferably, the method further includes: in the coordinate system of the city three-dimensional model, a navigation icon is provided within a second preset distance in front of the position of the head-mounted display device, the navigation icon is displayed on the display screen of the head-mounted display device, and moves with the user's displacement to continuously maintain the second preset distance from the user.

[0016] Preferably, in the coordinate system of the city's three-dimensional model, a navigation icon is provided within a second preset distance in front of the head-mounted display device. The navigation icon is displayed on the display screen of the head-mounted display device and moves with the user's displacement to continuously maintain the second preset distance from the user. This includes: when the navigation icon moves to a turning point, it stops moving, waits for the user to move to that turning point, and then turns and moves forward.

[0017] Compared with existing technologies, the above technical solution has the following advantages:

[0018] 1. Utilize digital cities built using urban information models for route planning and map actual routes to routes within the digital city; flexibly use different information at different stages for tracking and positioning, coordinate correction, and IMU error elimination, avoiding the real-time and simultaneous use of multi-source data, reducing navigation errors in head-mounted displays, reducing the power consumption of the head-mounted display navigation system, and increasing the battery life of the head-mounted display.

[0019] 2. Navigation icons can improve the user's AR navigation experience, increase interaction with the real world, and make it more vivid and engaging; the navigation method that combines cloud and head-mounted display devices reduces navigation latency and device power consumption, significantly improving the user's navigation experience. Attached Figure Description

[0020] Figure 1 A flowchart of the augmented reality-based navigation method provided by the present invention. Detailed Implementation

[0021] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0023] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0026] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0027] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0028] See appendix Figure 1 This invention discloses an augmented reality-based navigation method applied to outdoor AR navigation. After wearing an AR device (head-mounted display, typically AR glasses), the user observes the real world (such as their current city) through the device. As the user moves within this real world, their first-person perspective changes accordingly. The AR device provides a visualized navigation route, displaying the route overlaid on the real world. The user can then use this visualized route to move, thus achieving route navigation based on the head-mounted display.

[0029] The navigation method of the present invention includes the following steps:

[0030] S100: Obtain the initial location information and destination location information of the user wearing the head-mounted display device;

[0031] S200. Calculate the navigation path based on the initial location information and the destination location information, wherein the navigation path includes at least one path and multiple key point location information;

[0032] S300: Display the navigation route on the head-mounted display device;

[0033] S400: Obtain the current real-time location information of the head-mounted display device and transmit the real-time location information to the cloud;

[0034] S500: Obtain the current predicted position information calculated by the head-mounted display device;

[0035] S600 calculates the error between real-time location information and predicted location information, feeds the error back to the head-mounted display device, and updates the predicted location information with real-time location information.

[0036] The cloud stores a 3D city model, and a coordinate system is established based on this model. The 3D city model includes key points of the city and their location information, which is the 3D coordinates within the city's 3D model coordinate system. In steps S400-S500, the real-time location information of the head-mounted display device is acquired and transmitted to the cloud. The cloud then calculates the corresponding real-time coordinates within the city's 3D model coordinate system, which serve as the real-time location information. Finally, the predicted location information of the user within the city's 3D model coordinate system, calculated by the head-mounted display device, is also acquired.

[0037] Specifically, 3D city models can be acquired and stored in the cloud based on aerial oblique measurement, laser scanning measurement, structured light scanning measurement, stereo photogrammetry, satellite imagery and GIS data, and manual modeling. Laser scanning measurement: Laser scanning technology uses a laser to emit a laser beam and measures its reflection time to obtain accurate 3D point cloud data of the ground, buildings, and other objects. This point cloud data can be used to generate high-precision city information models. Structured light scanning measurement: Structured light scanning technology uses a light source and a camera to capture structured light patterns projected onto the surface of an object. By analyzing the deformation of the pattern, the depth information of the object's surface can be calculated, thereby generating a 3D model. Stereo photogrammetry: Stereo photogrammetry is a method of deriving 3D points by taking photographs of the same target from different locations and using image matching and triangulation techniques. Aerial photography can be carried out using drones or other flight platforms, and image processing and model generation can be performed using specialized software. Combining satellite imagery and GIS data: Information such as the height, shape, and location of buildings can be extracted using satellite imagery and GIS data and correlated with geographic coordinates to construct a 3D city information model. Manual modeling: For specific buildings or areas, 3D models can be manually created using computer-aided design (CAD) software or 3D modeling software. This requires modeling based on existing geographic data, floor plans, or on-site measurements.

[0038] Different cities correspond to different 3D city models, each with its own world coordinate system, denoted as the city 3D model coordinate system. The city 3D model includes key points (locations) and their location information (coordinate information). Key points include items such as shops, supermarkets, residential areas, and schools, as well as other path-related elements such as roads and traffic lights. Each key point and path-related element has coordinates relative to the same world coordinate system. Within this world coordinate system, these coordinates are relative; when the origin is different, the coordinates of each key point and path-related element will change accordingly.

[0039] In step S100, the initial location information and destination location information of the user wearing the head-mounted display device are obtained. Specifically, when the user queries a navigation route, the location where the user starts navigation is used as the initial location information, and the destination of the user's navigation is used as the destination location information. Preferably, the initial location information and the destination location information are coordinate points in the coordinate system of the city's three-dimensional model.

[0040] In step S200, a navigation path is calculated based on the initial location information and the destination location information. The navigation path includes at least one path and multiple key point location information. Specifically, the cloud determines at least one path from the initial location information to the destination location information using a navigation algorithm. Each navigation path includes the location information of corresponding key points (such as turning points (which can be the intersection of any two straight lines), traffic light points, overpass points, etc.) and sends the navigation path to the head-mounted display device.

[0041] In some embodiments, after calculating several navigation routes based on initial location information and destination location information, the user can further select one of the navigation routes according to different selection preferences, which include one or more of the following: shortest distance, fewest red lights, and fewest stairs. After the user makes a selection, the selected navigation route information (such as key point location information) is sent to the head-mounted display device. In step S300, the navigation route is displayed on the head-mounted display device. Specifically, after obtaining the navigation route information, the head-mounted display device can overlay and display part of the navigation route on the head-mounted display device to guide the user to move towards the destination. Obtaining the current real-time location information of the head-mounted display device and transmitting the real-time location information to the cloud further includes: dividing the navigation route into several segments, setting a preset distance before the head-mounted display device reaches the end of the current segment each time, obtaining the real-time location information of the head-mounted display device through a positioning device and sending it to the cloud, and having the cloud calculate the positioning coordinates in the city 3D model corresponding to the real-time location information as the real-time location information. A complete navigation path consists of numerous consecutive key point locations. The navigation path is divided into several segments. Each time the user reaches a predetermined distance before the end of the current segment, a positioning device (such as BeiDou or GPS) acquires the user's real-time location information from their head-mounted display and transmits it to the cloud. The cloud calculates the corresponding coordinates in the city's 3D model coordinate system and uses these coordinates as the real-time location information. The reason for correcting the location before reaching the end of the current segment, rather than upon (or after) reaching the end of the previous segment, is to perform the correction in the background, avoiding errors during actual navigation (i.e., premature correction) that could degrade the user's travel experience. Furthermore, by dividing the entire route into multiple continuous segments, the head-mounted display only needs to calculate the distance from the current location to the next target point to achieve real-time position and movement tracking. Ideally, the navigation path should be divided into segments using turning points, ensuring that each segment is a straight line. In other embodiments, a preset distance length can be used as the dividing point to divide the space into several segments.

[0042] Furthermore, if the length of a segment is too large (greater than the first preset distance, for example, 5km), that is, the distance is too far, the segment is divided into two or more sub-segments with the first preset distance as the interval, and error correction is performed on each sub-segment.

[0043] After the cloud calculates the location coordinates corresponding to the real-time location information in the city's 3D model coordinate system, the head-mounted display device needs to calculate the user's predicted coordinates (i.e., predicted position information) in the city's 3D model coordinate system. Specifically, between two consecutive points in the current segment, the inertial sensor of the head-mounted display device is used to obtain the user's motion speed, thereby calculating the user's motion displacement. Then, based on the starting position of the current segment and the user's motion displacement, the predicted coordinates of the user in the city's 3D model coordinate system are calculated.

[0044] It should be noted that the end point of the current segment is relative to the start point of the current segment, and the start point here should be understood as the position point of the last error correction, which is usually the end point of the previous segment (that is, the start point of the current segment).

[0045] In step S600, the error between the real-time location information and the predicted location information is calculated, and the error is fed back to the head-mounted display device. The predicted location information is then updated using the real-time location information. Real-time positioning information (positioning coordinates) is the user's actual coordinates, which is relatively accurate, while the predicted coordinates are the coordinates calculated by the head-mounted display device. If there is a difference between the two, it indicates that the head-mounted display device's calculation has an error. After obtaining the error, on the one hand, the calculated predicted coordinates can be replaced with the user's actual coordinates (i.e., real-time positioning information) to correct the navigation information within the current segment in a timely manner. On the other hand, the error also needs to be fed back to the head-mounted display device to correct its calculation algorithm, so that the head-mounted display device's next calculation can be more accurate. The head-mounted display device calculates the user's displacement using an inertial sensor, and this error is fed back into the inertial sensor's calculation algorithm, specifically into the error term of the inertial sensor's integrator algorithm, to correct the inertial sensor's calculation.

[0046] Preferably, the errors of the inertial sensor are continuously updated and eliminated. To further ensure the accuracy of the displacement calculated by the head-mounted display device, this invention also employs different calculation methods for different roads. As the number of uses increases, different error elimination algorithms are used for each error type, making it increasingly adaptable to various navigation routes. It should be noted that different road types correspond to different integrator algorithms.

[0047] Specifically, based on partial location information from key point location information on the navigation path, the road type of the road segment where the partial location information is located is determined. Road types include, but are not limited to, straight roads, uphill roads, downhill roads, and turning roads. The error is fed back to the integrator algorithm corresponding to different road types, thereby realizing error correction for different roads.

[0048] Preferably, in order to improve the user experience of the head-mounted display device, a navigation icon is also superimposed on the real world seen by the user after wearing the head-mounted display device. The navigation icon is displayed on the display screen of the head-mounted display device and is located within a second preset distance in front of the user's current position. AR technology can be used to superimpose the navigation icon on the path of the real world as the user moves forward.

[0049] The navigation icon moves with the user to maintain a second preset distance, typically at the same speed as the user. The navigation icon can be a human figure, arrow, animal, etc., and is not limited to any particular form; users can choose according to their preferences.

[0050] Ideally, when the navigation icon moves to a turning point, it stops moving, waits for the user to move to that turning point, and then turns before moving forward. After turning, the navigation icon moves at a speed greater than the normal speed (the user's moving speed) until it maintains a second preset distance from the user, then resumes the same moving speed as the user, preventing the navigation icon from being obscured after turning. Furthermore, during navigation, the navigation icon can emit corresponding voice prompts, such as: "Continue along the current new road for X meters," "Turn left in X meters," etc. The voice of the navigation icon can be freely selected from a pre-set voice pack.

[0051] Ideally, the distance between the navigation icon and the user should be adjustable by the user.

[0052] In some embodiments, when the navigation icon moves to the preset coordinates of the end point of the current segment (corresponding to the preset distance before the user reaches the end point of the current segment), the aforementioned coordinate correction and IMU error elimination will be performed (i.e., the current real-time positioning information of the user's head-mounted display device and the predicted position information of the head-mounted display device are obtained through the positioning device, the error between the real-time position information and the predicted position information is calculated, the error is fed back to the head-mounted display device for IMU error elimination, and the predicted position information is updated with the real-time position information).

[0053] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A navigation method based on augmented reality, characterized in that, Includes the following steps: Obtain the user's initial location information and destination location information while wearing a head-mounted display device; Based on the initial location information and the destination location information, a navigation path is calculated, wherein the navigation path includes at least one path and multiple key point location information; The navigation path is displayed on the head-mounted display device; Obtain the current real-time location information of the head-mounted display device and transmit the real-time location information to the cloud; Obtain the current predicted position information calculated by the head-mounted display device; Calculate the error between the real-time location information and the predicted location information, feed the error back to the head-mounted display device, and update the predicted location information with the real-time location information; The cloud stores a 3D model of the city, which includes key points of the city and the location information of each key point. The location information is the 3D coordinates in the coordinate system of the 3D model of the city. The steps of obtaining the current real-time location information of the head-mounted display device and transmitting the real-time location information to the cloud; and obtaining the user's current predicted location information calculated by the head-mounted display device include: The system obtains the current real-time location information of the head-mounted display device, transmits the real-time location information to the cloud, and calculates the positioning coordinates corresponding to the real-time location information in the coordinate system of the city's three-dimensional model as the real-time location information. Obtain the predicted location information of the user in the coordinate system of the city's three-dimensional model, calculated by the head-mounted display device; The process of obtaining the current real-time location information of the head-mounted display device includes: The navigation path is divided into several segments. When the head-mounted display device reaches a preset distance from the end of the current segment, the real-time positioning information of the head-mounted display device is obtained through the positioning device and sent to the cloud. The cloud calculates the positioning coordinates in the city's three-dimensional model coordinate system corresponding to the real-time positioning information as the real-time location information. The method further includes: In the coordinate system of the city's three-dimensional model, a navigation icon is positioned within a second preset distance in front of the head-mounted display device. This navigation icon is displayed on the screen of the head-mounted display device and moves with the user to maintain the second preset distance from the user. In the coordinate system of the city's three-dimensional model, a navigation icon is provided within a second preset distance in front of the head-mounted display device. The navigation icon is displayed on the screen of the head-mounted display device and moves with the user's movement to continuously maintain the second preset distance from the user. When the navigation icon moves to a turning point, it stops moving, waits for the user to move to that turning point, and then turns and moves forward. After turning, the navigation icon moves forward at a speed greater than the user's speed until it maintains a second preset distance from the user, and then resumes the same speed as the user.

2. The augmented reality-based navigation method according to claim 1, characterized in that, Dividing the navigation path into several segments includes any one of the following: The navigation path is divided into several segments by using the turning points as dividing points, so that each segment is a straight line; Divide the system into several segments using a preset distance as the dividing point.

3. The augmented reality-based navigation method according to claim 2, characterized in that, The step of dividing the navigation path into several segments using the turning points as dividing points, such that each segment is a straight line, includes: If the distance between the segments is greater than the first preset distance, then the segments are further divided into two or more sub-segments with the first preset distance as the interval.

4. The augmented reality-based navigation method according to claim 1, characterized in that, The step of obtaining the predicted current location information in the cloud calculated by the head-mounted display device includes: The user's motion speed is obtained through the inertial sensor of the head-mounted display device, thereby calculating the user's motion displacement; Based on the starting point of the current segment and the user's movement displacement, the predicted position information of the head-mounted display device in the coordinate system of the city's three-dimensional model is calculated.

5. The augmented reality-based navigation method according to claim 4, characterized in that, The step of feeding back the error to the head-mounted display device includes: The error is fed back to the head-mounted display device, and the calculations of the inertial sensor in the head-mounted display device are corrected based on the integrator algorithm.

6. The augmented reality-based navigation method according to claim 5, characterized in that, The step of feeding the error back to the head-mounted display device and correcting the calculations of the inertial sensor in the head-mounted display device based on the integrator algorithm includes: Based on partial location information from the key point location information on the navigation path, the road type of the road segment where the partial location information is located is determined. The road type includes one or more of the following: straight road, uphill road, downhill road, and turning road. The error is fed back to the integrator algorithm corresponding to different road types; The integrator algorithm is different for different road types.

Images