Off-line self-adaptive navigation method and system based on AR (Augmented Reality) glasses

Abstract

The invention relates to the technical field of tourism navigation, in particular to an off-line self-adaptive navigation method and system based on AR glasses, and the method comprises the steps: presetting an off-line map data packet at the local part of the AR glasses; after the AR glasses are started, providing a scenic spot overview for the user based on the data packet and receiving scenic spot selection input of the user; on the basis of the selected scenic spots and the communication relationship, locally planning an optimal touring path in real time on the AR glasses; positioning and tracking through a vision-inertial odometer of AR glasses, and rendering the optimal touring path on the real environment surface in the user view in real time in an augmented reality guidance form for navigation; in the navigation process, responding to a path modification instruction, locally and instantly planning a path again and updating augmented reality guidance; and correcting the positioning result of the vision-inertial odometer through a local beacon signal at a preset correction point. And vision-inertial odometer positioning and local beacon correction are combined, so that the navigation precision is improved.

Description

Technical Field

[0001] This application relates to the field of tourism navigation technology, specifically to an offline adaptive navigation method and system based on AR glasses. Background Technology

[0002] Existing augmented reality (AR) navigation solutions for scenic areas have significant limitations. Most solutions rely on mobile devices such as smartphones, requiring users to frequently interact with the screen to view maps or switch applications, severely disrupting the immersive experience. Furthermore, screen visibility is poor in bright outdoor light. Traditional AR navigation is passively triggered, often only activating when users reach fixed landmarks at attractions, making it difficult for tourists to gain a comprehensive understanding of the park's layout and tour routes beforehand.

[0003] Route planning functionality is generally weak, typically offering only pre-stored fixed recommended routes on servers, unable to dynamically adjust or personalize routes based on tourists' individual physical condition, real-time interests, or temporary changes in needs. Regarding positioning, in complex environments such as forests and building complexes, relying solely on GPS or Wi-Fi results in severe signal drift, leading to inaccurate AR guidance overlays and misleading users.

[0004] Furthermore, existing solutions heavily rely on a continuous and stable mobile network connection. When network coverage is poor or network congestion occurs during peak hours, map data downloads, cloud-based route planning, and audio guide services will be interrupted, rendering the guided tour experience completely unusable. For elderly people and children unfamiliar with smart device operation, the complex questionnaire completion, account login, and multi-level menu navigation also present a significant barrier to entry. Therefore, an AR guided tour solution that can operate offline, provides active panoramic previews, supports natural interaction and dynamic route planning, and possesses stable and high-precision positioning capabilities has become an urgent technical challenge. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an offline adaptive navigation method and system based on AR glasses. By pre-setting offline data packets, it fundamentally eliminates dependence on mobile networks, ensuring navigation continuity in network blind spots or congestion. Combining VIO positioning, real-time AR rendering, and local beacon correction, it achieves stable navigation in complex environments.

[0006] In a first aspect, the present invention provides an offline adaptive navigation method based on AR glasses, comprising the following steps: An offline map data package is pre-installed on the AR glasses, and the offline map data package contains spatial data and connectivity relationships of multiple attractions; After the AR glasses are activated, they provide the user with an overview of the scenic area based on the data packet and receive the user's input of attraction selection; Based on the selected attractions and their connectivity, the optimal tour route is planned locally and in real time on the AR glasses. The AR glasses use visual-inertial odometry for positioning and tracking, and render the optimal tour route in real time on the real environment surface in the user's field of vision as augmented reality guidance for navigation. During navigation, in response to the user's route modification command, the route is replanned locally in real time and the augmented reality guidance is updated. At a preset calibration point, the positioning results of the visual-inertial odometry are calibrated using local beacon signals.

[0007] The proactive interactive mode of first providing an overview and then allowing for self-selection breaks away from the passive triggering limitations of traditional guided tours; while the real-time route replanning that supports dynamic commands gives the route a high degree of flexibility and personalization, enhancing the autonomy, smoothness, and immersion of the tour.

[0008] As a preferred embodiment of the technical solution of the present invention, the offline map data package includes: A structured file used to describe the coordinates of attractions and the connectivity graph; Image files used for spot previews; And audio files used for voice explanations.

[0009] By employing an optimized design that separates structured files, images, and audio files for offline map data packages, the creation, updating, and management of data packages are greatly facilitated. This modular structure makes adding, deleting, modifying, and querying scenic area information more efficient, while ensuring that the system can quickly parse and call various resources, providing a lightweight and standardized data foundation for subsequent efficient route planning and multimedia navigation.

[0010] As a preferred embodiment of the technical solution of the present invention, the specific steps for providing a scenic area overview to the user based on the data packet include: Parse the offline map data package to obtain preview information for all attractions; A panoramic visualization interface containing thumbnails of all attractions is generated and displayed on the waveguide display of the AR glasses. At the same time, a voice summary containing the total number of attractions and the estimated total tour time is broadcast through bone conduction headphones.

[0011] By simultaneously outputting a panoramic visual interface and a voice summary, users are provided with an intuitive and efficient overall understanding of the scenic area. Without having to manually explore or consult materials, users can grasp the scale, layout, and overall time required for the scenic area within tens of seconds, laying a solid foundation for subsequent proactive route decisions and effectively solving the problem of aimlessness at the beginning of a tour.

[0012] As a preferred embodiment of the technical solution of the present invention, the step of receiving the scenic spot selection input is implemented in at least one of the following ways: Method A: Track the user's head movement, control the selection cursor to move in the visualization interface and align it with the thumbnail of the target attraction, and complete the selection through a preset confirmation action; Method B: Capture the name of the attraction or category keywords spoken by the user through the local speech recognition module, match the recognition results with the attraction information in the data packet, and add the successfully matched attractions to the selection list; Method C: The eye-tracking module captures the location where the user's gaze rests on the visualization interface. When the dwell time exceeds a preset threshold, the system automatically selects the location of the focal point.

[0013] By integrating three natural interaction methods—head posture control, voice commands, and eye tracking—the physical and cognitive barriers to route selection have been lowered. Users do not need to learn complex menus or perform precise touch operations; they can complete selections simply by natural head movements, voice commands, or eye contact. The interaction process is intuitive, efficient, and hands-free, ensuring a smooth tour experience.

[0014] As a preferred embodiment of the technical solution of the present invention, when method B is adopted, the method further includes: Based on a predefined tag library and user profiles, attractions are scored for interest and one or more tour routes consisting of highly interested attractions are automatically recommended to the user for confirmation.

[0015] Building upon voice interaction, the system incorporates intelligent recommendations based on tags and user profiles, transforming it from a passive response tool into a proactive tour guide assistant. It can understand users' potential interests and automatically generate multiple themed and personalized alternative routes, enhancing the intelligence and thoughtfulness of the tour guide service, making it especially suitable for tourists unfamiliar with the scenic area or those with specific preferences.

[0016] As a preferred embodiment of the technical solution of the present invention, the step of planning the optimal tour route locally and in real time on the AR glasses based on the selected scenic spots and the connectivity relationship specifically includes: The multiple scenic spots and their connectivity relationships are modeled as a weighted undirected graph, where the scenic spots are the nodes of the graph, the connecting paths are the edges of the graph, and the weight of the edges is determined based on the actual distance of the path or the estimated travel time. In the weighted undirected graph, starting from the user's current location or the first selected attraction, and ending at the last selected attraction or the exit, a graph search algorithm is used to find the shortest path to all selected attractions, which is the optimal tour path.

[0017] The core technical advantage of this solution lies in abstracting the complex route planning problem into a mathematical model for finding the shortest path in a weighted undirected graph and executing it locally. This method quantifies the spatial topology of attractions and path costs, ensuring that the planned route is optimal or near-optimal in terms of distance or time.

[0018] As a preferred embodiment of the technical solution of the present invention, the steps of positioning and tracking by the visual-inertial odometry of the AR glasses and rendering the optimal tour path in real time on the real environment surface in the user's field of vision for navigation include: By using the camera and inertial measurement unit integrated into the AR glasses, the pose calculation between consecutive frames is performed using the visual-inertial odometry method, and the six-degree-of-freedom pose information of the AR glasses in the environmental map is output in real time. The next direction of travel along the optimal tour route is converted into a three-dimensional virtual guide model; the virtual guide model is in the form of arrows, footprints, or light strips attached to the ground. The spatial coordinates of the virtual guide model are aligned with the plane in the real environment determined by the current pose information output by the visual-inertial odometry, and then fused and rendered through an optical waveguide display module to make the virtual guide model appear to fit the surface of the real environment.

[0019] By providing precise six-DOF pose through VIO, the virtual guidance model can be spatially aligned with the real world with millimeter-level accuracy. Binding and blending the guidance model (such as arrows) onto the real environment surface (such as the ground) solves the cognitive burden of traditional map navigation that requires users to perform map-to-ground matching. The guidance method is more natural, safe, and less affected by ambient light.

[0020] As a preferred embodiment of the technical solution of the present invention, the specific steps of responding to the user's path modification command, and instantly replanning the path and updating the augmented reality guidance locally are as follows: When an instruction to add or delete a attraction is received, a heuristic optimization algorithm is used to locally rearrange the path sequence based on the currently planned path and the connectivity relationship to generate a new optimal tour path, and the augmented reality guidance rendered on the real environment surface is updated immediately.

[0021] Employing a heuristic optimization algorithm for local rearrangement, the system can respond to changes in an extremely short time and smoothly update AR guidance. This allows tour plans to be flexibly adjusted according to mood, physical condition, or unexpected situations, achieving adaptive navigation and elevating the degree of interaction freedom and system response speed to a practical level.

[0022] As a preferred embodiment of the technical solution of the present invention, the step of correcting the positioning result of the visual-inertial odometry using local beacon signals at a preset correction point specifically includes: Low-power Bluetooth beacons are deployed at the entrance of the scenic spot or at key nodes along the route. The beacons continuously broadcast beacon signals containing their own unique ID and preset three-dimensional spatial coordinates. When the AR glasses move into the effective signal coverage area of ​​the beacon, its integrated Bluetooth module receives and parses the beacon signal to obtain the beacon's ID and coordinate information; The AR glasses compare the received beacon coordinates with their current pose calculated by the visual-inertial odometry to calculate a pose deviation. Using the beacon coordinates as an absolute reference, the current pose output by the visual-inertial odometry is subjected to a one-time coordinate translation and rotation correction, and the correction amount is applied to the subsequent visual-inertial odometry calculation process to reset the accumulated error.

[0023] As a preferred embodiment of the present invention, when the AR glasses detect an available wireless network connection, the method further includes at least one of the following enhancement steps: E1. Automatically or according to user instructions, download and update the local offline map data package from the server to obtain the latest attraction information, event notifications, or route changes.

[0024] E2. Anonymously upload users' tour routes, stay times, and preferences to the server for big data analysis of scenic spots or popular route recommendations; or receive anonymous heat information from other users broadcast by the server and gently indicate the flow of people in the area in the AR view.

[0025] E3 sends complex route planning requests (such as ultra-large-scale scenic areas and multi-person collaborative route planning) to the cloud server for calculation, and sends the results back to the glasses for local execution to share the local computing power.

[0026] E4. Obtain and broadcast real-time information such as queue times, show times, and weather warnings for attractions online, and integrate it into AR navigation prompts or route recommendation logic.

[0027] E5. Enable real-time audio and video call function, allowing scenic area customer service or companions to make AR annotations and guidance in the user's field of vision.

[0028] Secondly, the present invention also provides an offline adaptive navigation system based on AR glasses, used to implement the method described in the first aspect, comprising: an AR glasses terminal and multiple positioning and correction beacons deployed at preset correction points in the scenic area, used to broadcast the local beacon signals to the AR glasses terminal; the AR glasses terminal integrates: A memory for pre-storing the offline map data packets; The processing unit, connected to the memory, is used to perform tasks such as receiving attraction selections, planning routes, replanning routes, and controlling rendering. The visual-inertial odometry module is used to perform localization tracking and output pose information; The optical waveguide display module is used to receive rendering instructions from the processing unit and present augmented reality guidance to the user's field of vision; Audio output device, used to broadcast audio summaries and explanations; The input sensing module is used to capture the user's interaction commands; A wireless communication module is used to receive the local beacon signal; When performing path planning, the processing unit is configured to: model the multiple attractions and their connectivity as a weighted undirected graph, and use a graph search algorithm to solve for the shortest path to all selected attractions; When performing AR rendering, the processing unit is configured to: convert the next direction of travel of the planned path into a three-dimensional virtual guide model, align the spatial coordinates of the virtual guide model with the real environment plane determined by the current pose information, and perform fusion rendering through the optical waveguide display module; When responding to a path modification command, the processing unit is configured to: use a heuristic optimization algorithm to locally rearrange the current path sequence to generate a new optimal tour path, and control the optical waveguide display module to immediately update the augmented reality guidance.

[0029] As a preferred embodiment of the technical solution of the present invention, the input sensing module includes at least one of the following: An inertial measurement unit used to achieve head posture control; Microphone array and local speech recognition unit for implementing voice command input; Infrared camera module and eye-tracking analysis unit used for eye-tracking selection.

[0030] As a preferred embodiment of the technical solution of the present invention, the offline map data package specifically includes: A structured JSON file used to describe the coordinates of attractions and the connectivity graph; PNG thumbnail files used for attraction previews; And voice narration files using adaptive multi-rate narrowband coding.

[0031] As a preferred embodiment of the present invention, the wireless communication module is a low-power Bluetooth module, and the positioning correction beacon is a low-power Bluetooth beacon.

[0032] As can be seen from the above technical solutions, this application has the following advantages: by pre-installing offline map data packages and providing an overview of the scenic area after the AR glasses are activated, users can know the full picture of the scenic area and the connections between attractions before setting off, breaking away from the passive limitation of traditional tour guides who only know the location when they arrive at the attraction, helping users to make tour decisions quickly and improving planning efficiency.

[0033] By leveraging the local computing power of AR glasses to complete path planning and recalculation without relying on mobile networks, the navigation interruption problem caused by network congestion or blind spots is completely solved. Combined with visual-inertial odometry positioning and local beacon correction, accurate navigation is achieved and the clarity of guidance is guaranteed.

[0034] Users can initiate route modification commands through natural interaction, without relying on the mobile app to repeatedly unlock and view the map, thus avoiding interruptions to the tour rhythm; the route is rearranged locally in real time and AR guidance is updated synchronously. The operation process is simple and intuitive, and can be easily used by people of all ages, including the elderly and children.

[0035] The optimal tour route is rendered in real time on the surface of the real environment in AR form, and the guidance is deeply integrated with the real scene, reducing the frequency of users looking down at their devices, which not only enhances the immersive experience of the tour, but also reduces safety hazards during walking; route planning and modification can be quickly responded to locally throughout the entire process, ensuring navigation continuity and smoothness. Attached Figure Description

[0036] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0038] Figure 2 A block diagram of a system provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0041] like Figure 1 As shown, this embodiment of the invention provides an offline adaptive navigation method based on AR glasses, including the following steps: S1. A pre-installed offline map data package is stored locally on the AR glasses. The offline map data package contains spatial data and connectivity relationships of multiple attractions. The offline map data package includes: a structured file describing the coordinates and connectivity of attractions; an image file for attraction preview; and an audio file for voice narration.

[0042] S2. After the AR glasses are activated, an overview of the scenic area is provided to the user based on the data packet, and the user's attraction selection input is received. The specific steps for providing the overview of the scenic area to the user based on the data packet include: Parse the offline map data package to obtain preview information for all attractions; A panoramic visualization interface containing thumbnails of all attractions is generated and displayed on the waveguide display of the AR glasses. At the same time, a voice summary containing the total number of attractions and the estimated total tour time is broadcast through bone conduction headphones.

[0043] In this embodiment of the invention, the overview module provides the user with an overview of the scenic area based on the data packet. The specific implementation process is as follows: Step 2-1, Data Parsing and Interface Generation: Step 2-11: After the system starts, it reads the map_info.json file in the offline map data package and parses out metadata such as the name of all attractions, 3D coordinates, thumbnail file path, and audio file path.

[0044] Step 2-12: Based on the number of attractions N obtained from the analysis, automatically calculate and determine the layout of the panoramic visualization interface. Preferably, an M×M grid layout is used to ensure that all attraction thumbnails can be clearly displayed on one screen.

[0045] Steps 2-13: In the 3D rendering engine, create a virtual spherical or cylindrical surface as a background backdrop. After loading the thumbnail texture of each attraction as an independent pin object, project it spherically or azimuthally onto the backdrop based on its real geographic coordinates to generate a panoramic image wall with a sense of space. Users can rotate their heads to view this panoramic interface.

[0046] Step 2-2, Tour Duration Estimation and Voice Summary Generation: Step 2-21: Read the connectivity information between attractions from the same map_info.json file. Each edge contains the starting attraction ID, the ending attraction ID, and the weight, which is the estimated walking time of the path.

[0047] Step 2-22: Perform a standard tour route calculation: Assuming the tourist visits all attractions, starting from the entrance and ending at the exit, use Dijkstra's algorithm to find an approximately optimal Hamiltonian path that covers all attractions in the connected graph.

[0048] Step 2-23: Sum the weights of all edges in the above path to obtain the total walking time. A standard stay time is preset for each attraction. For example, 5 minutes for cultural attractions and 8 minutes for photo opportunities. Total stay time .

[0049] Steps 2-24: Estimated total tour duration Please use this calculation result. The total number of attractions, N, is used as a key variable to fill in the pre-set voice template. For example: This park presents you with N selected attractions. A complete tour will take approximately [time missing]. minute.

[0050] Steps 2-25: Call the local text-to-speech engine, or directly read the pre-stored AMR-NB format voice package corresponding to this summary, and broadcast it through bone conduction headphones.

[0051] The step of receiving the attraction selection input can be implemented in at least one of the following ways: Method A: Track the user's head movement, control the selection cursor to move in the visualization interface and align it with the thumbnail of the target attraction, and complete the selection through a preset confirmation action; Method B: Capture the name of the attraction or category keywords spoken by the user through the local speech recognition module, match the recognition results with the attraction information in the data packet, and add the successfully matched attractions to the selection list; Method C: The eye-tracking module captures the location where the user's gaze rests on the visualization interface. When the dwell time exceeds a preset threshold, the system automatically selects the location of the focal point.

[0052] S3. Based on the selected attractions and their connectivity, plan the optimal tour route locally and in real time on the AR glasses; specifically: The multiple scenic spots and their connectivity relationships are modeled as a weighted undirected graph, where the scenic spots are the nodes of the graph, the connecting paths are the edges of the graph, and the weight of the edges is determined based on the actual distance of the path or the estimated travel time. In the weighted undirected graph, starting from the user's current location or the first selected attraction, and ending at the last selected attraction or the exit, a graph search algorithm is used to find the shortest path to all selected attractions, which is the optimal tour path.

[0053] The path planning in this embodiment of the invention is implemented through a path planning engine, and the specific implementation process is as follows: Step 3-1: Construction of the weighted undirected graph model: Step 3-11: Read the connected graph description file from the offline map data package. Abstract each attraction in the file into a node v∈V in a weighted undirected graph G=(V,E). The attributes of node v include at least its unique ID and three-dimensional spatial coordinates (x,y,z).

[0054] Step 3-12: For each connected edge e defined in the description file, it is represented in the weighted undirected graph G as connecting two nodes ( Let e ​​be an edge ∈ E. Each edge e is associated with a weight w. The weight w is determined as follows: a. During the data package creation stage, calculations were performed based on the high-precision map data. and The actual walking path length between them is calculated, and this length is directly used as the edge weight w.

[0055] b. The weight w can also be represented as the estimated travel time. This time can be calculated as (path length / average walking speed) + (path type correction value). For example, a staircase path takes longer than a flat path, so a road condition coefficient can be added to the edge in the data packet to correct for this. This corresponds to the optimal path being the time-optimal one.

[0056] Step 3-2: Solve for the shortest path for the selected set of attractions: Step 3-21: The user selects M attractions through interaction, forming the target node set. The goal of path planning is to find a path starting from the origin (the node closest to the user's current position). or the first designated attraction Starting from point S, visit all nodes in set S exactly once, and finally reach the destination (the specified last attraction). or fixed exit node The loop or path with the smallest sum of edge weights along the entire path.

[0057] Step 3-22: Given that this system runs locally on the AR glasses, it needs to be solved within limited computing resources. The following steps are used for an efficient approximate solution: a. During system initialization or idle periods, Dijkstra's algorithm is used to pre-calculate the shortest path distance between all pairs of nodes in the weighted undirected graph G, and this distance is stored as a distance matrix D. Thus, the shortest distance between any two points can be determined by looking up table D[...]. ][ Get it now.

[0058] b. Based on the target node set S and the distance matrix D, construct a new complete graph with nodes in S as vertices and the shortest path distance between each pair of nodes as edge weights. In the full map The above employs an efficient heuristic algorithm to solve for the near-optimal access sequence. The preferred algorithm combines the Nearest Neighbor algorithm with 2-opt local search: i. Starting point To begin, find the nearest unvisited node in S to the current point and add it to the path.

[0059] ii. Repeat the previous step until all nodes in S have been visited, and finally connect to the destination. This forms an initial path.

[0060] iii. For this initial path, apply the 2-opt algorithm for local optimization: continuously try to swap the connection of two edges in the path. If the swap can reduce the total path length, then accept the swap, until no improvement can be made.

[0061] Step 3-23: The above steps are in the complete diagram The result shown is the order of visits between attractions. And the total distance. The system needs to restore this order to the original weighted undirected graph G: based on the distance matrix D, calculate the distances from... arrive , arrive ... arrive The specific travel path in the original road network G (consisting of a series of intermediate nodes). Connecting these specific paths end to end yields the final navigable optimal tour path that closely resembles the real roads.

[0062] S4. Positioning and tracking are performed using the visual-inertial odometry of the AR glasses, and the optimal tour route is rendered in real-time on the real environment surface in the user's field of vision as augmented reality guidance for navigation; specifically including: By using the camera and inertial measurement unit integrated into the AR glasses, the pose calculation between consecutive frames is performed using the visual-inertial odometry method, and the six-degree-of-freedom pose information of the AR glasses in the environmental map is output in real time. The next direction of travel along the optimal tour route is converted into a three-dimensional virtual guide model; the virtual guide model is in the form of arrows, footprints, or light strips attached to the ground. The spatial coordinates of the virtual guide model are aligned with the plane in the real environment determined by the current pose information output by the visual-inertial odometry, and then fused and rendered through an optical waveguide display module to make the virtual guide model appear to fit the surface of the real environment.

[0063] In this embodiment of the invention, the specific implementation process of AR real-time navigation and rendering is as follows: Step 1: Real-time pose calculation based on vision-inertial odometry (VIO); This step is executed by the VIO algorithm engine built into the AR glasses, and its specific operation process is as follows: The camera captures environmental images at a fixed frequency. Before being fed into the VIO algorithm, the image undergoes preprocessing such as distortion correction and denoising.

[0064] Inertial measurement unit synchronously acquires triaxial acceleration and triaxial angular velocity The instantaneous data is obtained and then subjected to zero bias correction and coordinate system alignment.

[0065] For the current image Extract feature points and compare them with the previous frame. Alternatively, feature points in the local map can be matched, and the rotation and translation (ΔR, Δt) of the camera from time t-1 to time t can be initially estimated using epipolar geometry or PnP (Perspective-n-Point) methods.

[0066] At the same time, using IMU data, the pose from the previous moment is integrated using the principle of inertial navigation to independently predict the pose at the current moment. ).

[0067] The core of the VIO algorithm lies in a nonlinear optimizer. This optimizer combines the geometric constraints of visual observation and the dynamic expectation constraints of the IMU into an optimization problem. Its state variables typically include the pose and velocity of the current frame and historical frames within the window, as well as the IMU bias. The optimizer solves for the optimal, smooth six-DOF pose by minimizing the visual reprojection error and the IMU measurement error. , where R is a 3x3 rotation matrix (pose) and t is a 3x1 translation vector (position). This pose describes the transformation relationship between the camera coordinate system and the world coordinate system.

[0068] The solution The data is output to the rendering pipeline in real time. At the same time, the VIO engine maintains a sparse local 3D feature point map, whose positions in the world coordinate system are continuously optimized as a reference for subsequent frame tracking and relocalization.

[0069] Step 2: Generation and binding of the virtual guide model; This step is performed by the rendering engine's processing unit, which transforms abstract path instructions into concrete AR visual elements: Extract the vector direction dir from the optimal tour path from the user's current location to the next key point on the path (such as the next turning point or target attraction).

[0070] Instantiate a pre-defined 3D model from the repository based on system settings or user preferences. For example, a cone-shaped arrow model or a sequence of footprint models. The model is initially located at the origin of the world coordinate system.

[0071] Set the coordinates of the next path key points in the world coordinate system. , serving as the target location for guidance.

[0072] Using the pose information output by the current VIO and combined with the ground plane equation estimated from the image, the ground plane in the world coordinate system is calculated. A simplified and efficient approach is to assume that the user is standing on a horizontal ground, then the ground plane equation can be defined as Y=h, where h is the user's foot height (a constant set according to the device's wearing height).

[0073] Place the anchor point of the virtual guide model at the target point. Vertical projection point on the ground On the coordinates ( At the same time, set the model's orientation (such as the direction of the arrow) to be aligned with the horizontal direction vector dir.

[0074] Step 3: Virtual-real fusion and optical waveguide display; This step is accomplished by the optical engine and display driver of the AR glasses: The coordinates of the virtual guide model in the world coordinate system Using the camera pose of the current frame inverse matrix Transform to the camera coordinate system: .

[0075] The projection matrix, constructed based on the inherent parameters of the spectacle lenses (focal length, optical center, etc.), will... The 3D camera coordinates are projected onto the 2D image plane coordinates (u, v), and the depth value z is calculated.

[0076] The geometric data, texture, and final pixel position and depth information of the virtual guide model are submitted to the graphics processing unit.

[0077] The GPU executes the standard graphics rendering pipeline, performing rasterization and shading. The key is depth testing: the depth z of the virtual pixel is compared to an estimated depth map of the real scene seen through a camera (obtained from sparse map interpolation by VIO or a monocular depth estimation network). Only virtual pixels located in front of real objects are ultimately displayed, ensuring the virtual arrow "fits" the ground and isn't incorrectly occluded by tree trunks or pedestrians in front.

[0078] The generated virtual image is fed into the optical waveguide display module. Light propagates within the waveguide sheet via total internal reflection and is ultimately coupled out in front of the user's glasses, precisely superimposed on the light from the real physical world seen through the waveguide. Due to the continuous updates of the VIO pose and the high frame rate of rendering (e.g., 60Hz), the virtual guide model remains stable in the user's field of vision as the user moves their head, appearing as if it were nailed to the real ground, resulting in a natural guiding effect.

[0079] S5. During navigation, in response to the user's path modification command, the system immediately replans the path locally and updates the augmented reality guidance; specifically, when receiving a command to add or delete attractions, based on the currently planned path and the connectivity relationship, a heuristic optimization algorithm is used to locally rearrange the path sequence to generate a new optimal tour path, and the augmented reality guidance rendered on the real environment surface is immediately updated.

[0080] S6. At a preset calibration point, the positioning result of the visual-inertial odometry is calibrated using local beacon signals. Specifically, this includes: Low-power Bluetooth beacons are deployed at the entrance of the scenic spot or at key nodes along the route. The beacons continuously broadcast beacon signals containing their own unique ID and preset three-dimensional spatial coordinates. When the AR glasses move into the effective signal coverage area of ​​the beacon, its integrated Bluetooth module receives and parses the beacon signal to obtain the beacon's ID and coordinate information; The AR glasses compare the received beacon coordinates with their current pose calculated by the visual-inertial odometry to calculate a pose deviation. Using the beacon coordinates as an absolute reference, the current pose output by the visual-inertial odometry is subjected to a one-time coordinate translation and rotation correction, and the correction amount is applied to the subsequent visual-inertial odometry calculation process to reset the accumulated error.

[0081] It should be noted that the effective signal coverage radius of the Bluetooth Low Energy beacon is 1.5 to 2.5 meters, and the calibration operation is triggered only when the AR glasses enter this range. The calibration operation is only performed when the user is relatively stationary or moving at low speed to ensure the accuracy of the calibration.

[0082] In some embodiments, when method B is used, the method further includes: Based on a predefined tag library and user profiles, attractions are scored for interest and one or more tour routes consisting of highly interested attractions are automatically recommended to the user for confirmation.

[0083] In this embodiment of the invention, the intelligent recommendation module recommends tour routes, and the specific implementation steps are as follows: 1. Data preparation and model initialization: Step a1: In the tags_config.json file of the offline map data package, predefine a multi-dimensional attraction attribute tag library. For example: {"Tag Category": "Interest Theme", "Optional Values": ["History and Culture", "Natural Scenery", "Family Entertainment", "Popular Spots"]}; {"Tag Category": "Physical Exertion", "Optional Values": ["Easy", "Moderate", "Challenging"]}. Each attraction is assigned one or more tag values ​​in the data package.

[0084] Step a2: Pre-define several typical user profile types, such as "Senior Tourists," "Families with Children," "Photography Enthusiasts," and "Cultural Study Tours." Each profile corresponds to a preference vector, which defines its preference weights for various tags. For example, in the preference vector of the "Families with Children" profile, the weight of the "Family Entertainment" tag is 0.9, the weight of "Physical Exercise: Relaxing" is 0.8, and the weight of "History and Culture" is 0.3.

[0085] 2. Interactive profile selection and interest rating: Step b1: When a user uses the recommendation feature for the first time or selects it, ask a simple question via voice interaction, such as, "Are you traveling with family or alone?", or let the user select an icon representing their identity in the visual interface. Based on the user's direct selection or answer to the question, the system matches the closest pre-defined user type profile.

[0086] Step b2: For each attraction within the scenic area Its interest score Calculated dynamically using the following formula:

[0087] in: It is a tourist attraction The labels it has The inherent weights. It is the preference vector of the current user profile, corresponding to the tag The preference weight of the category.

[0088] Iterate through all the tags of each attraction, perform a weighted sum, and calculate the value of each attraction relative to the current user. .

[0089] 3. Recommended route generation and presentation: Step c1: Score all attractions based on interest level. Sort the attractions from highest to lowest. Select the top-N attractions to form a candidate pool of high-interest attractions.

[0090] Step c2: Perform the following sub-steps: Based on the geographical coordinates of the attractions in the candidate pool, the K-means clustering algorithm is used to divide them into 3-4 groups according to spatial distance.

[0091] For each group, taking the attraction with the highest interest within the group as the core, a path planning algorithm is used to plan an optimal sub-path that connects all attractions within the group.

[0092] Considering the user's current location or entry point, these optimal sub-paths are connected according to the principle of spatial adjacency, and fine-tuned using a heuristic algorithm to ultimately form three recommended routes with different themes and reasonable paths; for example: "Group A: In-depth cultural tour", "Group B: Family leisure route", and "Group C: Panoramic sightseeing route".

[0093] Step c3: The system announces the recommended routes via voice, such as: "We recommend three routes: the first focuses on history and culture, including XX and YY, and takes approximately 1 hour; the second..." Simultaneously, the system highlights or numbers the key attractions along these routes on the AR interface. Users can confirm one of the routes via voice or nod, and the system will then add the attractions included in that route to their selection list and trigger subsequent navigation.

[0094] In some embodiments, when the AR glasses detect an available wireless network connection, the method further includes at least one of the following enhancement steps: E1. Automatically or according to user instructions, download and update the local offline map data package from the server to obtain the latest attraction information, event notifications, or route changes.

[0095] E2. Anonymously upload users' tour routes, stay times, and preferences to the server for big data analysis of scenic spots or popular route recommendations; or receive anonymous heat information from other users broadcast by the server and gently indicate the flow of people in the area in the AR view.

[0096] E3 sends complex route planning requests (such as ultra-large-scale scenic areas and multi-person collaborative route planning) to the cloud server for calculation, and sends the results back to the glasses for local execution to share the local computing power.

[0097] E4. Obtain and broadcast real-time information such as queue times, show times, and weather warnings for attractions online, and integrate it into AR navigation prompts or route recommendation logic.

[0098] E5. Enable real-time audio and video call function, allowing scenic area customer service or companions to make AR annotations and guidance in the user's field of vision.

[0099] like Figure 2 As shown, this embodiment of the invention also provides an offline adaptive navigation system based on AR glasses, used to implement the method described in the above embodiments, including: an AR glasses terminal and multiple positioning correction beacons deployed at preset correction points in the scenic area, used to broadcast the local beacon signals to the AR glasses terminal; the AR glasses terminal integrates: A memory for pre-storing the offline map data packets; The processing unit, connected to the memory, is used to perform tasks such as receiving attraction selections, planning routes, replanning routes, and controlling rendering. The visual-inertial odometry module is used to perform localization tracking and output pose information; The optical waveguide display module is used to receive rendering instructions from the processing unit and present augmented reality guidance to the user's field of vision; Audio output device, used to broadcast audio summaries and explanations; The input sensing module is used to capture the user's interaction commands; A wireless communication module is used to receive the local beacon signal; When performing path planning, the processing unit is configured to: model the multiple attractions and their connectivity as a weighted undirected graph, and use a graph search algorithm to solve for the shortest path to all selected attractions; When performing AR rendering, the processing unit is configured to: convert the next direction of travel of the planned path into a three-dimensional virtual guide model, align the spatial coordinates of the virtual guide model with the real environment plane determined by the current pose information, and perform fusion rendering through the optical waveguide display module; When responding to a path modification command, the processing unit is configured to: use a heuristic optimization algorithm to locally rearrange the current path sequence to generate a new optimal tour path, and control the optical waveguide display module to immediately update the augmented reality guidance.

[0100] In some embodiments, the input sensing module includes at least one of the following: An inertial measurement unit used to achieve head posture control; Microphone array and local speech recognition unit for implementing voice command input; Infrared camera module and eye-tracking analysis unit used for eye-tracking selection.

[0101] In some embodiments, the offline map data package specifically includes: A structured JSON file used to describe the coordinates of attractions and the connectivity graph; PNG thumbnail files used for attraction previews; And voice narration files using adaptive multi-rate narrowband coding.

[0102] In some embodiments, the wireless communication module is a Bluetooth Low Energy module, and the positioning correction beacon is a Bluetooth Low Energy beacon.

[0103] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0104] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An offline adaptive navigation method based on AR glasses, characterized in that, Includes the following steps: An offline map data package is pre-installed on the AR glasses, and the offline map data package contains spatial data and connectivity relationships of multiple attractions; After the AR glasses are activated, they provide the user with an overview of the scenic area based on the data packet and receive the user's input of attraction selection; Based on the selected attractions and their connectivity, the optimal tour route is planned locally and in real time on the AR glasses. The AR glasses use visual-inertial odometry for positioning and tracking, and render the optimal tour route in real time on the real environment surface in the user's field of vision as augmented reality guidance for navigation. During navigation, in response to the user's route modification command, the route is replanned locally in real time and the augmented reality guidance is updated. At a preset calibration point, the positioning results of the visual-inertial odometry are calibrated using local beacon signals.

2. The offline adaptive navigation method based on AR glasses according to claim 1, characterized in that, The offline map data package includes: A structured file used to describe the coordinates of attractions and the connectivity graph; Image files used for spot previews; And audio files used for voice explanations.

3. The offline adaptive navigation method based on AR glasses according to claim 1, characterized in that, The specific steps for providing a scenic area overview to the user based on the data packet include: Parse the offline map data package to obtain preview information for all attractions; A panoramic visualization interface containing thumbnails of all attractions is generated and displayed on the waveguide display of the AR glasses. At the same time, a voice summary containing the total number of attractions and the estimated total tour time is broadcast through bone conduction headphones.

4. The offline adaptive navigation method based on AR glasses according to claim 1, characterized in that, The step of receiving the attraction selection input can be implemented in at least one of the following ways: Method A: Track the user's head movement, control the selection cursor to move in the visual interface and align it with the thumbnail of the target attraction, and complete the selection through a preset confirmation action; Method B: Capture the name of the attraction or category keywords spoken by the user through the local speech recognition module, match the recognition results with the attraction information in the data packet, and add the successfully matched attractions to the selection list; Method C: The eye-tracking module captures the location where the user's gaze rests on the visualization interface. When the dwell time exceeds a preset threshold, the system automatically selects the location of the focal point.

5. The offline adaptive navigation method based on AR glasses according to claim 4, characterized in that, When method B is used, the method further includes: Based on a predefined tag library and user profiles, attractions are scored for interest and one or more tour routes consisting of highly interested attractions are automatically recommended to the user for confirmation.

6. The offline adaptive navigation method based on AR glasses according to claim 4, characterized in that, Based on the selected attractions and their connectivity, the specific steps for planning the optimal tour route locally and in real time on the AR glasses are as follows: The multiple scenic spots and their connectivity relationships are modeled as a weighted undirected graph, where the scenic spots are the nodes of the graph, the connecting paths are the edges of the graph, and the weight of the edges is determined based on the actual distance of the path or the estimated travel time. In the weighted undirected graph, starting from the user's current location or the first selected attraction, and ending at the last selected attraction or the exit, a graph search algorithm is used to find the shortest path to all selected attractions, which is the optimal tour path.

7. The offline adaptive navigation method based on AR glasses according to claim 4, characterized in that, The steps of positioning and tracking using the visual-inertial odometry of the AR glasses, and navigating by rendering the optimal tour path in real time onto the real environment surface in the user's field of vision as augmented reality guidance include: By using the camera and inertial measurement unit integrated into the AR glasses, the pose calculation between consecutive frames is performed using the visual-inertial odometry method, and the six-degree-of-freedom pose information of the AR glasses in the environmental map is output in real time. The next direction of travel along the optimal tour route is converted into a three-dimensional virtual guide model; the virtual guide model is in the form of arrows, footprints, or light strips attached to the ground. The spatial coordinates of the virtual guide model are aligned with the plane in the real environment determined by the current pose information output by the visual-inertial odometry, and then fused and rendered through an optical waveguide display module to make the virtual guide model appear to fit the surface of the real environment.

8. The offline adaptive navigation method based on AR glasses according to claim 7, characterized in that, The specific steps for responding to a user's route modification command, instantly replanning the route locally, and updating the augmented reality guidance are as follows: When an instruction to add or delete a attraction is received, a heuristic optimization algorithm is used to locally rearrange the path sequence based on the currently planned path and the connectivity relationship to generate a new optimal tour path, and the augmented reality guidance rendered on the real environment surface is updated immediately.

9. The offline adaptive navigation method based on AR glasses according to claim 1, characterized in that, The specific steps for correcting the positioning results of the visual-inertial odometry at a preset calibration point using local beacon signals include: Low-power Bluetooth beacons are deployed at the entrance of the scenic spot or at key nodes along the route. The beacons continuously broadcast beacon signals containing their own unique ID and preset three-dimensional spatial coordinates. When the AR glasses move into the effective signal coverage area of ​​the beacon, its integrated Bluetooth module receives and parses the beacon signal to obtain the beacon's ID and coordinate information; The AR glasses compare the received beacon coordinates with their current pose calculated by the visual-inertial odometry to calculate a pose deviation. Using the beacon coordinates as an absolute reference, the current pose output by the visual-inertial odometry is subjected to a one-time coordinate translation and rotation correction, and the correction amount is applied to the subsequent visual-inertial odometry calculation process to reset the accumulated error.

10. An offline adaptive navigation system based on AR glasses, characterized in that, For implementing the method of any one of claims 1-9, the system comprises: an AR glasses terminal and a plurality of positioning and calibration beacons deployed at preset calibration points in the scenic area, for broadcasting the local beacon signals to the AR glasses terminal; the AR glasses terminal integrates: A memory for pre-storing the offline map data packets; The processing unit, connected to the memory, is used to perform tasks such as receiving attraction selections, planning routes, replanning routes, and controlling rendering. The visual-inertial odometry module is used to perform localization tracking and output pose information; The optical waveguide display module is used to receive rendering instructions from the processing unit and present augmented reality guidance to the user's field of vision; Audio output device, used to broadcast audio summaries and explanations; The input sensing module is used to capture the user's interaction commands; A wireless communication module is used to receive the local beacon signal; When performing path planning, the processing unit is configured to: model the multiple attractions and their connectivity as a weighted undirected graph, and use a graph search algorithm to solve for the shortest path to all selected attractions; When performing AR rendering, the processing unit is configured to: convert the next direction of travel of the planned path into a three-dimensional virtual guide model, align the spatial coordinates of the virtual guide model with the real environment plane determined by the current pose information, and perform fusion rendering through the optical waveguide display module; When responding to a path modification command, the processing unit is configured to: use a heuristic optimization algorithm to locally rearrange the current path sequence to generate a new optimal tour path, and control the optical waveguide display module to immediately update the augmented reality guidance.

Images