Passive positioning and navigation method and system in complex cabin

Abstract

The application provides a passive positioning and navigation method and system in a complex cabin, the method comprising: processing inertial measurement data, visible light images and depth images, fusing depth information of visual feature points to generate initial pose estimation through visual-inertial tight coupling optimization; when cabin text identifiers or visual labels are detected in the visible light images, text features or label corner pixel coordinates are extracted, and inverse depth plane constraints and absolute pose constraints are respectively constructed; cabin text identifiers and visual labels in the complex cabin are used as passive and high-precision calibration sources to provide effective geometry and absolute pose constraints for pose calibration, and the defects of the existing pure visual-inertial positioning in lacking effective calibration mechanism are made up; the initial pose estimation is calibrated based on the inverse depth plane constraints and the absolute pose constraints to obtain target pose estimation; and navigation guide information is generated based on the target pose estimation to realize visual display, and high-precision and high-robustness positioning and navigation in the complex cabin environment.

Description

Technical Field

[0001] This invention relates to the field of navigation technology, and in particular to a passive positioning and navigation method and system for complex cabins. Background Technology

[0002] In complex environments such as ships, aircraft, underground tunnels, and nuclear power plants, traditional positioning technologies like GPS, Wi-Fi, and Bluetooth become ineffective due to the shielding effect of building structures on electromagnetic waves. Existing indoor positioning solutions largely rely on pre-installed beacons or infrastructure, such as Ultra-Wideband (UWB) base stations and Radio Frequency Identification (RFID) tags, which suffer from high deployment costs, maintenance difficulties, and susceptibility to interference. Furthermore, pure inertial navigation systems are prone to severe drift and cannot be used long-term, while visual SLAM (Simultaneous Localization and Mapping) systems exhibit poor stability in low-light or texture-deficient environments.

[0003] With the development of Augmented Reality (AR) technology, AR navigation has gained attention for its immersive interactive experience.

[0004] However, existing AR navigation systems generally rely on external positioning signals, such as GPS or Bluetooth beacons, making it difficult to operate independently in passive environments. Some studies have attempted to combine visual and inertial sensors to achieve passive positioning, but the lack of an effective pose calibration mechanism results in large accumulated errors and weak repositioning capabilities. Summary of the Invention

[0005] This invention provides a passive positioning and navigation method and system for complex cabins, which solves the defects of existing technologies that rely too much on external information sources or lack robust sensor fusion and closed-loop calibration mechanisms, resulting in easy positioning failure, error accumulation and difficulty in repositioning in passive or dynamic environments.

[0006] This invention provides a passive positioning and navigation method for complex cabins, comprising the following steps.

[0007] Acquire multi-source heterogeneous data collected by multi-source sensors mounted on an augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; The inertial measurement data is pre-integrated to obtain the inertial pre-integration increment, and the visible light image is feature extracted to obtain visual feature points. Depth information of the visual feature points is obtained based on the depth image. Target visual feature points are determined based on the depth information and the visual feature points. Tight coupling optimization is performed based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate. If a cabin text label is detected in the visible light image, the text features of the cabin text label are extracted, and an inverse depth plane constraint is constructed based on the text features; if a visual label is detected in the visible light image, the pixel coordinates of the label corner points are extracted, and an absolute pose constraint is constructed based on the pixel coordinates of the label corner points. Based on the inverse depth plane constraint and the absolute pose constraint, the initial pose estimate is dynamically calibrated to obtain the target pose estimate; Based on the target pose estimation, navigation guidance information is generated, and the navigation guidance information is visualized through the augmented reality device.

[0008] According to a passive positioning and navigation method for complex cabins provided by the present invention, the step of pre-integrating the inertial measurement data to obtain the inertial pre-integration increment includes: The first visible light image and the second visible light image acquired continuously in the visible light image are determined, the acquisition time of the first visible light image is determined as the first time, and the acquisition time of the second visible light image is determined as the second time. Acquire the acceleration and angular velocity from the inertial measurement data between the first time point and the second time point; Based on the acceleration and the angular velocity, the position pre-integral, velocity pre-integral, and rotational pre-integral are calculated respectively, and the motion pre-integral is determined based on the position pre-integral, velocity pre-integral, and rotational pre-integral. An inertial constraint term is constructed based on the motion pre-integral quantity, and the inertial pre-integral increment is determined based on the residual information corresponding to the inertial constraint term.

[0009] According to the passive positioning and navigation method for complex cabins provided by the present invention, the text features are multiple visual key points extracted within the boundary area of ​​the cabin text markings in the visible light image; The construction of inverse depth plane constraints based on the text features includes: Based on the pixel coordinates and inverse depth values ​​corresponding to the multiple visual key points, the three-dimensional spatial plane where the cabin text sign is located is fitted. The visible light image including the cabin text markings acquired at the current moment is taken as the current frame, and the visible light image including the cabin text markings acquired at a historical moment is taken as the reference frame. Determine the geometric error and photometric error corresponding to the visual key point; the geometric error is determined based on the geometric distance from the visual key point to the three-dimensional space plane; the photometric error is determined based on the photometric difference between the visual key point in the current frame and the matching point in the reference frame, wherein the matching point is the visual key point in the reference frame that matches the visual key point in the current frame; Based on the geometric and photometric errors corresponding to all the aforementioned visual keypoints, a point-to-surface distance error function is determined. The point-to-surface distance error function is used as the inverse depth plane constraint. Passive positioning and navigation method for complex cabins. According to the passive positioning and navigation method for complex cabins provided by the present invention, the step of constructing absolute pose constraints based on the pixel coordinates of the tag corner points includes: Obtain the spatial coordinates of the label corner points in the physical world coordinate system; the spatial coordinates of the label corner points are determined based on the actual physical size of the visual label; Based on a preset corner point index order, the pixel coordinates of the label corner points are matched with the spatial coordinates of the label corner points to construct a set of point pairs; Based on the set of point pairs and the intrinsic parameter matrix of the visible light camera, pose calculation is performed to obtain the relative pose transformation matrix of the visible light camera relative to the visual label; The relative pose transformation matrix is ​​converted into an absolute pose in the global coordinate system, and the absolute pose is used as the absolute pose constraint.

[0010] According to the passive positioning and navigation method for complex cabins provided by the present invention, the step of generating navigation guidance information based on the target pose estimation includes: Receive navigation requests that include preset start and destination anchor points; In the preset anchor point topology map, a main navigation path is generated based on the optimal path between the preset starting anchor point and the destination anchor point; An initial guidance path is generated based on the shortest path between the user's position estimated by the target pose and the preset starting anchor point. The navigation guidance information is generated based on the initial guidance path and the main navigation path.

[0011] According to a passive positioning and navigation method for complex cabins provided by the present invention, the step of constructing the anchor point topology map includes: Receive the first instruction to create an anchor point at the current location; In response to the first instruction, the pose estimation result of the augmented reality device at the current moment is obtained, and the three-dimensional spatial coordinates in the pose estimation result are determined as the spatial coordinates of the anchor point, and the spatial coordinates of the anchor point are stored. Receive the semantic name associated with the anchor point, and associate and store the semantic name with the spatial coordinates of the anchor point; Receive a second instruction for establishing a topological connection relationship between at least two of the anchor points that have been created, and determine the topological connection relationship between at least two of the anchor points; Based on the spatial coordinates of all the anchor points, the associated semantic names of all the anchor points, and the topological connection relationships, the anchor point topology map is generated.

[0012] The present invention also provides a passive positioning and navigation system for complex cabins, comprising the following modules: The acquisition module is used to acquire multi-source heterogeneous data collected by the multi-source sensors mounted on the augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; The processing module is used to perform pre-integration processing on the inertial measurement data to obtain the inertial pre-integration increment, and to extract visual feature points from the visible light image. The determination module is used to obtain depth information of the visual feature points based on the depth image, determine the target visual feature points based on the depth information and the visual feature points, and perform tight coupling optimization based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate. The construction module is used to extract the text features of the cabin text markings when cabin text markings are detected in the visible light image, and to construct inverse depth plane constraints based on the text features; and to extract the label corner pixel coordinates of the visual labels when visual labels are detected in the visible light image, and to construct absolute pose constraints based on the label corner pixel coordinates. The pose optimization module is used to dynamically calibrate the initial pose estimate based on the inverse depth plane constraint and the absolute pose constraint to obtain the target pose estimate. The navigation guidance module is used to perform anchor point matching in a pre-built anchor point topology map based on the target pose estimation to determine the current anchor point position, perform path planning based on the current anchor point position to generate a target path, generate navigation guidance information based on the target path, and visualize the navigation guidance information through the augmented reality device.

[0013] According to the present invention, a passive positioning and navigation system for complex cabins is provided, wherein the navigation guidance module includes a user interaction unit, and the user interaction unit includes an AR glasses visualization unit, an interactive interface, and a triggering unit; The AR glasses visualization unit is used to render the navigation guidance information into virtual visual elements and display the virtual visual elements on the display interface of the AR glasses to blend with the real-world scene; the virtual visual elements include directional arrows, navigation trajectory, cabin anchor frames, and arrival reminder effects; The interactive interface is used to generate visual interactive elements for navigation control; The triggering unit is used to trigger navigation control commands pre-associated with the visual interactive element in response to recognizing a user's gesture input or voice input to the visual interactive element.

[0014] According to the present invention, a passive positioning and navigation system for complex cabins is provided, the system is further used for: Based on the target pose estimation, the first unreached anchor point in the anchor point sequence of the target path is determined as the immediate target anchor point; The yaw angle of the directional arrow is calculated in real time based on the first three-dimensional spatial coordinates of the user's head pose currently acquired in real time, and the second three-dimensional spatial coordinates of the instantaneous target anchor point. The AR glasses visualization unit is also used to adjust the orientation of the directional indicator arrow on the display interface in real time based on the yaw angle.

[0015] According to the present invention, a passive positioning and navigation system for complex cabins is provided, the system is further used for: Determine the spatial distance between the user position of the target pose estimation and the position of the process anchor point located after the instantaneous target anchor point in the anchor point sequence; In response to the spatial distance being less than a preset arrival threshold, the next anchor point in the anchor point sequence is redefined as the immediate target anchor point.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the passive positioning and navigation method for complex cabins as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the passive positioning and navigation method for complex cabins as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the passive positioning and navigation method for complex cabins as described above.

[0019] The present invention provides a passive positioning and navigation method and system for complex cabins, which acquires inertial measurement data, visible light images, and depth images collected by multi-source sensors mounted on an augmented reality device. By processing the inertial measurement data and visible light images, and fusing depth information from visual feature points, a vision-inertial tight coupling optimization is performed to generate an initial pose estimate. When cabin text markings or visual labels are detected in the visible light image, their text features or label corner pixel coordinates are extracted to construct inverse depth plane constraints and absolute pose constraints, respectively. Thus, cabin text markings commonly found in complex cabins and visual labels that can be deployed on demand can be used as passive, high-precision external calibration sources, providing effective geometric and absolute pose constraints for subsequent pose calibration, thus overcoming the deficiency of existing pure vision-inertial positioning methods that lack an effective calibration mechanism. Based on the inverse depth plane constraints and absolute pose constraints, the accumulated error in the initial pose estimate is dynamically calibrated to obtain a high-precision target pose estimate. Finally, navigation guidance information is generated based on the target pose estimate and visualized through an augmented reality device. This invention introduces cabin text markings and visual tags in the environment as an effective pose calibration mechanism. In a passive environment that does not rely on external signals such as GPS, it can dynamically correct the cumulative drift generated by the visual-inertial odometry. This solves the technical problems of large cumulative errors and weak repositioning capabilities caused by the lack of an effective calibration mechanism in existing technologies. As a result, it achieves high-precision and robust positioning and navigation in complex cabin environments with changes in lighting and missing textures, ensuring the stability and accuracy of navigation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the passive positioning and navigation method for complex cabins provided by the present invention.

[0022] Figure 2 This is the second flowchart of the passive positioning and navigation method for complex cabins provided by the present invention.

[0023] Figure 3 This is an internal schematic diagram of the multi-source fusion positioning module provided by the present invention.

[0024] Figure 4 This is an internal schematic diagram of the path planning module provided by the present invention.

[0025] Figure 5This is a flowchart illustrating the process of visual interaction based on augmented reality devices provided by the present invention.

[0026] Figure 6 This is a structural schematic diagram of the passive positioning and navigation system for complex cabins provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0030] This invention provides a passive positioning and navigation method for complex cabins, particularly suitable for enclosed environments such as ships, aircraft, underground facilities, and large buildings where GPS signals are lacking, electromagnetic environments are complex, and spatial structures are variable. This method aims to solve the problems of low positioning accuracy, large cumulative errors, and poor robustness in passive environments of existing technologies, achieving high-precision and high-reliability augmented reality navigation. Figure 1 This is one of the flowcharts illustrating the passive positioning and navigation method for complex cabins provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 110: Acquire multi-source heterogeneous data collected by the multi-source sensors mounted on the augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images.

[0031] Specifically, firstly, multi-source heterogeneous data is acquired from the multi-source sensors mounted on the augmented reality device. Here, the augmented reality device refers to a wearable electronic device worn by the user that has the ability to acquire, process, and display data. In this embodiment, the augmented reality device specifically refers to augmented reality glasses. The augmented reality glasses integrate multi-source sensors for perception and a near-eye display module for display, thereby enabling virtual guidance to be overlaid directly in front of the user's line of sight during navigation, providing an immersive navigation visualization experience.

[0032] Here, multi-source heterogeneous data is used to reflect comprehensive information about the external environment and the motion state perceived by the augmented reality device. "Multi-source" refers to the data coming from at least three different types of sensors mounted on the augmented reality device, and "heterogeneous" refers to the differences in the physical meaning, data format, sampling frequency, and error characteristics of this data.

[0033] The multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images. Inertial measurement data is typically acquired by an Inertial Measurement Unit (IMU) integrated within the augmented reality device. The IMU typically includes a gyroscope and an accelerometer, used to measure the angular velocity and linear acceleration of the augmented reality device (i.e., the user's head) in three-dimensional space, respectively. This high-frequency data can accurately capture the motion changes of the augmented reality device over short periods. In some embodiments, the IMU may also include a magnetometer to provide an absolute heading angle reference, but this is not a requirement of the embodiments of the present invention.

[0034] Visible light images are typically captured by a visible light camera, such as an RGB camera, in the forward field of view of an augmented reality device. The visible light camera continuously captures a sequence of color images of the environment at a certain frame rate (e.g., 30 FPS). Visible light images are rich in texture, color, and semantic information about the scene, and are fundamental for identifying visual features and environmental landmarks, such as text signs and visual labels.

[0035] Here, depth images are typically captured by depth cameras mounted on augmented reality devices. Each pixel value in a depth image directly or indirectly corresponds to the physical distance of that pixel from the camera in the real world. Therefore, depth images provide information about the three-dimensional geometry of a scene, complementing visible light images.

[0036] Here, the process of acquiring multi-source heterogeneous data refers to the process by which the central processing unit or dedicated processor of the augmented reality device starts and controls the operation of multiple source sensors according to the preset program logic, reads measurement data from the sensors, and performs timestamp synchronization processing on the measurement data to ensure that data collected from different sensors at the same time can be correctly correlated.

[0037] Step 120: Perform pre-integration processing on the inertial measurement data to obtain the inertial pre-integration increment, and extract visual feature points from the visible light image.

[0038] Specifically, after obtaining inertial measurement data, the inertial measurement data can be pre-integrated to obtain the inertial pre-integration increment, and the visible light image can be feature extracted to obtain visual feature points.

[0039] Here, the inertial pre-integration increment represents the relative pose change of the augmented reality device calculated solely from inertial measurement data between two consecutive visible light image acquisition moments, including increments in position, velocity, and attitude. The purpose of pre-integrating the inertial measurement data is to avoid repeatedly integrating the IMU data from the initial moment each time during backend pose optimization, thus significantly reducing computational complexity. This process integrates a large number of IMU measurements between two keyframes into a single relative motion constraint, which depends only on the pose of these two keyframes and the IMU's zero bias.

[0040] Visual feature points refer to stable, distinguishable, and easily trackable pixels in a visible light image. Visual feature points can be corner points, edge points, or spots with unique textures of an object. This embodiment of the invention does not specifically limit them.

[0041] It should be understood that visual feature points are fundamental elements in visual odometry and simultaneous localization and mapping systems, used to match different images to calculate camera motion. Feature extraction from visible light images refers to using specific computer vision algorithms to detect and describe these feature points. For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm can be used, which is known for its high computational speed and good rotation invariance, making it well-suited for real-time applications in augmented reality devices. Other feature extraction algorithms, such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF), can also be employed.

[0042] Step 130: Obtain depth information of the visual feature points based on the depth image; determine target visual feature points based on the depth information and the visual feature points; perform tight coupling optimization based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate.

[0043] Specifically, after acquiring the depth image, depth information of visual feature points can be obtained based on the depth image, and target visual feature points can be determined based on the depth information and visual feature points.

[0044] In this context, the depth information of a visual feature point refers to the distance of that visual feature point in three-dimensional space relative to the origin of the camera coordinate system. The depth information is determined by projecting the two-dimensional pixel coordinates of a visual feature point in the visible light image onto the corresponding position in a depth image synchronized with its timestamp, and then reading the depth value at that position.

[0045] Here, target visual feature points refer to feature points extracted from visible light images and whose effective depth values ​​are obtained by association with a synchronized depth image. Target visual feature points reflect the three-dimensional spatial structure information in the scene that can be used for precise positioning. The process of determining target visual feature points is a data filtering process. Because depth cameras may have measurement blind spots, measurement noise, or fail on specific materials, such as reflective or transparent surfaces, not all extracted visual feature points can provide effective depth information. Therefore, this step discards visual feature points that lack corresponding depth information or whose depth information is unreliable, retaining only feature points with effective three-dimensional spatial coordinates as target visual feature points.

[0046] Then, tight coupling optimization can be performed based on the inertial pre-integration increment and the target visual feature points to generate the initial pose estimate.

[0047] Initial pose estimation refers to the position and orientation of the augmented reality device in three-dimensional space calculated by the system by fusing visual and inertial information. It is usually represented by a six-degree-of-freedom (6-DoF) variable.

[0048] Tightly coupled optimization based on inertial pre-integration increments and target visual feature points refers to jointly optimizing visual and inertial information within a unified optimization framework to obtain the optimal pose estimation. Specifically, tight coupling means simultaneously incorporating the raw measurements from the IMU (represented in pre-integration form) and the raw measurements from the vision (i.e., the pixel coordinates of the target visual feature points) into the objective function of a nonlinear optimization problem. This objective function typically consists of two parts: an inertial constraint residual constructed based on the inertial pre-integration model and a visual reprojection residual constructed based on the camera projection model. By minimizing this joint residual, the pose, velocity, and state variables such as the IMU's zero bias of the augmented reality device can be estimated simultaneously. Compared to loose coupling, which calculates the pose separately and then fuses them, this tight coupling approach achieves higher accuracy and robustness because the error models of the two sensors can mutually correct each other. For example, a vision-inertial navigation tight coupling scheme based on the ORB-SLAM3 framework can be used to achieve this step.

[0049] Step 140: If a cabin text label is detected in the visible light image, extract the text features of the cabin text label and construct an inverse depth plane constraint based on the text features; if a visual label is detected in the visible light image, extract the label corner pixel coordinates of the visual label and construct an absolute pose constraint based on the label corner pixel coordinates.

[0050] Specifically, when cabin text markings are detected in the visible light image, the text features of the cabin text markings are extracted, and inverse depth plane constraints are constructed based on the text features. When visual labels are detected in the visible light image, the pixel coordinates of the label corner points are extracted, and absolute pose constraints are constructed based on the label corner point pixel coordinates. This step is used to correct the unavoidable cumulative drift in the initial pose estimation mentioned above. This step is dynamically triggered, that is, it is only executed when a specific landmark is detected in the environment.

[0051] Here, "cabin text markings" refers to text signs commonly found inside cabins and attached to walls, equipment, or doors, such as electrical distribution boxes, cabin A-01, etc. This embodiment of the invention does not specifically limit this.

[0052] The text features are used to reflect the geometric distribution of the cabin text markings in the visible light image. The text features can be key points of the strokes of the characters constituting the cabin text markings, or boundary corner points of the text areas of the cabin text markings; this embodiment of the invention does not specifically limit these features.

[0053] Here, the inverse depth plane constraint is a geometric constraint imposed on the optimization problem. Its construction principle is based on a physical prior: all text feature points on the same cabin signage should be coplanar. The inverse depth plane constraint utilizes this property to correct depth estimation errors and pose drift caused by visual-inertial odometry (VIO). When the estimated 3D points do not meet the coplanarity condition, the inverse depth plane constraint generates an error term that drives the optimizer to adjust the pose, causing these points to re-converge into coplanarity.

[0054] Visual tags are artificial markers with known dimensions and patterns that are pre-placed in the environment. For example, AprilTag or QR codes can be used as visual tags. Visual tags essentially provide artificial GPS anchor points for the system. The pixel coordinates of the tag corners refer to the coordinates of the four corner points of the visual tag detected in a visible light image within the image coordinate system.

[0055] Here, absolute pose constraints are strong constraints that provide precise pose information in the global coordinate system. The construction of absolute pose constraints is based on visual labels with known real physical dimensions. When the four corner points of a visual label are detected by the image, the relative pose of the camera relative to the visual label can be accurately calculated using methods such as the Perspective-n-Point (PnP) algorithm. If the position of the visual label in the global map has been pre-calibrated, the calculated relative pose can be used as an absolute pose observation to correct the accumulated drift error of the visual-inertial odometry (VIO) in one step, thereby achieving global relocalization of the system. Even if the global position of the visual label is unknown, this observation can still serve as a high-precision local geometric constraint, effectively suppressing the accumulation of VIO drift.

[0056] Step 150: Based on the inverse depth plane constraint and the absolute pose constraint, the initial pose estimate is dynamically calibrated to obtain the target pose estimate.

[0057] Specifically, after obtaining the inverse depth plane constraints and absolute pose constraints, the initial pose estimate can be dynamically calibrated based on these constraints to obtain the target pose estimate. Here, a pose optimization function can be determined based on the inverse depth plane constraints and absolute pose constraints, and the initial pose estimate can be dynamically calibrated based on this function to obtain the target pose estimate.

[0058] Here, the pose optimization function is a mathematical objective function used to solve for the optimal pose. The pose optimization function is obtained by combining the visual reprojection residual, the inertial constraint residual, the point-to-surface distance error generated by the newly introduced inverse depth plane constraint, and the pose error generated by the absolute pose constraint.

[0059] Here, dynamic calibration of the initial pose estimation refers to minimizing the pose optimization function using a nonlinear optimization method. This optimization process minimizes the total residuals of various constraints, including visual reprojection residuals, inertial constraint residuals, point-to-surface distance errors introduced by inverse depth plane constraints, and pose errors generated by absolute pose constraints, by jointly adjusting the pose parameters of multiple keyframes within the sliding window and the 3D coordinates of map points. This calibration process is dynamic, meaning that each time the system continuously detects new environmental features during navigation, such as cabin text markings or visual labels, one or more rounds of optimization can be triggered, thereby achieving recursive correction and enhancement of the system pose.

[0060] The target pose estimation is the final pose result output after the above dynamic calibration. Compared with the initial pose estimation, the target pose estimation incorporates structured prior information from the environment, such as planes and landmarks. Therefore, it significantly improves global consistency, long-term operational stability and drift suppression capabilities, and is especially suitable for long-term, large-scale navigation tasks with high requirements for accuracy and robustness.

[0061] Step 160: Based on the target pose estimation, generate navigation guidance information and visualize the navigation guidance information through the augmented reality device.

[0062] Specifically, after obtaining the target pose estimate, navigation guidance information can be generated based on the target pose estimate, and the navigation guidance information can be visualized through augmented reality devices.

[0063] Here, navigation guidance information refers to visual instructions used to guide users on how to move from their current location to their destination. The process of generating navigation guidance information based on target pose estimation is as follows: First, the system determines the user's accurate location on a pre-built map based on high-precision target pose estimation; then, based on the target point specified by the user, an optimal or suboptimal path is planned on the pre-built map; finally, the path is converted into a series of intuitive navigation instructions, such as directional arrows pointing to the next turning point, the distance to be traveled, and virtual path lines.

[0064] Among them, the augmented reality device is preferably AR glasses, but it can also be a mobile phone or tablet.

[0065] Visualizing navigation guidance information using augmented reality (AR) devices involves rendering generated virtual navigation guidance information, such as 3D arrows and highlighted paths, and accurately overlaying it onto the real-world scene seen by the user through the AR device. For example, a virtual arrow can be suspended in the actual corridor space in front of the user's field of vision, precisely pointing to the intersection where a turn is expected.

[0066] It should be understood that, due to the system's high-precision pose estimation capabilities, the virtual navigation guidance information can achieve stable and accurate alignment with the real environment in three-dimensional space. Even if the user's head moves, the virtual guidance markers remain fixed in the expected physical position without noticeable drift or jitter. This precise virtual-real registration capability effectively enhances the intuitiveness and reliability of navigation guidance, providing users with an intuitive, accurate, and immersive navigation experience that seamlessly blends the virtual and real worlds.

[0067] The method provided in this invention acquires inertial measurement data, visible light images, and depth images collected by multi-source sensors mounted on an augmented reality device. It processes the inertial measurement data and visible light images, and fuses depth information from visual feature points for visual-inertial tight coupling optimization to generate an initial pose estimate. When cabin text markings or visual labels are detected in the visible light image, their text features or label corner pixel coordinates are extracted to construct inverse depth plane constraints and absolute pose constraints, respectively. This allows the use of cabin text markings commonly found in complex cabins and on-demand deployable visual labels as passive, high-precision external calibration sources, providing effective geometric and absolute pose constraints for subsequent pose calibration and overcoming the lack of an effective calibration mechanism in existing pure visual-inertial positioning systems. Based on the inverse depth plane constraints and absolute pose constraints, the accumulated error in the initial pose estimate is dynamically calibrated to obtain a high-precision target pose estimate. Finally, navigation guidance information is generated based on the target pose estimate and visualized through the augmented reality device. This invention introduces cabin text markings and visual tags in the environment as an effective pose calibration mechanism. In a passive environment that does not rely on external signals such as GPS, it can dynamically correct the cumulative drift generated by the visual-inertial odometry. This solves the technical problems of large cumulative errors and weak repositioning capabilities caused by the lack of an effective calibration mechanism in existing technologies. As a result, it achieves high-precision and robust positioning and navigation in complex cabin environments with changes in lighting and missing textures, ensuring the stability and accuracy of navigation.

[0068] Based on the above embodiments, step 120, which involves pre-integrating the inertial measurement data to obtain the inertial pre-integration increment, includes: Step 120-1: Determine the first visible light image and the second visible light image that are continuously acquired in the visible light image, and determine the acquisition time of the first visible light image as the first moment and the acquisition time of the second visible light image as the second moment; Step 120-2: Obtain the acceleration and angular velocity from the inertial measurement data between the first time point and the second time point; Step 120-3: Based on the acceleration and the angular velocity, calculate the position pre-integral, velocity pre-integral, and rotational pre-integral respectively; and determine the motion pre-integral based on the position pre-integral, velocity pre-integral, and rotational pre-integral. Step 120-4: Construct an inertial constraint term based on the motion pre-integral quantity, and determine the inertial pre-integral increment based on the residual information corresponding to the inertial constraint term.

[0069] Specifically, firstly, a first visible light image and a second visible light image acquired continuously in the visible light image are determined, the acquisition time of the first visible light image is determined as the first moment, and the acquisition time of the second visible light image is determined as the second moment.

[0070] Here, the first visible light image and the second visible light image refer to two adjacent frames in the time series. In visual inertial odometry, complex pose optimization is not typically performed on every frame; instead, a subset of images is selected as keyframes. Therefore, in a preferred embodiment, the first visible light image and the second visible light image can each refer to two consecutive keyframes. The first time point and the second time point refer to the precise timestamps at which these two frames were acquired by the augmented reality device's sensors. These two timestamps define a time interval [tk, t{k+1}], within which subsequent inertial data processing will be performed.

[0071] Then, the acceleration and angular velocity from the inertial measurement data between the first and second moments are acquired. Acceleration and angular velocity are used to reflect the instantaneous rate of change of the linear and angular motion of the augmented reality device in three-dimensional space, respectively. These are the raw measurement values ​​output by the inertial measurement unit. Here, acquiring the acceleration and angular velocity from the inertial measurement data means reading and caching all data samples generated by the IMU sensor within the time interval [tk, t{k+1}]. Since the sampling frequency of the IMU is much higher than that of a visible light camera (e.g., 200Hz for the IMU and 30Hz for the visible light camera), this time interval will contain multiple sets of acceleration and angular velocity measurements.

[0072] Next, based on acceleration and angular velocity, the position pre-integral, velocity pre-integral, and rotational pre-integral are calculated respectively, and the motion pre-integral is determined based on the position pre-integral, velocity pre-integral, and rotational pre-integral.

[0073] The position pre-integral, velocity pre-integral, and rotation pre-integral are used to reflect the relative changes in position, velocity, and attitude of the augmented reality device (ARD) caused solely by inertial measurement data within the interval [tk, t{k+1}], without considering the initial state (pose and velocity) at the first moment. Specifically, the calculation process involves integrating the acceleration and angular velocity over the time interval, using the ARD's attitude at the first moment tk as the reference coordinate system. For example, integrating the angular velocity yields the rotation pre-integral, integrating the acceleration after attitude transformation once yields the velocity pre-integral, and integrating twice yields the position pre-integral. The specific methods for calculating the position, velocity, and rotation pre-integral are existing techniques in this field and will not be elaborated upon here.

[0074] Among them, the motion pre-integral quantity refers to the set of position pre-integral quantity, velocity pre-integral quantity, and rotational pre-integral quantity. The motion pre-integral quantity is used to reflect the relative motion information from the first moment to the second moment.

[0075] Finally, inertial constraint terms are constructed based on the motion pre-integral quantities, and inertial pre-integral increments are determined based on the residual information corresponding to the inertial constraint terms.

[0076] Here, the inertial constraint term refers to a physical constraint connecting the augmented reality device's state (including pose, velocity, and IMU bias) at the first and second time points. Specifically, the inertial constraint term constructs a residual function that measures the difference between the state estimate at the first time point and the state at the second time point predicted by the motion pre-integral, and the state estimate at the second time point. This difference is the residual information. Ideally, if all state estimates are accurate, this residual information should be zero.

[0077] The residual information generated by the inertial constraint term will serve as an important component of the subsequent pose optimization function. During pose optimization, the optimizer adjusts the state estimates at the first and second time moments (as optimization variables) to minimize this residual information (i.e., the error). Therefore, the inertial pre-integration increment in this step is essentially a high-frequency, short-duration relative motion measurement value, providing the core basis for calculating the residual, thus effectively integrating the IMU measurement data into the global pose optimization framework that operates at a lower frequency.

[0078] The method provided in this invention introduces inertial pre-integration technology to preprocess a large amount of IMU data between two keyframes into a single relative motion constraint, avoiding repeated integration of IMU measurements during back-end pose optimization, greatly reducing computational complexity, and enabling tight coupling optimization of visual and inertial data to run in real time on augmented reality devices with limited computing resources.

[0079] Based on the above embodiments, the text features are multiple visual key points extracted within the boundary area of ​​the cabin text markings in the visible light image; Step 140, which involves constructing inverse depth plane constraints based on the text features, includes: Step 1401: Fit the three-dimensional spatial plane where the cabin text sign is located based on the pixel coordinates and inverse depth values ​​corresponding to the multiple visual key points respectively; Step 1402: Take the visible light image including the cabin text markings acquired at the current moment as the current frame, and take the visible light image including the cabin text markings acquired at a historical moment as the reference frame. Step 1403: Determine the geometric error and photometric error corresponding to the visual key point; the geometric error is determined based on the geometric distance from the visual key point to the three-dimensional space plane; the photometric error is determined based on the photometric difference between the visual key point in the current frame and the matching point in the reference frame, wherein the matching point is the visual key point in the reference frame that matches the visual key point in the current frame. Step 1404: Determine the point-to-surface distance error function based on the geometric and photometric errors corresponding to all the visual key points; Step 1405: Use the point-to-surface distance error function as the inverse depth plane constraint.

[0080] Specifically, text features are multiple visual keypoints extracted from the boundary region of the cabin text signage in a visible light image. These visual keypoints are image points with good repeatability extracted within the boundary region of the cabin text signage, such as corners, endpoints, or intersections of character strokes. The boundary region of the cabin text signage refers to the rectangular area surrounding the text, located in the visible light image by the text detection algorithm.

[0081] Then, based on the pixel coordinates and inverse depth values ​​corresponding to multiple visual key points, the three-dimensional spatial plane where the cabin text signage is located is fitted.

[0082] In this system, pixel coordinates reflect the precise location of each visual keypoint in the 2D visible light image, while the inverse depth value reflects the distance information of the visual keypoint in 3D space, specifically the reciprocal of the distance. It is important to emphasize that the inverse depth value is estimated and optimized by combining the keypoints in the visible light image with the depth information provided by the depth camera, and a visual odometry system is used. The inverse depth value has a one-to-one correspondence with the pixel coordinates of each keypoint.

[0083] Here, the three-dimensional spatial plane refers to the mathematical representation of the physical plane to which the cabin text markings are attached in three-dimensional space. The fitting process involves taking the three-dimensional coordinates of this series of visual key points (determined by pixel coordinates and inverse depth values) as input and calculating the optimal planar parameters using a robust estimation algorithm, such as Random Sampling Consensus (RANSAC).

[0084] Then, the visible light image including the cabin text markings captured at the current moment can be used as the current frame, and the visible light image including the cabin text markings captured at a historical moment can be used as the reference frame.

[0085] The current frame refers to the latest image frame that the system is currently processing. The reference frame represents an image taken at a specific point in the past, taken from a good angle, with high image quality, and also containing the text markings of the same compartment. The purpose of establishing a reference frame is to enable stable and reliable feature tracking within a continuous sequence of images.

[0086] Furthermore, the geometric error and photometric error corresponding to the visual keypoint can be determined, wherein the geometric error is determined based on the geometric distance from the visual keypoint to the three-dimensional spatial plane; the photometric error is determined based on the photometric difference between the visual keypoint in the current frame and the matching point in the reference frame, where the matching point is the visual keypoint in the reference frame that matches the visual keypoint in the current frame.

[0087] Here, photometric error is used to reflect the consistency of the appearance of the same visual keypoint across different images. Determining photometric error means that for a visual keypoint in the reference frame, the system projects it into the current frame based on the current pose estimate and compares the similarity of the keypoint with its surrounding small image patches in both the reference and current frames. This similarity can be measured using a robust measure to illumination changes, such as zero-mean normalized cross-correlation (ZNCC).

[0088] It should be understood that the smaller the photometric error, the more accurate the feature tracking.

[0089] The point-to-surface distance error function here is a joint optimization function, which is determined by defining it as a weighted combination of geometric and photometric errors, and introducing it as the optimization objective into the pose estimation framework. By minimizing this joint error function, the system simultaneously imposes dual constraints of geometric coplanarity and photometric consistency during the optimization process: ensuring that the 3D spatial distribution of key points conforms to the fitted planar model, and guaranteeing its 2D visual tracking accuracy across image frames.

[0090] By introducing photometric error as a joint optimization element on the basis of traditional geometric constraints, the robustness and stability of the planar constraint mechanism are significantly improved. The introduction of photometric error enables direct verification and screening of feature matching quality, effectively suppressing outlier constraints caused by mismatches. This ensures that the planar constraint terms maintain high reliability and estimation accuracy even in complex scenarios such as rapid camera movement, drastic changes in viewpoint, or fluctuations in lighting conditions.

[0091] Finally, based on the geometric and photometric errors corresponding to all visual keypoints, the point-to-surface distance error function is determined and used as an inverse depth plane constraint.

[0092] Here, geometric distance characterizes the degree of spatial deviation between each visual keypoint and its theoretical coplanar position; while the point-to-plane distance error function is an aggregate measure of the geometric distances of all keypoints, used to comprehensively quantify the consistency level between the set of visual keypoints and the fitted plane. Finally, the point-to-plane distance error function is introduced as an inverse depth plane constraint term into the overall pose optimization objective function of the system, and the pose estimation is corrected and optimized by minimizing this constraint term.

[0093] Based on the above embodiments, step 140, which involves constructing absolute pose constraints based on the pixel coordinates of the label corner points, includes: Step 140-1: Obtain the spatial coordinates of the label corner points in the physical world coordinate system; the spatial coordinates of the label corner points are determined based on the actual physical size of the visual label; Step 140-2: Based on the preset corner point index order, match the pixel coordinates of the label corner points with the spatial coordinates of the label corner points to construct a set of point pairs; Step 140-3: Based on the set of point pairs and the intrinsic parameter matrix of the visible light camera, perform pose calculation to obtain the relative pose transformation matrix of the visible light camera relative to the visual label; Step 140-4: Convert the relative pose transformation matrix into an absolute pose in the global coordinate system, and use the absolute pose as the absolute pose constraint.

[0094] Specifically, firstly, the spatial coordinates of the label corner points in the physical world coordinate system are obtained, where the spatial coordinates of the label corner points are determined based on the actual physical size of the visual label.

[0095] Here, the spatial coordinates of the tag corners reflect the three-dimensional positions of the four corners of the visual tag in a predefined global or world coordinate system. The method for obtaining the spatial coordinates of the tag corners is pre-measured and set during the system deployment phase. For example, an AprilTag can be affixed to the wall of a cabin, and then the coordinates of its four corners in a coordinate system with a corner of the cabin as the origin can be accurately determined using a total station or other measuring equipment, and this information can be stored as known information.

[0096] Here, the actual physical size refers to the design size of the visual label itself, for example, a side length of 10 centimeters, which is the basis for calculating the relative positions of its corners.

[0097] Then, based on the preset corner point index order, the pixel coordinates of the label corner points are matched with the spatial coordinates of the label corner points to construct a set of point pairs. Here, a set of point pairs refers to a set of matching pairs containing two-dimensional image points and their corresponding three-dimensional spatial coordinates.

[0098] Specifically, the detection algorithm for visual tags (such as AprilTag) can not only output the pixel coordinates of the four corner points of the tag, but also identify the tag's unique ID and its orientation in the image. Based on this, it can clearly determine the preset position order of each corner point in the image in the tag's body coordinate system. For example, starting from the top left corner and proceeding clockwise, the points are defined as indices 1 to 4. Based on the preset corner point index order, a unique mapping relationship can be established between the pixel coordinates of the i-th corner point detected in the image and the spatial coordinates of the i-th corner point stored in the tag's 3D model, thereby forming a point pair set consisting of four sets of matching pairs (two-dimensional pixel coordinates and three-dimensional spatial coordinates).

[0099] Next, pose calculation is performed based on the point pair set and the intrinsic parameter matrix of the visible light camera to obtain the relative pose transformation matrix of the visible light camera relative to the visual label.

[0100] Here, the intrinsic parameter matrix reflects the internal optical characteristics of the visible light camera, such as focal length and principal point coordinates. The intrinsic parameter matrix consists of known parameters obtained through a calibration procedure before the visible light camera leaves the factory or is used. The relative pose transformation matrix reflects the rotation and translation relationship between one coordinate system and another coordinate system, and is usually a 4x4 homogeneous transformation matrix.

[0101] Pose calculation refers to the process of solving the relative pose transformation matrix, which can be performed using the perspective n-point algorithm. For example, the solvePnP or solvePnP-Ransac functions in the OpenCV library can be used. Their inputs are the set of point pairs and the camera intrinsic matrix, and the outputs are the camera's rotation and translation vectors in the label coordinate system, which can be converted into the relative pose transformation matrix.

[0102] Finally, the relative pose transformation matrix can be converted into the absolute pose in the global coordinate system, and the absolute pose can be used as the absolute pose constraint.

[0103] Here, absolute pose reflects the pose of the visible light camera in the global coordinate system. The fixed pose of the visual label in the global coordinate system (world coordinate system) is known to be Tworld_tag. The relative pose transformation matrix Tcam_tag represents the transformation from the label coordinate system to the camera coordinate system. Therefore, the absolute pose Tworld_cam of the visible light camera in the global coordinate system can be calculated using the following formula: Tworld_cam = Tworld_tag ​​* inv(Tcam_tag); Using absolute pose as a constraint refers to constructing a pose prior constraint (or absolute observation factor) based on the calculated high-precision absolute pose observations and adding it to the system's global pose optimization objective function. The pose prior constraint defines the error between the current system-estimated camera pose and the observed pose. By minimizing the overall objective function including this constraint using optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method, the accumulated error of the system can be effectively corrected using this absolute observation information, improving the consistency accuracy between the global pose estimation and the map. Typically, the pose prior constraint is associated with a covariance matrix reflecting observation uncertainty to quantify its confidence level and apply appropriate weights in the optimization process.

[0104] The method provided in this invention, compared to VIO systems that rely solely on relative motion estimation, can provide globally consistent, drift-free pose truth, fundamentally solving the problem of accumulated error.

[0105] Based on the above embodiments, step 160, which involves generating navigation guidance information based on the target pose estimation, includes: Step 160-1: Receive a navigation request including a preset starting anchor point and a destination anchor point; Step 160-2: In the preset anchor point topology map, generate the main navigation path based on the optimal path between the preset starting anchor point and the destination anchor point; Step 160-3: Generate an initial guidance path based on the shortest path between the user's position estimated by the target pose and the preset starting anchor point; Step 160-4: Generate the navigation guidance information based on the initial guidance path and the main navigation path.

[0106] Specifically, first, a navigation request is received, including a preset start anchor point and a destination anchor point. A navigation request refers to a user-initiated navigation intention that specifies the starting point and destination. The preset start anchor point and destination anchor point are two nodes in a preset anchor point topology map.

[0107] Here, the pre-defined anchor point topology map refers to a simplified graph structure of a complex cabin environment, consisting of key location points (anchor points) and feasible paths (edges) between them. Anchor points can be doorways, stairwells, important equipment, etc., and this embodiment of the invention does not specifically limit them. The topology map is pre-built and stored before navigation. For example, a user can initiate a navigation request by selecting "A-01 cabin doorway" as the starting anchor point and "B-03 power cabin" as the destination anchor point through the interactive interface.

[0108] Then, in the preset anchor point topology map, the main navigation path is generated based on the optimal path between the preset starting anchor point and the destination anchor point.

[0109] Here, the main navigation path refers to the macroscopic path planned on the anchor point topology map, connecting the preset starting anchor point and the destination anchor point. The main navigation path is a sequence of anchor points that need to be traversed sequentially. The process of generating the main navigation path involves path searching on the anchor point topology map. Classic graph search algorithms, such as Dijkstra's algorithm or the A-STAR algorithm, can be used. In addition to considering the length of the path already traversed, the A-STAR algorithm uses a heuristic function (e.g., the straight-line distance to the destination) to evaluate the cost of future paths, and it can usually find the optimal path faster.

[0110] Next, an initial guidance path is generated based on the shortest path between the user's position estimated by the target pose and the preset starting anchor point.

[0111] Here, the initial guidance path refers to the connecting path that guides the user from any current location to the starting point of the main navigation path. The process of generating the initial guidance path is as follows: The system first uses high-precision target pose estimation to determine the user's current accurate 3D coordinates. Then, it calculates the shortest path from the accurate 3D coordinates to the first anchor point of the main navigation path (i.e., the preset starting anchor point). In a local space without obstacles, this shortest path is typically a straight line.

[0112] Finally, navigation guidance information is generated based on the initial guidance path and the main navigation path. This navigation guidance information is further defined as a complete navigation instruction composed of two path segments. The specific process for generating this navigation guidance information is as follows: First, the system provides step-by-step guidance for the initial guidance path based on the user's real-time location results, guiding the user to a preset starting anchor point on the main navigation path. Once the system detects that the user has arrived at the preset starting anchor point, it automatically switches to the segmented guidance mode of the main navigation path, continuously guiding the user along the topological path until they reach their final destination.

[0113] More specifically, the path planning strategy can present different modes based on user input. In one specific implementation, when a user initiates navigation but has not yet explicitly specified a destination anchor point, the system enters a proximity-based guidance mode. In this mode, the system uses Dijkstra's algorithm to calculate all paths to anchor points in a preset anchor point topology map, starting from the user's current location, and automatically selects the shortest path to generate the shortest path to the nearest anchor point as navigation guidance information. This design is well-suited for users in unfamiliar environments who only need to find a known reference point. In another specific implementation, when the user explicitly specifies a destination anchor point through a navigation request, the system initiates a goal-oriented path planning mode. In this case, the system uses the A-STAR algorithm, which, when planning a path, considers not only the actual cost (distance traveled) from the starting point to the current node but also incorporates a straight-line distance heuristic function to estimate the estimated cost from the current node to the destination anchor point. By combining these two cost components, the A-STAR algorithm can more efficiently search and calculate the optimal path, avoiding blind exploration, and is particularly suitable for long-distance navigation in large and complex cabin environments.

[0114] The navigation method provided in this invention is a two-stage path generation and guidance strategy that integrates real-time positioning and topology planning. This generation and guidance strategy combines real-time, high-precision free-space positioning capabilities with offline, structured topology path planning, achieving intelligent connection and smooth transition in the navigation process. Users do not need to pre-arrive at a fixed path starting point; the system can automatically generate initial connection segments from any real-time location and seamlessly guide them onto the macroscopic topology path. This significantly improves the system's flexibility, adaptability, and end-user experience in practical applications while ensuring navigation accuracy.

[0115] Based on the above embodiments, the steps for constructing the anchor point topology map include: Step 210: Receive a first instruction for creating an anchor point at the current location; Step 220: In response to the first instruction, obtain the pose estimation result of the augmented reality device at the current moment, determine the three-dimensional spatial coordinates in the pose estimation result as the spatial coordinates of the anchor point, and store the spatial coordinates of the anchor point; Step 230: Receive the semantic name associated with the anchor point, and associate and store the semantic name with the spatial coordinates of the anchor point; Step 240: Receive a second instruction for establishing a topological connection relationship between at least two of the anchor points that have been created, and determine the topological connection relationship between at least two of the anchor points; Step 250: Generate the anchor point topology map based on the spatial coordinates of all the anchor points, the associated semantic names of all the anchor points, and the topological connection relationships.

[0116] Specifically, the preset anchor point topology map is pre-generated using a specific mapping method. This mapping method serves as an offline preparatory step, providing foundational data support for subsequent online navigation. The specific implementation of the mapping method is as follows: First, a first instruction is received to create an anchor point at the current location. The current location refers to the real-time position of the augmented reality device in the global coordinate system during the mapping operation. The first instruction is a preset signal initiated by the user to trigger the anchor point creation action.

[0117] In one specific implementation, the first instruction can be a button signal triggered by a physical button on the augmented reality device, a specific gesture made by the user through a gesture recognition system (such as "air click"), or a voice command issued by the user through a voice recognition system (such as saying "mark anchor point"). The system receives the first instruction through the corresponding input interface.

[0118] Then, in response to the first command, the system obtains the pose estimation result of the augmented reality device at the current moment, and uses the three-dimensional spatial coordinates in the pose estimation result as the spatial coordinates of the anchor point, storing the anchor point's spatial coordinates. Specifically, upon receiving the first command, the system immediately invokes its internal pose estimation algorithm to obtain a high-precision pose estimation result for the current moment. The pose estimation result includes at least the device's three-dimensional spatial coordinates in the global coordinate system, such as (x, y, z), and pose information. The system extracts the three-dimensional spatial coordinates and uses them as the spatial coordinates of the newly created anchor point. These spatial coordinates are then associated with a unique anchor point ID automatically generated by the system and stored in memory. An anchor point is a logical node used to mark a key location with navigational significance in the physical world.

[0119] Furthermore, the system receives the semantic name associated with the anchor point and stores it in association with the anchor point's spatial coordinates. Specifically, after creating an anchor point and recording its spatial coordinates, the system will display an interactive interface prompting the user to name the anchor point. The semantic name refers to the text information entered by the user through this interface, describing the anchor point's physical location. For example, the user can enter "Entrance to Area A of Deck 1" via the virtual keyboard, or "Power Plant Electrical Distribution Box" via the speech-to-text function. Upon receiving this text information, the system will use it as the anchor point's semantic name and establish a connection between it and the anchor point's spatial coordinates.

[0120] Next, a second instruction can be received to establish a topological connection between at least two created anchor points, and the topological connection between at least two anchor points can be determined.

[0121] Specifically, after creating multiple anchor points, the user can specify the connectivity between them. The second instruction is a user-initiated preset signal used to confirm the existence of a passable path between two or more anchor points.

[0122] In one implementation, a user can simultaneously select two anchor points on the interactive interface, such as "Anchor A" and "Anchor-01," and then trigger an "Establish Connection" button. This button signal is the second instruction. In response to the second instruction, the system records a topological connection indicating that "Anchor A" and "Anchor-01" are directly traversable. This topological connection is represented in the data structure as an edge connecting the two anchor point nodes.

[0123] Finally, based on the spatial coordinates of all anchor points, their associated semantic names, and topological connections, an anchor point topology map is generated. The system integrates all anchor points created during the mapping process, their corresponding spatial coordinates, semantic names, and topological connections to generate a structured anchor point topology map. Logically, this map constitutes a graph with node attributes (spatial coordinates, semantic names) and edge information, and is serialized and stored for use during the navigation phase. This method completes the digital modeling of complex cabin environments, laying a solid foundation for subsequent high-precision path planning and navigation guidance.

[0124] Based on any of the above embodiments, the passive positioning and navigation method for complex cabins provided by this invention was tested. The test environment was set up in a complex environment simulating a three-story ship cabin, characterized by narrow passages, multiple rooms, stairs, and typical weak texture and lighting variations. Test results show that the system described in this invention can operate stably in this environment, with an average positioning error of 0.063 meters, achieving centimeter-level positioning accuracy. In multiple repositioning tests that intentionally introduced positioning loss (e.g., rapid rotation or camera obstruction), the system's repositioning success rate reached 100%. Furthermore, the overall response time of the system (from data acquisition to final pose output) is less than or equal to 33.9 milliseconds, enabling the entire system to operate smoothly at a frame rate of 30 FPS (30 frames per second). These test data fully demonstrate that the technical solution of this invention fully meets the navigation requirements of high precision, high robustness, and real-time performance in complex cabin environments.

[0125] In summary, this invention proposes a passive AR navigation system method for complex cabin environments. Through an innovative three-tiered architecture of fusion positioning, navigation planning, and user interaction, it first achieves high-precision fusion positioning by fusing multi-source heterogeneous data and combining cabin text and visual labels for pose calibration. Then, based on the high-precision pose, it performs efficient path planning on an anchor point topology map. Finally, it seamlessly integrates navigation information with the real world through AR glasses, providing a natural interaction method. This invention enables high-precision and robust AR navigation in the absence of external signal sources (such as GPS), effectively overcoming challenges such as varying lighting conditions and texture loss. It is particularly suitable for enclosed environments with complex structures and harsh electromagnetic environments, such as ships, aircraft, and underground facilities, and has a very broad application prospect.

[0126] Based on any of the above embodiments Figure 2 This is the second flowchart illustrating the passive positioning and navigation method for complex cabins provided by this invention, as shown below. Figure 2 As shown, this method is applied to a passive positioning and navigation system within a complex cabin. The system uses augmented reality devices, such as AR glasses integrating visible light cameras, inertial measurement units, and depth cameras, as data acquisition sources. The multi-source heterogeneous data collected by these sensors is first fed into a multi-source fusion positioning module. This module works in conjunction with a positioning calibration module to generate a high-precision local pose estimate. Specifically, the multi-source fusion positioning module is primarily responsible for implementing visual-inertial odometry, while the positioning calibration module utilizes cabin text signage and visual tag positioning information obtained from the environment to suppress cumulative odometry drift and feeds back calibration information to the multi-source fusion positioning module. Subsequently, the high-precision local pose estimate is used in two parallel processes: on one hand, the local pose estimate is input to an anchor point matching module to determine the user's current position in a pre-defined topology map, which is then used by a path planning module to generate a navigation path; on the other hand, it directly provides accurate pose references to the AR glasses visualization module. Finally, the path planning result is combined with the high-precision pose estimate, and the navigation commands are presented to the user through the AR glasses visualization module. Meanwhile, the system also includes a closed-loop interactive interface that receives user feedback (such as gestures or voice commands) and passes this feedback to modules such as route planning and AR glasses visualization to adjust the navigation process, thus realizing a complete AR navigation system from positioning and planning to interaction.

[0127] Figure 3 This is an internal schematic diagram of the multi-source fusion positioning module provided by the present invention, as shown below. Figure 3As shown, the module mainly comprises three units: signal processing, data fusion, and position calculation. In the signal processing unit, the system processes two data streams in parallel: one stream pre-integrates the input IMU data to generate inertial constraints; the other stream processes the input visible light image frames to extract ORB features. In the data fusion unit, the system first determines the subsequent process based on the tracking status: if tracking is normal, the pose is predicted based on the motion model; if tracking is lost, the pose is recovered using DPoW2 relocalization technology; if it is the initial stage of system startup, a new map is initialized. The poses obtained from these branches undergo a geometric verification pose step and are used to project map points. Subsequently, feature matching associates the ORB features of the current frame with the projected map points, and the matching results are fed back to fuse the IMU constraints. In the position calculation unit, constraints fused with visual and inertial information are used for the final position calculation, and tight-coupling optimization is used to jointly optimize state variables such as pose, velocity, and zero bias, ultimately outputting the pose.

[0128] Figure 4 This is an internal schematic diagram of the path planning module provided by the present invention, as shown below. Figure 4 As shown, this illustrates two main operating modes from system startup to navigation execution. After system startup, the user can choose to enter the mapping mode to construct the anchor point topology map mentioned in the previous embodiments. In this mode, the user can preset anchor points and input information, storing the results in the database for later use. Alternatively, when navigation is needed, the user can choose to enter the navigation mode. In this mode, the user first confirms the starting and ending points, and the system then calculates the path based on the anchor point information in the database, finally providing navigation through AR visualization or other methods.

[0129] Figure 5 This is a flowchart illustrating the process of visual interaction based on augmented reality devices provided by the present invention, such as... Figure 5As shown, upon initiating the application, the system first checks if any existing anchor points are detected. If the result is negative, the system guides the user through a pop-up window to create the first anchor point and generate a data table, thus initiating the mapping process. If the result is positive, the user is taken to the main menu selection interface, where they can choose between mapping mode and navigation mode. In mapping mode, the user selects the entry anchor point through the panel to insert the new anchor point into the offline anchor point library. This process, along with the creation of the first anchor point, is merged into subsequent anchor point visualization and information panel display. Users can fine-tune anchor points through gesture dragging and information entry, ultimately updating the local database. In navigation mode, the user selects the starting and ending points through the panel to confirm the route. The system displays progress indicators and provides a "Ready" prompt through a pop-up window before starting navigation. During navigation, the system provides navigation effects and prompts through dynamic anchor point color updates and floating arrow guidance, achieving an immersive AR navigation experience.

[0130] The passive positioning and navigation system for complex cabins provided by the present invention is described below. The passive positioning and navigation system for complex cabins described below can be referred to in correspondence with the passive positioning and navigation method for complex cabins described above.

[0131] Based on any of the above embodiments, the present invention provides a passive positioning and navigation system for complex cabins. Figure 6 This is a structural schematic diagram of the passive positioning and navigation system for complex cabins provided by the present invention, as shown below. Figure 6 As shown, the system includes: The acquisition module 610 is used to acquire multi-source heterogeneous data collected by the multi-source sensors mounted on the augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images and depth images; The processing module 620 is used to perform pre-integration processing on the inertial measurement data to obtain the inertial pre-integration increment, and to extract features from the visible light image to obtain visual feature points. The determination module 630 is used to obtain depth information of the visual feature points based on the depth image, determine target visual feature points based on the depth information and the visual feature points, and perform tight coupling optimization based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate. The construction module 640 is used to extract the text features of the cabin text markings when cabin text markings are detected in the visible light image, and to construct inverse depth plane constraints based on the text features; and to extract the label corner pixel coordinates of the visual labels when visual labels are detected in the visible light image, and to construct absolute pose constraints based on the label corner pixel coordinates. The pose optimization module 650 is used to dynamically calibrate the initial pose estimate based on the inverse depth plane constraint and the absolute pose constraint to obtain the target pose estimate. The navigation guidance module 660 is used to perform anchor point matching in a pre-built anchor point topology map based on the target pose estimation to determine the current anchor point position, perform path planning based on the current anchor point position to generate a target path, generate navigation guidance information based on the target path, and visualize the navigation guidance information through the augmented reality device.

[0132] The system provided in this invention acquires inertial measurement data, visible light images, and depth images collected by multi-source sensors mounted on an augmented reality device. It processes the inertial measurement data and visible light images, and fuses depth information from visual feature points for visual-inertial tight coupling optimization to generate an initial pose estimate. When cabin text markings or visual labels are detected in the visible light image, their text features or label corner pixel coordinates are extracted to construct inverse depth plane constraints and absolute pose constraints, respectively. This allows the use of cabin text markings commonly found in complex cabins and on-demand deployable visual labels as passive, high-precision external calibration sources, providing effective geometric and absolute pose constraints for subsequent pose calibration and overcoming the lack of an effective calibration mechanism in existing pure visual-inertial positioning systems. Based on the inverse depth plane constraints and absolute pose constraints, the system dynamically calibrates the accumulated errors in the initial pose estimate to obtain a high-precision target pose estimate. Finally, navigation guidance information is generated based on the target pose estimate and visualized through the augmented reality device. This invention introduces cabin text markings and visual tags in the environment as an effective pose calibration mechanism. In a passive environment that does not rely on external signals such as GPS, it can dynamically correct the cumulative drift generated by the visual-inertial odometry. This solves the technical problems of large cumulative errors and weak repositioning capabilities caused by the lack of an effective calibration mechanism in existing technologies. As a result, it achieves high-precision and robust positioning and navigation in complex cabin environments with changes in lighting and missing textures, ensuring the stability and accuracy of navigation.

[0133] Based on the above embodiments, the navigation guidance module includes a user interaction unit, which includes an AR glasses visualization unit, an interactive interface, and a triggering unit; The AR glasses visualization unit is used to render the navigation guidance information into virtual visual elements and display the virtual visual elements on the display interface of the AR glasses to blend with the real-world scene; the virtual visual elements include directional arrows, navigation trajectory, cabin anchor frames, and arrival reminder effects; The interactive interface is used to generate visual interactive elements for navigation control; The triggering unit is used to trigger navigation control commands pre-associated with the visual interactive element in response to recognizing a user's gesture input or voice input to the visual interactive element.

[0134] Specifically, the navigation guidance module may include a user interaction unit, which includes an AR glasses visualization unit, an interactive interface, and a triggering unit.

[0135] Here, the AR glasses visualization unit renders navigation guidance information as virtual visual elements and displays these virtual visual elements on the AR glasses' display interface to blend them with the real-world scene. These virtual visual elements are a series of computer-generated two-dimensional or three-dimensional graphic objects used to assist user navigation. The virtual visual elements are designed to intuitively convey path, direction, and status information.

[0136] The virtual visual elements include directional arrows, navigation tracks, cabin anchor frames, and arrival alert effects. A directional arrow is a three-dimensional, dynamic arrow model that points in real-time in the direction the user needs to proceed or to the location of the next critical waypoint (anchor point). A navigation track is a virtual light strip or path line extending on the ground or in the air, visually depicting the complete path planned by the system, which the user can follow. A cabin anchor frame is a highlighted virtual frame superimposed on a specific anchor point in the real world (such as a door or a piece of equipment), used to clearly identify key nodes or destinations on the path. Arrival alert effects are special visual effects triggered when the user reaches an anchor point or final destination, such as a flashing halo, a pop-up congratulatory message, or a changing color, used to provide clear status feedback.

[0137] Here, the interactive interface is used to generate visual interactive elements for navigation control. These visual interactive elements refer to a set of virtual buttons, menus, or panels displayed in the user's field of vision that the user can interact with. For example, the interactive interface can generate a virtual menu containing options such as "Start Navigation," "Pause," and "Reschedule."

[0138] It should be noted that the virtual menu generated by the interactive interface allows users to interact with the system through gestures (such as air clicks) or voice input (such as saying "pause navigation"), thereby enabling control such as pausing, restarting, or changing navigation destinations.

[0139] Here, the triggering unit is used to respond to the recognition of a user's gesture input or voice input to the visual interactive element, and to trigger the navigation control command pre-associated with the visual interactive element.

[0140] Gesture input refers to the way users interact with the system through hand gestures. For example, the front-facing camera of AR glasses can recognize the user's "air click" gesture to select a virtual button. Voice input refers to the way users interact with the system by speaking specific keywords. For example, if a user says "start navigation," the system's microphone will pick up and recognize the voice command. The triggering process refers to the process where, once valid gesture or voice input is recognized, the triggering unit executes the preset function corresponding to the gesture or voice input, such as starting route planning or pausing navigation guidance.

[0141] The system provided in this invention significantly enhances the usability and user experience of the navigation system through its user interaction unit. The AR glasses visualization unit transforms abstract navigation data into intuitive graphics that seamlessly integrate with the real world, reducing the user's cognitive load. Meanwhile, the interactive interface and trigger unit provide a natural and convenient hands-free interaction method, enabling users to easily control the navigation process while walking, thus improving the convenience of navigation in complex cabins.

[0142] Based on the above embodiments, the system is also used for: Based on the target pose estimation, the first unreached anchor point in the anchor point sequence of the target path is determined as the immediate target anchor point; The yaw angle of the directional arrow is calculated in real time based on the first three-dimensional spatial coordinates of the user's head pose currently acquired in real time, and the second three-dimensional spatial coordinates of the instantaneous target anchor point. The AR glasses visualization unit is also used to adjust the orientation of the directional indicator arrow on the display interface in real time based on the yaw angle.

[0143] Specifically, in order to achieve precise and dynamic guidance from the directional arrows, the system can also be configured to perform the following steps: First, based on target pose estimation, the first unreached anchor point in the target path's anchor point sequence is identified as the immediate target anchor point. The immediate target anchor point refers to the user's next and most direct target point during navigation. In the target path comprised of a series of anchor points, the system continuously tracks the user's movement, and identifying the immediate target anchor point means finding the first anchor point on this target path that the user has not yet traversed.

[0144] Here, the yaw angle of the directional arrow is calculated in real time based on the first three-dimensional spatial coordinates of the user's head pose currently acquired in real time, and the second three-dimensional spatial coordinates of the instantaneous target anchor point.

[0145] The first three-dimensional spatial coordinates reflect the current position of the user's head (i.e., the AR glasses) in the global coordinate system. These first three-dimensional spatial coordinates can be obtained directly from the system's target pose estimation results. The second three-dimensional spatial coordinates reflect the position of the instantaneous target anchor point in the global coordinate system; these second three-dimensional spatial coordinates are pre-set when constructing the anchor point topology map.

[0146] Here, the yaw angle reflects a rotation angle in a horizontal direction, determining which direction the directional arrow should point in the user's horizontal field of vision. A direction vector is established pointing to the second three-dimensional spatial coordinate system (target position) with the first three-dimensional spatial coordinate system (user position) as the origin. Then, the angle between this direction vector and the user's current head orientation vector on the horizontal plane is calculated; this is the yaw angle.

[0147] Finally, the AR glasses visualization unit is also used to adjust the orientation of the directional indicator arrow on the display interface in real time based on the yaw angle. This step means that the AR glasses visualization unit will repeat the above yaw angle calculation process at a very high frequency (e.g., once per frame), and immediately update and redraw the rotation angle of the directional indicator arrow on the screen based on the latest calculated yaw angle value.

[0148] The system provided in this invention provides dynamic and adaptive guidance to users by calculating and updating the yaw angle of the directional arrow in real time. No matter how the user turns their head, the virtual arrow can always accurately point to the physical direction of the next target anchor point, thus providing extremely clear and unambiguous directional guidance.

[0149] Based on the above embodiments, the system is also used for: Determine the spatial distance between the user position of the target pose estimation and the position of the process anchor point located after the instantaneous target anchor point in the anchor point sequence; In response to the spatial distance being less than a preset arrival threshold, the next anchor point in the anchor point sequence is redefined as the immediate target anchor point.

[0150] Specifically, in order to achieve automatic updates of navigation paths and seamless process integration, the system is also configured to perform the following steps: First, determine the user position for target pose estimation and the spatial distance between it and the position of the process anchor point located after the immediate target anchor point in the anchor point sequence.

[0151] Here, a process anchor point refers to the target of the next stage in the navigation path. Specifically, a process anchor point is the anchor point immediately following the current immediate target anchor point in a complete anchor point sequence. Spatial distance refers to the straight-line physical distance between the user's current location and a future target point, which can be obtained by calculating the Euclidean norm of the three-dimensional coordinates of these two points.

[0152] Then, in response to the spatial distance being less than a preset threshold, the next anchor point in the anchor point sequence of the process anchor point is redefined as the immediate target anchor point.

[0153] Here, the preset arrival threshold refers to a small distance value (e.g., 1.5 meters). The preset arrival threshold defines a virtual arrival area centered on the process anchor point. When the system determines that the user is close enough to the next anchor point, it considers the user to be about to complete the current stage of navigation.

[0154] In other words, if the process has not yet reached anchor point A, but has already reached another anchor point B further down the anchor point sequence, the next anchor point will be updated to the next anchor point of anchor point B. For example, if the order of the anchor point sequence is A, B, C, D..., if the user reaches B earlier during navigation, then navigation will continue to C and D.

[0155] The system provided in this invention automates and seamlessly switches the navigation process. Users do not need to perform any manual operations. When they approach a waypoint, the system will automatically switch the navigation guidance to the next waypoint, ensuring the continuity and smoothness of the navigation experience and avoiding the situation where users need to stop and wait for the next instruction after reaching a turning point.

[0156] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a passive positioning and navigation method for complex cabin environments. This method includes: acquiring multi-source heterogeneous data collected by multi-source sensors mounted on an augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; performing pre-integration processing on the inertial measurement data to obtain an inertial pre-integration increment; extracting features from the visible light images to obtain visual feature points; acquiring depth information of the visual feature points based on the depth images; determining target visual feature points based on the depth information and the visual feature points; and tightly coupling the inertial pre-integration increment and the target visual feature points. The system optimizes and generates an initial pose estimate. If a cabin text label is detected in the visible light image, the text features of the cabin text label are extracted, and an inverse depth plane constraint is constructed based on the text features. If a visual label is detected in the visible light image, the pixel coordinates of the label corner points are extracted, and an absolute pose constraint is constructed based on the label corner point pixel coordinates. Based on the inverse depth plane constraint and the absolute pose constraint, the initial pose estimate is dynamically calibrated to obtain a target pose estimate. Based on the target pose estimate, navigation guidance information is generated, and the navigation guidance information is visualized using the augmented reality device.

[0157] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the passive positioning and navigation method for complex cabins provided by the above methods. The method includes: acquiring multi-source heterogeneous data collected by multi-source sensors mounted on an augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; performing pre-integration processing on the inertial measurement data to obtain inertial pre-integration increments; performing feature extraction on the visible light images to obtain visual feature points; acquiring depth information of the visual feature points based on the depth images; and determining depth information based on the depth information and the visual feature points. The target visual feature points are tightly coupled and optimized based on the inertial pre-integration increment to generate an initial pose estimate. If a cabin text label is detected in the visible light image, the text features of the cabin text label are extracted, and an inverse depth plane constraint is constructed based on the text features. If a visual label is detected in the visible light image, the label corner pixel coordinates of the visual label are extracted, and an absolute pose constraint is constructed based on the label corner pixel coordinates. The initial pose estimate is dynamically calibrated based on the inverse depth plane constraint and the absolute pose constraint to obtain a target pose estimate. Based on the target pose estimate, navigation guidance information is generated and visualized using the augmented reality device.

[0159] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a passive positioning and navigation method for complex cabins provided by the methods described above. This method includes: acquiring multi-source heterogeneous data collected by multi-source sensors mounted on an augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; performing pre-integration processing on the inertial measurement data to obtain an inertial pre-integration increment; extracting features from the visible light images to obtain visual feature points; acquiring depth information of the visual feature points based on the depth images; determining target visual feature points based on the depth information and the visual feature points; and based on the inertial... The pre-integral increment and the target visual feature points are tightly coupled and optimized to generate an initial pose estimate. If a cabin text label is detected in the visible light image, the text features of the cabin text label are extracted, and an inverse depth plane constraint is constructed based on the text features. If a visual label is detected in the visible light image, the label corner pixel coordinates of the visual label are extracted, and an absolute pose constraint is constructed based on the label corner pixel coordinates. Based on the inverse depth plane constraint and the absolute pose constraint, the initial pose estimate is dynamically calibrated to obtain a target pose estimate. Based on the target pose estimate, navigation guidance information is generated, and the navigation guidance information is visualized using the augmented reality device.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A passive positioning and navigation method for complex cabins, characterized in that, include: Acquire multi-source heterogeneous data collected by multi-source sensors mounted on an augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; The inertial measurement data is pre-integrated to obtain the inertial pre-integration increment, and the visible light image is feature extracted to obtain visual feature points. Depth information of the visual feature points is obtained based on the depth image. Target visual feature points are determined based on the depth information and the visual feature points. Tight coupling optimization is performed based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate. If a cabin text label is detected in the visible light image, the text features of the cabin text label are extracted, and an inverse depth plane constraint is constructed based on the text features. When a visual label is detected in the visible light image, the pixel coordinates of the label corner points are extracted, and an absolute pose constraint is constructed based on the pixel coordinates of the label corner points. Based on the inverse depth plane constraint and the absolute pose constraint, the initial pose estimate is dynamically calibrated to obtain the target pose estimate; Based on the target pose estimation, navigation guidance information is generated, and the navigation guidance information is visualized through the augmented reality device.

2. The passive positioning and navigation method for complex cabins according to claim 1, characterized in that, The step of pre-integrating the inertial measurement data to obtain the inertial pre-integration increment includes: The first visible light image and the second visible light image acquired continuously in the visible light image are determined, the acquisition time of the first visible light image is determined as the first time, and the acquisition time of the second visible light image is determined as the second time. Acquire the acceleration and angular velocity from the inertial measurement data between the first time point and the second time point; Based on the acceleration and the angular velocity, the position pre-integral, velocity pre-integral, and rotational pre-integral are calculated respectively, and the motion pre-integral is determined based on the position pre-integral, velocity pre-integral, and rotational pre-integral. An inertial constraint term is constructed based on the motion pre-integral quantity, and the inertial pre-integral increment is determined based on the residual information corresponding to the inertial constraint term.

3. The passive positioning and navigation method for complex cabins according to claim 1, characterized in that, The text features are multiple visual key points extracted within the boundary area of ​​the cabin text markings in the visible light image; The construction of inverse depth plane constraints based on the text features includes: Based on the pixel coordinates and inverse depth values ​​corresponding to the multiple visual key points, the three-dimensional spatial plane where the cabin text sign is located is fitted. The visible light image containing the cabin text markings acquired at the current moment is taken as the current frame, and the visible light image containing the cabin text markings acquired at a historical moment is taken as the reference frame. Determine the geometric error and photometric error corresponding to the visual key point; the geometric error is determined based on the geometric distance from the visual key point to the three-dimensional space plane; the photometric error is determined based on the photometric difference between the visual key point in the current frame and the matching point in the reference frame, wherein the matching point is the visual key point in the reference frame that matches the visual key point in the current frame; Based on the geometric and photometric errors corresponding to all the aforementioned visual keypoints, a point-to-surface distance error function is determined. The point-to-surface distance error function is used as the inverse depth plane constraint.

4. The passive positioning and navigation method for complex cabins according to any one of claims 1 to 3, characterized in that, The construction of absolute pose constraints based on the pixel coordinates of the label corner points includes: Obtain the spatial coordinates of the label corner points in the physical world coordinate system; the spatial coordinates of the label corner points are determined based on the actual physical size of the visual label; Based on a preset corner point index order, the pixel coordinates of the label corner points are matched with the spatial coordinates of the label corner points to construct a set of point pairs; Based on the set of point pairs and the intrinsic parameter matrix of the visible light camera, pose calculation is performed to obtain the relative pose transformation matrix of the visible light camera relative to the visual label; The relative pose transformation matrix is ​​converted into an absolute pose in the global coordinate system, and the absolute pose is used as the absolute pose constraint.

5. The passive positioning and navigation method for complex cabins according to any one of claims 1 to 3, characterized in that, The step of generating navigation guidance information based on the target pose estimation includes: Receive navigation requests that include preset start and destination anchor points; In the preset anchor point topology map, a main navigation path is generated based on the optimal path between the preset starting anchor point and the destination anchor point; An initial guidance path is generated based on the shortest path between the user's position estimated by the target pose and the preset starting anchor point. The navigation guidance information is generated based on the initial guidance path and the main navigation path.

6. The passive positioning and navigation method for complex cabins according to claim 5, characterized in that, The steps for constructing the anchor point topology map include: Receive the first instruction to create an anchor point at the current location; In response to the first instruction, the pose estimation result of the augmented reality device at the current moment is obtained, and the three-dimensional spatial coordinates in the pose estimation result are determined as the spatial coordinates of the anchor point, and the spatial coordinates of the anchor point are stored. Receive the semantic name associated with the anchor point, and associate and store the semantic name with the spatial coordinates of the anchor point; Receive a second instruction for establishing a topological connection relationship between at least two of the anchor points that have been created, and determine the topological connection relationship between at least two of the anchor points; Based on the spatial coordinates of all the anchor points, the associated semantic names of all the anchor points, and the topological connection relationships, the anchor point topology map is generated.

7. A passive positioning and navigation system for complex cabins, characterized in that, include: The acquisition module is used to acquire multi-source heterogeneous data collected by the multi-source sensors mounted on the augmented reality device; the multi-source heterogeneous data includes inertial measurement data, visible light images, and depth images; The processing module is used to perform pre-integration processing on the inertial measurement data to obtain the inertial pre-integration increment, and to extract visual feature points from the visible light image. The determination module is used to obtain depth information of the visual feature points based on the depth image, determine the target visual feature points based on the depth information and the visual feature points, and perform tight coupling optimization based on the inertial pre-integration increment and the target visual feature points to generate an initial pose estimate. A construction module is used to extract the text features of the cabin text markings when cabin text markings are detected in the visible light image, and to construct inverse depth plane constraints based on the text features; When a visual label is detected in the visible light image, the pixel coordinates of the label corner points are extracted, and an absolute pose constraint is constructed based on the pixel coordinates of the label corner points. The pose optimization module is used to dynamically calibrate the initial pose estimate based on the inverse depth plane constraint and the absolute pose constraint to obtain the target pose estimate. The navigation guidance module is used to perform anchor point matching in a pre-built anchor point topology map based on the target pose estimation to determine the current anchor point position, perform path planning based on the current anchor point position to generate a target path, generate navigation guidance information based on the target path, and visualize the navigation guidance information through the augmented reality device.

8. The passive positioning and navigation system for complex cabins according to claim 7, characterized in that, The navigation guidance module includes a user interaction unit, which includes an AR glasses visualization unit, an interactive interface, and a triggering unit. The AR glasses visualization unit is used to render the navigation guidance information into virtual visual elements and display the virtual visual elements on the display interface of the AR glasses to blend with the real-world scene; the virtual visual elements include directional arrows, navigation trajectory, cabin anchor frames, and arrival reminder effects; The interactive interface is used to generate visual interactive elements for navigation control; The triggering unit is used to trigger navigation control commands pre-associated with the visual interactive element in response to recognizing a user's gesture input or voice input to the visual interactive element.

9. The passive positioning and navigation system for complex cabins according to claim 8, characterized in that, The system is also used for: Based on the target pose estimation, the first unreached anchor point in the anchor point sequence of the target path is determined as the immediate target anchor point; The yaw angle of the directional arrow is calculated in real time based on the first three-dimensional spatial coordinates of the user's head pose currently acquired in real time, and the second three-dimensional spatial coordinates of the instantaneous target anchor point. The AR glasses visualization unit is also used to adjust the orientation of the directional indicator arrow on the display interface in real time based on the yaw angle.

10. The passive positioning and navigation system for complex cabins according to claim 9, characterized in that, The system is also used for: Determine the spatial distance between the user position of the target pose estimation and the position of the process anchor point located after the instantaneous target anchor point in the anchor point sequence; In response to the spatial distance being less than a preset arrival threshold, the next anchor point in the anchor point sequence is redefined as the immediate target anchor point.

Images