A space positioning method and system based on light-inertial fusion

Abstract

The present application belongs to the field of virtual reality and human-computer interaction technology, and particularly relates to a space positioning method and system based on light-inertia fusion, which can reduce the layout density of marker points, utilize neural networks to improve the recognition speed and accuracy of marker points, and ensure the positioning accuracy and anti-occlusion robustness of the system. The present application discards the high-density ArUco code layout mode, adopts discrete feature point plane layout combined with active infrared imaging, maintains the spatial uniformity and local uniqueness of feature distribution while greatly reducing the marker density, and realizes robust decoding by directly learning the spatial correlation between points through a BiLSTM-CRF neural network, without relying on complex artificial rules and environment feature triangulation, so that the recognition is more stable and the calculation is more efficient in the occlusion and imaging failure scene.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality and human-computer interaction technology, specifically relating to a spatial positioning method and system based on optical-inertial fusion. Background Technology

[0002] Optical-inertial fusion positioning is an important multi-sensor fusion technology. It achieves high-precision continuous estimation of device motion states by jointly modeling and optimizing optical information and inertial measurement unit (IMU) data. Optical information provides rich environmental structural features and global constraints, helping to suppress long-term accumulated errors; while inertial measurement data, with its high frequency and low latency, can provide smooth and continuous motion estimates in a short time, effectively compensating for the instability of visual information under conditions of rapid movement, changes in lighting, or partial occlusion. Through the close integration of the two, the optical-inertial fusion positioning method significantly improves the robustness and real-time performance of the system while maintaining positioning accuracy, showing promising application prospects in fields such as virtual reality, augmented reality, and autonomous driving.

[0003] One existing technology offers a high-precision indoor positioning and attitude determination method based on visual QR code and inertial fusion. This method uses a strapdown inertial navigation algorithm as its core, employing MEMS-IMU and a monocular camera to collect data. It obtains absolute pose correction information by recognizing pre-deployed ArUco codes with known coordinates indoors, and uses extended Kalman filtering to achieve tight coupling and fusion of visual and inertial information. To address issues such as QR code detection failure, image blurring, or camera occlusion, the method simultaneously triangulates and stores environmental feature points in a sliding window during QR code recognition. When the QR code is unavailable, environmental features maintain continuous filtering calculations. Furthermore, a reverse smoothing algorithm is combined to fully utilize multi-time observation information to further improve pose estimation accuracy and system robustness, ultimately achieving stable and reliable indoor positioning and attitude determination. Compared to traditional visual positioning or visual-inertial odometry, which relies on continuous clear images, is prone to pose drift, and has poor robustness, this method combines inertial autonomous calculation with visual absolute correction, enabling continuous and stable operation even when visual observation is interrupted. This significantly improves the reliability and environmental adaptability of the indoor positioning and attitude determination system.

[0004] Another approach is a non-coded reference marker detection and recognition framework for harsh and constrained environments. This method designs novel reference markers adapted to specific scenarios, employs a hybrid detection process to extract candidate marker regions, and utilizes a shallow feedforward neural network to achieve non-coded marker recognition and matching, significantly improving robustness under occlusion conditions. Compared to traditional geometric rule-based marker encoding and recognition methods, this scheme relies on neural networks for marker decoding, does not depend on complex manually designed rules, requires fewer parameter adjustments, is easy to deploy, and can stably adapt to different target structures and various marker layouts.

[0005] Optical-inertial fusion tracking and localization has significant application value in fields such as virtual reality and robotics. An early method for high-precision indoor positioning and orientation determination based on visual QR code fusion was proposed, which improved the system's stability under visual occlusion to some extent. However, this method still has certain drawbacks. Its visual localization part relies on deploying a large number of QR code reference markers in space, which not only increases environmental deployment costs but also significantly increases the computational load of feature detection and data fusion. Furthermore, this scheme struggles to balance tracking accuracy and occlusion robustness; accuracy decreases when markers are sparse, while the computational burden increases dramatically when markers are dense, failing to meet the practical requirements of lightweight and highly robust systems. Summary of the Invention

[0006] In view of this, the present invention provides a spatial positioning method and system based on optical-inertial fusion, which is capable of...

[0007] To achieve the objectives of this invention, the following technical solutions are provided.

[0008] A spatial positioning method based on optical-inertial fusion includes the following steps:

[0009] Step 1: Place marker points in the tracking space. The marker points are active infrared LEDs, and are discretely arranged in a regular grid to form a planar marker field. Use the floor or ceiling of the tracking space as the marker placement plane and divide it into a regular grid. Use a set of points with fixed relative positions to form basic graphic elements. Each basic graphic element is arranged across two adjacent grid units. Multiple basic graphic elements are combined according to preset rules to form a complete planar marker. All basic graphic elements adopt the same internal point arrangement. Step 2: Rigidly connect the infrared camera to the inertial measurement unit, and synchronously acquire infrared feature images and inertial measurement data. The marker detection is completed by the image front end. Step 3: Decode the detected landmarks using a BiLSTM-CRF neural network: Construct the input feature sequence using the relative coordinates between landmarks, extract contextual information through a three-layer bidirectional long short-term memory network, concatenate the forward and backward hidden states and apply Dropout regularization to obtain a sequence representation with fused context; then pass through a fully connected layer to obtain the emission score matrix, introduce a conditional random field layer for global optimal decoding, and select the sequence with the highest score as the globally unique number of each feature point; combine the known 3D spatial coordinates to complete the PnP pose calculation to obtain the visual pose; Step 4: Construct an optical-inertial tightly coupled filtering framework and use extended Kalman filtering to fuse visual observations with IMU data: using IMU strapdown inertial navigation calculation as the prediction model, position, attitude, velocity, and inertial sensor zero bias are incorporated into the unified estimation of the state vector; using the marker observations obtained by neural network decoding as the update quantity, when vision is effective, the marker observations are used to correct inertial navigation drift; when the marker is occluded, lost, or imaging fails, the system automatically switches to pure inertial calculation mode, using the IMU to maintain continuous pose output.

[0010] In step three, the path score is calculated for all possible label sequences during global optimal decoding. ,in Indicates position The emission score is used to characterize the matching relationship between observed features and labels; The transition score between adjacent labels is represented by the learnable transition matrix A. The model calculates the path score of all possible label sequences and selects the sequence with the highest score as the final recognition result.

[0011] In step one, active infrared LEDs and narrowband filter imaging are used to suppress background stray light interference and obtain a high signal-to-noise ratio feature image.

[0012] This invention also proposes a spatial positioning system based on optical-inertial fusion, comprising: The planar marking field is formed by active infrared LED markers arranged in a regular grid and set on the floor or ceiling of the tracking space. The regular grid divides the marking plane into grids, and the basic graphic elements are composed of a set of points with fixed relative positions. Each basic graphic element is arranged across two adjacent grid cells, and multiple basic graphic elements are combined to form a complete planar marking. All basic graphic elements adopt the same internal point arrangement. An infrared camera and an inertial measurement unit are rigidly connected to synchronously acquire infrared feature images and inertial measurement data. The image processing and decoding module is used to detect marker points in the acquired infrared feature images and decode the marker points using a BiLSTM-CRF neural network to obtain the globally unique number and corresponding image coordinates of each feature point. Combined with the known three-dimensional spatial coordinates, the PnP pose is calculated to obtain the visual pose. The optical-inertial tightly coupled filtering module uses extended Kalman filtering to fuse visual observations with IMU data. It uses IMU strapdown inertial navigation calculations as the prediction model and neural network decoded marker observations as the update variables. It incorporates position, attitude, velocity, and inertial sensor zero bias into the state vector for unified estimation. When vision is effective, marker observations are used to correct inertial navigation drift. When vision fails, it switches to pure inertial calculation mode to maintain continuous pose output.

[0013] The BiLSTM-CRF neural network uses a three-layer bidirectional long short-term memory network to extract contextual information, concatenates the forward and backward hidden states and performs Dropout regularization, then passes through a fully connected layer to obtain the emission score matrix, introduces a conditional random field layer for global optimal decoding, and constructs the input feature sequence using the relative coordinates between the marker points.

[0014] The active infrared LED, in conjunction with a narrowband filter, is used to acquire high signal-to-noise ratio feature images under low illumination or complex lighting conditions.

[0015] Beneficial effects 1. This invention abandons the high-density ArUco code deployment method and adopts a discrete feature point planar layout + active infrared imaging. This significantly reduces marker density while maintaining the spatial uniformity and local uniqueness of feature distribution. Furthermore, it achieves robust decoding by directly learning the spatial correlation between points through a BiLSTM-CRF neural network, eliminating the need for complex manual rules and environmental feature triangulation. This results in more stable recognition and higher computational efficiency even in occlusion and imaging failure scenarios. Through optimized marker design and recognition strategies, the density of marker placement is reduced, and neural networks are used to improve the speed and accuracy of marker recognition. Simultaneously, visual marker localization and inertial measurement are integrated to construct a tightly coupled optical-inertial positioning system, ensuring system positioning accuracy and robustness against occlusion.

[0016] 2. This invention discloses a spatial positioning system combining optical and inertial fusion with neural networks. Infrared LED markers are arranged on a marked plane within the tracking space, and a neural network is used for identification and decoding of the known marker layout. Simultaneously, this method uses extended Kalman filtering for tight information coupling, enabling continuous and reliable tracking using an IMU even when visual images are lost or blurred. An end-to-end optical-inertial tightly coupled system architecture is constructed, deeply fusing the robust recognition results of the neural network with inertial measurement to uniformly estimate position, attitude, velocity, and sensor bias. This improves system positioning accuracy, tracking continuity, and environmental adaptability while maintaining lightweight deployment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an embodiment of the system of the present invention.

[0018] Figure 2 This is a schematic diagram of the planar marking layout in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0020] This invention proposes a spatial positioning method based on optical-inertial fusion: First, the floor or ceiling is selected as the marking plane in the tracking space, and it is divided into a regular grid. Basic primitives are constructed from a set of points with fixed relative positions. Each basic primitive spans two adjacent grid cells. Multiple basic primitives are combined according to preset rules to form a complete planar marker. All basic primitives use the same internal point arrangement, eliminating the need for differentiated coding structures. The marker points use active infrared LEDs, utilizing their active emission and narrowband filtering imaging to suppress background stray light interference and acquire high signal-to-noise ratio feature images. The infrared camera is rigidly connected to the IMU, simultaneously acquiring infrared feature images and inertial measurement data.

[0021] After the image front-end completes the marker detection, a BiLSTM-CRF neural network is used to decode the markers: the input feature sequence is constructed using the relative coordinates between markers, context information is extracted through a three-layer bidirectional long short-term memory network, the forward and backward hidden states are concatenated and regularized by Dropout to obtain a sequence representation with fused context; then, the emission score matrix is ​​obtained through a fully connected layer, and a conditional random field layer is introduced for global optimal decoding, that is, the path score is calculated for all possible label sequences. ,in Indicates position The emission score is used to characterize the matching relationship between observed features and labels; The transition score between adjacent labels is represented by the learnable transition matrix A. The model calculates the path score of all possible label sequences and selects the sequence with the highest score as the final recognition result. The sequence with the highest score is selected as the globally unique number of each feature point, and combined with the known 3D spatial coordinates, the PnP pose is calculated to obtain the visual pose.

[0022] Then, a tightly coupled optical-inertial filtering framework is constructed, and an extended Kalman filter is used to fuse visual observations with IMU data: the IMU strapdown inertial navigation calculation is used as the prediction model, and the position, attitude, velocity, and inertial sensor zero bias are incorporated into the unified estimation of the state vector; the marker observations obtained by neural network decoding are used as the update quantity, and when vision is effective, high-precision marker observations are used to correct inertial navigation drift; when the marker is occluded, lost, or imaging fails, the system automatically switches to pure inertial calculation mode, and uses the IMU to maintain continuous pose output, thereby achieving continuous, stable, and high-precision positioning and attitude determination.

[0023] The system of the present invention is as follows Figure 1 As shown, the system uses active infrared LED markers as a spatial reference, forming a planar marker field with a discrete layout according to a regular grid. The camera and IMU simultaneously acquire infrared feature images and inertial data. Marker detection is first performed by the image front-end, and then processed by BiLSTM... The CRF neural network models and globally optimizes the relative position sequence of marker points, obtaining a unique number and corresponding coordinates for each feature point. Then, visual observations and IMU data are fed into a tightly coupled optical-inertial filtering framework. Using inertial navigation calculations for prediction and marker point observations for correction, position, attitude, velocity, and sensor bias are uniformly estimated, achieving continuous, stable, and high-precision positioning and attitude determination output. The system adopts a planar marker layout design, specifically a planar marker layout based on a regular grid: First, the tracking space is determined, selecting the floor or ceiling as the marker placement plane and dividing it into a regular grid; basic primitives are constructed using a set of points with fixed relative positions, each basic primitive spanning two adjacent grid cells. Multiple basic primitives are combined according to preset rules to form a complete planar marker, and all basic primitives use the same internal point arrangement, eliminating the need for differentiated encoding structures. The planar marker layout in this embodiment is as follows: Figure 2 As shown.

[0024] Compared to traditional contour-based and coded block-based markers, the planar marker designed in this invention possesses stronger anti-occlusion capabilities and environmental adaptability. This marker is composed of discrete feature points; in the event of partial occlusion, only a sufficient number of valid primitives need to be retained to complete pose calculation. Furthermore, the missing point number can be inferred through geometric constraints, ensuring tracking continuity. Simultaneously, combined with an active infrared imaging scheme, employing active infrared LED emission and narrowband filter imaging, it effectively suppresses background stray light interference, acquires high signal-to-noise ratio feature images, simplifies the feature extraction process, reduces computational load, and enables the system to operate stably under low-light and complex lighting conditions.

[0025] Neural networks, with their powerful nonlinear feature modeling and parallel computing capabilities, significantly improve the efficiency and robustness of marker decoding. The DeepTag method, based on convolutional neural networks, achieves end-to-end marker detection and encoding recognition, simplifying algorithm design and improving deployment flexibility. Ribeiro et al. further employed shallow neural networks to model the relative geometric relationships of markers, effectively utilizing the spatial structure of the point set to achieve stable recognition in occluded scenarios. Based on this, and according to analysis, the system of this invention is based on BiLSTM. The CRF neural network decoding architecture models marker recognition as a sequence classification task. Input features are constructed using the relative coordinates between markers. The network employs a three-layer BiLSTM to extract contextual information, concatenates the forward and backward hidden states, and applies Dropout regularization to obtain a sequence representation with fused context. The emission score matrix is ​​then obtained through a fully connected layer, and subsequently introduced into the CRF layer for globally optimal decoding.

[0026] Given an input sequence and a predicted label, the path score is calculated as follows:

[0027] By introducing the CRF mechanism, the model can use label transition relationships to constrain and optimize the preliminary prediction results during the decoding stage, thereby improving the robustness and overall consistency of sequence prediction.

[0028] At the hardware level, the system uses active infrared LEDs to form discrete feature points, arranged on the floor or ceiling according to the aforementioned planar layout rules, creating a spatially uniform and locally unique reference field. The camera and IMU are rigidly connected, simultaneously acquiring infrared feature images and inertial measurement data to construct a visual-inertial co-source observation input. For detection and decoding, the image front-end first performs marker detection, then the aforementioned BiLSTM-CRF neural network decodes to obtain the globally unique number and corresponding image coordinates of each feature point. Combined with the known three-dimensional spatial coordinates, PnP pose calculation is performed to obtain a high-precision visual pose.

[0029] At the fusion level, a tightly coupled visual and inertial filtering framework is constructed. The IMU strapdown inertial navigation solution is used as the prediction model, and the landmark observations after neural network decoding are used as the update quantity. Position, attitude, velocity, and inertial sensor zero bias are incorporated into the state vector for unified estimation. When vision is effective, high-precision landmark observations are used to correct inertial navigation drift. When the landmark is occluded, lost, or the imaging fails, the system automatically switches to pure inertial inference mode to maintain continuous and uninterrupted pose output.

[0030] In summary, this invention is based on discrete feature point planar layout design and BiLSTM-CRF neural network decoding. It reduces the density of marker points through regular grids and unified primitive structures, and utilizes neural network modeling of relative position sequence features and globally optimal label decoding to achieve high-precision and robust marker point recognition. It constructs an overall architecture of optical-inertial tightly coupled positioning system, which tightly couples and fuses the marker point observations obtained by neural network decoding with IMU strapdown inertial navigation to uniformly estimate pose, velocity, and sensor zero bias. It also seamlessly switches to inertial calculation when vision fails, achieving continuous and stable high-precision positioning and attitude determination.

[0031] This invention includes, but is not limited to, the above embodiments. Any equivalent substitutions or partial improvements made under the spirit and principles of this invention shall be considered within the scope of protection of this invention.

Claims

1. A spatial positioning method based on optical-inertial fusion, characterized in that, Includes the following steps: Step 1: Place marker points in the tracking space. The marker points are active infrared LEDs, and are discretely arranged in a regular grid to form a planar marker field. Use the floor or ceiling of the tracking space as the marker placement plane and divide it into a regular grid. Use a set of points with fixed relative positions to form basic graphic elements. Each basic graphic element is arranged across two adjacent grid units. Multiple basic graphic elements are combined according to preset rules to form a complete planar marker. All basic graphic elements adopt the same internal point arrangement. Step 2: Rigidly connect the infrared camera to the inertial measurement unit, and synchronously acquire infrared feature images and inertial measurement data. The marker detection is completed by the image front end. Step 3: Decode the detected landmarks using a BiLSTM-CRF neural network: Construct the input feature sequence using the relative coordinates between landmarks, extract contextual information through a three-layer bidirectional long short-term memory network, concatenate the forward and backward hidden states and apply Dropout regularization to obtain a sequence representation with fused context; then pass through a fully connected layer to obtain the emission score matrix, introduce a conditional random field layer for global optimal decoding, and select the sequence with the highest score as the globally unique number of each feature point; combine the known 3D spatial coordinates to complete the PnP pose calculation to obtain the visual pose; Step 4: Construct an optical-inertial tightly coupled filtering framework and use extended Kalman filtering to fuse visual observations with IMU data: using IMU strapdown inertial navigation calculation as the prediction model, position, attitude, velocity, and inertial sensor zero bias are incorporated into the unified estimation of the state vector; using the marker observations obtained by neural network decoding as the update quantity, when vision is effective, the marker observations are used to correct inertial navigation drift; when the marker is occluded, lost, or imaging fails, the system automatically switches to pure inertial calculation mode, using the IMU to maintain continuous pose output.

2. The spatial positioning method based on optical-inertial fusion according to claim 1, characterized in that, In step three, during global optimal decoding, path scores are calculated for all possible label sequences. ,in Indicates position The emission score is used to characterize the matching relationship between observed features and labels; The transition score between adjacent labels is represented by the learnable transition matrix A; The model calculates the path score of all possible label sequences and selects the sequence with the highest score as the final recognition result.

3. The spatial positioning method based on optical-inertial fusion according to claim 1, characterized in that, In step one, active infrared LEDs and narrowband filter imaging are used to suppress background stray light interference and obtain a high signal-to-noise ratio feature image.

4. A spatial positioning system based on optical-inertial fusion, characterized in that, include: The planar marking field is formed by active infrared LED markers arranged in a regular grid and set on the floor or ceiling of the tracking space. The regular grid divides the marking plane into grids, and the basic graphic elements are composed of a set of points with fixed relative positions. Each basic graphic element is arranged across two adjacent grid cells, and multiple basic graphic elements are combined to form a complete planar marking. All basic graphic elements adopt the same internal point arrangement. An infrared camera and an inertial measurement unit are rigidly connected to synchronously acquire infrared feature images and inertial measurement data. The image processing and decoding module is used to detect marker points in the acquired infrared feature images and decode the marker points using a BiLSTM-CRF neural network to obtain the globally unique number and corresponding image coordinates of each feature point. Combined with the known three-dimensional spatial coordinates, the PnP pose is calculated to obtain the visual pose. The optical-inertial tightly coupled filtering module uses extended Kalman filtering to fuse visual observations with IMU data. It uses IMU strapdown inertial navigation calculations as the prediction model and neural network decoded marker observations as the update variables. It incorporates position, attitude, velocity, and inertial sensor zero bias into the state vector for unified estimation. When vision is effective, marker observations are used to correct inertial navigation drift. When vision fails, it switches to pure inertial calculation mode to maintain continuous pose output.

5. A spatial positioning system based on optical-inertial fusion according to claim 4, characterized in that, The BiLSTM-CRF neural network uses a three-layer bidirectional long short-term memory network to extract contextual information, concatenates the forward and backward hidden states and performs Dropout regularization, then passes through a fully connected layer to obtain the emission score matrix, introduces a conditional random field layer for global optimal decoding, and constructs the input feature sequence using the relative coordinates between the marker points.

6. A spatial positioning system based on optical-inertial fusion according to claim 4, characterized in that, The active infrared LED, in conjunction with a narrowband filter, is used to acquire high signal-to-noise ratio feature images under low illumination or complex lighting conditions.

Images