SLAM repositioning method and device based on feature point homogenization and multi-source information fusion and application
The SLAM relocalization method, which integrates feature point homogenization and multi-source information fusion, solves the instability problem of VR relocalization in large-space environments, and achieves stable relocalization with high accuracy and low computational load, which is suitable for cordless VR devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-27
- Publication Date
- 2026-04-07
AI Technical Summary
Existing VR relocalization technologies face problems such as uneven distribution of key points, low matching accuracy, and the impact of mismatched points on pose estimation in large spatial environments, leading to relocalization failure or instability.
The SLAM relocalization method, which employs feature point homogenization and multi-source information fusion, distributes feature points evenly through a quadtree data structure. It combines BEBLID descriptors and multi-source information fusion, and uses 3D point cloud and planar map for constraint optimization to output a stable camera pose.
It improves pose estimation accuracy, reduces computational load, enhances the robustness and stability of relocation, meets high real-time requirements, and is suitable for large-space VR experiences.
Smart Images
Figure CN121810797A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital data processing, and in particular to a SLAM relocalization method, device and application based on feature point homogenization and multi-source information fusion. Background Technology
[0002] With the development of virtual reality technology, large-space, cordless VR experiences have become an important trend. In such applications, VR headsets need to be able to achieve accurate and stable self-positioning in huge physical spaces, such as an entire room or exhibition hall, i.e., repositioning. This is crucial to ensuring an immersive user experience and avoiding motion sickness.
[0003] Current mainstream VR relocalization technologies are based on visual SLAM or visual odometry. A typical process includes extracting key points from an image, performing feature matching, and estimating camera pose by solving the Perspective-n-Point (PnP) problem. The mathematical essence of the PnP problem is to find a rotation matrix R and a translation vector t such that a set of 3D spatial points... Its 2D projection point on the image The following relationship must be satisfied:
[0004]
[0005] in, It is the camera intrinsic parameter matrix. It is a non-zero scaling factor.
[0006] However, in large-space application scenarios, existing technologies face many challenges, mainly including:
[0007] (1) Uneven distribution of key points; Traditional feature extractors, such as FAST, tend to concentrate the extracted key points in high-texture areas, resulting in few matching point pairs in low-texture areas and easy failure of relocalization.
[0008] (2) Although ORB features are fast to compute, their discrimination (matching accuracy) is relatively low, and they are prone to mismatches in large scenes. Meanwhile, features with high accuracy such as SIFT take too long to compute and cannot meet the high real-time requirements of VR. It is difficult to balance feature matching efficiency and accuracy.
[0009] (3) Due to the presence of more occlusion, light changes and dynamic object interference in large space environments, PnP solutions are easily affected by mismatched points (outside points), resulting in large jitter or even complete errors in pose estimation. When facing planar structures or weak texture regions, the scale uncertainty of the traditional monocular PnP solution is also a prominent problem, and the pose solution has poor robustness.
[0010] Therefore, there is an urgent need in this field for a relocation solution that can achieve fast, accurate and highly robust relocation in large-space environments. Summary of the Invention
[0011] This invention solves the problems existing in the prior art and provides a SLAM relocalization method, device and application based on feature point homogenization and multi-source information fusion.
[0012] The technical solution adopted in this invention is a SLAM relocalization method based on feature point homogenization and multi-source information fusion. The method acquires one or more frames of images captured by a multi-view camera, extracts key points, performs feature point homogenization processing, and extracts descriptors. Based on the descriptors and a pre-constructed SLAM map, multi-source information is fused, and the relocalized camera pose is output under constraints.
[0013] Preferably, the feature point homogenization process includes the following steps:
[0014] S1.1 Construct a quadtree data structure for spatial partitioning, where the root node corresponds to the entire image plane, and each internal node is divided into four non-overlapping sub-regions. This partitioning process is performed recursively until the area of the region corresponding to any sub-node is less than or equal to a preset area threshold.
[0015] S1.2 Within the image region corresponding to each child node, sort all the initially extracted key points according to their response intensity, retain the top few key points with the highest intensity, and discard the rest; the response intensity is the FAST corner score;
[0016] S1.3 Collect all key points retained from child nodes to form a set of feature points with uniform spatial distribution.
[0017] Preferably, the descriptor is a BEBLID descriptor, and the output is a binary string.
[0018] Preferably, multi-source information fusion includes the following steps:
[0019] S2.1 Based on the pre-built map, feature matching is performed with the descriptor of the current frame;
[0020] S2.2 Construct a nonlinear least squares optimization problem to solve for the rotation matrix R and translation vector t of the current camera pose;
[0021] S2.3 Based on feature matching results, pre-built maps and constraints, multi-source information is fused and pose is solved.
[0022] Preferably, in S2.1, the pre-built map is a SLAM map with 3D point cloud and planar structure, and a correspondence is established between 2D image points and 3D map points.
[0023] Preferably, in S2.3, the constraints are associated with point constraints, planar constraints, and multi-objective constraints;
[0024] Point constraints based on reprojection error ,
[0025]
[0026] in, For the i-th 3D map point, This corresponds to the i-th 2D image point;
[0027] Based on the current frame's camera pose, determine that points on the 3D plane should be projected onto their corresponding 2D contours in the image, and calculate the planar constraint error. ,
[0028]
[0029] Where n is the unit normal vector and d is the bias;
[0030] For multi-view constraints, calculate the reprojection error of each 3D point on sub-camera c.
[0031]
[0032] in, , and For the c-th camera, the fixed extrinsic parameters are... For point 3D coordinates in the coordinate system of sub-camera c For point The actual 2D pixel coordinates observed on the image plane of sub-camera c.
[0033] Preferably, the nonlinear least squares optimization problem is minimized.
[0034]
[0035] in, and It is a hyperparameter that balances the weights of various constraints. This is an adjustable parameter.
[0036] Preferably, the method includes a feedback optimization step:
[0037] Based on plane constraint error If the same 3D map point In a series of frames If all values are greater than the threshold, it is determined that the point may be located on a dynamic object or an incorrect planar association. In the feature matching and optimization of subsequent frames, the feature matching weight corresponding to the point is reduced.
[0038] And / or,
[0039] Reprojection error based on multi-view constraints If the same map point If the difference in reprojection error under different camera views exceeds the preset range, the point is marked as a dynamic suspicious point, and its corresponding constraint weights are reduced in the optimization.
[0040] A SLAM relocalization device based on feature point homogenization and multi-source information fusion, the device comprising a processor and a memory:
[0041] The memory is used to store program code and transmit the program code to the processor;
[0042] The processor is used to execute the SLAM relocalization method based on feature point homogenization and multi-source information fusion according to the instructions in the program code.
[0043] An application of the SLAM relocalization method based on feature point homogenization and multi-source information fusion is shown in a VR headset for cordless VR devices.
[0044] This invention relates to a SLAM relocalization method, device, and application based on feature point homogenization and multi-source information fusion. The method involves acquiring one or more frames of images captured by a multi-view camera, extracting key points, performing feature point homogenization, and extracting descriptors. Based on the descriptors and a pre-constructed SLAM map, multi-source information is fused to output the relocalized camera pose under constraints. The method is implemented in a device, and applied to a VR headset in a cordless VR device.
[0045] The beneficial effects of this invention are as follows:
[0046] (1) The uniform distribution of feature points provides a better geometric basis for pose solving. Combined with the enhanced optimization model that integrates planar and multi-view constraints, the uncertainty of the solution is significantly reduced and the accuracy of pose estimation is improved.
[0047] (2) The homogenization feature reduces the risk of matching failure due to local texture loss. The multi-source information fusion model effectively suppresses mismatch and noise interference through multiple geometric constraints. Even when the matching quality is average, it can output a stable pose, greatly reducing tracking jitter and loss, and has strong robustness.
[0048] (3) By filtering through quadtrees, the number of redundant feature points is reduced from the source, and the efficient BEBLID binary descriptor is used, which greatly reduces the overall computational load of feature processing and matching, and can meet the stringent requirements of high real-time applications such as VR (≥60Hz).
[0049] (4) By closely combining the features of rich planar structures in large-space VR scenes with the characteristics of multi-view camera hardware, it effectively solves a series of problems such as feature distribution, matching speed and solution stability in large-scale environments. The technology is highly practical and can provide reliable technical support for offline large-space VR experiences, museum tours and other applications. Attached Figure Description
[0050] Figure 1 This is a flowchart of the method of the present invention;
[0051] Figure 2 This is a schematic diagram illustrating the implementation process of the method of the present invention. Detailed Implementation
[0052] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.
[0053] This invention relates to a SLAM relocalization method based on feature point homogenization and multi-source information fusion, the method comprising the following steps:
[0054] (1) Acquire one or more frames of images captured by a multi-view camera, extract key points, perform feature point homogenization processing, and extract descriptors;
[0055] (2) Multi-source information fusion is performed based on descriptors and pre-built SLAM maps, and the repositioned camera pose is output under constraints.
[0056] The steps are explained below with reference to specific implementation methods.
[0057] (1) Acquire one or more frames of images captured by a multi-view camera, extract key points, perform feature point homogenization processing, and extract descriptors;
[0058] In this invention, a feature detector (FAST) is generally used to initially extract key points.
[0059] Subsequently, the feature point homogenization process includes the following steps:
[0060] (1-1) Construct a quadtree data structure for spatial partitioning, where the root node corresponds to the entire image plane, and each internal node is divided into four non-overlapping sub-regions. This partitioning process is performed recursively until the area of the region corresponding to any sub-node is less than or equal to a preset area threshold.
[0061] (1-2) Within the image region corresponding to each child node, sort all the initially extracted key points according to their response intensity, retain the first few key points with the highest intensity, generally 1 or 2, and discard the rest; the response intensity is the FAST corner score;
[0062] (1-3) Collect all the key points retained by the child nodes to form a set of feature points with uniform spatial distribution.
[0063] In this invention, the feature point homogenization process ensures that the feature points cover the entire field of view, thus avoiding aggregation.
[0064] The descriptor is a BEBLID descriptor, and the output is a binary string.
[0065] In this invention, for each uniformly distributed key point selected, the BEBLID descriptor algorithm is used to calculate its feature descriptor vector. It generates a binary string by comparing a set of predefined intensity value integration regions within the image block. Each comparison result generates a binary bit (0 or 1), and finally forms a compact binary string as the descriptor of the feature point. This binary descriptor is not only fast to calculate, but also can use the processor's low-level instructions to quickly calculate the Hamming distance when performing feature matching, which greatly improves the matching efficiency.
[0066] (2) Multi-source information fusion is performed based on descriptors and pre-built SLAM maps, and the repositioned camera pose is output under constraints.
[0067] Multi-source information fusion includes the following steps:
[0068] (2-1) Based on the pre-built map, perform feature matching with the descriptor of the current frame;
[0069] The pre-built map is a SLAM map with 3D point cloud and planar structure, establishing the correspondence between 2D image points and 3D map points.
[0070] Specifically, the BEBLID descriptor of the current frame is matched with pre-built map points (with 3D locations and descriptors), generally using the BOW bag-of-words model to establish the correspondence between 2D image points and 3D map points.
[0071] (2-2) Construct a nonlinear least squares optimization problem, usually based on graph optimization or bundle adjustment principle, to solve the rotation matrix R and translation vector t of the current camera pose;
[0072] (2-3) Based on feature matching results, pre-built maps and constraints, multi-source information is fused and pose is solved.
[0073] Multi-source information is closely related to constraints; these constraints are associated with point constraints, planar constraints, and multi-view constraints.
[0074] Point constraints refer to using the traditional reprojection error—the difference between the position of the matched 3D map point projected onto the current image plane and the actual position of the 2D feature point—as part of the optimization objective. This is based on the reprojection error. ,
[0075]
[0076] in, For the i-th 3D map point, This corresponds to the i-th 2D image point;
[0077] Plane constraints refer to identifying the main planes in a scene, typically through pre-scanning mapping or real-time detection. The 2D contour of the detected plane in the image is then correlated with the corresponding 3D plane in the map. The geometric constraint that "points on the 3D plane should be projected into the corresponding 2D contour in the image under the current camera pose" is added as a penalty term to the optimization objective. Let a plane in the map be represented by a unit normal vector n and a bias d, i.e. +d=0, then for the observed map points Obtaining error That is, based on the camera pose of the current frame, determine that a point on the 3D plane should be projected into the corresponding 2D contour in the image, and calculate the planar constraint error. ,
[0078]
[0079] Where n is the unit normal vector and d is the bias;
[0080] For multi-camera systems, the same 3D map point can be observed simultaneously by multiple cameras (such as the left and right cameras). During optimization, the reprojection error of this 3D point across all camera views is considered, which naturally provides scale information and more observation constraints. For multi-camera constraints, the reprojection error of each 3D point on sub-camera c is calculated.
[0081]
[0082] in, , and For the c-th camera, the fixed extrinsic parameters are... For point 3D coordinates in the coordinate system of sub-camera c For point The actual 2D pixel coordinates observed on the image plane of sub-camera c.
[0083] Minimize the nonlinear least squares optimization problem.
[0084]
[0085] in, and It is a hyperparameter that balances the weights of various constraints. This is an adjustable parameter.
[0086] In this invention, the Levenberg-Marquardt nonlinear optimization algorithm is used to solve the comprehensive optimization objective. The comprehensive optimization objective, which integrates multi-point, planar, and multi-view information, finally outputs the optimal 6-DOF camera pose with strong noise resistance.
[0087] To further improve the method, a feedback optimization step is also included:
[0088] Based on plane constraint error If the same 3D map point In a series of frames If all values are greater than the threshold, it is determined that the point may be located on a dynamic object or an incorrect planar association. In the feature matching and optimization of subsequent frames, the feature matching weight corresponding to the point is reduced.
[0089] And / or,
[0090] Reprojection error based on multi-view constraints If the same map point If the difference in reprojection error under different camera views exceeds the preset range, the point is marked as a dynamic suspicious point, and its corresponding constraint weights are reduced in the optimization.
[0091] This invention proposes a "closed-loop feedback" optimization framework, introducing a "feedback" mechanism that guides the front-end and data association from the optimization results, forming a closed-loop system; specifically including:
[0092] Firstly, planar constraints guide feature selection:
[0093] During the optimization iteration, the planar constraint residual of each feature point is calculated in real time. For points with persistently large residuals, they are likely to be points of dynamic objects that are incorrectly associated with the plane. Therefore, low-confidence feedback is generated and fed back to the front end to reduce the weight of such points in the next frame or the next optimization, or to guide the extraction of more features in a more reliable region.
[0094] Secondly, dynamic points for multi-view consistency verification:
[0095] By utilizing the geometric constraints of a multi-view system, dynamic objects can be detected more robustly. If the reprojection error of the same 3D point differs greatly between the main camera and the sub-camera, or if its multi-view triangulation position contradicts the position of existing map points, it can be marked as a "suspicious dynamic point" and its point constraint weight can be significantly reduced in subsequent optimizations, or even a dedicated dynamic object processing thread can be triggered.
[0096] By setting up a feedback mechanism, the method of this invention is adaptive and has a clear synergistic enhancement effect.
[0097] The present invention also relates to a SLAM relocalization device based on feature point homogenization and multi-source information fusion, the device comprising a processor and a memory:
[0098] The memory is used to store program code and transmit the program code to the processor;
[0099] The processor is used to execute the SLAM relocalization method based on feature point homogenization and multi-source information fusion according to the instructions in the program code.
[0100] This invention also relates to the application of the aforementioned SLAM relocalization method based on feature point homogenization and multi-source information fusion, in a VR headset for cordless VR devices.
[0101] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0102] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0105] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0106] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A SLAM relocalization method based on feature point homogenization and multi-source information fusion, characterized in that: The method acquires one or more frames of images captured by a multi-view camera, extracts key points, performs feature point homogenization processing, and extracts descriptors; based on the descriptors and a pre-built SLAM map, it performs multi-source information fusion and outputs the repositioned camera pose under constraints.
2. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 1, characterized in that: The feature point homogenization process includes the following steps: S1.1 Construct a quadtree data structure for spatial partitioning, where the root node corresponds to the entire image plane, and each internal node is divided into four non-overlapping sub-regions. This partitioning process is performed recursively until the area of the region corresponding to any sub-node is less than or equal to a preset area threshold. S1.2 Within the image region corresponding to each child node, sort all the initially extracted key points according to their response intensity, retain the top few key points with the highest intensity, and discard the rest; the response intensity is the FAST corner score; S1.3 Collect all key points retained from child nodes to form a set of feature points with uniform spatial distribution.
3. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 2, characterized in that: The descriptor is a BEBLID descriptor, and the output is a binary string.
4. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 1, characterized in that: Multi-source information fusion includes the following steps: S2.1 Based on the pre-built map, feature matching is performed with the descriptor of the current frame; S2.2 Construct a nonlinear least squares optimization problem to solve for the rotation matrix R and translation vector t of the current camera pose; S2.3 Based on feature matching results, pre-built maps and constraints, multi-source information is fused and pose is solved.
5. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 4, characterized in that: In S2.1, the pre-built map is a SLAM map with 3D point cloud and planar structure, and the correspondence between 2D image points and 3D map points is established.
6. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 5, characterized in that: In S2.3, the constraints are associated with point constraints, planar constraints, and multi-objective constraints; Point constraints based on reprojection error , ， in, For the i-th 3D map point, This corresponds to the i-th 2D image point; Based on the current frame's camera pose, determine that points on the 3D plane should be projected onto their corresponding 2D contours in the image, and calculate the planar constraint error. , ， Where n is the unit normal vector and d is the bias; For multi-view constraints, calculate the reprojection error of each 3D point on sub-camera c. ， in, , and For the c-th camera, the fixed extrinsic parameters are... For point 3D coordinates in the coordinate system of sub-camera c For point The actual 2D pixel coordinates observed on the image plane of sub-camera c.
7. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 6, characterized in that: Minimize the nonlinear least squares optimization problem. ， in, and It is a hyperparameter that balances the weights of various constraints. This is an adjustable parameter.
8. The SLAM relocalization method based on feature point homogenization and multi-source information fusion according to claim 6, characterized in that: The method includes a feedback optimization step: Based on plane constraint error If the same 3D map point In a series of frames If all values are greater than the threshold, it is determined that the point may be located on a dynamic object or an incorrect planar association. In the feature matching and optimization of subsequent frames, the feature matching weight corresponding to the point is reduced. And / or, Reprojection error based on multi-view constraints If the same map point If the difference in reprojection error under different camera views exceeds the preset range, the point is marked as a dynamic suspicious point, and its corresponding constraint weights are reduced in the optimization.
9. A SLAM relocalization device based on feature point homogenization and multi-source information fusion, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the SLAM relocalization method based on feature point homogenization and multi-source information fusion as described in any one of claims 1 to 8 according to the instructions in the program code.
10. An application of the SLAM relocalization method based on feature point homogenization and multi-source information fusion as described in any one of claims 1 to 8, characterized in that: It is used in VR headsets for cordless VR devices.