Pure vision monocular space repositioning method and electronic equipment

Abstract

The invention discloses a pure vision monocular space repositioning method and electronic equipment. In order to solve the problems of accumulative drifting, insufficient repositioning precision and the like in the prior art, the method adopts a two-stage architecture: in a cold start repositioning stage, features are extracted from an image based on a deep neural network and are matched with a three-dimensional point cloud map, and an initial high-precision pose is obtained; in the continuous positioning and error correction stage, real-time SLAM pose data, inertial measurement unit (IMU) data and image feature data are fused, and an error credibility evaluation model is introduced. And when the error credibility score is lower than a threshold value, triggering vision-geometry joint optimization to carry out pose constraint, and smoothly updating the global pose through multi-layer time sequence filtering processing. According to the method, the positioning precision, the track smoothness and the robustness are remarkably improved, and the problem of accumulated drifting is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology. Specifically, it relates to a method for performing high-precision and high-robust six-degree-of-freedom (6-DoF) pose (position and attitude) estimation in a pre-constructed three-dimensional point cloud map using a monocular vision sensor, and an electronic device for implementing the method. Background Art

[0002] Accurate spatial positioning is a core technology for modern intelligent systems, including autonomous driving, augmented reality (AR), and embodied intelligence. Existing satellite navigation technologies, such as the Global Positioning System (GPS), suffer from severely limited signals in critical scenarios, such as indoors, in urban canyons, and in tunnels, making them inadequate for these applications. Therefore, vision-based positioning technology has become a key development direction.

[0003] While simultaneous localization and mapping (SLAM) methods have been widely studied in visual positioning technology, they suffer from inherent cumulative drift when operating over long periods of time or distances. This problem makes it difficult to achieve centimeter-level accuracy and stability when relocalizing in scenes with existing maps.

[0004] To address the drift problem in SLAM, existing technologies have proposed other purely visual localization solutions. One approach relies on geometric matching of image keypoints with 3D point clouds (such as the PnP algorithm), but this suffers from poor robustness in environments with sparse textures or varying lighting. Another approach relies on deep learning to directly regress the camera pose from the image. However, these methods suffer from insufficient model generalization, strong data dependence, and significantly increase localization error with increasing spatial scale.

[0005] Therefore, existing technologies still have technical bottlenecks that need to be addressed in using monocular vision to achieve real-time relocalization with high precision, high robustness and effective suppression of cumulative errors in large-scale known environments. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a high-precision, high-efficiency and smooth-trajectory pure vision monocular spatial relocalization method and electronic equipment to address the common defects of existing pure vision or visual-inertial navigation technologies, such as accumulated drift, insufficient relocalization accuracy, large consumption of computing resources and discontinuous positioning results.

[0007] To solve the above technical problems, the present invention provides a pure vision monocular spatial relocalization method, the system architecture of which is divided into two core stages: a cold start relocalization stage and a continuous positioning and error correction stage.

[0008] Furthermore, the cold-start relocalization phase is dedicated to initial system positioning or pose recovery after tracking loss. Its goal is to accurately obtain a globally consistent initial pose. This phase specifically includes the following steps: acquiring images using a monocular vision sensor; extracting image features using a pre-trained deep neural network; performing high-dimensional point cloud feature matching using the pre-trained deep neural network, and solving the pose based on the matching results.

[0009] Furthermore, the system enters the continuous localization and error correction phase after obtaining the initial pose during the cold-start relocalization phase. This phase continuously tracks the device state at a high frequency and low latency, aiming to combat inherent cumulative drift. This phase specifically includes the following steps: recording real-time SLAM pose data and inertial measurement unit (IMU) data with time stamps; estimating the error confidence score using an error confidence assessment model; determining a relocalization strategy based on the error confidence score to obtain a relocalization result; and performing multi-source pose fusion and temporal smoothing constraints based on joint vision-geometry optimization.

[0010] Furthermore, after all processes in the continuous positioning and error correction stage are completed, all steps in this stage are repeated to achieve continuous positioning and cumulative drift error correction during the movement of the device.

[0011] Furthermore, the error confidence score in the continuous positioning and error correction stage is calculated using a weighted evaluation model that integrates posture changes, inertial measurement unit (IMU) dynamic information, and visual feature matching quality. This error confidence score constitutes the key technical basis of the present invention and is used to dynamically reflect the reliability of the positioning results.

[0012] Furthermore, the multi-source pose fusion in the continuous positioning and error correction stage adopts a nonlinear solution method based on visual-geometric joint optimization. This method comprehensively considers SLAM estimation, relocalization results and historical trajectory information, and dynamically adjusts the weight of each source data under the guidance of error confidence. This fusion method is one of the technical cores of the present invention.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. Positioning accuracy is significantly improved.

[0015] By combining the advantages of deep learning feature extraction with classical geometric algorithms, this method uses high-dimensional features based on deep neural networks for initial positioning during the cold-start relocalization phase. Compared to traditional hand-crafted features (such as SIFT and ORB), this method extracts more robust and discriminative image features, resulting in smaller deviations in positional and angular errors. During the continuous positioning and error correction phase, the application of multi-source pose fusion methods through joint vision-geometric optimization further improves positioning accuracy, effectively controlling positioning errors in large-scale point cloud spaces and meeting centimeter-level accuracy requirements.

[0016] 2. Positioning continuity and trajectory smoothness are significantly improved.

[0017] This invention introduces an error confidence score and combines SLAM pose data, inertial measurement unit (IMU) data, and visual feature data to intelligently determine when to trigger relocalization corrections, thus avoiding frequent rigid changes in positioning results. In the pose fusion module, a smooth update strategy is employed to effectively reduce trajectory inconsistencies caused by sudden changes in positioning results.

[0018] 3. The applicable scenarios and robustness have been significantly expanded.

[0019] This paper integrates information from multiple sensors (vision, IMU) and the strengths of multiple algorithms (deep learning, classical geometry, and SLAM) to build a highly robust positioning system. This system maintains stable positioning performance in complex environments, such as those with varying lighting, sparse textures, and dynamic occlusion. Its application extends beyond traditional structured indoor environments to complex environments with limited GPS signal access, such as large-scale outdoor spaces, underground parking lots, and tunnels.

[0020] 4. Effective solution to the cumulative drift problem.

[0021] Traditional SLAM systems are prone to cumulative drift errors after long-term or long-distance movement. This invention uses active error monitoring and cumulative drift error correction to evaluate error confidence in real time and trigger repositioning correction in a timely manner, effectively combating the inherent cumulative drift problem.

[0022] The present invention also provides an electronic device for executing the above-mentioned pure vision monocular spatial relocalization method, wherein the electronic device includes: (a) at least one monocular vision sensor for acquiring image frames; (b) at least one memory on which computer program instructions, point cloud map data and a deep neural network model are stored; and (c) at least one processor; the processor implements all the steps in the above-mentioned pure vision monocular spatial relocalization method by executing the computer program instructions stored in the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the overall system architecture of the pure visual monocular space relocalization method of the present invention.

[0024] Figure 2 1 is a process flow chart of the cold start relocation phase of the present invention, wherein S101 and S102 are two sub-steps of S1, S201 and S202 are two sub-steps of S2, and S301, S302 and S303 are three sub-steps of S3.

[0025] Figure 3 4 is a process flow chart of the continuous positioning and error correction stage of the present invention, wherein S401, S402, S403 and S404 are four sub-steps of S4, S601 and S602 are two sub-steps of S6, and S701 and S702 are two sub-steps of S7.

[0026] Figure 4 It is a schematic diagram of the hardware structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0028] Overall solution overview

[0029] See also Figure 1 The pure vision monocular spatial relocalization method proposed in this paper adopts a two-stage architecture design, including a cold-start relocalization phase (S1-S3) and a continuous localization and error correction phase (S4-S7). This architecture combines the feature extraction capabilities of deep learning algorithms with the computational efficiency of classical geometric algorithms. Through intelligent phase switching and error monitoring mechanisms, it achieves high-precision spatial localization with smooth trajectories.

[0030] The core design concept of the system is to use deep learning methods for feature extraction and matching when high-precision initial positioning is required, rely on SLAM and IMU fusion solutions for high-frequency tracking during continuous tracking, and intelligently determine the triggering time of repositioning correction through a trust evaluation mechanism.

[0031] Detailed implementation of the cold start relocation phase

[0032] See also Figure 2 ,The detailed implementation process of the cold start relocation phase is as follows:

[0033] Step S1: Monocular vision sensor acquires image

[0034] The S101 uses a CMOS or CCD monocular vision sensor with a resolution of at least 1280x720 pixels to capture video at a frame rate of at least 30 fps. During image acquisition, the system automatically adjusts exposure parameters and focus distance to accommodate varying lighting conditions and scene depth.

[0035] S102 extracts frame images by parsing the video stream data.

[0036] Step S2: Extract image features using a pre-trained deep neural network

[0037] S201 processes the frame image data, adjusts it to a standard size of 512x512, filters out low-quality images such as those that are too blurry, and completes data normalization. The data normalization process requires normalizing pixel values ​​to the range [0, 1].

[0038] S202 uses a pre-trained deep neural network to extract visual features and fixes the dimension through full connectivity and feature mapping. The feature descriptor is rotation-invariant and scale-invariant. The fixed dimension is [1, 512].

[0039] Step S3: Use the pre-trained deep neural network to perform high-dimensional point cloud feature matching and solve the pose based on the matching results.

[0040] S301 performs feature matching between the extracted features and a pre-built point cloud map using a deep high-dimensional multi-modal matching algorithm. The point cloud map contains the coordinate information of each 3D point in space and the corresponding feature descriptors. The deep high-dimensional multi-modal matching algorithm is used to establish a correspondence between the image features and the point cloud features.

[0041] S302 uses the image features and the matching points of the point cloud map to perform pose estimation using the PnP algorithm combined with RANSAC.

[0042] S303 performs nonlinear optimization on all poses to finally obtain an initial pose. The nonlinear optimization is to select the candidate solution with the largest number of inliers as the initial pose and perform multiple iterations.

[0043] Detailed implementation of the continuous positioning and error correction phase

[0044] See also Figure 3 ,The detailed implementation process of the continuous positioning and error correction phase is as follows:

[0045] Step S4: Record real-time SLAM pose data and inertial measurement unit (IMU) data with time scale

[0046] S401 obtains real-time SLAM pose data.

[0047] S402: Acquire real-time inertial measurement unit (IMU) data. The real-time inertial measurement unit (IMU) data includes three-axis acceleration and three-axis angular velocity information, with a sampling frequency of 100 Hz.

[0048] S403 records the precise time information of each data frame to ensure time synchronization of multi-sensor data. The precise time information of the data frame is the single data callback time of the above-mentioned SLAM pose data and the above-mentioned inertial measurement unit (IMU) data.

[0049] S404: Acquire real-time visual feature data. The visual feature data refers to image features extracted using a pre-trained deep neural network after acquiring an image using a monocular vision sensor.

[0050] Step S5: Use the error confidence evaluation model to estimate the error confidence score

[0051] The error confidence evaluation model calculates the error confidence score of the current positioning result based on multi-source information The multi-source information includes SLAM pose data, inertial measurement unit (IMU) data, and visual feature data. The error confidence score The calculation adopts weighted fusion method, detects abnormal motion patterns by analyzing the posture changes between consecutive frames, verifies the rationality of the motion state by analyzing the changing trends of acceleration and angular velocity, and evaluates the reliability of visual tracking by the success rate of statistical feature matching to predict the credibility of the current positioning. Its mathematical expression is:

[0052]

[0053] Where:

[0054] is the trust component based on SLAM pose data, and the calculation formula is: Where, Indicates the three-dimensional displacement modulus between the current frame and the previous frame; is the posture change sensitivity factor, which is used to adjust the impact of posture change on confidence. The larger the value, the more sensitive the system is to posture changes.

[0055] is the trust component based on IMU data, reflecting the consistency of the motion state. The calculation formula is: Where, The modulus of the acceleration rate of change reflects the stability of the current motion state; It is the IMU data sensitivity factor, which is used to control the influence of acceleration change rate on confidence.

[0056] is the trust component based on image feature matching, and the calculation formula is: Where, Indicates the number of feature points that have been successfully matched in the current frame; Indicates the total number of feature points involved in matching.

[0057] is the weight coefficient of each component, satisfying .

[0058] Step S6: Determine the relocation strategy based on the error confidence score and obtain the relocation result

[0059] S601 Determine the relocation strategy based on the error confidence score. Less than If relocation is required, step S602 is executed, otherwise step S701 is executed. is the minimum acceptable error confidence score.

[0060] S602 uses image features to perform similar steps as S3 to obtain visual pose solution (refer to Figure 2 ).

[0061] Step S7: Multi-source pose fusion and temporal smoothing constraints based on vision-geometry joint optimization

[0062] S701 adopts the vision-geometry joint optimization method to construct a multi-source pose constraint objective function and performs multi-source pose fusion by minimizing the objective function.

[0063] The multi-source pose constraint objective function is defined as: .

[0064] Where, is the pose transformation matrix to be solved, and the constraints are defined as follows:

[0065] SLAM pose constraints: Where, The pose output by the SLAM system; represents the Frobenius norm.

[0066] Repositioning pose constraints: Where, The pose output by the relocalization system; It is the relocation trigger flag. When relocation is executed ,otherwise .

[0067] Historical pose constraints: Where, Positioning the history.

[0068] The weights of the constraints are based on the error confidence score Dynamic calculation:

[0069] SLAM pose constraint weights:

[0070] Relocalize pose constraint weights:

[0071] Historical pose constraint weight:

[0072] Where, 、 、 are the benchmark values ​​of each weight, is the posture change sensitive factor, is the SLAM pose change, is the number of successfully matched feature points, is the total number of feature points involved in matching, is the number of inliers.

[0073] The nonlinear optimization algorithm is used to solve the joint optimization objective function and obtain the fusion pose .

[0074] S702 performs multi-layer temporal filtering on the fused pose results to eliminate visual jitter and ensure trajectory continuity, generating the final smooth pose output.

[0075] The multi-layer temporal filtering mechanism is:

[0076] (a) First-order time domain filtering:

[0077] .

[0078] Where, is the time filter coefficient, is the posture at the previous moment.

[0079] (b) Velocity constrained smoothing:

[0080] Calculate the pose change speed:

[0081]

[0082] When the speed modulus exceeds the threshold, it is limited:

[0083]

[0084] Where, is the maximum permissible speed.

[0085] (c) Acceleration constrained smoothing:

[0086] Calculate an acceleration estimate:

[0087]

[0088] When the acceleration exceeds the limit, perform secondary smoothing:

[0089]

[0090] Where, is the maximum permissible acceleration, is the final smoothed pose output.

[0091] Step S8: Loop S4-S7 and continue to apply the posture to achieve continuous positioning and cumulative drift error correction during the device movement

[0092] The system executes steps S4 to S7 in a loop to achieve continuous positioning and cumulative drift error correction during the movement of the device.

[0093] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A purely visual monocular spatial relocalization method, characterized in that: It includes the cold start repositioning phase and the continuous positioning and error correction phase.

2. The method according to claim 1, characterized in that The cold start relocalization phase includes: (a) acquiring images using a monocular vision sensor; (b) extracting image features using a pre-trained deep neural network; and (c) performing high-dimensional point cloud feature matching using the pre-trained deep neural network, and solving the pose based on the matching results.

3. The method according to claim 1, characterized in that The continuous positioning and error correction stage includes: (a) recording real-time SLAM pose data and inertial measurement unit (IMU) data with time scale; (b) estimating the error confidence score using an error confidence evaluation model; (c) determining a relocalization strategy based on the error confidence score: triggering relocalization when the error confidence score falls below a preset threshold to obtain a relocalization result; (d) performing multi-source pose fusion and temporal smoothing constraints based on vision-geometry joint optimization; and (e) repeating all steps of this stage to achieve continuous positioning and cumulative drift error correction during device motion.

4. The method according to claim 3, characterized in that The error trust evaluation model uses a weighted fusion method to evaluate multi-source information to calculate the error trust score. The multi-source information includes: (a) a trust component based on SLAM pose data; (b) a trust component based on inertial measurement unit (IMU) data; and (c) a trust component based on image feature matching. The sum of the weight coefficients corresponding to each trust component is equal to 1.

5. The method according to claim 3, characterized in that The multi-source pose fusion adopts a nonlinear optimization algorithm to solve the minimization of the multi-source pose constraint objective function, which includes: (a) SLAM pose constraint item; (b) relocalization pose constraint item; (c) historical pose constraint item; the weight of each constraint item is dynamically adjusted based on the error confidence score.

6. The method according to claim 3, characterized in that The time series smoothing constraint adopts multi-layer time series filtering processing, and the multi-layer time series filtering processing includes: (a) first-order time domain filtering; (b) velocity constraint smoothing; (c) acceleration constraint smoothing.

7. An electronic device, characterized in that: Used to execute the pure vision monocular spatial relocalization method described in any one of claims 1-6, the electronic device includes: (a) at least one monocular vision sensor for acquiring image frames; (b) at least one memory on which computer program instructions, point cloud map data and a deep neural network model are stored; (c) at least one processor; the processor implements all the steps in the pure vision monocular spatial relocalization method described in any one of claims 1-6 by executing the computer program instructions stored in the memory.

Images