Pure visual monocular space relocation method and electronic device

Abstract

The application discloses a pure visual monocular space relocation method and electronic equipment. In view of problems such as cumulative drift and insufficient relocation accuracy in the prior art, the method adopts a two-stage architecture: in a cold start relocation stage, a deep neural network is used to extract features from an image and match the features with a three-dimensional point cloud map to obtain an initial high-precision pose; in a continuous positioning and error correction stage, real-time SLAM pose data, inertial measurement unit (IMU) data and image feature data are fused, and an error trustworthiness evaluation model is introduced. When the error trustworthiness score is lower than a threshold value, visual-geometric joint optimization is triggered to constrain the pose, and then multi-layer time series filtering is performed to update the global pose smoothly. The application significantly improves the positioning accuracy, trajectory smoothness and robustness, and effectively solves the cumulative drift problem.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and artificial intelligence, and in particular, relates to a method for high-precision and high-robustness six-degree-of-freedom (6-DoF) pose (position and attitude) estimation in a pre-constructed three-dimensional point cloud map using a monocular vision sensor and an electronic device implementing the method. BACKGROUND

[0002] Accurate spatial positioning is a core technology of modern intelligent systems such as autonomous driving, augmented reality (AR), and embodied agents. Existing satellite navigation technologies such as the global positioning system (GPS) are severely limited in key scenarios such as indoors, urban canyons, and tunnels, and cannot meet application requirements, so vision-based positioning technology has become an important development direction.

[0003] In vision-based positioning technology, simultaneous localization and mapping (SLAM) methods have been widely studied, but they have inherent cumulative drift problems when running for a long time or over long distances. This problem leads to difficulties in achieving centimeter-level high-precision requirements in terms of accuracy and stability when repositioning in a scene with an existing map.

[0004] To solve the drift problem of SLAM, existing technologies propose other pure vision positioning schemes. One category is based on geometric matching between image key points and three-dimensional point clouds (such as PnP algorithms), but it has poor robustness in sparse texture or changing light environments. Another category is based on deep learning to directly regress camera poses from images, but this type of method has problems such as insufficient model generalization ability, strong data dependency, and positioning errors that increase significantly with the increase in spatial scale.

[0005] Therefore, the existing technology still has technical bottlenecks to be solved in terms of achieving high-precision, high-robustness, and effectively suppressing cumulative errors in real-time repositioning using monocular vision in large-scale known environments. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a pure vision monocular spatial repositioning method and electronic device with high precision, high efficiency, and smooth trajectory, which addresses the cumulative drift, insufficient repositioning accuracy, high computational resource consumption, and discontinuous positioning results that are common in existing pure vision or vision-inertial navigation technologies.

[0007] To solve the above technical problems, the present application provides a pure vision monocular spatial repositioning method, and the system architecture of the method is divided into two core stages: a cold start repositioning stage and a continuous positioning and error correction stage.

[0008] Further, the cold start relocalization stage is dedicated to the initial localization of the system or the pose recovery after tracking loss, and the goal is to accurately obtain a globally consistent initial pose. This stage specifically includes the following steps: monocular vision sensor acquires images; image features are extracted using a pre-trained deep neural network; high-dimensional point cloud feature matching is performed using a pre-trained deep neural network, and pose solving is performed through the matching results.

[0009] Further, the continuous localization and error correction stage is entered after the initial pose is obtained through the cold start relocalization stage, and the system continuously tracks the device state in a high-frequency and low-delay manner, and the goal is to resist inherent cumulative drift. This stage specifically includes the following steps: recording real-time SLAM pose data and inertial measurement unit (IMU) data with time scale; performing error confidence score estimation using an error confidence evaluation model; determining a relocalization strategy based on the error confidence score to obtain a relocalization result; performing multi-source pose fusion and temporal smoothing constraint based on visual-geometric joint optimization.

[0010] Further, after all the processes of the continuous localization and error correction stage are completed, all the steps of the stage are repeatedly executed to realize continuous localization and cumulative drift error correction during device motion.

[0011] Further, the error confidence score of the continuous localization and error correction stage is calculated using a weighted evaluation model that fuses pose changes, inertial measurement unit (IMU) dynamic information, and visual feature matching quality. This error confidence score is the key technical basis of the present application and is used to dynamically reflect the reliability of the localization result.

[0012] Further, the multi-source pose fusion of the continuous localization and error correction stage uses a nonlinear solving method based on visual-geometric joint optimization. This method considers SLAM estimation, relocalization results, and historical trajectory information, and dynamically adjusts the weights of each source data under the guidance of error confidence. This fusion method is one of the technical cores of the present application.

[0013] Compared with the prior art, the present application has the following beneficial effects:

[0014] 1. The positioning accuracy is significantly improved.

[0015] The application combines the advantages of deep learning feature extraction and classical geometric algorithms, uses high-dimensional features based on deep neural networks for initial positioning in the cold start relocalization stage, and can extract more robust and discriminative image features compared to traditional manual features (such as SIFT, ORB, etc.), thereby achieving smaller deviations in position error and angle error. In the continuous positioning and error correction stage, the application of a multi-source pose fusion method through visual-geometric joint optimization further improves the positioning accuracy, effectively controls the positioning error in large-scale point cloud space, and meets the centimeter-level precision requirement.

[0016] 2. Positioning continuity and trajectory smoothness are significantly improved.

[0017] The application introduces an error trust score, combines SLAM pose data, inertial measurement unit (IMU) data, and visual feature data, intelligently determines when to trigger repositioning correction, and avoids frequent hard positioning result changes. In the pose fusion module, a smoothing update strategy is used to effectively reduce the problem of discontinuous trajectory caused by sudden changes in positioning results.

[0018] 3. The applicable scene and robustness are significantly expanded.

[0019] The application fuses multiple sensor information (vision, IMU) and the advantages of multiple algorithms (deep learning, classical geometry, SLAM) to construct a highly robust positioning system. The system can maintain stable positioning performance in complex environmental conditions such as changes in light, sparse texture, and dynamic occlusion, and is applicable to scenes from traditional structured indoor environments to large-scale outdoor spaces, underground parking lots, tunnels, and other complex environments with limited GPS signal.

[0020] 4. Effective solution to the problem of cumulative drift.

[0021] Traditional SLAM systems are prone to cumulative drift error after long time or long distance movement. The application actively monitors and corrects cumulative drift error, evaluates error trust in real time, triggers repositioning correction in time, and effectively counteracts the inherent cumulative drift problem.

[0022] The application also provides an electronic device for executing the above-mentioned pure visual monocular space repositioning method, which comprises: (a) at least one monocular vision sensor for collecting image frames; (b) at least one memory having computer program instructions, point cloud map data and deep neural network model stored thereon; (c) at least one processor; the processor realizes all steps in the pure visual monocular space repositioning method as described above by executing the computer program instructions stored in the memory. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the system overall architecture schematic diagram of the pure visual monocular space relocation method of the present application.

[0024] Figure 2 is the processing flow chart of the cold start relocation phase of the present application. Among them, S101 and S102 are two sub-steps of S1, S201 and S202 are two sub-steps of S2, S301, S302 and S303 are three sub-steps of S3.

[0025] Figure 3 is the processing flow chart of the continuous positioning and error correction phase of the present application. Among them, S401, S402, S403 and S404 are four sub-steps of S4, S601 and S602 are two sub-steps of S6, S701 and S702 are two sub-steps of S7.

[0026] Figure 4 is the hardware structure schematic diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0028] Overall scheme overview

[0029] Referring to Figure 1 , the pure visual monocular space relocation method provided by the present application adopts a two-stage architecture design, including a cold start relocation phase (S1-S3) and a continuous positioning and error correction phase (S4-S7). The architecture combines the feature extraction capability of deep learning algorithm with the calculation efficiency of classical geometric algorithm, and realizes high-precision and smooth trajectory space positioning through intelligent stage switching and error monitoring mechanism.

[0030] The core design concept of the system is: when high-precision initial positioning is needed, deep learning method is used for feature extraction and matching, in the process of continuous tracking, SLAM and IMU fusion scheme is relied on for high-frequency tracking, and the triggering time of relocation correction is intelligently decided through the trustworthiness evaluation mechanism.

[0031] Detailed implementation of the cold start relocation phase

[0032] Referring to Figure 2 , the detailed implementation process of the cold start relocation phase is as follows:

[0033] Step S1: monocular vision sensor acquires image

[0034] S101 A CMOS or CCD monocular vision sensor with a resolution of no less than 1280x720 pixels is used to collect a video with a frame rate of no less than 30 fps. During the image collection process, the system automatically adjusts the exposure parameters and focusing distance to adapt to different lighting conditions and scene depths.

[0035] S102 Frame images are extracted by analyzing the video stream data.

[0036] Step S2: Extract image features using a pre-trained deep neural network

[0037] S201 The frame images are processed and adjusted to a standard size of 512x512, and low-quality images such as excessively blurred images are filtered out. At the same time, data normalization processing is completed. The data normalization processing needs to normalize the pixel value to the [0, 1] interval.

[0038] S202 The pre-trained deep neural network is used to extract its visual features, and the fixed dimension is achieved through full connection and feature mapping. The feature descriptor has rotation invariance and scale invariance. The fixed dimension is [1, 512].

[0039] Step S3: High-dimensional point cloud feature matching is performed using a pre-trained deep neural network, and pose solving is performed based on the matching results

[0040] S301 The extracted features are matched with the pre-constructed point cloud map using a deep high-dimensional multi-modal matching algorithm. The point cloud map contains the coordinate information of each three-dimensional point in space and the corresponding feature descriptor. The deep high-dimensional multi-modal matching algorithm is used to establish the correspondence between image features and point cloud features.

[0041] S302 The matching points of image features and point cloud maps are used to estimate the pose using the PnP algorithm combined with RANSAC.

[0042] S303 All poses are nonlinearly optimized to obtain the initial pose. The nonlinear optimization selects the candidate solution with the most inliers as the initial pose and performs multiple iterations.

[0043] Detailed implementation of continuous positioning and error correction phase

[0044] Referring to Figure 3 , the detailed implementation process of the continuous positioning and error correction phase is as follows:

[0045] Step S4: Record real-time SLAM pose data and inertial measurement unit (IMU) data with time scale

[0046] S401 Real-time SLAM pose data is obtained.

[0047] S402 Obtain real-time inertial measurement unit (IMU) data. The real-time inertial measurement unit (IMU) data includes three-axis acceleration and three-axis angular velocity information, and the sampling frequency is 100 Hz.

[0048] S403 Record the precise time information of each data frame to ensure the time synchronization of multi-sensor data. The precise time information of the data frame is the single data callback time of the SLAM pose data and the inertial measurement unit (IMU) data mentioned above.

[0049] S404 Obtain real-time visual feature data. The visual feature data refers to the image features extracted by using a pre-trained deep neural network after obtaining an image using a monocular vision sensor.

[0050] Step S5: Use the error confidence evaluation model to estimate the error confidence score The error confidence evaluation model calculates the error confidence score of the current positioning result based on multi-source information

[0051] The multi-source information includes SLAM pose data, inertial measurement unit (IMU) data, and visual feature data. The error confidence score is calculated using a weighted fusion method, which detects abnormal motion patterns by analyzing the pose changes between consecutive frames, verifies the rationality of the motion state by analyzing the change trend of acceleration and angular velocity, and evaluates the reliability of visual tracking by statistically analyzing the success rate of feature matching to predict the credibility of the current positioning. The mathematical expression is:

[0052]

[0053] In the formula:

[0054] is the trust score component based on SLAM pose data, and the calculation formula is: In the formula, represents the three-dimensional displacement modulus value between the current frame and the previous frame; is the pose change sensitivity factor, which is used to adjust the influence degree of pose change on confidence, The larger the value, the higher the sensitivity of the system to pose change.

[0055] is the trust score component based on IMU data, which reflects the consistency of the motion state, and the calculation formula is: In the formula, represents the modulus value of the acceleration change rate, which reflects the stability of the current motion state; This is the IMU data sensitivity factor, used to control the degree of influence of the rate of change of acceleration on the confidence level.

[0056] The confidence component based on image feature matching is calculated using the following formula: In the formula, This indicates the number of feature points that were successfully matched in the current frame; This represents the total number of feature points involved in the matching.

[0057] Let be the weighting coefficients of each component, satisfying... .

[0058] Step S6: Determine the relocation strategy based on the error confidence score and obtain the relocation result.

[0059] S601 determines the relocation strategy based on the error confidence score. If the relocation strategy is based on the error confidence score... Less than If relocation is required, proceed to step S602; otherwise, proceed to step S701. The minimum acceptable error confidence score.

[0060] S602 utilizes image features to perform similar steps to S3 to obtain the visual pose solution (refer to...). Figure 2 ).

[0061] Step S7: Perform multi-source pose fusion and temporal smoothing constraints based on vision-geometric joint optimization.

[0062] The S701 employs a vision-geometric joint optimization method to construct a multi-source pose constraint objective function, and performs multi-source pose fusion by minimizing this objective function.

[0063] The objective function for the multi-source pose constraint is defined as follows: .

[0064] In the formula, Given the pose transformation matrix to be solved, the constraint terms are defined as follows:

[0065] SLAM pose constraints: In the formula, The pose output by the SLAM system; This represents the Frobenius norm.

[0066] Repositioning pose constraints: In the formula, The pose output by the relocation system; This is a relocation trigger flag, used when a relocation is performed. , otherwise .

[0067] Historical pose constraint term: . In which, is the historical positioning pose.

[0068] The weight of each constraint term is based on the error trust score Dynamic calculation:

[0069] SLAM pose constraint weight:

[0070] Repositioning pose constraint weight:

[0071] Historical pose constraint weight:

[0072] In which, , , are the baseline values of each weight, is the pose change sensitivity factor, is the SLAM pose change amount, is the number of successfully matched feature points, is the total number of feature points participating in matching, is the number of inliers.

[0073] The above joint optimization objective function is solved by using a nonlinear optimization algorithm to obtain the fusion pose .

[0074] S702 The fusion pose result is processed by multi-layer time series filtering to eliminate visual jitter and ensure trajectory continuity, generating the final smooth pose output.

[0075] The multi-layer time series filtering mechanism is:

[0076] (a) First-order time domain filtering:

[0077] .

[0078] In which, is the time filtering coefficient, is the pose at the previous time.

[0079] (b) Velocity constraint smoothing:

[0080] Calculate the pose change velocity:

[0081]

[0082] When the velocity modulus exceeds the threshold value, it is limited:

[0083]

[0084] wherein, is the maximum allowed velocity.

[0085] (c) Acceleration constraint smoothing:

[0086] Calculate acceleration estimate:

[0087]

[0088] Secondary smoothing when acceleration is out of limits:

[0089]

[0090] wherein, is the maximum allowed acceleration, is the final smoothed pose output.

[0091] Step S8: Loop S4-S7, and continuously apply the pose to achieve continuous positioning and cumulative drift error correction during device motion

[0092] The system loops steps S4 to S7 to achieve continuous positioning and cumulative drift error correction during device motion.

[0093] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A pure visual monocular space relocalization method, characterized in that, The method comprises a cold start relocalization stage and a continuous localization and error correction stage; the cold start relocalization stage comprises: (a) acquiring an image by a monocular vision sensor; (b) extracting image features by using a pre-trained deep neural network; (c) performing high-dimensional point cloud feature matching by using the pre-trained deep neural network, and solving a pose by a matching result; the continuous localization and error correction stage comprises: (a) recording real-time SLAM pose data and inertial measurement unit (IMU) data with a time scale; (b) estimating an error trust score by using an error trust evaluation model; (c) determining a relocalization strategy according to the error trust score: triggering relocalization when the error trust score is lower than a preset threshold, and obtaining a relocalization result; (d) performing multi-source pose fusion and time sequence smoothing constraint based on visual-geometric joint optimization; (e) repeatedly performing all steps of the stage to realize continuous localization and cumulative drift error correction in the process of device movement.

2. The method of claim 1, wherein, The error trust evaluation model adopts a weighted fusion manner to evaluate multi-source information to calculate an error trust score, and the multi-source information comprises: (a) a trust score component based on SLAM pose data; (b) a trust score component based on inertial measurement unit (IMU) data; (c) a trust score component based on image feature matching; wherein the sum of the weight coefficients corresponding to each trust score component is equal to 1.

3. The method of claim 1, wherein, The multi-source pose fusion adopts a nonlinear optimization algorithm to solve a minimum multi-source pose constraint target function, and the multi-source pose constraint target function comprises: (a) a SLAM pose constraint term; (b) a relocalization pose constraint term; (c) a historical pose constraint term; the weight of each constraint term is dynamically adjusted based on the error trust score.

4. The method of claim 1, wherein, The time sequence smoothing constraint adopts a multi-layer time sequence filtering process, and the multi-layer time sequence filtering process comprises: (a) first-order time domain filtering; (b) velocity constraint smoothing; (c) acceleration constraint smoothing.

5. An electronic device, comprising: An electronic device for performing the pure visual monocular space relocalization method of any one of claims 1-4, the electronic device comprising: (a) at least one monocular vision sensor for collecting image frames; (b) at least one memory having computer program instructions, point cloud map data and a deep neural network model stored thereon; (c) at least one processor; the processor realizes all steps in the pure visual monocular space relocalization method of any one of claims 1-4 by executing the computer program instructions stored in the memory.

Images