A mapping method and system integrating GNSS and multi-view vision

Through the mapping construction method of integrating GNSS and multi-view vision, GNSS timestamp constraints on visual image updates, combined with consistency constraints, the problems of low positioning accuracy and poor robustness in the existing technology are solved, and high-precision and real-time global Pose optimization and globally consistent map construction.

CN114966789BActive Publication Date: 2024-11-05HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210526760.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-12
Publication Date
2024-11-05
Estimated Expiration
2042-05-12

AI Technical Summary

Technical Problem

The prior art GNSS positioning accuracy is not high in indoor and outdoor scenarios, visual SLAM loopback detection is time-consuming and not real-time, and single-view visual mapping is affected by occlusion, resulting in low positioning accuracy, poor robustness and poor global consistency.

Method used

Using a mapping method that integrates GNSS and multi-view vision, the visual image update is updated through the GNSS measurement value timestamp, pose estimation and global optimization are performed, and the multi-view local map is integrated with time consistency and spatial consistency constraints to establish a global Consistent map.

Benefits of technology

The accuracy and robustness of pose estimation are improved, real-time global pose optimization is achieved, closed-loop missed detection of visual due to limited viewing angles and missing single-view features, and a globally consistent map is established to adapt to dynamically changing sites.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114966789B_ABST
    Figure CN114966789B_ABST
Patent Text Reader

Abstract

The present invention discloses a mapping method and system integrating GNSS and multi-view vision, wherein the method comprises: constraining the update time of multi-view visual images by the timestamp of GNSS measurement values ​​so as to align the GNSS measurement values ​​with the multi-view visual images, and then performing pose estimation on the multi-view visual images to obtain key frame poses; converting the current key frame poses and the previous key frame poses under each view to the station center coordinate system, calculating the relative poses, taking the sum of the current key frame poses and the relative poses as the visual observation value of the current key frame, and solving the global optimized pose with the goal of minimizing the difference between the GNSS measurement value and the visual observation value of the current key frame; updating the single-view local map by using the global optimized pose under a single view, and fusing the multi-view local maps when the time consistency constraint and the space consistency constraint are met to obtain a globally consistent map. The present invention can improve the real-time positioning accuracy and robustness, and establish a globally consistent map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, unmanned driving and mobile robot technology and multi-sensor fusion positioning and mapping, and more specifically, to a mapping method and system integrating GNSS and multi-view vision. Background Art

[0002] GNSS signals can provide high-precision global positioning information, but in indoor scenes, satellite signals may be blocked, and in outdoor scenes, satellite signals are easily lost, affecting positioning accuracy or even tracking loss. Visual information is not affected by satellite signals, and only visual features in the environment need to be extracted to complete pose estimation well. However, visual features are easily affected by factors such as lighting and tracking loss, and eliminating accumulated errors depends on loop detection. If the motion cannot form a loop, the accuracy of pose estimation is average.

[0003] In addition, the optimization process of loop detection in the visual SLAM system is time-consuming, the optimized posture is not real-time, and the posture optimization will produce a large jump in order to eliminate the cumulative error when the loop is detected, so it has no good estimation effect on the motion control and decision-making of the mobile robot. At the same time, the use of a single-view visual sensor for mapping is easily blocked by the limitation of the viewing angle, resulting in poor mapping results.

[0004] It can be seen that the existing technology has technical problems such as low real-time positioning accuracy, weak robustness and poor global consistency. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a mapping method and system that integrates GNSS and multi-perspective vision, thereby solving the technical problems of low real-time positioning accuracy, weak robustness and poor global consistency in the prior art.

[0006] To achieve the above object, according to one aspect of the present invention, a mapping method integrating GNSS and multi-view vision is provided, comprising the following steps:

[0007] (1) The update time of the multi-view visual image is constrained by the timestamp of the GNSS measurement value so that the GNSS measurement value is aligned with the multi-view visual image. Then, the pose of the multi-view visual image is estimated to obtain the initial key frame pose of each view and the key frame pose of each subsequent moment.

[0008] (2) The current key frame pose and the previous key frame pose at each viewing angle are converted to the station center coordinate system, the relative pose is calculated, and the sum of the current key frame pose and the relative pose is taken as the visual observation value of the current key frame. The global optimal pose of the current key frame at each viewing angle is solved with the goal of minimizing the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value;

[0009] (3) The single-view local map is updated using the globally optimized pose of the current key frame under a single view. When the multi-view visual images are aligned, the difference in the key frame depth values ​​between the multiple views is within the visual error range, and the pose translation relationship of the current key frame of the multiple views relative to the initial key frame is within the view position error range, the temporal consistency constraints and spatial consistency constraints are met, and the multi-view local maps are fused to obtain a globally consistent map.

[0010] Furthermore, the specific implementation of the spatial consistency constraint is as follows:

[0011]

[0012]

[0013] in, is the visual error, is the key frame depth value of a certain perspective in multiple perspectives, is the key frame depth value of other perspectives in the multi-perspective, b is the camera baseline corresponding to other perspectives, f is the focal length of the camera corresponding to other perspectives, σ is the parallax between other perspectives and a certain perspective, is the position error, 0<b<1, is the position translation relationship of the current key frame relative to the initial key frame at a certain perspective, is the position translation relationship of the current key frame of other perspectives relative to the initial key frame of a certain perspective, is the external position parameter relationship between other perspectives and a certain perspective, and C is the relative posture of a certain perspective.

[0014] Furthermore, the alignment method in step (1) is:

[0015] The tolerance time is set by the update frequency f of the GNSS measurement value

[0016] Update time t of GNSS measurement value gnss Get the timestamp [t gnss -ts,t gnss +ts], the multi-view visual images are updated within the timestamp, then the GNSS measurement values ​​are aligned with the multi-view visual images for subsequent operations.

[0017] Furthermore, the alignment method in step (3) is:

[0018] The tolerance time is set by the update frequency f of the visual image of a certain perspective in the multi-perspective

[0019] The update time t of a visual image from a certain perspective among multiple perspectives gnss Get the timestamp [t gnss -ts,t gnss +ts], if the visual images of other perspectives in the multi-perspective are updated within the timestamp, the other perspectives are aligned with the visual image of a certain perspective for subsequent operations.

[0020] Furthermore, the relative position and posture are obtained by:

[0021] The current key frame pose and the previous key frame pose under each viewing angle are converted from the local coordinate system to the station-centered coordinate system. The local coordinate system is the camera coordinate system corresponding to multiple viewing angles, and its z axis is aligned with the station-centered coordinate system. The x, y, and z axes of the station-centered coordinate system point to the east, north, and sky respectively. The origin of the station-centered coordinate system is a point on the global coordinate system. It is a semi-global coordinate system used to connect the global coordinate system and the local coordinate system. The global coordinate system takes the center of mass of the earth as its origin, the xy plane coincides with the equatorial plane, the x axis points to the prime meridian, and the z axis points to the North Pole. Then the relative pose of the current key frame pose and the previous key frame pose is calculated.

[0022] The station center coordinate system is also called the site coordinate system and the east-north-sky coordinate system ENU.

[0023] Furthermore, the global optimization pose solution also includes:

[0024] Convert the GNSS measurement value from the global coordinate system to the station center coordinate system. The coordinate value of the GNSS measurement value in the global coordinate system is The corresponding coordinate value in the station center coordinate system is [enu] T , the conversion between the two is as follows:

[0025]

[0026] Among them, the coordinate difference Obtained by the following formula:

[0027]

[0028] Among them, e is the eccentricity of the ellipsoid of the global coordinate system, N is the radius of curvature of the reference ellipsoid, and the longitude, latitude and altitude of the origin of the station center coordinate system are λ0, and h0, N0 are the radius of curvature of the reference ellipsoid corresponding to the origin of the station-centered coordinate system.

[0029] Furthermore, the global optimization pose solution also includes:

[0030] The difference between the aligned current key frame GNSS measurement value and the current key frame pose is taken as the GNSS residual term;

[0031] The difference between the global optimized pose of the current keyframe and the global optimized pose of the previous keyframe is taken as the pose increment after vision and GNSS optimization, the difference between the pose of the current keyframe and the pose of the previous keyframe is taken as the pose increment under vision, and the difference between the pose increment after vision and GNSS optimization and the pose increment under vision is taken as the residual term of visual odometer;

[0032] First, the pose of the multi-view visual images is estimated. When the GNSS measurement value is updated, the visual odometer residual term and the GNSS residual term are fused to solve the least squares problem and obtain the global optimal pose of different viewpoints.

[0033] The visual odometry residual term and the GNSS residual term are integrated to solve the least squares problem, that is, to minimize the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value, and solve the global optimal pose.

[0034] Furthermore, the multiple perspectives include a front perspective, a rear perspective, a left perspective and a right perspective. When the method is applied to a mobile robot, during the forward movement of the mobile robot, a dynamically changing site map is constructed based on visual images of different perspectives such as the front, back, left and right perspectives to obtain a site map. If the site from the front perspective is dynamically changing, the visual image from the front perspective does not participate in the mapping, and the site map is constructed based on the visual images from the rear perspective, the left perspective and the right perspective. During the backward movement of the mobile robot, the visual image from the front perspective is integrated into the site map.

[0035] According to another aspect of the present invention, there is provided a mapping system integrating GNSS and multi-view vision, comprising: a camera, a GNSS and a controller;

[0036] There are multiple cameras, each located at a different viewing angle, for collecting multi-view visual images;

[0037] The GNSS is used to collect GNSS measurements;

[0038] The processor comprises:

[0039] A pose estimation module is used to constrain the update time of the multi-view visual image through the timestamp of the GNSS measurement value so that the GNSS measurement value is aligned with the multi-view visual image, and then perform pose estimation on the multi-view visual image to obtain the initial key frame pose of each view and the key frame pose of each subsequent moment;

[0040] The global optimization module is used to convert the current key frame pose and the previous key frame pose under each viewing angle to the station center coordinate system, calculate the relative pose, and take the sum of the current key frame pose and the relative pose as the visual observation value of the current key frame. The global optimization pose of the current key frame under each viewing angle is solved with the goal of minimizing the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value.

[0041] The mapping module is used to update the single-view local map using the global optimized posture of the current key frame under a single view. When the multi-view visual images are aligned, the difference in the key frame depth values ​​between the multiple views is within the visual error range, and the posture translation relationship of the current key frame of the multiple views relative to the initial key frame is within the view position error range, the temporal consistency constraint and the spatial consistency constraint are met, and the multi-view local maps are fused to obtain a globally consistent map.

[0042] Furthermore, the multiple cameras are installed on different parts of a single ground mobile robot, or the multiple cameras are installed on multiple ground mobile robots respectively, or some of the multiple cameras are installed on the ground mobile robot and some are installed on the aerial drone.

[0043] Furthermore, the multiple cameras are installed at different positions of a single aerial drone, or the multiple cameras are dispersedly installed on multiple aerial drones, or the multiple cameras are partially installed on the aerial drone and partially installed on the ground mobile robot.

[0044] Furthermore, when the system is applied to a mobile robot or an aerial drone, during the forward movement of the mobile robot or the aerial drone, a dynamically changing site map is constructed based on visual images from different perspectives such as front, back, left and right to obtain a site map. If the site from the front perspective is dynamically changing, the visual image from the front perspective does not participate in the mapping, and the site map is constructed based on visual images from the rear perspective, left perspective and right perspective. During the backward movement of the mobile robot or the aerial drone, the visual image from the front perspective is integrated into the site map.

[0045] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0046] (1) The present invention first focuses on the pose estimation of multi-view visual images; then GNSS and multi-view vision are fused in a loosely coupled manner to perform global pose optimization, so that the pose estimation does not rely on loop detection, thereby improving the robustness of the system and the pose estimation accuracy. The global optimization is performed with the goal of minimizing the difference between the GNSS measurement value and the visual observation value, which reduces the amount of calculation, ensures the accuracy of the pose estimation in a short time, and can achieve the purpose of visual estimation of one frame and global optimization solution of one frame, with good real-time performance. The pose estimated by visual SLAM is inconsistent with the reference coordinate system of the GNSS signal, so it is necessary to perform global optimization after coordinate transformation. The timestamp of the GNSS measurement value is used to align the visual images of different perspectives, ensuring the consistency of the GNSS and visual image frames, and then the multi-view local maps are fused under the premise of satisfying the time consistency constraint and the space consistency constraint to establish a globally consistent map. The multi-view images can play a role in complementing each other, avoiding the closed loop missed detection due to the limited perspective and the lack of single-view features.

[0047] (2) The depth value constraint of the present invention can effectively eliminate some abnormal depth values ​​in the depth image measured by the camera, making the map information more credible. The position relationship constraint is based on the key frame postures of different perspectives, combined with the cameras of different perspectives to constrain according to the relative position relationship, and the maps of different perspectives are fused to make the map globally consistent.

[0048] (3) The method of the present invention aligns different camera image frames and GNSS measurement values, which belongs to external alignment between different sensors. The first alignment is mainly to estimate the pose of a single-view camera and GNSS fusion to obtain a high-precision real-time key frame pose. After obtaining the high-precision key frame pose, the maps of different viewpoints are aligned. According to the update time, a second alignment is performed. The local maps of different viewpoints are updated in real time and merged with the global map to ensure the global consistency of the local maps of different viewpoints. At the same time, the first alignment uses the update time and frequency of GNSS for alignment, and the second alignment uses the update time and frequency of the camera. Both alignments are automatic alignments, which effectively improve the robustness of the system.

[0049] (4) The coordinate transformation is mainly to integrate the relative unification of different coordinate systems during GNSS optimization, which is conducive to solving the least squares problem. In the map fusion stage, the transformation and alignment between different camera coordinate systems ensure the global consistency of maps from different perspectives. At the same time, the spherical coordinate system is inconsistent with the planning and control of the robot's plane movement. The conversion to the ENU (station center) plane coordinate system is conducive to the robot's motion planning and control.

[0050] (5) GNSS constraints and visual feature constraints are fused in a loosely coupled manner. When there is no GNSS signal update, the visual odometry pose estimation is mainly used; whenever there is a GNSS signal update, the two are fused for global pose optimization, so that the pose estimation does not rely on loop closure detection, which improves the robustness of the system and the accuracy of pose estimation. During the fusion optimization, the visual odometry residual term is constructed by the pose increment after vision and GNSS optimization and the pose increment under vision, and the GNSS residual term is constructed by the current key frame pose and GNSS measurement value, which reduces the amount of calculation and ensures the accuracy of the visual odometry pose estimation in a short time. It can achieve the goal of visual odometry estimation for one frame and global optimization solver optimization for one frame, and has good real-time performance.

[0051] (6) The present invention uses multi-perspective visual images for mapping, and can flexibly configure images from different perspectives for fusion to adapt to the dynamic changes of the site. If the site changes dynamically due to the robot's operation or manipulation during the robot's forward movement, the forward-perspective visual image will not be used in mapping to prevent interference with the mapping and affect the mapping precision and accuracy. At this time, the site is mapped based on the visual images from other perspectives such as the rearward and left and right sides. During the robot's backward movement, the forward-perspective visual image is added to perceive the changed forward-perspective scene. Based on the fusion of the visual images from several different perspectives, the dynamic site area map is incrementally updated, and the updated site map is used for the next robot path planning, thereby improving the mapping adaptability to dynamically changing site conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a mapping method integrating GNSS and multi-view vision provided by an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of coordinate system transformation provided by an embodiment of the present invention;

[0054] Figure 3 (a) is the mapping result of sequence 00 provided in an embodiment of the present invention;

[0055] Figure 3 (b) is the mapping result of sequence 09 provided in an embodiment of the present invention;

[0056] Figure 3 (c) is the mapping result of sequence 08 provided in an embodiment of the present invention;

[0057] Figure 3 (d) is the mapping result of sequence 10 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] The present invention provides a mapping method that integrates GNSS and multi-perspective vision, which can be applied to urban streets, indoor corridors, outdoor construction scenes, etc., to achieve high-precision pose estimation, and still has high robustness in environments with poor lighting conditions. The present invention aims to improve the accuracy and robustness of robot pose estimation without looping the motion, while also ensuring the real-time nature of the pose. The present invention uses multi-perspective depth vision images for mapping and optimizes key frame poses. Multi-perspective images can complement each other and improve the temporal consistency check and spatial consistency check of closed-loop frames, avoiding closed-loop missed detections due to limited visual angles and missing single-perspective features. Figure 1 As shown, a mapping method integrating GNSS and multi-view vision includes the following steps:

[0060] (1) The update time of the multi-view visual image is constrained by the timestamp of the GNSS measurement value so that the GNSS measurement value is aligned with the multi-view visual image. Then, the pose of the multi-view visual image is estimated to obtain the initial key frame pose of each view and the key frame pose of each subsequent moment.

[0061] (2) The current key frame pose and the previous key frame pose at each viewing angle are converted to the station center coordinate system, the relative pose is calculated, and the sum of the current key frame pose and the relative pose is taken as the visual observation value of the current key frame. The global optimal pose of the current key frame at each viewing angle is solved with the goal of minimizing the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value;

[0062] (3) The single-view local map is updated using the globally optimized pose of the current key frame under a single view. When the multi-view visual images are aligned, the difference in the key frame depth values ​​between the multiple views is within the visual error range, and the pose translation relationship of the current key frame of the multiple views relative to the initial key frame is within the view position error range, the temporal consistency constraints and spatial consistency constraints are met, and the multi-view local maps are fused to obtain a globally consistent map.

[0063] The alignment is:

[0064] The tolerance time is set by the update frequency f of the GNSS measurement value Preferably, a is 0.5.

[0065] Update time t of GNSS measurement value gnss Get the timestamp [t gnss -ts,t gnss +ts], if the multi-view visual images are updated within the timestamp, they are considered to be measured in the same frame, and the GNSS measurement values ​​are aligned with the multi-view visual images for subsequent operations. Otherwise, the GNSS measurement values ​​are discarded. In subsequent fusion optimization, the GNSS frames are the most recently updated values, ensuring the consistency of the GNSS and visual image frames.

[0066] In the present invention, the GNSS data update frequency is 20 Hz, and the fusion optimization frequency is 10 Hz, so that the GNSS frames are all the most recently updated values, ensuring the consistency of the GNSS and visual image frames.

[0067] GNSS constraints and visual feature constraints are fused in a loosely coupled manner. When there is no GNSS position signal update, the visual odometer is used as the main pose estimation. Whenever the GNSS position signal is updated, the two are fused for global pose optimization, making the pose estimation independent of loop closure detection, thereby improving the system's robustness and pose estimation accuracy.

[0068] Pose estimation of multi-view visual images includes:

[0069] The binocular vision SLAM system is used to perform initial pose estimation, and the pose estimation of the two frames is performed according to the homography matrix model or the basic matrix model to obtain the initial map point and the initial image frame is defaulted as the key frame. The reference key frame model tracking or constant speed model tracking is used to obtain the key frame pose.

[0070] The initial keyframe pose can be used to build the initial map, and the subsequent global optimized pose is used to determine the map points and then update the map.

[0071] Fusion optimization also requires transforming the pose estimated by visual SLAM and GNSS signals into the same reference coordinate system. The coordinate system transformation diagram is shown in Figure 2As shown, the coordinate system of the present invention mainly includes ECEF coordinate system, ENU coordinate system, local map coordinate system, camera coordinate system and robot chassis control coordinate system. Among them, ECEF (Earth-Centered, Earth-Fixed) global coordinate system takes the center of mass of the earth as the origin, the xy plane coincides with the equatorial plane, the x-axis points to the prime meridian, the z-axis points to the North Pole, and the ECEF reference coordinate system uses WGS84 (World Geodetic System-1984) coordinate system. The x, y, and z axes of the ENU (East-North-Up) coordinate system point to the east, north, and sky respectively, and the origin of the ENU coordinate system is a point on the ECEF coordinate system, which is a semi-global coordinate system used to connect the ECEF global coordinate system and the local map coordinate system. The local coordinate system is a relative coordinate system for pose estimation and mapping of the visual SLAM system, and its z-axis is aligned with the ENU coordinate system. The camera coordinate system is a relative coordinate system formed by the global coordinate system under camera observation. The robot chassis control coordinate system is the coordinate system in which the mobile robot makes relevant decisions, and in this invention, it is consistent with the global coordinate system. (·) WE represents the ECEF global coordinate system, (·) WN represents the ENU semi-global coordinate system, (·) WL represents the local coordinate system, (·) WC represents the camera coordinate system, (·) WR Represents the robot chassis control coordinate system. The transformation relationship between different coordinate systems is as follows:

[0072]

[0073] in, Represents the coordinate system To coordinate system The rotation matrix of Representing coordinate system The origin of the coordinate system The present invention has the following conversion relationship:

[0074] Camera coordinate system (·) WC and the robot chassis coordinate system (·) WR : The transformation matrix between the two is mainly determined by the installation position of the camera on the chassis.

[0075] Local coordinate system (·) WL and ENU coordinate system (·) WN : The transformation matrix between the two is mainly determined by the absolute coordinate value of the camera's 0th frame in the ENU coordinate system.

[0076] ENU coordinate system (·) WN and the ECEF global coordinate system (·) WE: The transformation matrix between the two is mainly determined by the GPS positioning information (longitude, latitude, and elevation) of the camera in frame 0.

[0077] When fusing GNSS with visual SLAM, it is necessary to transform the coordinate system of GNSS positioning information and visual SLAM pose estimation so that the coordinate system is unified and convenient for calculation. Convert the pose result in the local coordinate system to the ENU coordinate system, and then convert the GNSS positioning information from the WGS spherical coordinate system to the ENU plane coordinate system. The WGS84 coordinate system is an ellipsoidal coordinate system with a major axis radius a of 6378137.0m, a minor axis radius b of 6356752.3142m, and an eccentricity e of 0.0818. Assume that the measurement values ​​of longitude, latitude, and altitude of GNSS positioning information in the WGS84 coordinate system are The corresponding coordinate value in the ECEF coordinate system is [xyz]T, and the conversion relationship between the two is as follows:

[0078]

[0079] Where N is the radius of curvature of the reference ellipsoid, which is calculated as follows:

[0080]

[0081] Assume that the coordinate value of the positioning information in the ENU coordinate system is [enu] T , the longitude, latitude and altitude of the coordinate origin are The radius of curvature of the corresponding reference ellipsoid The conversion relationship between the WGS coordinate system and the ENU coordinate system is as follows:

[0082]

[0083]

[0084]

[0085]

[0086] After completing the calibration of the sensor's external parameters, as long as the initial pose of the camera relative to the ENU coordinate system and the corresponding coordinate value of the GNSS positioning sensor in the ENU coordinate system are obtained, the relative pose obtained by visual SLAM can be fused with the pose of the GNSS positioning sensor. Then, based on the initial coordinate value of the GNSS positioning sensor in the ECEF coordinate system, the absolute pose of the mobile robot in the world coordinate system during actual movement can be converted.

[0087] The difference between the aligned current key frame GNSS measurement value and the current key frame pose is taken as the GNSS residual term;

[0088] The difference between the global optimized pose of the current keyframe and the global optimized pose of the previous keyframe is taken as the pose increment after vision and GNSS optimization, the difference between the pose of the current keyframe and the pose of the previous keyframe is taken as the pose increment under vision, and the difference between the pose increment after vision and GNSS optimization and the pose increment under vision is taken as the residual term of visual odometer;

[0089] First, the pose of the multi-view visual images is estimated. When the GNSS measurement value is updated, the visual odometer residual term and the GNSS residual term are fused to solve the least squares problem and obtain the global optimal pose of different viewpoints.

[0090] The visual odometry residual term and the GNSS residual term are integrated to solve the least squares problem, that is, to minimize the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value, and solve the global optimal pose.

[0091] Calculate the global optimization pose and get the following formula:

[0092]

[0093] Among them, χ * represents the optimal solution of the global optimization pose solver, represents the measurement value of GNSS at the kth key frame, h(·) is the observation equation obtained by the front-end visual odometry. In the method of the present invention, the observation equation is obtained by converting the pose of the kth key frame of different viewing angles plus the kth frame and the k-1th frame obtained by the front-end visual odometry to the relative pose in the ENU coordinate system.

[0094] Using forward vision as a reference value, the relative posture relationship is obtained and spatial consistency constraints are performed. The constraint method is as follows:

[0095]

[0096]

[0097] in, is the key frame depth value obtained by forward vision, is the key frame depth value obtained by the camera from other perspectives, b is the camera baseline, f is the focal length of the camera, σ is the parallax, is the position translation relationship between the current key frame and the initial key frame obtained by forward vision, is the position translation relationship of the current key frame obtained from other perspectives relative to the initial key frame of the forward perspective, is the external position parameter relationship between other perspectives and perspectives, and C is the relative pose estimated by the current perspective. If the above relationship is satisfied, it is considered that other perspectives and the forward perspective meet the global consistency constraint, and the map is fused according to the relative relationship between the key frame poses to obtain a globally consistent map.

[0098] Example 1

[0099] The pose estimation experiment was conducted on the public dataset KITTI sequence 00, sequence 08, sequence 09, and sequence 10. The real trajectories of sequence 00 and sequence 09 form a loop, while the real trajectories of sequence 08 and sequence 10 do not form a loop. The CPU model of the computing platform used is AMD R7-4800H, 16-core CPU, and the GPU model is NVIDIA GeForce RTX2060, 6G video memory. The map construction results are shown in Figure 2. Figure 3 Middle (a), Figure 3 Middle (b), Figure 3 Middle (c), Figure 3 As shown in (d), the experimental results of pose estimation are shown in Table 1.

[0100] Table 1 Absolute trajectory error results of different sequences of KITTI dataset (unit: m)

[0101]

[0102] The absolute trajectory error RMSE of different sequences and its corresponding statistical results (mean value, median value, root mean square error STD, minimum value, maximum value Max) are recorded. It can be seen from the experimental results that the pose estimation method of the present invention can effectively improve the accuracy of pose estimation in urban road scenes with and without loop closures, so that pose estimation does not rely on loop closure detection.

[0103] At the same time, the running time of each thread of different methods is recorded to obtain Table 2. The pure visual pose estimation method has three threads: front-end tracking, local mapping, and back-end optimization. The method of the present invention adds a GNSS optimization thread on this basis to integrate GNSS for global pose optimization. It can be seen from the experimental results that the back-end optimization threads of sequence 00 and sequence 09 take a long time, making the optimized pose not real-time. Sequence 08 and sequence 10 do not form a trajectory loop, so there is no back-end optimization. When optimizing, the method of the present invention only needs to add the global pose of the current key frame and the relative pose of the current key frame generated by the front-end tracking thread relative to the previous key frame to the solver, and its average time is about 20ms, while the front-end tracking thread needs to extract feature points of image information, initialize map points, select key frames, etc., which takes a long time, and its average time is greater than 40ms. Therefore, the method of the present invention can achieve the purpose of front-end tracking one frame and GNSS optimization one frame, while improving the accuracy of pose estimation, it also has good real-time performance.

[0104] Table 2 Dynamic performance results of different sequences of KITTI dataset (unit: ms)

[0105] Dataset sequence number Front-end tracking thread Local Mapping Thread Backend optimization thread GNSS optimization thread 00 42.368 185.573 6640.289 22.710 08 42.971 176.485 0 27.290 09 41.441 157.320 3673.453 24.127 10 40.134 151.876 0 15.679

[0106] Example 2

[0107] The present invention is applied to the field of unmanned engineering machinery construction, and the unmanned engineering machinery includes unmanned bulldozers, unmanned graders, unmanned rollers, etc. During the forward operation of the unmanned engineering machinery, because the construction site of the forward perspective is dynamically changing, the forward perspective visual image does not participate in the mapping to prevent interference with the mapping and affect the mapping precision and accuracy. At this time, the site mapping is carried out based on the visual images of other perspectives of the rear, left and right sides; during the backward process of the unmanned engineering machinery, the forward perspective visual image is added to perceive the changed forward perspective scene, and based on the fusion of the visual images of several different perspectives of the front, back, left and right, the dynamic site area map is incrementally updated, and the updated site map is used for the next unmanned engineering machinery path planning to complete the construction work to adapt to the dynamic changes of the construction site.

[0108] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A mapping method integrating GNSS and multi-view vision, characterized in that: The steps include: (1) The update time of the multi-view visual image is constrained by the timestamp of the GNSS measurement value so that the GNSS measurement value is aligned with the multi-view visual image. Then, the pose of the multi-view visual image is estimated to obtain the initial key frame pose of each view and the key frame pose of each subsequent moment. (2) The current key frame pose and the previous key frame pose at each viewing angle are converted to the station center coordinate system, the relative pose is calculated, and the sum of the current key frame pose and the relative pose is taken as the visual observation value of the current key frame. The global optimal pose of the current key frame at each viewing angle is solved with the goal of minimizing the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value; (3) The single-view local map is updated using the globally optimized pose of the current key frame under a single view. When the multi-view visual images are aligned, the difference in the key frame depth values ​​between the multiple views is within the visual error range, and the pose translation relationship of the current key frame of the multiple views relative to the initial key frame is within the view position error range, the temporal consistency constraints and spatial consistency constraints are met, and the multi-view local maps are fused to obtain a globally consistent map.

2. A mapping method integrating GNSS and multi-view vision as claimed in claim 1, characterized in that: The specific implementation of the spatial consistency constraint is as follows: in, is the visual error, is the key frame depth value of a certain perspective in multiple perspectives, is the key frame depth value of other perspectives in the multi-perspective, b is the camera baseline corresponding to other perspectives, f is the focal length of the camera corresponding to other perspectives, σ is the parallax between other perspectives and a certain perspective, is the position error, 0<b<1, is the position translation relationship of the current key frame relative to the initial key frame at a certain perspective, is the position translation relationship of the current key frame of other perspectives relative to the initial key frame of a certain perspective, is the external position parameter relationship between other perspectives and a certain perspective, and C is the relative posture of a certain perspective.

3. A mapping method integrating GNSS and multi-view vision as claimed in claim 1 or 2, characterized in that: The alignment method in step (1) is: The tolerance time is set by the update frequency f of the GNSS measurement value Update time t of GNSS measurement value gnss Get the timestamp [t gnss -ts,t gnss +ts], the multi-view visual images are updated within the timestamp, then the GNSS measurement values ​​are aligned with the multi-view visual images for subsequent operations.

4. A mapping method integrating GNSS and multi-view vision as claimed in claim 1 or 2, characterized in that: The alignment method in step (3) is: The tolerance time is set by the update frequency f of the visual image of a certain perspective in the multi-perspective 0<a<1; The update time t of a visual image from a multi-viewpoint gnss Get the timestamp [t gnss -ts,t gnss +ts], if the visual images of other perspectives in the multi-perspective are updated within the timestamp, the other perspectives are aligned with the visual image of a certain perspective for subsequent operations.

5. A mapping method integrating GNSS and multi-view vision as claimed in claim 1 or 2, characterized in that: The relative position is obtained as follows: The current key frame pose and the previous key frame pose under each viewing angle are converted from the local coordinate system to the station-centered coordinate system. The local coordinate system is the camera coordinate system corresponding to multiple viewing angles, and its z axis is aligned with the station-centered coordinate system. The x, y, and z axes of the station-centered coordinate system point to the east, north, and sky respectively. The origin of the station-centered coordinate system is a point on the global coordinate system. It is a semi-global coordinate system used to connect the global coordinate system and the local coordinate system. The global coordinate system takes the center of mass of the earth as its origin, the xy plane coincides with the equatorial plane, the x axis points to the prime meridian, and the z axis points to the North Pole. Then the relative pose of the current key frame pose and the previous key frame pose is calculated.

6. A mapping method integrating GNSS and multi-view vision as claimed in claim 5, characterized in that: The global optimization pose solution also includes: Convert the GNSS measurement value from the global coordinate system to the station center coordinate system. The coordinate value of the GNSS measurement value in the global coordinate system is The corresponding coordinate value in the station center coordinate system is [enu] Τ , the conversion between the two is as follows: Among them, the coordinate difference It is obtained by the following formula: Among them, e is the eccentricity of the ellipsoid of the global coordinate system, N is the radius of curvature of the reference ellipsoid, and the longitude, latitude and altitude of the origin of the station center coordinate system are λ0, and h0, N0 are the radius of curvature of the reference ellipsoid corresponding to the origin of the station-centered coordinate system.

7. A mapping method integrating GNSS and multi-view vision as claimed in claim 1 or 2, characterized in that: The multiple perspectives include a front perspective, a rear perspective, a left perspective and a right perspective. When the method is applied to a mobile robot, during the forward movement of the mobile robot, a dynamically changing site map is constructed based on visual images of different perspectives of front, back, left and right to obtain a site map. If the site of the front perspective is dynamically changing, the visual image of the front perspective does not participate in the mapping, and the site map is constructed based on the visual images of the rear perspective, left perspective and right perspective. During the backward movement of the mobile robot, the visual image of the front perspective is integrated into the site map.

8. A mapping method integrating GNSS and multi-view vision as claimed in claim 1 or 2, characterized in that: The global optimization pose solution also includes: The difference between the aligned current key frame GNSS measurement value and the current key frame pose is taken as the GNSS residual term; The difference between the global optimized pose of the current keyframe and the global optimized pose of the previous keyframe is taken as the pose increment after vision and GNSS optimization, the difference between the pose of the current keyframe and the pose of the previous keyframe is taken as the pose increment under vision, and the difference between the pose increment after vision and GNSS optimization and the pose increment under vision is taken as the residual term of visual odometer; First, the pose of the multi-view visual images is estimated. When the GNSS measurement value is updated, the visual odometer residual term and the GNSS residual term are fused to solve the least squares problem and obtain the global optimal pose of different viewpoints.

9. A mapping system integrating GNSS and multi-view vision, characterized in that: include: Camera, GNSS and processor; There are multiple cameras, each located at a different viewing angle, for collecting multi-view visual images; The GNSS is used to collect GNSS measurements; The processor comprises: A pose estimation module is used to constrain the update time of the multi-view visual image through the timestamp of the GNSS measurement value so that the GNSS measurement value is aligned with the multi-view visual image, and then perform pose estimation on the multi-view visual image to obtain the initial key frame pose of each view and the key frame pose of each subsequent moment; The global optimization module is used to convert the current key frame pose and the previous key frame pose under each viewing angle to the station center coordinate system, calculate the relative pose, and take the sum of the current key frame pose and the relative pose as the visual observation value of the current key frame. The global optimization pose of the current key frame under each viewing angle is solved with the goal of minimizing the difference between the aligned current key frame GNSS measurement value and the current key frame visual observation value. The mapping module is used to update the single-view local map using the global optimized posture of the current key frame under a single view. When the multi-view visual images are aligned, the difference in the key frame depth values ​​between the multiple views is within the visual error range, and the posture translation relationship of the current key frame of the multiple views relative to the initial key frame is within the view position error range, the temporal consistency constraints and spatial consistency constraints are met, and the multi-view local maps are fused to obtain a globally consistent map.

10. A mapping system integrating GNSS and multi-view vision as claimed in claim 9, characterized in that: The multiple cameras are installed on different parts of a single ground mobile robot, or the multiple cameras are respectively installed on multiple ground mobile robots, or some of the multiple cameras are installed on the ground mobile robot and some are installed on the aerial drone, or the multiple cameras are installed on different parts of a single aerial drone, or the multiple cameras are dispersedly installed on multiple aerial drones.