A Fast Differential Latent AGV Dense Three-Dimensional Reconstruction Method Based on Multi-Sensor Fusion

Abstract

A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion and its working method disclosed by the present invention include the following steps: obtaining a mobile scene RGB image, preprocessing the RGB image, extracting RGB image features through a feature extraction algorithm, adding key frames to the image frames, and screening the key frames; detecting and generating two-dimensional poses through a pose measurement unit; constraining the three-dimensional poses with the two-dimensional poses of the screened key frames to obtain the constrained poses, performing three-dimensional mapping through the poses and depth information of the key frames, and optimizing the three-dimensional mapping through a bag-of-words model; removing discrete point clouds, removing the key frames corresponding to the discrete point clouds, and forming a final dense three-dimensional reconstruction model; constructing an experimental platform for a fast moving scene according to the final dense three-dimensional reconstruction model.

Description

Technical Field

[0001] The present invention relates to the field of map reconstruction based on multi-sensor fusion, and more specifically, to a fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion. Background Art

[0002] In recent years, with the rapid development of computer software and hardware, artificial intelligence has increasingly appeared in people's daily lives. Three-dimensional reconstruction, as a special technology, plays an important role in the field of artificial intelligence. This is usually applied in indoor robot navigation, outdoor driverless, augmented reality, and game scene production, etc. Devices such as lidar and RGB-D cameras (depth cameras) are the most critical sensors in the three-dimensional reconstruction system. Advanced algorithms enable them to construct three-dimensional maps and models of the surrounding environment while perceiving their own positions.

[0003] As an important branch of robotics, mobile robots first appeared in the late 1960s. The first practical mobile robot was developed by the Artificial Intelligence Research Center of the Stanford Research Institute. Its most important purpose is to apply artificial intelligence technology in complex environments, enabling robots to perceive the world, autonomously plan, and execute and control their own behaviors.

[0004] Map reconstruction methods based on lidar can be divided into two types, namely methods with visual information and methods without visual information. In order to reconstruct three-dimensional map information, it is often necessary to fuse vision to achieve it. The simultaneous localization and mapping technology based on a monocular camera has obvious advantages. The sensor is small, convenient, and inexpensive. In addition to tracking the trajectory, there are also many methods that can meet the three-dimensional reconstruction requirements indoors and outdoors. Before the rise of RGB-D cameras, in order to obtain the depth information of images, researchers mostly chose binocular cameras. A binocular camera is composed of two separate RGB cameras. The distance between the two cameras is called the baseline. The depth information of each pixel grid can be estimated through the baseline, so the inherent scale uncertainty problem of a monocular camera can be solved. Through a binocular camera, a more accurate and dense three-dimensional map can be reconstructed. An RGB-D camera is a device that emerged around 2010 and measures the depth information of an object through the principle of infrared structured light or Time-of-Flight (ToF).

[0005] Different from binocular cameras, RGB-D cameras directly measure the depth information of images by physical means, without the need to rely on software calculation, thus saving some hardware resources. Three-dimensional reconstruction methods based on RGB-D cameras can be divided into two types, one is three-dimensional reconstruction with dynamic scenes, and the other is pure static three-dimensional reconstruction. Summary of the Invention

[0006] To solve at least one of the above technical problems, the present invention proposes a fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion.

[0007] The first aspect of the present invention provides a fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion, including the following steps:

[0008] Obtain RGB images of the moving scene, preprocess the RGB images, and extract RGB image features through a feature extraction algorithm;

[0009] Add key frames to the image frames and screen the key frames;

[0010] Detect and generate two-dimensional poses through a pose measurement unit;

[0011] Constrain the three-dimensional poses with the two-dimensional poses of the screened key frames to obtain the constrained poses;

[0012] Perform three-dimensional mapping through the poses and depth information of the key frames;

[0013] Optimize the three-dimensional mapping through a bag-of-words model;

[0014] Remove discrete point clouds, remove the key frames corresponding to the discrete point clouds, and form a final dense three-dimensional reconstruction model;

[0015] Construct an experimental platform for a fast moving scene according to the final dense three-dimensional reconstruction model.

[0016] In a preferred embodiment of the present invention, the pose measurement unit includes one or a combination of two or more of a two-dimensional lidar, an inertial navigation sensor, and a wheel speed meter.

[0017] In a preferred embodiment of the present invention, the image feature extraction specifically includes an ORB feature point detection algorithm, which is formed by combining a FAST detector and a BRIEF descriptor. Use FAST for feature point detection, and then extract the maximum response feature points of Harris corner points from the obtained candidate FAST feature points. The response function of the Harris corner point is as follows:

[0018] R = detM - α(traceM) 2

[0019] Among them, R is defined as the corner point response function, and it is judged whether a pixel is a corner point by judging the size of R. α is an empirical constant, usually taking a value of 0.04 - 0.06. The M matrix is the covariance matrix with the means of each dimension averaged.

[0020] In a preferred embodiment of the present invention, it further includes calculating Sim3 after optimizing the bag-of-words model for global optimization, removing discrete point clouds, eliminating the key frames corresponding to the discrete point clouds, and forming a final dense three-dimensional reconstruction model.

[0021] In a preferred embodiment of the present invention, the ORB feature point detection method is corrected by using the gray centroid method. The gray centroid method includes the geometric center P and the gray center Q. Taking the geometric center P as the zero point in this coordinate system, the coordinates of the gray centroid are obtained as follows:

[0022]

[0023] where M 10 is the sum of the gray values of all X axes, M 01 is the sum of the gray values of all Y axes, and M 00 is the sum of all gray values within this pixel block;

[0024] The angle calculation formula for ORB feature points is: θ = atan2(m 01 , m 10 ).

[0025] In a preferred embodiment of the present invention, it further includes a carrier vehicle, which is a differential omnidirectional automatic guided vehicle. The RGB-D camera and the lidar are mounted on the differential omnidirectional automatic guided vehicle.

[0026] In a preferred embodiment of the present invention, the carrier vehicle includes an IMU inertial navigation sensor, an industrial computer, and a single-chip microcomputer. The quaternion of the vehicle movement is transmitted to the industrial computer through the IMU inertial navigation sensor. At the same time, the two-dimensional point cloud is transmitted to the industrial computer through the two-dimensional lidar. The single-chip microcomputer sends control commands to the motor driver and simultaneously receives the wheel speed data of the motor driver;

[0027] The wheel speed data is sent to the industrial computer through the serial port to establish a two-dimensional topological map.

[0028] In a preferred embodiment of the present invention, the carrier vehicle carrying the camera reduces the three-dimensional movement to two-dimensional movement for adding ground constraints.

[0029] For the orthonormal basis (e 1 , e 2 , e 3 ) with a unit length, after one transformation, it becomes (e 1 ′, e 2 ′, 0) on the plane. For the rotation matrix R in three dimensions:

[0030]

[0031] After adding the ground constraint, e in the rotation matrix R 3 ′ is Therefore, the rotation matrix R' under the plane constraint is:

[0032]

[0033] In a preferred embodiment of the present invention, the translation vector t' under the plane constraint is:

[0034]

[0035] Therefore, SE'(3) under the plane constraint is shown as follows:

[0036]

[0037] Among them, T is the transformation matrix, SE'(3) refers to the special Euclidean group composed of transformation matrices, and SO(3) refers to the special orthogonal group composed of three-dimensional rotation matrices. By performing ground constraint processing on the optimized pose, the overall camera motion is reduced in dimension.

[0038] The above technical solutions of the present invention have the following advantages compared with the prior art:

[0039] (1) Aiming at the problem that the current use of only visual sensors cannot cope with the loss of robots, and the use of only lidar to obtain less information and difficult to reposition, this patent proposes a method of fusing visual sensors and two-dimensional lidar, combining the information collected by lidar and depth cameras, having more accurate scale information on the basis of obtaining rich feature information, and inputting it into the three-dimensional reconstruction system, greatly improving the speed and accuracy of traditional three-dimensional reconstruction pose calculation.

[0040] (2) Aiming at the high requirements of the handheld three-dimensional reconstruction method for the collector, avoiding the need to train the collector and being unable to control the stable and continuous acquisition of information during the acquisition process and generating unnecessary errors, this patent configures a wide-angle depth camera and a two-dimensional lidar on the same vertical line, which can accurately control the acquisition rate and movement angular velocity of the experimental platform, quantitatively control the video stream acquisition rate, so that the system can stably obtain continuous acquisition information while moving quickly.

[0041] (3) Aiming at the problem that most three-dimensional reconstruction algorithms focus on the reconstruction effect and ignore the accuracy of the motion pose, this patent proposes a ground constraint method to reduce the dimension of the system from three dimensions to two dimensions, reducing the camera motion from three-dimensional motion to two-dimensional motion, reducing the algorithm complexity of the overall system and improving the calculation efficiency, so that the system can better cope with fast-moving scenarios. Description of the Drawings

[0042] Figure 1It is a schematic diagram of a fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion in a preferred embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of the principle of the gray centroid method in a preferred embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of the internal principle structure of the carrier vehicle of the RGB-D camera in a preferred embodiment of the present invention.

[0045] Figure 4 It is the large-scale two-dimensional topological map and its scale generated by the two-dimensional lidar carrier vehicle in the embodiment of the present invention.

[0046] Figure 5 It is a statistical table of the wall spacing measurement results and average errors in the multi-scene fast moving and rotating three-dimensional reconstruction of the carrier vehicle in the embodiment of the present invention. Detailed implementation manners

[0047] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific implementation manners. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0048] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0049] A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion includes the following steps:

[0050] Obtain the RGB images of the moving scene, preprocess the RGB images, and extract the features of the RGB images through the feature extraction algorithm.

[0051] Add key frames to the image frames and screen the key frames.

[0052] Detect and generate two-dimensional poses through the pose measurement unit.

[0053] Use the two-dimensional poses of the screened key frames to constrain their three-dimensional poses to obtain the constrained poses.

[0054] Perform three-dimensional mapping through the poses and depth information of the key frames.

[0055] Optimize the three-dimensional mapping through the bag-of-words model.

[0056] Remove the discrete point cloud, eliminate the key frames corresponding to the discrete point cloud, and form the final dense three-dimensional reconstruction model;

[0057] Construct an experimental platform for a fast-moving scene based on the final dense three-dimensional reconstruction model.

[0058] Specifically, ground constraint means that the three-dimensional pose is optimized through the "ground constraint" optimization algorithm to obtain the optimized three-dimensional pose. The three-dimensional reconstruction method of the present invention first calculates the pose estimation for three-dimensional reconstruction of a fast-moving scene → studies the image feature extraction algorithm for RGB images, and uses a feature extraction algorithm with rapidity as the first element and accuracy as the second element → adds key frames to the image frames → screens the key frames → screens the local key frames through local optimization → generates two-dimensional poses through a two-dimensional lidar, an inertial navigation sensor, and a wheel speedometer → constrains the three-dimensional pose with the two-dimensional poses of the screened key frames to obtain the constrained pose → filters the depth map → constructs a moving cut-off depth map through the poses and depth information of the key frames → optimizes the bag-of-words model → calculates Sim3 → performs global optimization → removes the discrete point cloud → eliminates the key frames corresponding to the discrete point cloud → forms the final dense three-dimensional reconstruction model → designs an experimental platform for three-dimensional reconstruction map construction of a fast-moving scene → verifies the results through experiments.

[0059] Among them, Sim3 is to solve the similarity transformation using 3 pairs of matching points, and then solve the rotation matrix R, translation vector t, and scale between two coordinate systems. In particular, a depth camera is adopted, and the scale factor is set to 1.

[0060] The ORB (Oriented FAST and Rotated BRIEF) feature point detection algorithm is formed by combining FAST and BRIEF. This algorithm first uses FAST for feature point detection, and then extracts the feature points with the maximum response of Harris corners from the obtained candidate FAST feature points. The response function for Harris corners is as follows:

[0061] R = detM - α(traceM) 2 (1)

[0062] Among them, the M matrix is the covariance matrix with the means of each dimension averaged. Since the FAST feature point detection does not have the property of scale invariance, when performing ORB feature point detection, corner points are often detected on the Gaussian pyramid. This can achieve scale correlation for each layer of the image, thereby achieving scale-invariant results. On the other hand, the FAST feature point detection method not only lacks scale invariance but also lacks directionality. To address this defect, the ORB feature point detection method uses the gray centroid method for complementation. This method utilizes the characteristic that the gray centroid of most corner points does not coincide with their geometric center, obtaining the direction from the geometric center to the gray center, which is defined as the direction information of the ORB feature point. Since it is from the geometric center P to the gray center Q, the geometric center P is used as the zero point in this coordinate system, as Figure 2 shown.

[0063] Then the coordinates of the gray centroid are obtained as follows:

[0064]

[0065] where M 10 is the sum of the gray values of all X axes, M 01 is the sum of the gray values of all Y axes, and M 00 is the sum of all gray values within this pixel block. Therefore, the angle of the ORB feature point can be obtained as:

[0066] θ = atan2(m 01 , m 10 ) (3)

[0067] In the BRIEF descriptor, there is also no property of rotation invariance. Therefore, we also need to endow it with rotational feature information. Taking the θ angle obtained above as the main direction, and then discretizing θ into 12 sub-angles. For the 12 sub-angles, the BRIEF descriptors are solved respectively, thereby improving the rotational invariance robustness of the ORB feature points.

[0068] The dimension reduction method based on ground constraint is as follows:

[0069] For the RGB-D three-dimensional reconstruction system with a carrier vehicle added, the camera motion is reduced from three-dimensional motion to two-dimensional motion. Therefore, SE3 needs to be converted to SE2 to add ground constraint.

[0070] For the orthonormal basis (e 1 , e 2 , e 3 ) with unit length, after one transformation, it becomes (e 1 ′, e 2′, 0), for the rotation matrix R in three dimensions:

[0071]

[0072] After adding the ground constraint, e in the rotation matrix R 3 ′ is Therefore, the rotation matrix R′ under the plane constraint is:

[0073]

[0074] Similarly, the translation vector t′ under the plane constraint is:

[0075]

[0076] Therefore, SE′(3) under the plane constraint is shown as follows:

[0077]

[0078] By performing ground constraint processing on the optimized pose, the overall dimension of the camera movement is reduced, making it closer to the real scene, reducing unnecessary trajectory errors, and thus enabling a more perfect three-dimensional reconstruction. At the same time, during the three-dimensional reconstruction process, less pose information is transmitted, which can bring a faster three-dimensional reconstruction speed.

[0079] The experimental platform construction plan is as follows:

[0080] For the requirements of a dexterous and portable device for large-scale three-dimensional reconstruction, this patent selects an RGB-D camera with a small volume that can meet the needs of large-scale three-dimensional reconstruction. The RGB-D camera selected in this patent is the RealSense-D455 camera from Intel RealSense. The effective depth range of this camera is 0.6 - 6m, which can meet the "large" requirement of the large scene. At the same time, its maximum depth and RGB frame rate are 30 frames per second, which can meet the speed requirements of the fast and rotational large-scale three-dimensional reconstruction targeted by this patent. At the same time, due to its small volume: 124mm * 26mm * 29mm, it can be fixed on any movable device for faster large-scale three-dimensional reconstruction.

[0081] For fast rotational large - scale 3D reconstruction under different scene transformations, such as changes in lighting, scene size, etc., it will lead to inconsistent scales. Therefore, this patent designs a differential - drive AGV (Automated Guided Vehicle) equipped with an RGB - D camera and a lidar for fast 3D reconstruction of large remote scenes. This AGV is equipped with a SlamOpto two - dimensional lidar, an IMU (Inertial Measurement Unit), and a wheel speed sensor. By controlling the motor, the speed of image acquisition can be quantitatively compared. At the same time, since it builds a two - dimensional topological map through the lidar, the entire reconstruction system can move quickly with higher precision and stability. And because the motion dimension is reduced, the operating efficiency of the system can be greatly improved. The scale error of the 3D reconstruction result can also be compared through the two - dimensional topological map.

[0082] The internal principle structure of the carrier vehicle is as Figure 3 shown. The quaternion of the vehicle's motion is transmitted to the industrial computer through the IMU inertial sensor, and at the same time, the two - dimensional point cloud is transmitted to the industrial computer through the two - dimensional lidar. On the other side, the STM32F407 single - chip microcomputer sends control commands to the motor driver and receives the wheel speed data of the motor driver. And the wheel speed data is sent to the industrial computer through the 232 serial port. After receiving this information, the industrial computer builds a two - dimensional topological map. The two - dimensional topological map is used as the true scale basis for the 3D reconstruction model.

[0083] In terms of hardware: mainly use the Intel RealSense D455 camera. The effective depth range of this camera is 0.6 - 6m, which can meet the requirements of large - scale scenes and can directly obtain color images and depth images simultaneously; the SlamOpto two - dimensional lidar is based on the ToF ranging principle, with a scanning frequency of 15Hz (15r / s), an angular resolution of 0.33°, and a maximum effective working distance of 25 meters;

[0084] For software programming: mainly use C / C++ and Python, and mainly conduct experimental programming under the Ubuntu system;

[0085] 3D map construction strategy: Draw on current advanced SLAM algorithms to design a high - precision 3D map construction experimental platform for fast - moving scenes. This experimental platform integrates real - time data acquisition, real - time data processing, post - processing of data optimization, and display of 3D reconstruction results, etc. This experimental platform can be used for 3D reconstruction of various fast and complex scenes

[0086] This patent first reconstructs the two - dimensional topological map of the experimental scene with the carrier vehicle equipped with a two - dimensional lidar, as Figure 4 shown. After actual measurement, the difference between this two - dimensional topological map and the measurement result of the actual meter ruler is only 5mm. Therefore, this patent selects this two - dimensional topological map as the true value in scale comparison, asFigure 4 as shown

[0087] To achieve the condition of ground constraint and obtain accurate speed data, this patent mounts an RGB-D camera on a carrier cart. Through the IMU and wheel speedometer on the carrier cart, real-time angular velocity and linear velocity are obtained, so as to prove that this patent can obtain a better three-dimensional reconstruction model during fast movement and rotation in a large scene.

[0088] First, at the center of a large room, adjust the carrier cart to spin in place and adjust the wheel speed to 0.3 m / s.

[0089] The technical solution of the present invention has the following beneficial effects:

[0090] Aiming at the problem that the current use of only visual sensors cannot cope with robot loss, and the information obtained by only lidar is scarce and difficult to relocalize, this patent proposes a method of fusing a visual sensor and a two-dimensional lidar, combining the information collected by the lidar and the depth camera, having more accurate scale information on the basis of obtaining rich feature information, and inputting it into the three-dimensional reconstruction system, greatly improving the speed and accuracy of pose calculation in traditional three-dimensional reconstruction.

[0091] Aiming at the high requirements for collectors in hand-held three-dimensional reconstruction methods, avoiding the need to train collectors and the inability to control the stable continuity of information during the acquisition process, resulting in unnecessary errors, this patent configures a wide-angle depth camera and a two-dimensional lidar on the same vertical line, which can accurately control the acquisition rate and movement angular velocity of the experimental platform, quantitatively control the video stream acquisition rate, so that the system can stably obtain continuous acquisition information while moving quickly.

[0092] Aiming at the problem that most three-dimensional reconstruction algorithms focus on the reconstruction effect while ignoring the accuracy of the motion pose, this patent proposes a ground constraint method that reduces the dimension of the system from three dimensions to two dimensions, reducing the camera motion from three-dimensional motion to two-dimensional motion, reducing the algorithm complexity of the overall system and improving the calculation efficiency, enabling the system to better handle fast-moving scenarios.

[0093] The wheel spacing is 0.52 m. Through the real-time data feedback of the IMU, the average angular velocity of the carrier cart is 64 deg / s. At 64 deg / s in place, collect the RGB-D data set and input it into the algorithm of this patent and two other algorithms for comparative experiments;

[0094] Next, in order to test the optimization effect of ground constraints on the three-dimensional reconstruction model when reconstructing different large scenarios simultaneously, the wheel speed of the trolley in this patent was adjusted to 3 m / s and 1 m / s, and the angular velocity was adjusted to 66 deg / s. Among them, when the trolley was on the long straight road in the corridor, the running wheel speed was 3 m / s; when the trolley was in the large room, the running wheel speed was 1 m / s. According to the log file of the driver, the average wheel speed during this period was obtained as 1.671 m / s

[0095] The final measurement data and the true value data are as Figure 5 shown Figure 5 Statistical table of wall spacing measurement results and average errors in the three-dimensional reconstruction of a multi-scene fast-moving and rotating carrier trolley.

[0096] From Figure 5 it can be seen that the algorithm in this patent has an absolute advantage compared with the improved ORBSLAM2 dense reconstruction algorithm. At the same time, after adding ground constraints, the accuracy of the three-dimensional reconstruction model obtained by the algorithm in this patent has been significantly improved, almost doubling. Especially in the three measurement line segments with relatively long true distances, namely Line 1, Line 2, and Line 8, the accuracy advantage of the algorithm in this patent with ground constraints added is more significant. At the same time, in the four measurement line segments with relatively short true distances, namely Line 2, Line 3, Line 4, and Line 5, the algorithm in this patent with ground constraints added also has a certain advantage.

[0097] Aiming at the difficulty of a single-sensor three-dimensional reconstruction algorithm in ensuring pose accuracy, this patent proposes a three-dimensional reconstruction method for complex scenarios based on multi-sensor fusion. This method mainly fuses a two-dimensional lidar and a wide-angle depth sensor to obtain the AGV coordinates and attitude angles to obtain more accurate scale information, as well as color and depth video stream information, so that the system can avoid problems such as model misalignment and drift that most algorithms are prone to fall into with more accurate scale and feature information, and the final three-dimensional reconstruction model is closer to the true scale.

[0098] Aiming at the difficulty of a handheld sensor in stably obtaining acquisition information in a moving scenario, this patent proposes a three-dimensional reconstruction method for a fast-moving scenario based on a differential latent AGV. By controlling the AGV, accurate speed data can be obtained to verify that the system can obtain an accurate three-dimensional reconstruction model in the fast movement of a large scenario. In addition, in order to reduce the pose conversion calculation amount, a depth camera and a two-dimensional lidar are mounted on the same vertical line.

[0099] In order to improve the operation efficiency in a complex and fast-moving scenario, this patent proposes a three-dimensional reconstruction method with added ground constraint conditions. Since the sensors are all on the same vertical line, the system does not need to perform matrix transformation in the X and Y directions, and for the Z direction, no rotation transformation is required, only translation transformation is needed. Since the motion dimension is reduced from three-dimensional to two-dimensional, the operation and calculation complexity of the system can be greatly reduced, enabling the system to cope with the challenges brought by fast-moving scenarios.

[0100] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0101] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to the above embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the above embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0102] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion, characterized in that, it includes the following steps: Obtain the RGB image of the moving scene, preprocess the RGB image, and extract the RGB image features through the feature extraction algorithm, Add key frames to the image frames and screen the key frames; Detect and generate two-dimensional poses through the pose measurement unit; Use the two-dimensional poses of the screened key frames to constrain their three-dimensional poses to obtain the constrained poses; Perform three-dimensional mapping through the poses and depth information of the key frames; Optimize the three-dimensional mapping through the bag-of-words model; Remove the discrete point clouds, remove the key frames corresponding to the discrete point clouds, and form the final dense three-dimensional reconstruction model; Construct an experimental platform for the fast moving scene according to the final dense three-dimensional reconstruction model; It also includes a carrier vehicle, which is a differential latent automatic guided vehicle, and an RGB-D camera and a lidar are mounted on the differential latent automatic guided vehicle; The carrier vehicle includes an IMU inertial sensor, an industrial computer and a single-chip microcomputer. The quaternion of the vehicle movement is transmitted to the industrial computer through the IMU inertial sensor. At the same time, the two-dimensional point cloud is transmitted to the industrial computer through the two-dimensional lidar. The single-chip microcomputer sends control commands to the motor driver and receives the wheel speed data of the motor driver at the same time; Send the wheel speed data to the industrial computer through the serial port to establish a two-dimensional topological map; The carrier vehicle carrying the camera reduces from three-dimensional movement to two-dimensional movement and adds ground constraints, For an orthogonal basis (e 1 , e 2 , e 3 ) of unit length, after one transformation, it becomes (e 1 ′, e 2 ′, 0) on the plane. For the rotation matrix R in three dimensions: After adding the ground constraint, e in the rotation matrix R 3 ′ is Therefore, the rotation matrix R' under the plane constraint is: The translation vector t' under the plane constraint is: Therefore, SE'(3) under the plane constraint is shown as the following formula: where T is the transformation matrix, SE'(3) refers to the special Euclidean group composed of transformation matrices, and SO(3) refers to the special orthogonal group composed of three-dimensional rotation matrices; Reduce the overall dimension of the camera movement by performing ground constraint processing on the optimized poses.

2. A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion according to claim 1, characterized in that, The pose measurement unit includes one or a combination of two or more of a two-dimensional lidar, an inertial sensor, and a wheel speed meter.

3. A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion according to claim 1, characterized in that, The image feature extraction specifically includes the ORB feature point detection algorithm, which is formed by combining the FAST detector and the BRIEF descriptor. Use FAST for feature point detection, and then extract the maximum response feature points of the Harris corners from the obtained candidate FAST feature points. The response function of the Harris corners is as follows: R = detM - α(traceM) 2 where Define R as the corner response function, and judge whether the pixel is a corner by judging the size of R. α is an empirical constant, usually taking a value of 0.04 - 0.06, and the M matrix is the covariance matrix with the mean of each dimension averaged.

4. A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion according to claim 1, characterized in that, It also includes calculating Sim3 after optimizing the bag-of-words model, performing global optimization, removing discrete point clouds, eliminating the key frames corresponding to the discrete point clouds, and forming a final dense three-dimensional reconstruction model.

5. A fast differential latent AGV dense three-dimensional reconstruction method based on multi-sensor fusion according to claim 4, characterized in that the ORB feature point detection method is corrected by using the gray centroid method. The gray centroid method includes the geometric center P and the gray center Q. Taking the geometric center P as the zero point in the coordinate system, the coordinates of the gray centroid are obtained as: where M 10 is the sum of the grayscale values of all X - axes, M 01 is the sum of the grayscale values of all Y - axes, M 00 is the sum of all grayscale values within the pixel block; The angle calculation formula for ORB feature points is: θ = atan2(m 01 , m 10 ).

Images