An unmanned aerial vehicle real-time dense mapping method based on visual SLAM and deep learning

Abstract

The application relates to the field of computer vision, and discloses a real-time dense mapping method for a UAV based on visual SLAM and deep learning, which comprises the following steps: S1, calibrating camera parameters and correcting camera distortion; S2, acquiring image frames and GPS information; S3, tracking the acquired image frames by using a visual odometer; S4, performing sliding window optimization on the tracking results; S5, performing depth completion on the sliding window optimization results by using an SCCVA-MVSNet network; S6, optimizing the trajectory after the sliding window optimization by using GPS information; and S7, fusing the optimized trajectory and a depth map obtained by using the SCCVA-MVSNet to obtain a map. According to the application, a dense map can be constructed in real time only by using image and GPS information, compared with other methods, the application can be quickly deployed, and has relatively good accuracy.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary integration of computer vision and deep learning, specifically to a real-time dense mapping method for UAVs based on visual SLAM and deep learning. Background Technology

[0002] In recent years, with the rapid development of drone technology, drone application systems utilizing airborne sensors have gradually increased. For example, in agriculture, drones can be used to observe and identify ground plants from the air, enabling unmanned monitoring of plant growth in fields and significantly reducing labor costs. In rescue operations, due to complex terrain, severe weather, and damaged roads, the on-site environment is difficult to survey, seriously threatening the lives of rescue personnel. Drones can quickly survey the terrain of disaster-stricken areas and simultaneously search for trapped personnel, improving the safety of rescuers while providing rich information support for rescue operations, facilitating early rescue of those trapped. In the military field, battlefield reconnaissance is needed to monitor the battlefield situation in real time to avoid casualties. Using drones for high-altitude reconnaissance to quickly acquire surrounding terrain and enemy information can help reconnaissance personnel understand the overall terrain and locate key sensitive targets. Therefore, research on drone-based environmental perception systems is of great significance. Environmental perception systems mainly consist of two tasks: environmental map reconstruction and target search. The purpose of mapping is to acquire static environmental information such as surrounding terrain, buildings, and vegetation in unknown environments, and to create a map of the surrounding environment to help humans understand the surrounding environment. It can also be used to further develop automated tasks such as navigation and decision-making.

[0003] SLAM (Simultaneous Localization and Aspect-Oriented Mapping) is a challenging problem in the field of mobile robotics within artificial intelligence, and it has been extensively studied for over two decades. SLAM technology has developed rapidly, with various algorithms based on different sensors, such as ultrasonic sensors, LiDAR, visible light cameras, and infrared cameras. Among these, camera-only visual SLAM, due to its simple sensor configuration and technical challenges, has become a widely discussed research topic in recent years and has been applied in many different fields. Visual SLAM requires continuously estimating the camera pose of each frame in a continuous stream of image inputs while simultaneously reconstructing a 3D map of the scene. However, due to current computing power limitations, visual SLAM struggles to perform dense visual reconstruction in real time. Furthermore, in practical systems, sensor latency, timestamp misalignment, and other factors can lead to incomplete or even significant discrepancies between GPS and image timestamps, resulting in inaccurate reconstructed 3D maps. Summary of the Invention

[0004] This invention designs a fast dense map reconstruction system based on the Simultaneous Localization and Mapping (SLAM) method combined with deep learning. Through the cooperation between modules, a dense map reconstruction system based on UAV images is realized, solving various problems of existing solutions in specific scenarios.

[0005] To achieve the above objectives, this invention provides a real-time dense mapping method for UAVs based on visual SLAM and deep learning, comprising the following steps:

[0006] S1. Calibrate camera parameters and correct camera distortion;

[0007] S2. Acquire image frames and GPS information;

[0008] S3. Track the acquired image frames using a visual odometry system;

[0009] S4. Optimize the tracking results using a sliding window;

[0010] S5. The sliding window optimization results are deeply completed using the SCCVA-MVSNet network;

[0011] S6. Optimize the sliding window-optimized trajectory using GPS information;

[0012] S7. The optimized trajectory and the depth map obtained by SCCVA-MVSNet are fused to obtain the map.

[0013] Preferably, step S1 specifically includes the following steps:

[0014] S11. Project the 3D spatial point P(X,Y,Z) of the camera coordinate system onto the normalized image plane to form the normalized coordinates of that point as [x,y]. T ;

[0015] S12. For points [x,y] on the normalized plane... T Radial and tangential distortion corrections are achieved using the following formulas:

[0016]

[0017] Where [x] distorted y distorted ] T These are the corrected point coordinates, p1 and p2 are the tangential distortion coefficients of the camera, k1, k2, and k3 are the radial distortion coefficients of the camera, and r is the distance of point P from the origin of the coordinate system.

[0018] S13, adjust the corrected point [x] distorted ydistorted ] T The correct position [u,v] of a pixel is obtained by projecting it onto the pixel plane using the intrinsic parameter matrix. T This can be achieved through the following formula:

[0019] u = f x x distorted +c x

[0020] v = f y y distorted +c y

[0021] Where fx, fy, cx, and cy are the camera's intrinsic parameters.

[0022] Preferably, the image frames and GPS information are acquired using a DJI MAVIC 2 Pro drone. A camera is mounted on a 3-axis stabilization ring below the drone's nose, and the images and caption files are saved directly using the drone's own storage function. The GPS information can be obtained directly from the caption files.

[0023] Preferably, step S3 specifically includes the following steps:

[0024] S31. Construct an image pyramid for the acquired image frames, calculate the gradient for each layer, and select a certain number of points in each layer using a dynamic grid strategy.

[0025] S32. Generate a priori position through camera motion, that is, generate an estimate of the motion of the current frame based on the amount of motion of the previous frame plus fine-tuning.

[0026] S33. Calculate the corresponding residuals based on the estimation, and calculate the Jacobian matrix with respect to the 6-dimensional Lie algebra based on the residuals and gradients, and then iteratively optimize the estimated pose.

[0027] S34. Determine whether the current frame is a keyframe based on the pose relationship. If it is, add it to the sliding window for optimization.

[0028] Preferably, step S4 specifically includes the following steps:

[0029] S41. Based on the degree of co-visibility between the keyframes that are about to be added to the sliding window and the old keyframes, and the positional distribution of the keyframes within the sliding window, determine which frames need to be marginalized and marked.

[0030] S42. Add the latest keyframe to the sliding window and construct residual terms, perform iterative optimization, and optimize all keyframes and activation points within the sliding window;

[0031] S43. The marked frame and the activation points of the frame are edge-trimmed simultaneously;

[0032] S44. Select certain key points as pre-selections in the latest keyframe.

[0033] Preferably, the SCCVA-MVSNet network consists of two parts: a first part is a depth completion network, and the second part is a CVA-MVSNet network. The input of the first part of the depth completion network is a sparse depth map formed by projecting the feature points of the effective depth of all keyframes across all frames and the photometric error of each feature point. The input of the second part of the CVA-MVSNet network is a concatenation of the image in the sliding window and the output of the first part of the depth completion network. The output of the second part of the CVA-MVSNet network is the depth map of the latest keyframe.

[0034] Preferably, step S6 specifically includes the following steps:

[0035] S61. The error of the edges between vertices in the graph is calculated by comparing the relative pose obtained through sliding window optimization with the relative pose calculated based on the global pose. The 7-dimensional Lie algebra of this error is taken as the edge error, and the error term is defined as:

[0036]

[0037] Where S ij For the pose transformation from frame i to frame j, S ij The relative pose is optimized using a sliding window calculated from the edges between vertices in the graph. The relative pose is calculated based on the global pose.

[0038] S62. Add new vertices to integrate the GPS pose into the pose graph. Connect the GPS pose vertices to the pose vertices in the graph through a special edge. Constrain the translation amount of the visual pose and GPS pose on the same keyframe to be the same. Minimize the error of all edges in the graph and optimize the pose of all vertices.

[0039] Preferably, step S7 specifically includes the following steps:

[0040] S71. Calculate the vertex poses and corresponding points in the depth map for the keyframes:

[0041] p j =π j (R i→j φ(p i ,d i ,K i )+t i→j ,K)

[0042] Where φ(p) i ,d i ,K i ) is the homogeneous coordinate pi The pixels in the given estimated depth d i 3D point inverse projection, π j (p i K) represents the projection of a 3D point onto the image plane, R i→j , t i→j It is the rotation and translation from the source frame to the target frame, and K is an inherent characteristic of the camera;

[0043] S72. Calculate the forward reprojection error based on the geometric consistency between the two views ||p i -p j || 2 and depth error ||d i -d j || 2 , where d j It is the depth in reference frame j;

[0044] S73. Find consistent pixel clusters in multiple frames and fuse them, wherein the pixel clusters satisfy the reprojection error. and depth error ||d i -d j || 2 <ε constraint.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] The present invention provides a real-time dense mapping method for UAVs based on visual SLAM and deep learning, which can build dense maps in real time with only images and GPS information. Compared with other current methods, it can be deployed quickly and has relatively good accuracy. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the pinhole model of the present invention;

[0049] Figure 3 This is a schematic diagram of the radial distortion of the camera in this invention;

[0050] Figure 4 This is a schematic diagram of the tangential distortion of the camera in this invention;

[0051] Figure 5 This is a schematic diagram of the SCCVA-MVSNet network constructed in this invention;

[0052] Figure 6 for Figure 2 These are the original aerial images and the depth estimation results from SCCVA-MVSNet, as presented in this invention.

[0053] Figure 7 This is a schematic diagram of the back-end pose graph of the present invention;

[0054] Figure 8 This is a rendering of the design for this invention.

[0055] Figure 9 The map obtained by the mapping in this invention is obtained by surface reconstruction. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To address the problems and shortcomings of existing technologies, this invention proposes a real-time dense mapping method for UAVs based on visual SLAM (Simultaneous Localization and Mapping) and deep learning. It uses deep learning combined with a semi-dense SLAM algorithm model to achieve high-altitude online dense mapping.

[0058] This invention proposes a real-time dense mapping method for UAVs based on visual SLAM and deep learning, such as... Figure 1 The above includes the following steps:

[0059] S1. Calibrate camera parameters and correct camera distortion;

[0060] S2. Acquire image frames and GPS information;

[0061] S3. Track the acquired image frames using a visual odometry system;

[0062] S4. Optimize the tracking results using a sliding window;

[0063] S5. The sliding window optimization results are deeply completed using the SCCVA-MVSNet network;

[0064] S6. Optimize the sliding window-optimized trajectory using GPS information;

[0065] S7. The optimized trajectory and the depth map obtained by SCCVA-MVSNet are fused to obtain the map.

[0066] The following is a detailed explanation of each step.

[0067] Step S1: Calibrate camera parameters and correct camera distortion.

[0068] Calibration of camera calibration geometry is necessary regardless of whether the method is direct or indirect, and the accuracy of the calibration parameters directly determines the accuracy of subsequent tasks. This invention uses a pinhole camera model and a Rantan distortion model. The pinhole camera model is used by cameras that utilize lenses for focused light imaging, and its formula is as follows:

[0069]

[0070] Where, [X,Y,Z] T These are the coordinate values ​​of a 3D point in the camera coordinate system; u, v are the pixel coordinates on the image, such as... Figure 2 As shown, the left side is a 3D schematic diagram of the visual point mapped onto the normalized plane, and the right side is a schematic diagram of the mapped point on the image.

[0071] The Randan distortion model includes radial distortion and tangential distortion. Radial distortion is caused by the shape of the lens; essentially, the lens's imaging plane is not actually a plane, but rather approximates one. Since camera sensors are generally flat, this causes barrel or pincushion distortion (determined by the lens or lens group). Distortions such as... Figure 3 As shown.

[0072] Tangential distortion is primarily caused by the misalignment of the photosensitive element of the camera sensor with the lens. For example... Figure 4 The diagram illustrates the cause of tangential distortion. Tangential distortion can be artificially reduced. Currently, the tangential distortion calibration parameters of many cameras are almost zero, meaning they are almost parallel.

[0073] Simultaneous correction of radial and tangential distortion is achieved using the following formula:

[0074]

[0075] Where [x] distorted y distorted ] T These are the corrected point coordinates, p1 and p2 are the tangential distortion coefficients of the camera, k1, k2, and k3 are the radial distortion coefficients of the camera, and r is the distance of point P from the origin of the coordinate system.

[0076] The corrected point [x] distorted y distorted ] T The correct position [u,v] of a pixel is obtained by projecting it onto the pixel plane using the intrinsic parameter matrix. T This can be achieved through the following formula:

[0077] u = f x xdistorted +c x

[0078] v = f y y distorted +c y

[0079] Where fx, fy, cx, and cy are the camera's intrinsic parameters.

[0080] This invention uses the MATLAB toolbox for calibration.

[0081] Step S2: Obtain image frames and GPS information.

[0082] This invention utilizes a DJI Mavic 2 Pro drone for data acquisition. Its camera is suspended on a 3-axis stabilization ring beneath the drone's nose. During flight, the drone's camera automatically adjusts to achieve stability, ensuring stable and smooth image data. Furthermore, this drone can reach speeds up to 72 km / h, allowing it to capture data with significant inter-frame parallax even in large-scale environments. This invention directly utilizes the drone's built-in storage to save image and caption files, while GPS values ​​can be directly obtained from the caption files.

[0083] Step S3: Track the acquired image frames using a visual odometry system.

[0084] The tracking section mainly involves three tasks:

[0085] (1) Determine the frames and points used by the backend sliding window, i.e. which frames and points are used in window optimization, and also handle abnormal data to ensure that window optimization can be carried out robustly.

[0086] (2) Provide an initial optimization value for window optimization to ensure that the state variables can gradually converge during iterative optimization, or in other words, to ensure that the objective function can be successfully minimized;

[0087] (3) Determine under what circumstances a frame and point should be removed from the sliding window (marginalization) to ensure that dimensional explosion does not occur within the sliding window.

[0088] The main steps are as follows:

[0089] S31. Construct an image pyramid for the acquired image frames, calculate the gradient for each layer, and select a certain number of points in each layer using a dynamic grid strategy.

[0090] S32. Generate a priori position through camera motion, that is, generate an estimate of the motion of the current frame based on the amount of motion of the previous frame plus fine-tuning.

[0091] S33. Calculate the corresponding residuals based on the estimation, and calculate the Jacobian matrix with respect to the 6-dimensional Lie algebra based on the residuals and gradients, and then iteratively optimize the estimated pose.

[0092] S34. Determine whether the current frame is a keyframe based on the pose relationship. If it is, add it to the sliding window for optimization.

[0093] Step S4: Optimize the tracking results using a sliding window.

[0094] The sliding window contains several keyframes, each carrying some map point information. New keyframes also perform pose estimation and immature point operations on non-keyframes, and are subsequently added to the sliding window composed of keyframes, linking the currently active points in the sliding window to themselves. The process within the sliding window includes:

[0095] S41. Based on the degree of co-visibility between the keyframes to be added to the sliding window and the old keyframes, and the positional distribution of the keyframes within the sliding window, determine which frames need to be marginalized and marked.

[0096] There are three conditions for determining whether a frame needs to be marginalized. First, the two newest keyframes are always retained based on the order they were added. This ensures the sliding window optimizes the trajectory the camera has just traversed. However, this condition also prevents the system from remaining stationary for too long; otherwise, all keyframes within the optimization window will overlap, causing significant errors or even system failure. Second, based on common-view information, frames with insufficient common-view with the newest keyframe are removed. That is, if the percentage of points in a frame that can be seen by the newest frame is less than a certain value, that frame is marginalized. Third, based on positional relationships, a heuristic algorithm is used to remove a frame with a relatively poor position when the number of keyframes exceeds the maximum number of frames set for the sliding window. This condition ensures a better distribution of frames within the optimization window in 3D space. This method is superior to the approach of directly removing the oldest frame, which is common in many frameworks.

[0097] S42. Add the latest keyframe to the sliding window and construct residual terms, perform iterative optimization, and optimize all keyframes and activation points within the sliding window.

[0098] This application utilizes GN iterative optimization. During the iteration of GN, it is necessary to calculate H = J. T WJ and B = -J T Wr, where It is an information matrix. It is the residual vector. It is the Jacobian matrix. During GN iterative optimization, three types of Jacobian matrices need to be calculated:

[0099] (1) Image Jacobi (J) I), i.e., image gradient;

[0100] (2) Geometric Jacobi (J) geo That is, the reciprocal of each geometric quantity, including the reciprocal of the rotation matrix and translation matrix, as well as the reciprocal of the inverse depth value and the camera geometric parameters;

[0101] (3) Luminous Jacobi (J) photo That is, the reciprocal of the photometric parameter, which in this system is the affine transformation parameter of each frame;

[0102] Considering only one residual term r k In the case of Jacobian matrix, the formula is defined by the following equation:

[0103]

[0104] Where δ geo Represents "geometric parameters (T)" i ,T j ,d,c);δ photo Represents photometric parameters (a) i ,b j ,b i ,b j ); ζ0 represents the evaluation point in the tangent space; x is the accumulated incremental update; δ is calculated using the Gauss-Newton step; symbol This represents the update of the state variable.

[0105] S43. The marked frame and the activation points of the frame are edged simultaneously.

[0106] S44. Select certain key points as pre-selections in the latest keyframe.

[0107] Step S5: Perform deep completion on the sliding window optimization results using the SCCVA-MVSNet network.

[0108] The results of the SCCVA-MVSNet (Sparse and Confidence with Cascade View-Aggregation Multi-view Stereo Network) network are as follows: Figure 5As shown, the specific network consists of two parts: the first part is a depth completion network, and the second part is a CVA-MVSNet (Cascade View-Aggregation Multi-view Stereo Network) network. The input to the first part is a sparse depth map formed by interpolating the feature points of the effective depths of all keyframes across all frames, and the photometric error of each feature point. Confidence is obtained through parameterization, and the initial dense depth and the confidence corresponding to each depth point are interpolated through a normalization network. The confidence is forward propagated using the following formula:

[0109]

[0110] It is the output of pixel position i,j in the l-th layer network. The confidence level is the value of layer l-1, and n and m are the kernel sizes. These are the weights of the convolution kernel. Assuming that the neighbors in the depth map are locally similar, the sparse depth map is interpolated into the coarse depth map. The pixels with smaller photometric errors have higher confidence and contribute more to the surrounding pixels, resulting in more accurate depth.

[0111]

[0112] The second part of the network takes as input the image from the sliding window and concatenates it with the output of the first-stage network. Through a hierarchical multi-view reconstruction (MVS) network, it outputs depth data from low-resolution, low-precision to high-resolution, high-precision. The final output of the second part is the depth map of the latest keyframe, as shown below. Figure 6 As shown.

[0113] Step S6: Optimize the sliding window-optimized trajectory using GPS information.

[0114] In this invention, the SLAM backend utilizes GPS to optimize the attitude map (attitude map as shown in the figure). Figure 7 As shown), this is a special graph structure for a general pose graph, where vertices represent the pose at a certain moment, and the edges connecting two vertices represent the relative pose transformation between the two vertices, with a defined computational error. Graph optimization optimizes all vertices, i.e., all poses, in the graph by iteratively updating the pose values ​​of nodes and minimizing the sum of errors defined by their adjacent edges. This invention calculates the error between edges connecting front-end visual vertices using the relative pose optimized by a sliding window and the relative pose calculated based on the global pose. The 7-dimensional Lie algebra of this error is used as the edge error, and the error term is defined as:

[0115]

[0116] Where Sij For the pose transformation from frame i to frame j, S ij The relative pose is optimized using a sliding window calculated from the edges between vertices in the graph. This refers to the relative pose calculated based on the global pose.

[0117] For the newly added GPS pose, considering that its actual physical meaning is the same as the pose of the visual vertex, this invention integrates the GPS pose into the pose graph by adding a new vertex. To ensure that the GPS pose should be the same as the visual pose at any given time, we connect the GPS pose vertex to the front-end pose vertex through a special edge, constraining the translation amount of the visual pose and GPS pose on the same keyframe to be the same. The error only concerns the translation amount of the pose, defined as follows:

[0118] e i =|t vi -t gi |

[0119] Considering that in practical systems, due to sensor delays, timestamp misalignment, and other reasons, the timestamps of GPS information and image information may not be perfectly aligned, or even have significant deviations in some cases, this invention adds edges between adjacent GPS vertices as constraints to prevent such situations from affecting the optimization results. The type of these edges is the same as that between visual vertices. Finally, by constructing such pairwise connected pose graphs, the error of all edges in the graph is minimized, thereby achieving optimization of the poses of all vertices, i.e., the global pose.

[0120] Step S7: The optimized trajectory and the depth map obtained by SCCVA-MVSNet are fused to obtain a map.

[0121] After obtaining the camera pose and depth map for each keyframe, we need to fuse the results. To reduce filtering time, this invention employs a five-frame filtering method. For points corresponding to two frames, we calculate:

[0122] p j =π j (R i→j φ(p i ,d i ,K i )+t i→j ,K)

[0123] Where φ(p) i ,d i ,K i ) is the homogeneous coordinate p i The pixels in the given estimated depth d i 3D point inverse projection, π j (p iK) represents the projection of a 3D point onto the image plane, R i→j , t i→j This represents the rotation and translation from the source frame to the target frame, where K is an inherent characteristic of the camera. The forward reprojection error ||p is calculated by considering the geometric consistency between the two views. i -p j || 2 and depth error ||d i -d j || 2 , where d j The depth in reference frame j is used to find consistent pixel clusters across multiple frames for fusion, where the pixel clusters satisfy the reprojection error. And depth error ||d i -d j || 2 <ε constraint, the fusion point is the median of the cluster elements. The median position is used to avoid artifacts when averaging multiple adjacent pixels with large depth discontinuities. The final image of the constructed map is shown below. Figure 8 As shown, further 3D surface reconstruction can also be performed (such as...). Figure 9 (As shown).

[0124] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1.A method for real-time dense mapping of unmanned aerial vehicle based on visual SLAM and deep learning, characterized in that, The method comprises the following steps: S1, calibrate camera parameters and correct camera distortion; S2, acquire image frames and GPS information; S3, track the acquired image frames using a visual odometer; S4, perform sliding window optimization on the tracking results; S5, perform depth completion on the sliding window optimization results through an SCCVA-MVSNet network; The SCCVA-MVSNet network comprises two parts, a first part of a depth completion network and a second part of a CVA-MVSNet network; The input of the first part of the depth completion network is a sparse depth map formed by mutual projection of feature points of all key frames in all frames and photometric errors of each feature point; The input of the second part of the CVA-MVSNet network is spliced from the pictures in the sliding window and the output of the first part of the depth completion network, and the output of the second part of the CVA-MVSNet network is a depth map of the latest key frame; S6, optimize the trajectory optimized by the sliding window using GPS information; The step S6 specifically comprises the following steps: S61, calculate the error of the edge between the vertices in the graph by the relative pose obtained through the sliding window optimization and the relative pose calculated based on the global pose, and take the 7-dimensional Lie algebra of the error as the edge error; S62, add a new vertex to integrate the GPS pose into the pose graph, connect the GPS pose vertex and the pose vertex in the graph through a special edge, and constrain the translation of the visual pose and the GPS pose on the same key frame to be the same and minimize the error of all edges in the graph to optimize all vertex poses; S7, fuse the optimized trajectory and the depth map obtained by the SCCVA-MVSNet to obtain a map; The step S7 specifically comprises the following steps: S71, calculate the corresponding points of the vertex pose and the depth map of the key frame: ； wherein is the homogeneous coordinate of a pixel in the image inverse projection of a 3D point of given estimated depth denotes the projection of the 3D point on the image plane, is the rotation and translation from the source frame to the target frame, is the intrinsic properties of the camera; S72, compute forward re-projection error for geometric consistency between two views and depth error where is the depth in the reference frame ​ S73, finding consistent pixel clusters in multiple frames to fuse, the pixel clusters satisfying reprojection error and depth error constraints. 2.The visual SLAM and deep learning based real-time dense mapping method for UAVs according to claim 1, characterized in that, The step S1 specifically comprises the following steps: S11, project the three-dimensional space point P (X, Y, Z) of the camera coordinate system to the normalized image plane to form the normalized coordinates of the point as ; S12, normalizing the points on the plane Radial and tangential distortion correction is performed by the following equations: ； wherein is the corrected point coordinate, p1, p2 are tangential distortion coefficients of the camera, k1, k2, k3 are radial distortion coefficients of the camera, and r is the distance of the point P from the origin of the coordinate system. S13, the corrected point By the intrinsic parameter matrix, project to the pixel plane to get its correct position on the image , by the following formula: ； Wherein, fx, fy, cx, cy are the internal parameters of the camera. 3.The visual SLAM and deep learning based real-time dense mapping method for UAVs according to claim 1, characterized in that, The image frames and GPS information are acquired from a storage function module of the unmanned aerial vehicle. 4.The visual SLAM and deep learning based real-time dense mapping method for UAV according to claim 1, wherein, The step S3 specifically comprises the following steps: S31, construct an image pyramid for the acquired image frames, calculate the gradient of each layer, and select a certain number of points at each layer using a dynamic grid strategy; S32, generate an initial position through camera operation, that is, generate an estimate of the current frame motion based on the motion amount of the last time plus a fine adjustment; S33, calculate the corresponding residual according to the estimate, and calculate the Jacobian matrix of the 6-dimensional Lie algebra based on the residual and the gradient, and then iteratively optimize the estimated pose; S34, determine whether the current frame is a key frame according to the position relationship, and if so, add it to the sliding window for optimization. 5.The visual SLAM and deep learning based real-time dense mapping method for UAVs according to claim 4, characterized in that, The step S4 specifically comprises the following steps: S41, determine which frames need to be marginalized and mark according to the degree of co-visibility of the key frame about to be added to the sliding window and the old key frame and the position distribution of the key frames in the sliding window; S42, add the latest key frame to the sliding window and construct a residual term, and perform iterative optimization to optimize all key frames and active points in the sliding window; S43, marginalize the marked frames and the active points carried by the frames at the same time. S44, select certain key points in the latest key frame as preselection.

Images