Method and system for generating dense global point cloud images through depth completion

Abstract

The present invention relates to a method and system for generating a dense global point cloud map, which obtains a denser global point cloud map by utilizing an image-guided depth completion method, extracts image, point cloud and pose information, converts the image point cloud into a sparse depth map, extracts color information from the initial image to predict the depth, and outputs a color branch dense depth map; extracts depth information from the sparse depth map of each point cloud, and outputs a depth branch dense depth map; and fuses the color branch dense depth map and the depth branch dense depth map to obtain more accurate depth and fine texture dense depth information in a short time, thereby generating a dense global point cloud map.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing, and in particular to a method and system for generating a dense global point cloud image through image-guided depth completion. Background Art

[0002] Three-dimensional (3D) city models are indispensable in a wide range of applications such as digital cities, urban planning and geographic mapping, urban operation simulation, smart cities and environmental protection. The reconstruction methods of 3D city models include using satellite remote sensing images or drone remote sensing images, but there are many disadvantages: cumbersome planning stages, limited resolution and time-consuming reconstruction process, which are inconvenient.

[0003] LiDAR plays a key role in 3D city reconstruction, and the synergy of LiDAR, cameras, and drones will become the ultimate viable technology for real-time reconstruction of 3D city models with accurate geometry and detailed textures in the future. Driven by the latest advances in solid-state technology, solid-state LiDAR can provide many more data points than traditional mechanical LiDAR by using non-repeating scanning patterns. New and exciting applications are emerging with the support of this latest solid-state LiDAR (such as Livox) at lower costs and higher performance. For example, LiDAR odometry and mapping (LOAM), one of the most popular LiDAR simultaneous localization and mapping (SLAM) implementations, was recently ported to Livox. Based on this LOAM adapted to Livox, a LiDAR SLAM was developed for Livox Horizon. In addition, CamVox integrates Livox and a monocular camera into an RGB-D camera, and proposes a modified ORB-SLAM2 for the new RGB-D camera. Alternatively, the scanning time of the LiDAR can be increased to generate dense point clouds.

[0004] However, practical applications are limited in processing time. For example, in the reconstruction of 3D models of post-disaster environments, scanning must be completed in a short time because the post-disaster environment is unstable or even dangerous. Under this limitation, it is feasible to improve the reconstruction of 3D models by utilizing dense depth information. Accurate image-guided depth completion will be able to better utilize images taken by the camera to generate dense depth information. For example, PENet (Precise and Effective Image-Guided Depth Completion) is one of the most mature depth completion methods. In PENet, the KITTI dataset is used to train the neural network, and the depth information in KITTI is collected by a Velodyne lidar with a limited field of view. Recently, DenseLivox provided a new dataset collected by Livox lidar. The new dataset is said to be able to obtain dense depth truth values ​​that cover 88.3% of the field of view, far exceeding KITTI, which makes the evaluation of depth completion methods have more accurate results.

[0005] Therefore, how to obtain more accurate depth and fine texture in a short time, how to generate a dense global point cloud map and avoid the risks that exist in practice are urgent problems to be solved. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for obtaining a denser global point cloud image by using an image-guided depth completion method.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for generating a dense global point cloud map, comprising the following steps:

[0008] S1. Collect multiple initial images and their corresponding point clouds of the target that needs to generate a dense global point cloud map;

[0009] S2. Cloud computing and correcting each point to obtain its corresponding posture information;

[0010] S3. Each point cloud is converted into a sparse depth map;

[0011] S4. Extract color information from the initial image and predict depth, and output a color branch dense depth map;

[0012] Extract depth information from the sparse depth map of each point cloud and output a dense depth map of the depth branch;

[0013] S5. fuse the color branch dense depth map and the depth branch dense depth map of each point cloud using a confidence weight method to obtain multiple fused dense depth maps of the target;

[0014] S6. adding color information to each fused dense depth map by using a texture rendering method to generate a color point cloud frame and its pose information;

[0015] S7. According to each obtained color point cloud frame and its pose information, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

[0016] Furthermore, the step S2 adopts an iterative posture optimization method to update the posture information.

[0017] Further, the step S4 includes:

[0018] S41) constructing an encoder-decoder network of a color branch to extract color features from an initial image of the target;

[0019] S42) Depth prediction is performed based on color features, and a color branch dense depth map D is output. cp , Confidence map C cp And output steps;

[0020] S43) constructing an encoder-decoder network of a deep branched encoder-decoder network;

[0021] S44) Upsample the sparse depth map of each point cloud and output a dense depth map D of the depth branch dp , Confidence map C dp And output the steps.

[0022] Furthermore, the step S5 comprises:

[0023] S51) Calculate the depth data D of each point (u, v) in the depth map using the following formula (1): fp (u,v);

[0024]

[0025] S52) Output the target fused dense depth map.

[0026] Further, the step S6 includes:

[0027] S61) converting each sampling point into a point cloud coordinate system according to the depth information and coordinates of the dense depth map;

[0028] S62) adding (r, g, b) color information to the point cloud;

[0029] S63) Match the generated point cloud with the initial image to obtain a color point cloud frame of the target.

[0030] The present invention also provides a system for generating a dense global point cloud image, including an acquisition module, a depth completion module, and a texture rendering module;

[0031] The acquisition module comprises:

[0032] Acquisition unit: collects multiple initial images and their corresponding point clouds required to generate dense global point cloud map targets;

[0033] Posture information unit: Cloud computing and correction of each point get its corresponding posture information;

[0034] The depth completion module includes:

[0035] Depth map conversion unit: Each point cloud is converted into a sparse depth map;

[0036] Color depth branch output unit: extracts color information from the initial image and predicts the depth, and outputs a color branch dense depth map;

[0037] Extract depth information from the sparse depth map of each point cloud and output a dense depth map of the depth branch;

[0038] Fusion unit: The confidence weight method is used to fuse the color branch dense depth map and the depth branch dense depth map of each point cloud to obtain multiple fused dense depth maps of the target;

[0039] The texture rendering module comprises:

[0040] Color rendering unit: Use texture rendering method to add color information to each fused dense depth map to generate color point cloud frame and its pose information;

[0041] Global point cloud map generation unit: According to each color point cloud frame and its pose information obtained, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

[0042] Furthermore, the acquisition module adopts an iterative posture optimization method to update the posture information.

[0043] Furthermore, the color depth branch output unit specifically includes:

[0044] Color extraction subunit: constructs an encoder-decoder network of the color branch to extract color features from the initial image of the target;

[0045] Color branch output subunit: performs depth prediction based on color features and outputs a color branch dense depth map D cp , Confidence map C cp And output steps;

[0046] Deep information extraction subunit: an encoder-decoder network that constructs a deep branched encoder-decoder network;

[0047] Depth branch output subunit: Upsample the sparse depth map of each point cloud and output the dense depth map D of the depth branch dp, Confidence map C dp And output the steps.

[0048] Furthermore, the fusion unit specifically includes: using the following formula (1) to calculate the depth data D of each point (u, v) in the depth map: fp (u,v);

[0049]

[0050] Output target fused dense depth map.

[0051] Furthermore, the color rendering unit specifically includes:

[0052] Conversion point cloud subunit: According to the depth information and coordinates of the dense depth map, each sampling point is converted into a point cloud coordinate system;

[0053] Color adding subunit: add (r,g,b) color information to the point cloud;

[0054] Color point cloud frame generation subunit: matches the generated point cloud with the initial image to obtain the target color point cloud frame.

[0055] The present invention provides a method for generating a dense global point cloud map. The method obtains a denser global point cloud map by utilizing an image-guided depth completion method, extracts image, point cloud and posture information, converts the image point cloud into a sparse depth map, extracts color information from the initial image to predict the depth, and outputs a color branch dense depth map; extracts depth information from the sparse depth map of each point cloud, and outputs a depth branch dense depth map; fuses the color branch dense depth map and the depth branch dense depth map to obtain a dense depth map of the target, adds color information through texture rendering, generates a colored point cloud frame, thereby adding posture information and generating a dense global point cloud map, and obtains more accurate depth and fine texture dense depth information in a short time, thereby generating a dense global point cloud map.

[0056] The present invention also provides a system for generating a dense global point cloud map, including an acquisition module, a depth completion module, and a texture rendering module. The acquisition module acquires the required initial image and its corresponding point cloud and posture information; the depth completion module extracts color information to predict depth, extracts color information to predict depth on the initial image, and outputs a color branch dense depth map; extracts depth information from the sparse depth map of each point cloud, and outputs a depth branch dense depth map; the color branch dense depth map and the depth branch dense depth map of each point cloud are fused; the texture rendering module adds color information to generate a colored point cloud frame, thereby adding posture information to generate a dense global point cloud map, and more accurate depth and fine texture dense depth information are obtained in a short time, thereby generating a dense global point cloud map and improving the reconstruction of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the present invention;

[0058] Figure 2 This is a network structure diagram of the present invention. DETAILED DESCRIPTION

[0059] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0060] The most critical concept of the present invention is: using an image-guided depth completion method to obtain a denser global point cloud map method, extracting image, point cloud and pose information, converting the image point cloud into a sparse depth map, extracting color information from the initial image to predict the depth, and outputting a color branch dense depth map; extracting depth information from the sparse depth map of each point cloud, and outputting a depth branch dense depth map; fusing the color branch dense depth map and the depth branch dense depth map to obtain a dense depth map of the target, adding color information through texture rendering, and generating a colored point cloud frame, thereby adding pose information and generating a dense global point cloud map.

[0061] The present invention proposes a method for generating a dense global point cloud map.

[0062] S1. Collect multiple initial images and their corresponding point clouds of the target that needs to generate a dense global point cloud map;

[0063] S2. Cloud computing and correcting each point to obtain its corresponding posture information;

[0064] S3. Each point cloud is converted into a sparse depth map;

[0065] S4. Extract color information from the initial image to predict depth, and output a dense depth map of the color branch; extract depth information from the sparse depth map of each point cloud, and output a dense depth map of the depth branch;

[0066] S5. fuse the color branch dense depth map and the depth branch dense depth map of each point cloud using a confidence weight method to obtain multiple fused dense depth maps of the target;

[0067] S6. adding color information to each fused dense depth map by using a texture rendering method to generate a color point cloud frame and its pose information;

[0068] S7. According to each obtained color point cloud frame and its pose information, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

[0069] During S1 acquisition, the solid-state LiDAR and the monocular camera are combined into an integrated fusion sensor system. The monocular camera acquires the initial image, the LiDAR acquires point cloud information, and the LiDAR-IMU odometer uses the method provided in the LiDAR SLAM simultaneous localization and mapping to track and estimate positioning information, such as posture and direction. The IMU is integrated into the LiDAR and the monocular camera as input. When obtaining posture information in S2, the LiDAR-IMU odometer uses the point cloud acquired by the LiDAR to calculate the posture, and the IMU data is used to correct the posture to make it more accurate.

[0070] In step S4, a dual-branch backbone network is designed, which includes a color branch and a depth branch. The encoder-decoder network of the color branch extracts features from the initial image of the target and outputs a dense depth map D of the color branch. cp , Confidence map C cp And output; build a deep branch encoder-decoder network, upsample the sparse depth map of each point cloud, extract features, and the deep branch dense depth map D dp , Confidence map C dp And output.

[0071] The overall structure of this network is as follows Figure 2 As shown in Figure 2. The color branch mainly uses color information to predict the dense depth map, while the depth branch relies more on depth information. Finally, the two depth maps are fused through the learned confidence weights to render the texture of the dense point cloud by obtaining the color information of each point.

[0072] Completion From the above description, it can be seen that the beneficial effect of the present invention is that a dense global point cloud map is generated by embedding image-guided depth completion into LiDAR SLAM (livox_horizon_loam), thereby obtaining a denser three-dimensional color map. By utilizing the fusion of LiDAR-IMU-Camera, the reconstructed three-dimensional color model obtained is proved to be more accurate and more informative, that is, accurate depth and fine texture. In addition, compared with the SLAM method, due to the addition of image-guided depth completion under a limited scanning time window, this method can generate a denser global point cloud map. The experimental results also confirm that this method is very suitable for fast and dense three-dimensional color reconstruction, and can be applied to a wide range of fields such as digital cities, urban planning and geographic surveying and mapping.

[0073] Embodiment 1:

[0074] The step S2 adopts an iterative posture optimization method to update the posture information.

[0075] The solid-state LiDAR is used to obtain pose and orientation information using the methods provided in livox_horizon_loam. In livox_horizon_loam, the odometer uses an iterative pose optimization method to estimate the pose of the LiDAR. When the operator performs feature extraction and selection, it uses the iterative closest point (ICP) edge and plane algorithms to extract surface and line features, respectively. Let Pi be the i-th closest point of Pw, from the feature point set of the w-th frame. The edge-to-edge and plane-to-plane residuals are calculated according to formula (5) and formula (6). Using the inertial measurement unit (IMU) inside Livox Avia, the odometer can eliminate rotational distortion.

[0076]

[0077]

[0078] Embodiment 2:

[0079] The step S4 comprises:

[0080] S41) constructing an encoder-decoder network of a color branch to extract color features from an initial image of the target;

[0081] S42) Depth prediction is performed based on color features, and a color branch dense depth map D is output. cp , Confidence map C cp And output steps;

[0082] S43) constructing an encoder-decoder network of a deep branched encoder-decoder network;

[0083] S44) Upsample the sparse depth map of each point cloud and output a dense depth map D of the depth branch dp , Confidence map C dp And output the steps.

[0084] Based on ENet, a dual-branch backbone network is designed, which includes a color branch and a depth branch. The overall structure of this network is as follows Figure 2 As shown in Figure 2. The color branch mainly uses color information to predict a dense depth map, while the depth branch relies more on depth information. Finally, the two depth maps are fused together through the learned confidence weights.

[0085] The color branch predicts a dense depth map from the color image by leveraging the sparse depth map. In particular, this branch has an encoder-decoder network with symmetric skip connections. There is one convolutional layer and 10 residual blocks (ResBlocks) in the encoder, while the decoder contains five deconvolutional layers and one convolutional layer. In addition, the output of each convolutional layer is processed by a BN layer and a ReLU activation layer. The color branch extracts features from the color image so that the depth of the object edge can be predicted.

[0086] The depth branch is mainly used to create a denser depth map by upsampling the sparse depth map. This branch has a similar encoder-decoder network. In addition, the decoder features of the color branch are fused into this branch and connected with the corresponding encoder features of this branch. The results of the color branch are also input to help prediction. Example 3:

[0087] The step S5 comprises:

[0088] S51) Calculate the depth data D of each point (u, v) after fusion using the following formula (1): fp (u,v); follow the same strategy of ENet to fuse the output results of the two branches;

[0089]

[0090] S52) Output the target fused dense depth map.

[0091] Color branch dense depth map D cp , Confidence map C cp , the depth branch dense depth map D dp , Confidence map C dp , (u, v) represents the coordinates of a sampling point in the depth map, and follows the same strategy of ENet to fuse the output results of the two branches: the color branch dense depth map D cp , Confidence map C cp , the depth branch dense depth map D dp , Confidence map C dp . Thus, the two sparse depth maps are merged into a dense depth map. fp is the fused depth map, D gt (the dataset in denselivox) is the true value used for supervision, and the following loss function formula (4) is used during training:

[0092] L(D fp )=‖D fp -D gt ‖ 2 ,D gt >0 (4)

[0093] Data points without valid ground-truth depth values ​​are discarded during training.

[0094] Embodiment 4:

[0095] The step S6 comprises:

[0096] S61) converting each sampling point into a point cloud coordinate system according to the depth information and coordinates of the dense depth map;

[0097] S62) adding (r, g, b) color information to the point cloud;

[0098] S63) Match the generated point cloud with the initial image to obtain a color point cloud frame of the target.

[0099] Wherein, in S61), a method for obtaining a dense point cloud by converting a dense depth map into a point cloud is as follows: for a given coordinate Y=(u,v,1) T And the depth D of a point in the image coordinate system, the coordinates of the point in the point cloud coordinate system can be calculated using formula (5) and formula (6) X = (X, Y, Z, 1) T .

[0100]

[0101] DY=PYX (6)

[0102] In S62), each point in the dense point cloud is represented by the coordinates (x, y, z), and (r, g, b) color information is added to each point in the point cloud. The obtained data points are recorded in the format of (x, y, z, r, g, b). In order to obtain accurate color information, the lidar and camera collect data synchronously at a frequency of 10 Hz. Therefore, all data in the point cloud and image have timestamps, and the dense point cloud inherits the timestamp of the original point cloud. The point cloud and image with similar timestamps are matched by using the nearest time registration algorithm.

[0103] In S63), using Formula 1 and Formula 2, the texture rendering system matches each pixel in the image with a data point in the point cloud, and then adds color information (r, g, b) to the matched data point. After rendering the textures of all points in a frame, a colored point cloud frame is obtained. By applying this rendering process to each pair of point cloud and image, a dense colored point cloud frame sequence is obtained.

[0104] The present invention also proposes a system for generating a dense global point cloud map, including an acquisition module, a depth completion module, and a texture rendering module;

[0105] The acquisition module comprises:

[0106] Acquisition unit: collects multiple initial images and their corresponding point clouds required to generate dense global point cloud map targets;

[0107] Posture information unit: Cloud computing and correction of each point get its corresponding posture information;

[0108] The depth completion module includes:

[0109] Depth map conversion unit: Each point cloud is converted into a sparse depth map;

[0110] Color depth branch output unit: extracts color information from the initial image and predicts the depth, and outputs a color branch dense depth map;

[0111] Extract depth information from the sparse depth map of each point cloud and output a dense depth map of the depth branch;

[0112] Fusion unit: The confidence weight method is used to fuse the color branch dense depth map and the depth branch dense depth map of each point cloud to obtain multiple fused dense depth maps of the target;

[0113] The texture rendering module comprises:

[0114] Color rendering unit: Use texture rendering method to add color information to each fused dense depth map to generate color point cloud frame and its pose information;

[0115] Global point cloud map generation unit: According to each color point cloud frame and its pose information obtained, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

[0116] Embodiment 5: The acquisition module adopts an iterative posture optimization method to update the posture information.

[0117] The solid-state LiDAR is used to obtain pose and orientation information using the methods provided in livox_horizon_loam. In livox_horizon_loam, the odometer uses an iterative pose optimization method to estimate the pose of the LiDAR. When the operator performs feature extraction and selection, it uses the Iterative Closest Point (ICP) edge and plane algorithms to extract surface and line features, respectively. Let Pi be the i-th closest point of Pw, from the feature point set of the w-th frame. The edge-to-edge and plane-to-plane residuals are calculated according to formula (2) and formula (3). Using the inertial measurement unit (IMU) inside Livox Avia, the odometer can eliminate rotational distortion.

[0118]

[0119]

[0120] Embodiment 6: The color depth branch output unit specifically includes:

[0121] Color extraction subunit: constructs an encoder-decoder network of the color branch to extract color features from the initial image of the target;

[0122] Color branch output subunit: performs depth prediction based on color features and outputs a color branch dense depth map D cp , Confidence map C cp And output steps;

[0123] Deep information extraction subunit: an encoder-decoder network that constructs a deep branched encoder-decoder network;

[0124] Depth branch output subunit: Upsample the sparse depth map of each point cloud and output the dense depth map D of the depth branch dp , Confidence map C dp And output the steps.

[0125] Based on ENet, a dual-branch backbone network is designed, which includes a color branch and a depth branch. The overall structure of this network is as follows Figure 2 As shown in Figure 2. The color branch mainly uses color information to predict a dense depth map, while the depth branch relies more on depth information. Finally, the two depth maps are fused together through the learned confidence weights.

[0126] The color branch predicts a dense depth map from the color image by leveraging the sparse depth map. In particular, this branch has an encoder-decoder network with symmetric skip connections. There is one convolutional layer and 10 residual blocks (ResBlocks) in the encoder, while the decoder contains five deconvolutional layers and one convolutional layer. In addition, the output of each convolutional layer is processed by a BN layer and a ReLU activation layer. The color branch extracts features from the color image so that the depth of the object edge can be predicted.

[0127] The depth branch is mainly used to create a denser depth map by upsampling the sparse depth map. This branch has a similar encoder-decoder network. In addition, the decoder features of the color branch are fused into this branch and concatenated with the corresponding encoder features of this branch. The result of the color branch is also input to help prediction.

[0128] Embodiment 7:

[0129] The fusion unit specifically includes: calculating the depth data D of each point (u, v) in the depth map fp(u,v);

[0130]

[0131] Output target fused dense depth map.

[0132] Color branch dense depth map D cp , Confidence map C cp , the depth branch dense depth map D dp , Confidence map C dp , (u,v) represents the coordinates of a sampling point in the depth map, and follows the same strategy of ENet to fuse the output results of the two branches: the color branch dense depth map D cp , Confidence map C cp , the depth branch dense depth map D dp , Confidence map C dp . Thus, the two sparse depth maps are merged into a dense depth map. fp is the fused depth map, D gt (the dataset in denselivox) is the true value used for supervision, and the following loss function is used during training:

[0133] L(D fp )=‖D fp -D gt ‖ 2 ,D gt >0 (4)

[0134] Data points without valid ground-truth depth values ​​are discarded during training.

[0135] Embodiment 8:

[0136] The color rendering unit specifically includes:

[0137] Conversion point cloud subunit: According to the depth information and coordinates of the dense depth map, each sampling point is converted into a point cloud coordinate system;

[0138] Color adding subunit: add (r,g,b) color information to the point cloud;

[0139] Color point cloud frame generation subunit: matches the generated point cloud with the initial image to obtain the target color point cloud frame.

[0140] Among them, the method of obtaining a dense point cloud by converting a dense depth map into a point cloud, for a given coordinate Y = (u, v, 1)^T and the depth D of a point in the image coordinate system, the coordinate of the point in the point cloud coordinate system can be calculated using the formula X = (X, Y, Z, 1) T .

[0141]

[0142] DY=PYX (6)

[0143] Each point in the dense point cloud is represented by the coordinates (x, y, z), and (r, g, b) color information is added to each point in the point cloud. The resulting data points are recorded in the format of (x, y, z, r, g, b). In order to obtain accurate color information, the lidar and camera collect data synchronously at a frequency of 10 Hz. Therefore, all data in the point cloud and image have timestamps, and the dense point cloud inherits the timestamp of the original point cloud. The point cloud and image with similar timestamps are matched by using the nearest time registration algorithm.

[0144] The texture rendering system matches each pixel in the image with a data point in the point cloud, and then adds color information (r, g, b) to the matched data point. After rendering the textures of all points in a frame, a colored point cloud frame is obtained. By applying this rendering process to each pair of point cloud and image, a sequence of dense colored point cloud frames is obtained.

[0145] Embodiment 9:

[0146] The present invention proposes a method for generating a dense global point cloud map, which is a sensor fusion system composed of a solid-state laser radar, an inertial measurement unit (IMU) and a monocular camera. The monocular camera obtains the initial image, the laser radar obtains the point cloud information, and the LiDAR-IMU odometer uses the method provided in the laser radar SLAM simultaneous localization and mapping to track and estimate the positioning information, such as posture and direction. The IMU is integrated in the laser radar and the monocular camera as input. When obtaining the posture information, the LiDAR-IMU odometer uses the point cloud collected by the laser radar to calculate the posture, and the IMU data is used to correct the posture to make it more accurate.

[0147] The method provided in livox_horizon_loam is used to obtain the pose and direction information using a solid-state lidar. In livox_horizon_loam, the odometer uses an iterative pose optimization method to estimate the pose of the lidar. When the operator performs feature extraction and selection, it uses the iterative closest point (ICP) edge and plane algorithms to extract surface and line features respectively. Let Pi be the i-th closest point of Pw, from the feature point set of the w-th frame. It can be obtained according to formula (2) formula (3)

[0148]

[0149]

[0150] The edge-to-edge and plane-to-plane residuals are calculated. Using the inertial measurement unit (IMU) inside Livox Avia, the odometry is able to eliminate rotational distortion.

[0151] Each collected point cloud is projected onto the image plane and converted into a sparse depth map. A dual-branch backbone network is designed, which includes a color branch and a depth branch. The color branch predicts a dense depth map from the color image by utilizing the sparse depth map. In particular, this branch has an encoder-decoder network with symmetric skip connections. There is a convolutional layer and 10 residual blocks (ResBlocks) in the encoder, while the decoder contains five deconvolutional layers and one convolutional layer. In addition, the output of each convolutional layer is processed by a BN layer and a ReLU activation layer. The color branch extracts features in the color image so that the depth of the object edge can be predicted.

[0152] The depth branch is mainly used to create a denser depth map by upsampling the sparse depth map. This branch has a similar encoder-decoder network. In addition, the decoder features of the color branch are fused into this branch and they are concatenated with the corresponding encoder features of this branch. In addition, the results of the color branch are also input to help prediction. The dense depth map D of the color branch is obtained. cp , Confidence map C cp , the depth branch dense depth map D dp , Confidence map C dp And output.

[0153] Follow the same strategy of ENet to fuse the output results of the two branches and calculate the depth data D of each point (u,v) in the fused depth map fp (u,v);

[0154]

[0155] Thus, the two sparse depth maps are merged into a dense depth map. fp is the fused depth map, D gt (for the denselivox dataset) is the true value used for supervision. Data points without valid true depth values ​​are removed during training. The following loss function is used during training:

[0156] L(D fp )=‖D fp -D gt ‖ 2 ,D gt >0 (4)

[0157] The method of converting a dense depth map into a point cloud to obtain a dense point cloud, for a given coordinate Y = (u, v, 1)^T and the depth D of a point in the image coordinate system, the coordinate X = (X, Y, Z, 1) of the point in the point cloud coordinate system can be calculated using formula (5) and formula (6) T .

[0158]

[0159] DY=PYX (6)

[0160] Each point in the dense point cloud is represented by the coordinates (x, y, z), and (r, g, b) color information is added to each point in the point cloud. The resulting data points are recorded in the format of (x, y, z, r, g, b). In order to obtain accurate color information, the lidar and camera collect data synchronously at a frequency of 10 Hz. Therefore, all data in the point cloud and image have timestamps, and the dense point cloud inherits the timestamp of the original point cloud. The point cloud and image with similar timestamps are matched by using the nearest time registration algorithm.

[0161] The texture rendering system matches each pixel in the image with a data point in the point cloud, and then adds color information (r, g, b) to the matched data point. After rendering the textures of all points in a frame, a colored point cloud frame is obtained. By applying this rendering process to each pair of point cloud and image, a dense colored point cloud frame sequence and its pose information are obtained. According to each obtained colored point cloud frame and its pose information, all colored point cloud frames are spliced ​​in the coordinate system to generate the target dense global point cloud map.

[0162] Embodiment 10:

[0163] In addition, CamVox (a low-cost, high-precision lidar-assisted visual SLAM system) integrates Livox and a monocular camera into an RGB-D camera, and proposes an improved ORB-SLAM2 for the new RGB-D camera. In addition, the scanning time of the lidar can also be increased to generate dense point clouds. Accurate image-guided depth completion will be able to better utilize the images taken by the camera to generate dense depth information.

[0164] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating a dense global point cloud map, characterized in that: The steps include: S1, collecting multiple initial images and their corresponding point clouds of the target that need to generate a dense global point cloud map; S2, perform cloud computing correction on each point to obtain its corresponding posture information; S3, each point cloud is converted into a sparse depth map; S4. Predict the depth of the color information of the initial image and output a dense depth map of the color branch; extract the depth information of the sparse depth map of each point cloud and output a dense depth map of the depth branch: S41), constructing an encoder-decoder network of a color branch to extract color features of the initial image of the target; S42), perform depth prediction based on color features and output a color branch dense depth map , Confidence map ; S43), constructing an encoder-decoder network of a deep branched encoder-decoder network; S44) Upsample the sparse depth map of each point cloud and output a dense depth map of the depth branch , Confidence map ; S5. Use the confidence weight method to fuse the color branch dense depth map and the depth branch dense depth map of each point cloud to obtain multiple fused dense depth maps of the target: S51), calculate the depth data of each point (u, v) in the depth map ; S52), outputting the dense depth map after target fusion; S6, adding color information to each fused dense depth map using a texture rendering method to generate a color point cloud frame and its pose information; S7. According to each obtained color point cloud frame and its pose information, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

2. The method for generating a dense global point cloud map according to claim 1, characterized in that: The step S2 adopts an iterative posture optimization method to update the posture information.

3. A method for generating a dense global point cloud map according to claim 2, characterized in that: The step S6 comprises: S61), according to the depth information and coordinates of the dense depth map, convert each sampling point into a point cloud coordinate system; S62), adding (r, g, b) color information to the point cloud; S63), matching the generated point cloud with the initial image to obtain a color point cloud frame of the target.

4. A system for generating a dense global point cloud map, characterized in that: Including acquisition module, depth completion module, texture rendering module; The acquisition module comprises: Acquisition unit: collects multiple initial images and their corresponding point clouds required to generate dense global point cloud map targets; Posture information unit: Cloud computing and correction of each point get its corresponding posture information; The depth completion module includes: Depth map conversion unit: Each point cloud is converted into a sparse depth map; A color depth branch output unit: performs predicted depth extraction on the color information of the initial image and outputs a color branch dense depth map; the color depth branch output unit specifically includes: Color extraction subunit: constructs an encoder-decoder network of the color branch to extract color features from the initial image of the target; Color branch output subunit: performs depth prediction based on color features and outputs a dense depth map of the color branch , Confidence map ; Deep information extraction subunit: an encoder-decoder network that constructs a deep branched encoder-decoder network; Depth branch output subunit: Upsample the sparse depth map of each point cloud and output the dense depth map of the depth branch , Confidence map ; Extract depth information from the sparse depth map of each point cloud and output a dense depth map of the depth branch; Fusion unit: The color branch dense depth map and the depth branch dense depth map of each point cloud are fused by the confidence weight method to obtain multiple fused dense depth maps of the target; the fusion unit specifically includes: calculating the depth data of each point (u, v) in the depth map ; Output the target fused dense depth map; The texture rendering module comprises: Color rendering unit: Use texture rendering method to add color information to each fused dense depth map to generate color point cloud frame and its pose information; Global point cloud map generation unit: According to each color point cloud frame and its pose information obtained, all color point cloud frames are spliced ​​in the coordinate system to generate a dense global point cloud map of the target.

5. A system for generating a dense global point cloud map according to claim 4, characterized in that: The acquisition module adopts an iterative posture optimization method to update the posture information.

6. A system for generating a dense global point cloud map according to claim 5, characterized in that: The color rendering unit specifically includes: Conversion point cloud subunit: According to the depth information and coordinates of the dense depth map, each sampling point is converted into a point cloud coordinate system; Color adding subunit: add (r,g,b) color information to the point cloud; Color point cloud frame generation subunit: matches the generated point cloud with the initial image to obtain the target color point cloud frame.

Images