Depth completion method and device based on point cloud densification

Abstract

The invention relates to the technical field of automatic driving, in particular to a depth completion method and device based on point cloud densification, and the method comprises the steps: obtaining laser point cloud data of a target scene and a target image under the same timestamp as the laser point cloud data, and preprocessing the laser point cloud data and the target image; based on the preprocessed laser point cloud data and the target image, constructing a target 3D Gaussian sputtering model; and based on the target scene and the target 3D Gaussian sputtering model, outputting the dense point cloud after three-dimensional reconstruction to obtain a point cloud densified three-dimensional reconstruction point cloud image of the target scene, and determining a camera depth image of the target scene according to the three-dimensional reconstruction point cloud image. According to the method and the device, the target 3D Gaussian sputtering model can be constructed by using the laser point cloud data and the images of multiple visual angles to complete depth completion, so that the obtained camera depth map is fine and vivid, errors caused by calibration of display equipment are reduced, and the generalization capability is relatively high.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a depth completion method and device based on point cloud densification. Background Art

[0002] Deep completion plays an important role in computing platforms and development systems (CBDES), data closure, unsupervised automatic labeling, and dense reconstruction. It relies on the support of a powerful computing platform, obtains feedback on the repair results through data closure, accelerates model training and optimization with unsupervised automatic labeling, and provides richer image information with the help of dense reconstruction, thereby achieving high-quality repair and generation of image data. The development of deep completion technology has also promoted the continuous innovation and improvement of technologies such as computing platforms, data closure, unsupervised automatic labeling, and dense reconstruction, and jointly promoted the progress in the fields of machine learning and computer vision.

[0003] In the related technologies, depth completion and point cloud densification methods and corresponding technologies are all based on projecting the laser point cloud onto the camera and then using deep learning technology to complete the depth map and perform back-projection.

[0004] However, the depth completion and point cloud densification methods in related technologies are sensitive to hardware calibration errors and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar. At the same time, the current point cloud densification still has problems such as scene generalization, which need to be solved urgently. Summary of the invention

[0005] The present application provides a depth completion method, device, electronic device and storage medium based on point cloud densification to solve the problems that the depth completion and point cloud densification methods in the related technologies are sensitive to hardware calibration errors and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar. At the same time, the current point cloud densification also has problems such as scene generalization.

[0006] The first aspect of the present application provides a depth completion method based on point cloud densification, comprising the following steps: acquiring laser point cloud data of a target scene and a target image at the same timestamp as the laser point cloud data, preprocessing the laser point cloud data and the target image to obtain preprocessed laser point cloud data and the target image; constructing a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image; outputting a dense point cloud after three-dimensional reconstruction based on the target scene and the target 3D Gaussian sputtering model, so as to obtain a three-dimensional reconstructed point cloud map with densified point clouds of the target scene according to the three-dimensional reconstructed dense point cloud, and determining a camera depth map of the target scene according to the three-dimensional reconstructed point cloud map.

[0007] Optionally, in an embodiment of the present application, acquiring the laser point cloud data of the target scene and the target image at the same timestamp as the laser point cloud data, and preprocessing the laser point cloud data and the target image to obtain the preprocessed laser point cloud data and the target image includes: acquiring the RGB information of each three-dimensional point in the laser point cloud data according to the laser point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera internal parameters of each camera, so as to obtain the laser point cloud data with RGB information; converting the initial laser point cloud coordinates of the laser point cloud data with RGB information to the global coordinate system to obtain the converted laser point cloud data; calculating the normal vector of the converted laser point cloud data, and performing pose conversion on the pose corresponding to the target image in the converted laser point cloud data, so as to complete the preprocessing of the laser point cloud data and the target image.

[0008] Optionally, in an embodiment of the present application, constructing the target 3D Gaussian splatting model based on the preprocessed laser point cloud data and the target image includes: constructing a corresponding three-dimensional Gaussian distribution according to the preprocessed laser point cloud data to obtain a Gaussian ellipsoid under the three-dimensional Gaussian distribution; based on the target rasterizer and the Gaussian ellipsoid, performing block processing on the target image to obtain each image block and its corresponding 3D Gaussian model; according to the overlapping degree of the 3D Gaussian model of each image block with other image blocks except its corresponding image block, assigning corresponding key values to the 3D Gaussian model of each image block; based on the key values, performing Splatting splatting processing on the Gaussian ellipsoid blocks to construct the target 3D Gaussian splatting model.

[0009] Optionally, in an embodiment of the present application, constructing the target 3D Gaussian splatting model based on the preprocessed laser point cloud data and the target image further includes: acquiring the scale information of the 3D Gaussian model; correcting the 3D Gaussian model based on the scale information to construct the target 3D Gaussian splatting model.

[0010] Optionally, in an embodiment of the present application, the camera depth map includes at least one of the point cloud map and the projection map of the target scene.

[0011] The second aspect of the embodiments of the present application provides a depth completion device based on point cloud densification, including: a preprocessing module, configured to obtain the lidar point cloud data of a target scene and a target image at the same timestamp as the lidar point cloud data, and preprocess the lidar point cloud data and the target image to obtain the preprocessed lidar point cloud data and target image; a construction module, configured to construct a target 3D Gaussian sputtering model based on the preprocessed lidar point cloud data and target image; a reconstruction module, configured to output a densely sampled point cloud after three-dimensional reconstruction based on the target scene and the target 3D Gaussian sputtering model, to obtain a three-dimensional reconstruction point cloud map of the point cloud densification of the target scene according to the densely sampled point cloud after three-dimensional reconstruction, and determine the camera depth map of the target scene according to the three-dimensional reconstruction point cloud map.

[0012] Optionally, in an embodiment of the present application, the preprocessing module includes: a collection unit, configured to collect the RGB information of each three-dimensional point in the lidar point cloud data according to the lidar point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera internal parameters of each camera, so as to obtain the lidar point cloud data with RGB information; a conversion unit, configured to convert the initial lidar point cloud coordinates of the lidar point cloud data with RGB information into the global coordinate system to obtain the converted lidar point cloud data; a calculation unit, configured to calculate the normal vector of the converted lidar point cloud data, and perform pose conversion on the pose corresponding to the target image in the converted lidar point cloud data, so as to complete the preprocessing of the lidar point cloud data and the target image.

[0013] Optionally, in an embodiment of the present application, the construction module includes: a first construction unit, configured to construct a corresponding three-dimensional Gaussian distribution according to the preprocessed lidar point cloud data to obtain a Gaussian ellipsoid under the three-dimensional Gaussian distribution; a first allocation unit, configured to perform block processing on the target image based on a target rasterizer and the Gaussian ellipsoid to obtain each image block and its corresponding 3D Gaussian model; a second allocation unit, configured to assign corresponding key values to the 3D Gaussian models of each image block according to the overlapping degree of the 3D Gaussian model of each image block with other image blocks except its corresponding image block; a second construction unit, configured to perform Splatting sputtering processing on the Gaussian ellipsoid blocks based on the key values to construct the target 3D Gaussian sputtering model.

[0014] Optionally, in an embodiment of the present application, the construction module further includes: an acquisition unit, configured to acquire the scale information of the 3D Gaussian model; a correction unit, configured to correct the 3D Gaussian model based on the scale information to construct the target 3D Gaussian sputtering model.

[0015] Optionally, in an embodiment of the present application, the camera depth map includes at least one of a point cloud map and a projection map of the target scene.

[0016] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the depth completion method based on point cloud densification as described in the above embodiment.

[0017] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium storing a computer program, and when the program is executed by a processor, it implements the depth completion method based on point cloud densification as described above.

[0018] An embodiment of the fifth aspect of the present application provides a computer program product including a computer program, and when the computer program is executed, it is used to implement the depth completion method based on point cloud densification as described above.

[0019] Embodiments of the present application can use the preprocessed laser point cloud data and camera images to construct a target 3D Gaussian sputtering model, and then use the target 3D Gaussian sputtering model to obtain a three-dimensional reconstructed point cloud map, so as to determine the camera depth map of the target scene according to the three-dimensional reconstructed point cloud map. Thus, it realizes the depth completion of the image by using the information represented by the laser point cloud data and images from multiple perspectives to construct a target 3D Gaussian sputtering model, can better handle the effect of three-dimensional reconstruction of overlapping parts, and makes the obtained camera depth map have a delicate and realistic shape effect and texture effect; and using the target 3D Gaussian sputtering model to estimate the depth in the target scene can make the application more flexible, not restricted by the sensor type and without the need to retrain for each specific scene, can adapt to depth completion tasks of different scenes and objects, reduces the error caused by the calibration of the display device, and has strong generalization ability. Thus, it solves the problems that the depth completion and point cloud densification methods in the related art are sensitive to the calibration error of the hardware, and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar, and at the same time, the current point cloud densification also has problems such as scene generalization.

[0020] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. Description of the Drawings

[0021] The above and / or additional aspects and advantages of the present application will become apparent and be easily understood from the following description of the embodiments in conjunction with the drawings, where:

[0022] Figure 1Schematic framework diagram of the depth completion method based on point cloud densification in the embodiments of the present application;

[0023] Figure 2 Flowchart of a depth completion method based on point cloud densification provided according to an embodiment of the present application;

[0024] Figure 3 Schematic diagram of the training process of the 3D Gaussian sputtering model in an embodiment of the present application;

[0025] Figure 4 Schematic structural diagram of a depth completion device based on point cloud densification provided according to an embodiment of the present application;

[0026] Figure 5 Schematic structural diagram of an electronic device provided according to an embodiment of the present application.

[0027] Reference numerals:

[0028] 10 - Depth completion device based on point cloud densification: 100 - Preprocessing module, 200 - Construction module, and 300 - Reconstruction module; 501 - Memory, 502 - Processor, and 503 - Communication interface. Detailed description of specific embodiments

[0029] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0030] The method and device for depth completion based on point cloud densification according to the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the problems in the related art that the depth completion and point cloud densification methods are relatively sensitive to the calibration errors of hardware, and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar, and at the same time, the current point cloud densification also has the problem of scene generalization, the present application provides a depth completion method based on point cloud densification. In this method, a target 3D Gaussian sputtering model can be constructed by using the pre-processed laser point cloud data and camera images, and then a three-dimensional reconstructed point cloud map can be obtained by using the target 3D Gaussian sputtering model, so as to determine the camera depth map of the target scene according to the three-dimensional reconstructed point cloud map. Thus, it is realized to construct a target 3D Gaussian sputtering model by using the information represented by the laser point cloud data and images from multiple perspectives to complete the depth completion of the image, which can better process the effect of three-dimensional reconstruction of the overlapping part, so that the obtained camera depth map has a delicate and realistic shape effect and texture effect; and estimating the depth in the target scene by using the target 3D Gaussian sputtering model can make the application more flexible, without being restricted by the type of sensor and without the need for re-training for each specific scene, and can adapt to the depth completion tasks of different scenes and objects, reducing the error caused by the calibration of the display device and having strong generalization ability. Thus, the problems in the related art that the depth completion and point cloud densification methods are relatively sensitive to the calibration errors of hardware, and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar, and at the same time, the current point cloud densification also has problems such as scene generalization are solved.

[0031] Before explaining the depth completion method based on point cloud densification according to the embodiments of the present application, the computing platform and development system involved in the embodiments of the present application and the technologies involved in constructing the target 3D Gaussian sputtering model will be explained first.

[0032] The computing platform and development system (CBDES) is the key core of autonomous driving technology and undertakes the mission of realizing intelligent driving. CBDES provides powerful computing capabilities for the autonomous driving system, and constructs an integrated end-to-end autonomous driving large model through advanced algorithms and deep learning technologies to achieve efficient perception and accurate recognition of complex driving scenes. This platform not only supports real-time data processing, high-precision map construction and environmental understanding, but also is an innovative tool for developing, optimizing and integrating autonomous driving algorithms. And the densification technology for the collected raw point cloud data greatly affects the completion effect of all downstream tasks and ensures the efficient operation of the entire system.

[0033] Unsupervised automated annotation refers to the use of unsupervised learning and automated techniques to automatically annotate large-scale data, thus eliminating the dependence on manual annotation and improving the efficiency and scale of data annotation. Dense reconstruction refers to the use of computer vision and 3D reconstruction techniques to reconstruct the geometric structure and appearance information of a scene or object from large-scale image or point cloud data. This technology has important applications in the fields of virtual reality, 3D modeling, map making, etc.

[0034] Deep learning technology is increasingly widely used in the field of computer vision. Among them, depth completion, as an important technology, shows strong potential in image restoration, processing, and generation. Depth completion aims to use deep learning models to perform pixel-level restoration and completion of missing, damaged, or incomplete images, thereby achieving the restoration and reconstruction of image information.

[0035] At the same time, technologies and methods such as computing platforms and development systems, data closed-loop, unsupervised automated annotation, and dense reconstruction also play important roles in the fields of deep learning and computer vision. First of all, the computing platform and development system (CBDES) provide powerful computing capabilities and distributed computing support, enabling the training and optimization of deep learning models to achieve efficient and fast computing on large-scale data sets, thus providing the necessary technical foundation for the training and optimization of depth completion models. The data closed-loop refers to the process of continuously collecting, analyzing, and applying data in the fields of machine learning and artificial intelligence to continuously improve and optimize models. In the field of depth completion, the data closed-loop can provide feedback on the restoration results, enabling the depth completion model to continuously improve and adapt to different image restoration tasks, thereby improving the accuracy and robustness of the restoration.

[0036] The unsupervised automated annotation technology can quickly and accurately provide large-scale training data for deep learning models, thus accelerating the training and optimization process of depth completion models, reducing the dependence on manually annotated data, and improving the data utilization efficiency.

[0037] The dense reconstruction technology can provide more context information and image structure for depth completion, thus helping the depth completion model to better understand the image content during restoration and improving the quality and realism of the restoration results.

[0038] Figure 1 It is a framework schematic diagram of the depth completion method based on point cloud densification in the embodiments of this application. As Figure 1As shown in the figure, it can be divided into two parts: data preprocessing and model training. Among them, data preprocessing can be divided into several parts such as obtaining RGB values, converting coordinate systems, and calculating normal vectors, which can provide available data for model training. The model training part is mainly divided into several parts such as inputting the preprocessed data, constructing a 3D Gaussian sputtering model, and outputting the point cloud map and projection map after 3D reconstruction. After training, it can be applied in practice.

[0039] Specifically, Figure 2 This is a flowchart of a depth completion method based on point cloud densification provided by an embodiment of the present application.

[0040] As Figure 2 shown, the depth completion method based on point cloud densification includes the following steps:

[0041] In step S201, obtain the lidar point cloud data of the target scene and the target image at the same timestamp as the lidar point cloud data, and preprocess the lidar point cloud data and the target image to obtain the preprocessed lidar point cloud data and the target image.

[0042] It can be understood that the target scene here refers to the scene that needs to be sensed and recognized by CBDES. For example, the buildings and road scenes in front of the vehicle. The lidar point cloud data here refers to the dataset of spatial points scanned by the vehicle's 3D lidar device.

[0043] Depth completion plays an important role in aspects such as the computing platform and development system (CBDES), data closed-loop, unsupervised automatic annotation, and dense reconstruction. The point cloud data of the 3D lidar, that is, the lidar point cloud data, can provide strong and important data information for image depth completion.

[0044] In some embodiments, before performing 3D reconstruction on the target scene, the lidar point cloud data of the target scene can be obtained first. Among them, the lidar point cloud data in the embodiments of the present application includes but is not limited to lidar and various cameras on the vehicle to make full use of the shape and distance information collected by the 3D lidar and provide strong data support for the reconstruction of the target scene.

[0045] In view of the situation that the point cloud data directly collected by the 3D lidar may have inconsistent data benchmarks or low information accuracy, the embodiments of the present application can perform certain preprocessing on the lidar point cloud data so as to obtain the preprocessed lidar point cloud data that can meet the usage conditions in the subsequent process.

[0046] Optionally, in an embodiment of the present application, the laser point cloud data of the target scene and the target image at the same timestamp as the laser point cloud data are obtained, and the laser point cloud data and the target image are preprocessed to obtain the preprocessed laser point cloud data and target image, including: collecting the RGB information of each three-dimensional point in the laser point cloud data according to the laser point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera internal parameters of each camera, so as to obtain the laser point cloud data with RGB information; converting the initial laser point cloud coordinates of the laser point cloud data with RGB information to the global coordinate system to obtain the converted laser point cloud data; calculating the normal vector of the converted laser point cloud data, and performing pose conversion on the pose corresponding to the target image in the converted laser point cloud data, so as to complete the preprocessing of the laser point cloud data and the target image.

[0047] In the actual execution process, when preprocessing the laser point cloud data of the target scene, the present application mainly but not limited to collecting the RGB information corresponding to each three-dimensional point in the laser point cloud data, then converting the laser point cloud coordinates with RGB information to the global coordinate system, and finally, calculating the normal vector of the converted laser point cloud data to complete the preprocessing of the laser point cloud data and obtain the preprocessed laser point cloud data.

[0048] Specifically, the preprocessing process of the laser point cloud data can be but not limited to being expressed as follows:

[0049] (1) First, in the embodiment of the present application, the point cloud with only coordinate positions can be projected onto the image according to the pose data of the lidar and each camera on the target vehicle relative to the vehicle body coordinate system and the camera internal parameters. Thus, each three-dimensional point in the laser point cloud data can obtain the corresponding initial RGB color information, preparing for subsequent image processing and making the model rendering quality better. Here, the target vehicle refers to the vehicle corresponding to the collected laser point cloud data.

[0050] (2) Then, in the embodiment of the present application, according to the pose data of the lidar and each camera on the vehicle relative to the vehicle body coordinate system, the initial laser point cloud coordinates with RGB information obtained in step (1) can be converted to the global coordinate system, so as to unify all data to the same coordinate system and provide a unified data basis for subsequent processing.

[0051] (3) Finally, for the converted laser point cloud data, the normal vector is calculated, and the pose corresponding to the target image at the same timestamp is converted. By calculating the normal vector of each point cloud, the geometric features of the environment can be better understood, providing richer information for subsequent tasks such as model training.

[0052] Finally, based on the normal vector and RGB information, the preprocessed lidar point cloud data can be generated, and the pose of the target image can be transformed to obtain a transformation matrix that meets the requirements, which is convenient for subsequent calculation of the projection matrix and the clipping coordinate system matrix, so as to project the points in the three-dimensional space onto the two-dimensional image plane and achieve fast differentiable rasterization. It should be noted that in the embodiments of the present application, the preprocessing of the target image can be understood as transforming the pose corresponding to the image to obtain the form of the transformation matrix required by the 3D Gaussian method in the subsequent process as the input, and the target image will not change. That is, in the preprocessing process of the lidar point cloud data and the target image, all that changes is that the pose corresponding to each target image becomes the corresponding transformation matrix, and the lidar point cloud is transformed to the global coordinate system.

[0053] The embodiments of the present application make full use of the information represented by the radar and the image to realize high-quality three-dimensional scene reconstruction from images from multiple perspectives, including but not limited to information such as the geometric shape and surface texture of objects, so that the depth completion task can provide detailed and realistic results.

[0054] Step S202: Based on the preprocessed lidar point cloud data and the target image, construct a target 3D Gaussian sputtering model.

[0055] It can be understood that in addition to the three-dimensional lidar on the vehicle, there are also multiple cameras. The target image here can be understood as the images of each perspective collected by each camera installed on the same vehicle as the three-dimensional lidar that collects the lidar point cloud data. The target 3D Gaussian sputtering model here refers to the 3D Gaussian sputtering model used to complete depth completion to complete the three-dimensional reconstruction of the target scene.

[0056] As a possible implementation, in the process of optimizing point cloud densification and performing three-dimensional reconstruction of the scene, the embodiments of the present application can, but are not limited to, training the target 3D Gaussian sputtering model based on the preprocessed lidar point cloud data and the target image with the same timestamp as the lidar point cloud data to adapt to and complete the depth completion tasks of different scenes and objects.

[0057] The target 3D Gaussian sputtering model in the embodiments of the present application is not restricted by the type of sensor, which can make the application more flexible and reduce the error caused by the calibration of the display device. Further, the target 3D Gaussian sputtering model in the embodiments of the present application also uses images from multiple perspectives for training, so that the model can better handle the effect of three-dimensional reconstruction of the overlapping part, thereby improving the consistency and accuracy of depth completion.

[0058] Embodiments of the present application can construct a target 3D Gaussian splatting model based on preprocessed laser point cloud data and a target image with the same timestamp as the laser point cloud data to complete the depth completion task of the target scene, thereby realizing the three-dimensional reconstruction of the scene. Moreover, the target 3D Gaussian splatting model in the embodiments of the present application can be unrestricted by the type of sensor, can be used in various applications, and does not need to be retrained for each specific scene, which can make the application of the model more flexible, reduce the error caused by the calibration of the display device while improving the point cloud densification accuracy and the point cloud densification effect.

[0059] Optionally, in an embodiment of the present application, constructing a target 3D Gaussian splatting model based on preprocessed laser point cloud data and a target image includes: constructing a corresponding three-dimensional Gaussian distribution according to the preprocessed laser point cloud data to obtain a Gaussian ellipsoid under the three-dimensional Gaussian distribution; based on the target rasterizer and the Gaussian ellipsoid, performing block processing on the target image to obtain each image block and its corresponding 3D Gaussian model; according to the overlapping degree of the 3D Gaussian model of each image block with other image blocks except its corresponding image block, assigning a corresponding key value to the 3D Gaussian model of each image block; based on the key value, performing Splatting splatting processing on the Gaussian ellipsoid blocks to construct the target 3D Gaussian splatting model.

[0060] In some embodiments, when constructing the target 3D Gaussian splatting model in the present application, the preprocessed laser point cloud data can be first initialized to construct a corresponding three-dimensional Gaussian distribution, and a Gaussian ellipsoid under the three-dimensional Gaussian distribution can be obtained (the Gaussian ellipsoid is an intuitive geometric representation of the three-dimensional Gaussian distribution, and its shape, size, and direction can be described by the mean vector and the covariance matrix).

[0061] Next, the embodiments of the present application can use the target rasterizer for fast differentiable rasterization, that is, perform block processing on the Gaussian ellipsoid, thereby completing the block processing of the target image. After obtaining each image block and its corresponding 3D Gaussian model, according to the overlapping degree of the Gaussian model of each image block with other image blocks except its corresponding image block, assign a corresponding key value to the Gaussian model of each image block; based on the key value, perform Splatting splatting processing on the Gaussian ellipsoid blocks, thereby constructing the target 3D Gaussian splatting model in the embodiments of the present application.

[0062] Figure 3 Schematic diagram of the training process of the 3D Gaussian splatting model for an embodiment of the present application. As Figure 3As shown, the target 3D Gaussian sputtering model in the embodiments of the present application can, but is not limited to, have three initialization input modes: randomly generated points, SfM (Structure from Motion) points, and lidar point clouds. Since randomly generated points have no geometric prior, the worst results will occur. When the model fits the scene and outputs the point cloud, there will be a circle of "bad points". And due to the sparsity of the initialized SfM points and the intolerable structural errors, the accurate geometric shape of the scene cannot be fully restored. For the model initialized with lidar point cloud priors, although there may be problems with the loss of geometric information in local areas, by continuously rotating or moving the lidar sensor, point cloud data within a complete 360 degrees or three-dimensional space can be obtained, which can retain relatively accurate structural priors, thus exceeding SfM.

[0063] Therefore, the input of the target 3D Gaussian sputtering model in the embodiments of the present application can, but is not limited to, be set as a static scene image with color information at the same timestamp, and preprocessed lidar point cloud data. First, the embodiments of the present application can initialize the lidar point cloud to create a set of three-dimensional Gaussian distributions. Among them, the 3D Gaussian definition formula can, but is not limited to, be expressed as follows:

[0064]

[0065] Where Σ is the covariance matrix. The Gaussian distribution is centered on each lidar point, and the means of the coordinates x, y, and z respectively correspond to the positions of the point cloud, and its mean μ is set to 0. Since the scale coefficient in front of the exponential part does not affect the geometry of the ellipsoid, in the embodiments of the present application, the coefficient can be removed, so that the size of the entire distribution can be freely controlled. This processing method makes the Gaussian distribution more flexible and convenient in terms of rotation and scaling. By removing the scale coefficient, the embodiments of the present application can also more flexibly adjust the size and shape of the Gaussian distribution without changing its geometric characteristics, so as to better adapt to different application scenarios and requirements. Based on the above content, the 3D Gaussian in the embodiments of the present application can, but is not limited to, be expressed as follows:

[0066]

[0067] ∑=AA T

[0068] A=RS

[0069] ∑=RSS T R T

[0070] Among them, Σ is the covariance matrix, and matrix A can be constructed by the product of the rotation matrix R and the scaling matrix S. During the iterative optimization process, the embodiments of the present application can change the shape of the Gaussian ellipsoid by continuously optimizing the parameters of matrix A. For example, by adjusting the parameters of the rotation matrix R, the embodiments of the present application can change the direction of the Gaussian ellipsoid; by adjusting the parameters of the scaling matrix S, the embodiments of the present application can change the size of the ellipsoid. In this way, the embodiments of the present application can ensure that when the target 3D Gaussian sputtering model performs splatting, the desired effect can be correctly projected.

[0071] Among them, Splatting is a technique for rasterizing 3D objects in computer graphics. The 2D graphics obtained by mapping a 3D object onto a projection plane are called splats. The principle of this technique is similar to the mark left by a snowball hitting a wall. The energy of the snowball spreads out from the center and gradually weakens. During the splatting process, each point in the point cloud is projected onto the two-dimensional image plane, forming an effect similar to a mark, and these marks can produce a visually obvious effect and finally appear in the rendered image. This process can be processed in parallel on the GPU because each Splat is independent of each other, and this parallel processing can significantly improve the rendering speed and efficiency.

[0072] Furthermore, the embodiments of the present application can also perform fast differentiable rasterization on the 3D Gaussian model by using a rasterizer to achieve fast overall rendering. During this process, the embodiments of the present application perform block processing on the Gaussian ellipsoid. First, the entire input target image can be but is not limited to being divided into 16×16 blocks. Then, the embodiments of the present application can screen the visible 3D Gaussian models within the viewing frustum corresponding to each image block, and assign a key value to each 3D Gaussian model according to the degree of overlap of each 3D Gaussian model with other image blocks. This key value includes but is not limited to the ID of the block where the Gaussian ellipsoid is located and the depth of the corresponding viewing field.

[0073] Subsequently, the embodiments of the present application can sort the 3D Gaussian models according to the depth of the viewing field, and can perform splatting on the corresponding blocks in the order from near to far of the sorted 3D Gaussian models to obtain the final target 3D Gaussian sputtering model. That is, project the points generated by the 3D Gaussian model on each image block and perform parallel processing on each image block. This process will perform a series of splat operations on each image block, stack the projections of the 3D Gaussian models together, and stop the corresponding thread until the opacity of all pixels reaches saturation to obtain the target 3D Gaussian sputtering model.

[0074] Also, in each iteration, the embodiment of the present application randomly selects an input image at a certain timestamp and uses a rasterizer to render the image from this perspective. Then, the rendered image is compared with the input image to calculate the loss, and the formula can be but is not limited to the following:

[0075] L = (1 - α)L 1 + αL D-SSIM

[0076] In this way, the embodiment of the present application can index the Gaussian by blocks, backpropagate the error, and continuously adjust the parameters of the 3D Gaussian model by optimizing the loss, thereby changing the overall rendering effect. Through such an iterative optimization process, fine adjustment of the 3D Gaussian model can be achieved, and finally an overall rendering result that meets the requirements can be obtained. On the basis of parallel processing, the speed and efficiency of rendering are improved throughout the process.

[0077] Optionally, in an embodiment of the present application, based on the preprocessed laser point cloud data and the target image, to construct a target 3D Gaussian sputtering model, it further includes: obtaining the scale information of the 3D Gaussian model; and correcting the 3D Gaussian model based on the scale information to construct the target 3D Gaussian sputtering model.

[0078] In other embodiments, considering that the adaptive density control link enables the system to fit a relatively good model from sparse and not very high-quality initial laser point clouds, therefore, in the rasterization process of the 3D Gaussian model in the embodiment of the present application, the representation of the scene is achieved by the superposition of multiple 3D Gaussian models.

[0079] However, in the earlier number of iterations, it is very likely to encounter the problem of insufficient scene reconstruction. That is to say, the 3D Gaussian model fails to completely cover small-scale geometries, resulting in an unsatisfactory scene reconstruction effect. To solve this problem, the embodiment of the present application can obtain the scale information of the 3D Gaussian model and correct the 3D Gaussian model based on the scale information.

[0080] For example, when the 3D Gaussian model fails to completely cover small-scale geometries, the present application can increase the coverage by copying the 3D Gaussian model. However, in this case, in subsequent iterations, there may again be a problem of over-reconstruction. That is to say, the 3D Gaussian model exceeds the range of small-scale geometries. To solve this problem, the embodiment of the present application can divide the 3D Gaussian model into two parts.

[0081] Starting from a Gaussian model initialized as sparse laser point cloud, embodiments of the present application can gradually transition from a set of sparse 3D Gaussian models to a denser set that can better represent the scene by adaptively adjusting the number of 3D Gaussian models and their density per unit volume. During this process, due to insufficient reconstruction in some regions, there are often large gradients. The magnitude of the gradient can be regarded as an indication of the error at that position. If the gradient is too large, it means the error is large and the amount that needs to be corrected is also large. Once the gradient exceeds the threshold, a densify operation will be performed. By increasing the number of 3D Gaussian models or adjusting their density, the accuracy and precision of scene reconstruction are further improved. Among them, the operation process of densify can be but is not limited to the following:

[0082] (1) For those 3D Gaussian models that do not fully cover small-scale geometries, it is necessary to copy the 3D Gaussian models and move them along the position gradient direction to cover the geometries;

[0083] (2) For those 3D Gaussian models that exceed the range of small-scale geometries, it is necessary to split the 3D Gaussian models so that they only cover the geometries.

[0084] It should be noted that if there are floating objects near the camera during the optimization process, it will lead to some incorrect reconstructions that do not belong to the 3D Gaussian models. At this time, it is necessary to periodically reset the opacity to 0 to remove the floating objects. Similarly, it is also necessary to periodically remove larger Gaussians to avoid overlap.

[0085] Step S203, based on the target scene and the target 3D Gaussian sputtering model, output the dense point cloud after three-dimensional reconstruction, so as to obtain the three-dimensional reconstruction point cloud map of the point cloud densification of the target scene according to the dense point cloud after three-dimensional reconstruction, and determine the camera depth map of the target scene according to the three-dimensional reconstruction point cloud map.

[0086] Based on the related descriptions of other embodiments, it can be understood that the present application can construct a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image at the same timestamp as the laser point cloud data. Then, according to the input laser point clouds and images at different timestamps, embodiments of the present application can train different target 3D Gaussian sputtering models.

[0087] After completing the construction of the target 3D Gaussian sputtering model, when a fixed pose is input to the target 3D Gaussian sputtering model, embodiments of the present application can use this pose to render the image after three-dimensional reconstruction from this perspective. Similarly, embodiments of the present application can use the target 3D Gaussian sputtering model to output the dense point cloud after three-dimensional reconstruction, and then obtain the point cloud densification camera depth map of the target scene or at different poses from the dense point cloud.

[0088] For example, the present application can first obtain the coordinate values of the positions of each point cloud in the camera coordinate system. This process can be implemented but is not limited to using the coordinate transformation matrix from radar to camera and the point cloud coordinates, that is, multiplying the coordinate transformation matrix from radar to camera by the point cloud coordinates, and the coordinate values of the positions of each point cloud in the camera coordinate system can be obtained.

[0089] Next, the embodiments of the present application can convert the coordinate values of each point cloud into pixel coordinates on the image by applying the camera internal parameters to the coordinate values of each point cloud.

[0090] Finally, the embodiments of the present application can set the pixel values corresponding to the pixel positions of these points in the image as the corresponding depth values according to the pixel coordinates corresponding to each point cloud, so as to generate a camera depth map. For the background area in the point cloud, the depth value of the corresponding pixel position can be set to a marker value (such as 0 or a negative value) to distinguish the background and foreground areas.

[0091] In addition, the camera depth map in the embodiments of the present application includes but is not limited to two formats: point cloud map and projection map. Herein, the point cloud map refers to the point cloud map after three-dimensional reconstruction, and the projection map can be understood as the image obtained by projecting the point cloud with coordinate positions onto the target image.

[0092] According to the depth completion method based on point cloud densification proposed by the embodiments of the present application, a target 3D Gaussian sputtering model can be constructed by using the preprocessed laser point cloud data and camera images, and then the three-dimensional reconstructed point cloud map can be obtained by using the target 3D Gaussian sputtering model, so as to determine the camera depth map of the target scene according to the three-dimensional reconstructed point cloud map. Thus, the depth completion of the image is realized by using the information represented by the laser point cloud data and images from multiple perspectives to construct the target 3D Gaussian sputtering model, which can better process the effect of three-dimensional reconstruction of the overlapping part, so that the obtained camera depth map has a delicate and realistic shape effect and texture effect; and using the target 3D Gaussian sputtering model to estimate the depth in the target scene can make the application more flexible, not limited by the sensor type and without the need to retrain for each specific scene, and can adapt to the depth completion tasks of different scenes and objects, reducing the error caused by the calibration of the display device and having strong generalization ability. Thus, the problems in the related art that the depth completion and point cloud densification methods are sensitive to the calibration error of the hardware, and cannot fully utilize the color texture information provided by the camera and the shape distance information provided by the radar, and at the same time, the current point cloud densification also has problems such as scene generalization are solved.

[0093] Secondly, a depth completion device based on point cloud densification proposed by the embodiments of the present application is described with reference to the accompanying drawings.

[0094] Figure 4It is a schematic structural diagram of a depth completion device based on point cloud densification according to an embodiment of the present application.

[0095] As Figure 4 shown, the depth completion device 10 based on point cloud densification includes: a preprocessing module 100, a construction module 200, and a reconstruction module 300.

[0096] Among them, the preprocessing module 100 is used to obtain the laser point cloud data of the target scene and the target image at the same timestamp as the laser point cloud data, and preprocess the laser point cloud data and the target image to obtain the preprocessed laser point cloud data and target image;

[0097] The construction module 200 is used to construct a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image;

[0098] The reconstruction module 300 is used to output the densely populated point cloud after three-dimensional reconstruction based on the target scene and the target 3D Gaussian sputtering model, so as to obtain the three-dimensional reconstruction point cloud map of the point cloud densification of the target scene according to the densely populated point cloud after three-dimensional reconstruction, and determine the camera depth map of the target scene according to the three-dimensional reconstruction point cloud map.

[0099] Optionally, in an embodiment of the present application, the preprocessing module 100 includes: a collection unit, a conversion unit, and a calculation unit.

[0100] Among them, the collection unit is used to collect the RGB information of each three-dimensional point in the laser point cloud data according to the laser point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera internal parameters of each camera, so as to obtain the laser point cloud data with RGB information.

[0101] The conversion unit is used to convert the initial laser point cloud coordinates of the laser point cloud data with RGB information to the global coordinate system to obtain the converted laser point cloud data.

[0102] The calculation unit is used to calculate the normal vector of the converted laser point cloud data and perform pose conversion on the pose corresponding to the target image in the converted laser point cloud data, so as to complete the preprocessing of the laser point cloud data and the target image.

[0103] Optionally, in an embodiment of the present application, the construction module 200 includes: a first construction unit, a first distribution unit, a second distribution unit, and a second construction unit.

[0104] Among them, the first construction unit is used to construct a corresponding three-dimensional Gaussian distribution according to the preprocessed laser point cloud data to obtain a Gaussian ellipsoid under the three-dimensional Gaussian distribution.

[0105] The first allocation unit is configured to perform block processing on a target image based on a target rasterizer and a Gaussian ellipsoid to obtain each image block and its corresponding 3D Gaussian model.

[0106] The second allocation unit is configured to assign a corresponding key value to the 3D Gaussian model of each image block according to the overlapping degree between the 3D Gaussian model of each image block and other image blocks except its corresponding image block.

[0107] The second construction unit is configured to perform Splatting sputtering processing on Gaussian ellipsoid blocks based on the key values to construct a target 3D Gaussian sputtering model.

[0108] Optionally, in an embodiment of the present application, the construction module 200 further includes: an acquisition unit and a correction unit.

[0109] The acquisition unit is configured to acquire the scale information of the 3D Gaussian model.

[0110] The correction unit is configured to correct the 3D Gaussian model based on the scale information to construct a target 3D Gaussian sputtering model.

[0111] Optionally, in an embodiment of the present application, the camera depth map includes at least one of a point cloud map and a projection map of the target scene.

[0112] It should be noted that the foregoing explanation of the embodiments of the depth completion method based on point cloud densification also applies to the depth completion device based on point cloud densification of this embodiment, which will not be elaborated here.

[0113] According to the depth completion device based on point cloud densification proposed in the embodiments of the present application, a target 3D Gaussian sputtering model can be constructed by using the preprocessed laser point cloud data and camera images, and then a three-dimensional reconstructed point cloud map can be obtained by using the target 3D Gaussian sputtering model, so as to determine the camera depth map of the target scene according to the three-dimensional reconstructed point cloud map. Thus, the depth completion of the image is realized by using the information represented by the laser point cloud data and images from multiple perspectives to construct a target 3D Gaussian sputtering model, which can better handle the effect of three-dimensional reconstruction of overlapping parts, so that the obtained camera depth map has a delicate and realistic shape effect and texture effect; and using the target 3D Gaussian sputtering model to estimate the depth in the target scene can make the application more flexible, not restricted by the type of sensor and without the need to retrain for each specific scene, and can adapt to the depth completion tasks of different scenes and objects, reducing the error caused by the calibration of the display device while having strong generalization ability. Thus, the problems in the related art that the depth completion and point cloud densification methods are sensitive to the calibration error of the hardware, and cannot make full use of the color texture information provided by the camera and the shape distance information provided by the radar, and there are also problems such as scene generalization in the current point cloud densification are solved.

[0114] Figure 5 This is a schematic structural diagram of the electronic device provided by the embodiment of the present application. The electronic device may include:

[0115] A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0116] When the processor 502 executes the program, it implements the depth completion method based on point cloud densification provided in the above embodiment.

[0117] Furthermore, the electronic device further includes:

[0118] A communication interface 503 for communication between the memory 501 and the processor 502.

[0119] The memory 501 is used to store a computer program executable on the processor 502.

[0120] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0121] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 may be interconnected through a bus and complete communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 5 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a chip, the memory 501, the processor 502, and the communication interface 503 may complete communication with each other through an internal interface.

[0123] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0124] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-described depth completion method based on point cloud densification is implemented.

[0125] The embodiments of the present application also provide a computer program product, including a computer program, and the computer program can run computer instructions, and when the computer instructions are executed by a processor, the depth completion method based on point cloud densification provided by the embodiments of the present application is implemented.

[0126] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0127] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0128] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or N executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a manner that is not shown or discussed, including in a substantially simultaneous manner according to the involved functions or in a reverse order, which should be understood by those skilled in the art of the embodiments of the present application.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0130] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0131] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0132] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0133] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A depth completion method based on point cloud densification, characterized in that: The following steps are involved: Acquire laser point cloud data of a target scene and a target image at the same time stamp as the laser point cloud data, pre-process the laser point cloud data and the target image, and obtain pre-processed laser point cloud data and the target image; Based on the preprocessed laser point cloud data and the target image, construct a 3D Gaussian sputtering model of the target; Based on the target scene and the target 3D Gaussian sputtering model, a dense point cloud after three-dimensional reconstruction is output to obtain a three-dimensional reconstructed point cloud map with densified point clouds of the target scene according to the dense point cloud after three-dimensional reconstructed, and a camera depth map of the target scene is determined according to the three-dimensional reconstructed point cloud map.

2. The method according to claim 1, characterized in that The step of acquiring laser point cloud data of a target scene and a target image at the same time stamp as the laser point cloud data, and preprocessing the laser point cloud data and the target image to obtain preprocessed laser point cloud data and the target image comprises: Collecting RGB information of each three-dimensional point in the laser point cloud data according to the laser point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera intrinsic parameters of each camera to obtain laser point cloud data with RGB information; Converting the initial laser point cloud coordinates of the laser point cloud data with RGB information into a global coordinate system to obtain converted laser point cloud data; The normal vector of the converted laser point cloud data is calculated, and the posture corresponding to the target image in the converted laser point cloud data is transformed to complete the preprocessing of the laser point cloud data and the target image.

3. The method according to claim 1, characterized in that The method of constructing a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image comprises: Constructing a corresponding three-dimensional Gaussian distribution according to the preprocessed laser point cloud data to obtain a Gaussian ellipsoid under the three-dimensional Gaussian distribution; Based on the target rasterizer and the Gaussian ellipsoid, the target image is processed into blocks to obtain each image block and its corresponding 3D Gaussian model; According to the degree of overlap between the 3D Gaussian model of each image block and other image blocks except the image block corresponding to the 3D Gaussian model of each image block, a corresponding key value is assigned to the 3D Gaussian model of each image block; Based on the key value, a splatting process is performed on the Gaussian ellipsoid block to construct the target 3D Gaussian sputtering model.

4. The method according to claim 3, characterized in that The step of constructing a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image further includes: Obtaining scale information of the 3D Gaussian model; The 3D Gaussian model is modified based on the scale information to construct the target 3D Gaussian sputtering model.

5. The method according to claim 1, characterized in that The camera depth map includes at least one of a point cloud map and a projection map of the target scene.

6. A depth completion device based on point cloud densification, characterized in that: include: A preprocessing module, used to obtain laser point cloud data of a target scene and a target image at the same time stamp as the laser point cloud data, and preprocess the laser point cloud data and the target image to obtain preprocessed laser point cloud data and target image; A construction module, used to construct a target 3D Gaussian sputtering model based on the preprocessed laser point cloud data and the target image; A reconstruction module is used to output a dense point cloud after three-dimensional reconstruction based on the target scene and the target 3D Gaussian sputtering model, so as to obtain a three-dimensional reconstructed point cloud map with densified point clouds of the target scene according to the dense point cloud after three-dimensional reconstructed, and to determine a camera depth map of the target scene according to the three-dimensional reconstructed point cloud map.

7. The device according to claim 6, characterized in that The preprocessing module comprises: An acquisition unit is used to acquire RGB information of each three-dimensional point in the laser point cloud data according to the laser point cloud data, the target image, the pose data of each camera on the target vehicle relative to the vehicle body coordinate system, and the camera intrinsic parameters of each camera, so as to obtain laser point cloud data with RGB information; A conversion unit, used for converting the initial laser point cloud coordinates of the laser point cloud data with RGB information into a global coordinate system to obtain converted laser point cloud data; A calculation unit is used to calculate the normal vector of the converted laser point cloud data, and perform posture conversion on the posture corresponding to the target image in the converted laser point cloud data to complete the preprocessing of the laser point cloud data and the target image.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the depth completion method based on point cloud densification as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the depth completion method based on point cloud densification as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, it is used to implement the depth completion method based on point cloud densification as described in any one of claims 1-5.

Images