Training method and device, depth completion method, device and system, unmanned vehicle

Abstract

The present disclosure relates to a training method and device, a depth completion method, device and system, an unmanned vehicle, and relates to the field of computer vision, and particularly to the field of unmanned driving. The method of the present disclosure comprises: obtaining a current frame, a previous frame of the current frame and a next frame of the current frame from a video sequence for training, and grouping the current frame and the previous frame into a current image pair and grouping the next frame and the current frame into a next image pair; inputting a sparse depth map corresponding to the current image pair and the current frame into a depth completion model, adjusting parameters of the depth completion model according to a first result output by the depth completion model; inputting a sparse depth map corresponding to the next image pair and the next frame into the depth completion model, adjusting the parameters of the depth completion model again according to a second result output by the depth completion model; and repeating the above process until the training of the depth completion model is completed.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, particularly to the field of autonomous driving, and specifically to a training method and apparatus, a depth completion method, apparatus and system, and an unmanned vehicle. Background Technology

[0002] Currently, autonomous vehicles are used to automatically transport people or goods from one location to another. These vehicles collect environmental information through onboard sensors and complete the automated transport. Autonomous delivery vehicles controlled by autonomous driving technology greatly improve the convenience of production and daily life, and save labor costs.

[0003] In the field of autonomous driving, depth information from images is essential for completing autonomous driving tasks. Currently, acquiring depth information relies on various depth sensors, such as LiDAR. However, LiDAR and other depth sensors can only acquire depth information for a subset of points, resulting in a sparse depth map. Therefore, the depth completion task has been proposed, which aims to complete all depth information by combining the sparse depth map with red-green-blue (RGB) images.

[0004] The relevant technologies mainly focus on using Convolutional Neural Networks (CNNs) to build regression models to predict depth and complete depth information. Summary of the Invention

[0005] The inventors discovered that depth completion models also suffer from a significant performance drop when the test set differs from the training set. For many tasks requiring accurate depth information, this performance degradation in different environments is unacceptable. For example, in autonomous driving, the environment encountered by a car is often changing, including weather, lighting, terrain, and traffic conditions. This means that when depth completion methods are applied to autonomous driving, the acquired RGB images and sparse depth maps are difficult to keep consistent with those used during model training. If the model cannot quickly adapt to these environmental changes, it may predict incorrect depth values, potentially leading to serious traffic accidents.

[0006] One of the technical problems this disclosure aims to solve is: how to improve the generalization ability and accuracy of deep completion models.

[0007] According to some embodiments of this disclosure, a training method is provided, comprising: obtaining the current frame, the previous frame, and the next frame from a video sequence used for training; forming a current image pair with the current frame and the previous frame; forming a next image pair with the next frame and the current frame; inputting the current image pair and the sparse depth map corresponding to the current frame into a depth completion model; adjusting the parameters of the depth completion model according to a first result output by the depth completion model; inputting the next image pair and the sparse depth map corresponding to the next frame into the depth completion model; adjusting the parameters of the depth completion model again according to a second result output by the depth completion model; repeating the above process until the training of the depth completion model is completed.

[0008] In some embodiments, the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. The first result includes: a first sub-result and a second sub-result. The process involves inputting the sparse depth map corresponding to the current image pair and the current frame into the depth completion model, and adjusting the parameters of the depth completion model based on the first result output by the model. This includes: inputting the sparse depth map corresponding to the current image pair and the current frame into the depth completion model; adjusting the parameters of the depth compaction module based on the first sub-result output by the depth completion model; obtaining the second sub-result output by the depth completion model again based on the adjusted parameters of the depth compaction module; and adjusting the parameters of the pose estimation module and the depth prediction module based on the second sub-result.

[0009] In some embodiments, inputting the current image pair and the sparse depth map corresponding to the current frame into the depth completion model includes: inputting the current image pair into the pose estimation module to obtain the output relative pose of the camera when the current frame was captured and the previous frame; inputting the sparse depth map corresponding to the current frame into the depth compaction module to obtain the output first dense depth map corresponding to the current frame; determining the first reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first dense depth map; wherein the first dense depth map and the first reconstructed image are used as the first sub-result.

[0010] In some embodiments, obtaining the second sub-result output by the depth completion model based on the adjusted parameters of the depth compaction module includes: inputting the second dense depth map corresponding to the current frame output by the adjusted depth compaction module and the current frame into the depth prediction module to obtain the first predicted depth map corresponding to the current frame; determining the second reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first predicted depth map; and using the first predicted depth map and the second reconstructed image as the second sub-result.

[0011] In some embodiments, adjusting the parameters of the depth compaction module based on the first sub-result output by the depth completion model includes: determining a first loss function based on the difference between the first dense depth map and the sparse depth map corresponding to the current frame, and the difference between the first reconstructed image and the current frame; and adjusting the parameters of the depth compaction module based on the first loss function.

[0012] In some embodiments, adjusting the parameters of the pose estimation module and the depth prediction module according to the second sub-result includes: determining a second loss function based on the difference between the first predicted depth map and the sparse depth map corresponding to the current frame and the difference between the second reconstructed image and the current frame; and adjusting the parameters of the pose estimation module and the depth prediction module according to the second loss function.

[0013] In some embodiments, the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. Inputting the next image pair and the sparse depth map corresponding to the next frame into the depth completion model includes: inputting the next image pair into the pose estimation module to obtain the output relative pose of the camera when the next frame was captured and the current frame; inputting the sparse depth map corresponding to the next frame into the depth compaction module to obtain the output third dense depth map corresponding to the next frame; inputting the next frame and the third dense depth map into the depth prediction module to obtain the second predicted depth map corresponding to the next frame; and determining the third reconstructed image corresponding to the next frame based on the current frame, the next frame, the relative pose of the camera when the current frame was captured, and the second predicted depth map; wherein the second predicted depth map and the third reconstructed image are used as the second result.

[0014] In some embodiments, adjusting the parameters of the depth completion model again based on the second result output by the depth completion model includes: determining a third loss function based on the difference between the second predicted depth map and the sparse depth map corresponding to the next frame, and the difference between the third reconstructed image and the next frame; and adjusting the parameters of the depth compaction module, the pose estimation module, and the depth prediction module based on the third loss function.

[0015] According to some other embodiments of this disclosure, a depth completion method is provided, comprising: acquiring the current online frame of an online video and the sparse depth map corresponding to the current online frame, and forming a current online image pair by combining the current online frame with the previous online frame; inputting the current online image pair and the sparse depth map corresponding to the current online frame into a depth completion model, and adjusting the parameters of the depth completion model online according to a third result output by the depth completion model; and using the predicted depth map corresponding to the current online frame output by the depth completion model as the depth map corresponding to the current online frame.

[0016] In some embodiments, the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. The third result includes: a third sub-result and a fourth sub-result. The process involves inputting the sparse depth map corresponding to the current online image pair and the current online frame into the depth completion model, and adjusting the parameters of the depth completion model online based on the third result output by the model. This includes: inputting the sparse depth map corresponding to the current online image pair and the current online frame into the depth completion model; adjusting the parameters of the depth compaction module based on the third sub-result output by the depth completion model; obtaining the fourth sub-result output by the depth completion model again based on the adjusted parameters of the depth compaction module; and adjusting the parameters of the pose estimation module and the depth prediction module based on the fourth sub-result.

[0017] In some embodiments, inputting the current online image pair and the sparse depth map corresponding to the current online frame into the depth completion model includes: inputting the current online image pair into the pose estimation module to obtain the output relative pose of the current online frame and the camera when the previous online frame was captured; inputting the sparse depth map corresponding to the current online frame into the depth compaction module to obtain the output fourth dense depth map corresponding to the current online frame; and using the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the fourth dense depth map to determine the fourth reconstructed image corresponding to the current online frame; wherein the fourth dense depth map and the fourth reconstructed image are used as a third sub-result.

[0018] In some embodiments, obtaining the fourth sub-result output by the depth completion model based on the adjusted parameters of the depth compaction module includes: inputting the fifth dense depth map corresponding to the current online frame output by the adjusted depth compaction module and the current online frame into the depth prediction module to obtain the third predicted depth map corresponding to the current online frame; determining the fifth reconstructed image corresponding to the current frame by using the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the third predicted depth map; and using the third predicted depth map and the fifth reconstructed image as the fourth sub-result.

[0019] In some embodiments, adjusting the parameters of the depth compaction module based on the third sub-result output by the depth completion model includes: determining a fourth loss function based on the difference between the fourth dense depth map and the sparse depth map corresponding to the current online frame, and the difference between the fourth reconstructed image and the current online frame; and adjusting the parameters of the depth compaction module based on the fourth loss function.

[0020] In some embodiments, adjusting the parameters of the pose estimation module and the depth prediction module according to the fourth sub-result includes: determining a fifth loss function based on the difference between the third predicted depth map and the sparse depth map corresponding to the current online frame and the difference between the fifth reconstructed image and the current online frame; and adjusting the parameters of the pose estimation module and the depth prediction module according to the fifth loss function.

[0021] In some embodiments, the method further includes at least one of the following: controlling the autonomous vehicle to drive based on the depth map corresponding to each online frame of the online video; reconstructing a three-dimensional scene based on the depth map corresponding to each online frame of the online video; and identifying three-dimensional targets in each online frame based on the depth map corresponding to each online frame of the online video.

[0022] According to some embodiments of this disclosure, a training apparatus is provided, comprising: an acquisition unit, configured to acquire the current frame, the previous frame, and the next frame from a trained video sequence, and to form a current image pair with the current frame and the previous frame, and to form a next image pair with the next frame; a first adjustment unit, configured to input the sparse depth map corresponding to the current image pair and the current frame into a depth completion model, and to adjust the parameters of the depth completion model according to a first result output by the depth completion model; a second adjustment unit, configured to input the sparse depth map corresponding to the next image pair and the next frame into the depth completion model, and to adjust the parameters of the depth completion model again according to a second result output by the depth completion model; and a loop triggering unit, configured to trigger the acquisition unit, the first adjustment unit, and the second adjustment unit to repeatedly execute the above process until the training of the depth completion model is completed.

[0023] According to further embodiments of this disclosure, a depth completion device is provided, comprising: an acquisition unit, configured to acquire a current online frame of an online video and a sparse depth map corresponding to the current online frame, and to form a current online image pair by combining the current online frame with the previous online frame; an adjustment unit, configured to input the current online image pair and the sparse depth map corresponding to the current online frame into a depth completion model, and to adjust the parameters of the depth completion model online according to a third result output by the depth completion model; and a depth completion unit, configured to use the predicted depth map corresponding to the current online frame output by the depth completion model as the depth map corresponding to the current online frame.

[0024] According to some other embodiments of this disclosure, a training apparatus is provided, comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform a training method as described in any of the foregoing embodiments.

[0025] According to further embodiments of this disclosure, a deep completion apparatus is provided, comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform a deep completion method as described in any of the foregoing embodiments.

[0026] According to some further embodiments of this disclosure, a deep completion system is provided, including: a training device of any of the foregoing embodiments and a deep completion device of any of the foregoing embodiments.

[0027] According to further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the training method or deep completion method of any of the foregoing embodiments.

[0028] According to some other embodiments of this disclosure, an unmanned vehicle is provided, including: the depth completion device of any of the foregoing embodiments.

[0029] In some embodiments, the unmanned vehicle further includes: a camera device for acquiring online video and sending it to a depth completion device; and a sensor for acquiring sparse depth maps corresponding to each frame in the online video and sending them to the depth completion device.

[0030] In some embodiments, the driverless vehicle further includes a training device as described in any of the foregoing embodiments.

[0031] This disclosure obtains the current image pair and the next image pair from the video used for training. First, the current image pair and the sparse depth map corresponding to the current frame are input into a depth completion model. Based on the first output result, the parameters of the depth completion model are adjusted. Then, the next image pair and the sparse depth map corresponding to the next frame are input into the depth completion model, and based on the second output result, the parameters of the depth completion model are adjusted again. Each iteration includes two parameter adjustment processes. The first parameter adjustment process allows the depth completion model to learn how to predict depth for depth completion, while the second parameter adjustment process allows the depth completion model to learn "how to learn depth prediction," enhancing the model's ability to adapt quickly and improving its generalization ability and accuracy.

[0032] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a training method according to some embodiments of this disclosure is shown.

[0035] Figure 2 A schematic diagram of the structure of a depth completion model according to other embodiments of this disclosure is shown.

[0036] Figure 3A flowchart illustrating a training method according to other embodiments of this disclosure is shown.

[0037] Figure 4 A flowchart illustrating a deep completion method according to some embodiments of this disclosure is shown.

[0038] Figure 5 A flowchart illustrating a depth completion method according to other embodiments of this disclosure is shown.

[0039] Figure 6 A schematic diagram of the structure of a training apparatus according to some embodiments of the present disclosure is shown.

[0040] Figure 7 A schematic diagram of the structure of a training apparatus according to other embodiments of this disclosure is shown.

[0041] Figure 8 A schematic diagram of the structure of a training apparatus according to further embodiments of the present disclosure is shown.

[0042] Figure 9 A schematic diagram of the structure of a depth completion device according to some embodiments of the present disclosure is shown.

[0043] Figure 10 A schematic diagram of the structure of a depth completion device according to other embodiments of this disclosure is shown.

[0044] Figure 11 A schematic diagram of the structure of a depth completion device according to further embodiments of the present disclosure is shown.

[0045] Figure 12 A schematic diagram of the structure of a depth completion system according to some embodiments of the present disclosure is shown.

[0046] Figure 13 A schematic diagram of the structure of an unmanned vehicle according to some embodiments of the present disclosure is shown.

[0047] Figure 14 The diagram shows the structural schematics of an unmanned vehicle according to other embodiments of this disclosure.

[0048] Figure 15 A schematic diagram of the structure of an unmanned vehicle according to further embodiments of the present disclosure is shown. Detailed Implementation

[0049] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0050] This disclosure proposes a training method for deep completion models, which is described below in conjunction with... Figures 1-3 Describe it.

[0051] Figure 1 Flowcharts are shown for some embodiments of the training method disclosed herein. Figure 1 As shown, the method of this embodiment includes steps S102 to S108.

[0052] In step S102, the current frame, the previous frame, and the next frame are obtained from the video sequence used for training. The current frame and the previous frame are combined to form the current image pair, and the next frame and the current frame are combined to form the next image pair.

[0053] The training method disclosed herein is based on improvements to meta-learning algorithms, such as improvements to the MAML (Model-Agnostic Meta-Learning) algorithm. Multiple video sequences are used for training, each containing multiple frames. The video sequences used for training can be resampled versions of the original videos; for example, the time interval between two frames in the resampled video sequence may be longer than that in the original video. However, the difference between two consecutive frames in a video sequence cannot exceed a preset value. For each video sequence, the current image pair can be used as the meta-training set, and the next image pair as the meta-test set, forming a batch of training samples. Training samples from different batches can come from different video sequences. If two consecutive batches of training samples come from the same video sequence, it is not required that the training samples in these two batches use consecutive frames; it is only necessary to ensure that the frames in the current image pair and the next image pair within the same batch are consecutive. For example, the current image pair in the current batch may be frames t-1 and t of the video sequence, and the current image pair in the next batch may be frames t+n-1 and t+n of the video sequence. Each frame in the video sequence can be an RGB image, not limited to the examples given.

[0054] In step S104, the current image pair and the sparse depth map corresponding to the current frame are input into the depth completion model, and the parameters of the depth completion model are adjusted according to the first result output by the depth completion model.

[0055] The sparse depth map corresponding to the current frame is acquired by a sensor, such as a LiDAR. The sparse depth map includes depth information from multiple points, and the number of points is less than a threshold. The current frame and its corresponding sparse depth map can be acquired simultaneously.

[0056] In some embodiments, the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. The depth compaction module (also called a coarse depth estimation network, Coarse Net) is primarily used to process the input sparse depth map and generate a coarse dense depth map based on it. The dense depth map has more depth information for each point compared to the sparse depth map. The depth prediction module (also called Depth Net) is mainly responsible for predicting the depth map from the coarse dense depth map and the image input; for example, generating the predicted depth map for the current frame based on the coarse dense depth map and the current frame. The pose estimation module (also called Pose Net) is responsible for estimating the camera's relative pose from two consecutive frames of images; for example, estimating the relative pose of the camera when the current frame and the previous frame were captured based on the images of the current frame and the previous frame.

[0057] In some embodiments, the depth compaction module, pose estimation module, and depth prediction module are constructed based on a convolutional neural network. For example, the depth compaction module uses normalized convolution, the pose estimation module uses an encoder structure, and the depth prediction module uses an encoder-decoder structure.

[0058] In some embodiments, the depth completion model further includes a reconstruction module, configured to reconstruct the image based on the previous frame, the current frame, the relative pose of the camera at the time of capture of the previous frame, and the predicted depth map, to obtain the reconstructed image of the current frame. For example, image reconstruction can be achieved using image affine transformation.

[0059] like Figure 2 The diagram shown is a structural diagram of the depth completion model. For example, For the image of frame t, This is the sparse depth map corresponding to frame t. This is a rough density depth map. The relative pose of the camera when shooting frames t and t-1 is taken. The predicted depth map corresponding to frame t is the final predicted dense depth map. Let be the reconstructed image of frame t. and The input pose estimation module (PoseNet) can obtain... , Input a deep compact module (Coarse Net) to obtain , and Input the depth prediction module (Depth Net) to obtain , , , Input the refactoring module (warp) to get .

[0060] Since the input to the depth completion task is multimodal (e.g., RGB images and depth maps belong to different modalities), and the variations of inputs from different modalities are often different, the adjustment process of model parameters is decoupled to improve the accuracy of the depth completion model. In some embodiments, the first result includes: a first sub-result and a second sub-result. The current image pair and the sparse depth map corresponding to the current frame are input into the depth completion model. Based on the first sub-result output by the depth completion model, the parameters of the depth compaction module are adjusted. Based on the adjusted parameters of the depth compaction module, the second sub-result output by the depth completion model is obtained again. Based on the second sub-result, the parameters of the pose estimation module and the depth prediction module are adjusted.

[0061] The process of adjusting the parameters of the deep completion model based on the first result output by the deep completion model can be understood as an inner loop process based on meta-learning improvement. The inner loop process is divided into two stages: the first stage is to adjust the parameters of the deep compaction module using the first sub-result; the second stage is to obtain the second sub-result based on the adjusted parameters of the deep compaction module, and then adjust the parameters of the pose estimation module and the depth prediction module based on the second sub-result.

[0062] In some embodiments, the first stage includes: inputting the current image to the pose estimation module to obtain the output of the current frame and the relative pose of the camera when the previous frame was captured; inputting the sparse depth map corresponding to the current frame to the depth compaction module to obtain the output of the first dense depth map corresponding to the current frame; determining the first reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first dense depth map; wherein the first dense depth map and the first reconstructed image are used as the first sub-result.

[0063] Furthermore, based on the difference between the first dense depth map and the sparse depth map corresponding to the current frame, as well as the difference between the first reconstructed image and the current frame, a first loss function is determined; and based on the first loss function, the parameters of the depth compaction module are adjusted.

[0064] To reduce the reliance on dense depth annotations during model training, this approach employs a self-supervised training method. The self-supervised loss function is determined as the optimization objective by comparing the differences between the predicted depth map and its corresponding sparse depth map, as well as the differences between the reconstructed image and the original image. For example, the first loss function is determined based on the differences between the first dense depth map and the sparse depth map corresponding to the current frame, and the differences between the first reconstructed image and the current frame. The difference between the first dense depth map and the sparse depth map corresponding to the current frame can be determined by calculating the difference between the depth values ​​of each sparse point in the sparse depth map of the current frame and the depth values ​​of the corresponding points in the first dense depth map. The larger the difference, the larger the value of the first loss function. The difference between the first reconstructed image and the current frame can be determined by calculating the image similarity between them, for example, using photometric loss. Higher image similarity results in a smaller value for the first loss function.

[0065] The parameters of a deep compact module can be adjusted using stochastic gradient descent, and are not limited to the examples given. For instance, the parameters of a deep compact module can be adjusted using the following formula.

[0066] (1)

[0067] in, This represents the parameters of the deep compact module (Coarse Net) at time t. This represents the parameters of the deep compaction module at time t+1. Indicates the learning rate, Denotes the first loss function. This represents the parameters of the pose estimation module at time t. This represents the input at time t. , This represents the gradient.

[0068] After the first stage, the parameters of the depth compaction module are adjusted. In some embodiments, the second stage includes: inputting the adjusted second dense depth map corresponding to the current frame output by the depth compaction module and the current frame into the depth prediction module to obtain the first predicted depth map corresponding to the current frame; determining the second reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first predicted depth map; and using the first predicted depth map and the second reconstructed image as the second sub-result. Further, a second loss function is determined based on the difference between the first predicted depth map and the sparse depth map corresponding to the current frame, and the difference between the second reconstructed image and the current frame; and the parameters of the pose estimation module and the depth prediction module are adjusted based on the second loss function.

[0069] The second loss function is similar to the first, also employing a self-supervised loss function, which will not be elaborated further. The parameters of the pose estimation and depth prediction modules can be adjusted using stochastic gradient descent, and are not limited to the examples given. For instance, the parameters of the pose estimation and depth prediction modules can be adjusted using the following formula.

[0070] (2)

[0071] in, This represents the parameters of the pose estimation module (Pose Net) and the depth prediction module (Depth Net) at time t. This represents the parameters of the pose estimation module and the depth prediction module at time t+1. This represents the parameters of the deep compaction module at time t+1. Indicates the learning rate, This represents the second loss function. This represents the input at time t. , This represents the gradient.

[0072] In the above embodiments, the first stage calculates a self-supervised loss function based on the coarse dense depth map provided by the depth compaction module and the relative pose estimated by the pose estimation module, and adjusts the parameters of the depth compaction module using stochastic gradient descent. The second stage uses the adjusted depth compaction module in conjunction with the depth prediction module to predict the final depth, calculates a self-supervised loss function based on the relative pose estimated by the pose estimation module, and adjusts the parameters of both the depth prediction and pose estimation modules. The first stage aims to train the depth completion model's adaptability to sparsity and provide a dense depth map for the second stage. After the depth input has become dense, the second stage focuses on training the depth completion model's adaptability to (RGB) image variations. By decoupling the inner loop into two stages, the depth completion model can better learn and adapt to the features of images with different modalities, improving its adaptability and accuracy.

[0073] In step S106, the next image pair and the sparse depth map corresponding to the next frame are input into the depth completion model. Based on the second result output by the depth completion model, the parameters of the depth completion model are adjusted again.

[0074] This step can be understood as an improved outer loop process based on meta-learning. In some embodiments, the next image is input to the pose estimation module to obtain the output of the next frame and the relative pose of the camera when the current frame was captured; the sparse depth map corresponding to the next frame is input to the depth compaction module to obtain the output of the third dense depth map corresponding to the next frame; the next frame and the third dense depth map are input to the depth prediction module to obtain the second predicted depth map corresponding to the next frame; based on the current frame, the next frame, the relative pose of the camera when the current frame was captured, and the second predicted depth map, the third reconstructed image corresponding to the next frame is determined; wherein, the second predicted depth map and the third reconstructed image are used as the second result. Further, based on the difference between the second predicted depth map and the sparse depth map corresponding to the next frame, and the difference between the third reconstructed image and the next frame, a third loss function is determined; based on the third loss function, the parameters of the depth compaction module, the pose estimation module, and the depth prediction module are adjusted.

[0075] The third loss function is similar to the first and second loss functions, employing a self-supervised loss function, which will not be elaborated upon here. During the outer loop, the parameters of the three modules can be directly adjusted. For example, the stochastic gradient descent algorithm can be used to adjust the parameters of the three modules, and this is not limited to the examples given. For instance, the parameters of the depth compaction module, pose estimation module, and depth prediction module can be adjusted using the following formula.

[0076] (3)

[0077] in, This represents the parameters of the depth completion model, including the parameters of the depth compaction module, pose estimation module, and depth prediction module. This indicates the parameters of the depth completion model after adjustment according to formula (2). Indicates the learning rate, This represents the third loss function. This represents the input at time t. , This represents the gradient.

[0078] like Figure 3 The diagram illustrates the parameter adjustment process of the deep completion model in one iteration, comprising an inner loop and an outer loop. The inner loop includes a first stage and a second stage. The first stage is based on the first loss function. Adjusting the parameters of the depth compaction module yields... The second stage is based on the second loss function. Adjusting the parameters of the pose estimation module and the depth prediction module yields... , During the outer loop, based on the third loss function... Adjust the parameters of the depth completion model to obtain .

[0079] The parameter tuning process of the depth completion model in the above embodiments includes an inner loop and an outer loop. For a video sequence, the purpose of online tuning of the depth completion model in the current frame is to achieve good performance in the next frame. Therefore, the model parameters are first adjusted in the inner loop for the input of the current frame, and then the model performance is tested and optimized using the adjusted parameters in the outer loop for the next frame. This training method can tap the model's potential for online tuning, enabling the model to adapt quickly to new video sequences.

[0080] In step S108, determine whether training is complete. If yes, end the process; otherwise, return to step S102 and start execution again.

[0081] Steps S104 and S106 complete one training iteration of the depth completion model. The completion of training can be determined by checking if preset convergence conditions are met. These preset convergence conditions include, for example, reaching a preset number of iterations, or the loss function value reaching a threshold or minimum value, and are not limited to the examples given. Returning to step S102, the current image pair, the next image pair, the sparse depth map corresponding to the current frame, and the sparse depth map corresponding to the next frame are reacquired. These images can come from the same video sequence as the previous iteration or from different video sequences.

[0082] The method described above obtains the current image pair and the next image pair from the video used for training. First, it inputs the current image pair and the sparse depth map corresponding to the current frame into the depth completion model. Based on the first output result, it adjusts the parameters of the depth completion model. Then, it inputs the next image pair and the sparse depth map corresponding to the next frame into the depth completion model. Based on the second output result, it adjusts the parameters of the depth completion model again. Each iteration includes two parameter adjustment processes. The first parameter adjustment process allows the depth completion model to learn how to predict depth for depth completion, while the second parameter adjustment process allows the depth completion model to learn "how to learn depth prediction," enhancing the model's ability to adapt quickly and improving its generalization ability and accuracy.

[0083] After training the depth completion model according to the method of the above embodiments, when a new video and its corresponding sparse depth map arrive, the depth completion model can be directly used for prediction to obtain a predicted depth map. For example, the sparse depth map corresponding to the current online frame of the online video is input into the depth compaction module to obtain a coarse dense depth map. The coarse dense depth map and the current online frame are then input into the depth prediction module to obtain a predicted depth map.

[0084] To improve the adaptability of depth completion models to different environments and enhance their accuracy, the model can be adjusted online when new videos arrive before predicting depth. The following section combines... Figure 4 and Figure 5 Describe the process of online adjustment.

[0085] Figure 4 Flowcharts are provided for some embodiments of the depth completion method disclosed herein. For example... Figure 4 As shown, the method of this embodiment includes steps S402 to S408.

[0086] In step S402, the current online frame of the online video and the sparse depth map corresponding to the current online frame are obtained, and the current online frame and the previous online frame are combined to form the current online image pair.

[0087] In online videos, the frames are continuous, and they can be processed sequentially according to the order in which they arrive.

[0088] In step S404, the current online image pair and the sparse depth map corresponding to the current online frame are input into the depth completion model to obtain the third result output by the depth completion model.

[0089] In step S406, the predicted depth map corresponding to the current online frame is extracted from the third result and used as the depth map corresponding to the current online frame.

[0090] In step S408, the parameters of the depth completion model are adjusted online based on the third result output by the depth completion model.

[0091] Since the images and corresponding sparse depth maps in the video are of different modalities, the online adjustment process can be divided into a first node and a second stage to improve the accuracy of depth completion. In some embodiments, the third result includes: a third sub-result and a fourth sub-result. The current online image pair and the sparse depth map corresponding to the current online frame are input into the depth completion model. Based on the third sub-result output by the depth completion model, the parameters of the depth compaction module are adjusted. Based on the adjusted parameters of the depth compaction module, the fourth sub-result output by the depth completion model is obtained again. Based on the fourth sub-result, the parameters of the pose estimation module and the depth prediction module are adjusted.

[0092] In some embodiments, the first stage includes: inputting the current online image to the pose estimation module to obtain the output of the relative pose of the camera when the current online frame was captured and the camera pose when the previous online frame was captured; inputting the sparse depth map corresponding to the current online frame to the depth compaction module to obtain the output of the fourth dense depth map corresponding to the current online frame; determining the fourth reconstructed image corresponding to the current online frame by using the relative pose of the camera when the previous online frame was captured, the current online frame, and the camera pose when the previous online frame was captured, as well as the fourth dense depth map; wherein the fourth dense depth map and the fourth reconstructed image are used as a third sub-result. Further, a fourth loss function is determined based on the difference between the fourth dense depth map and the sparse depth map corresponding to the current online frame, and the difference between the fourth reconstructed image and the current online frame; the parameters of the depth compaction module are adjusted based on the fourth loss function. The fourth dense depth map can be used as the depth map corresponding to the current online frame.

[0093] In some embodiments, the second stage includes: inputting the adjusted fifth dense depth map corresponding to the current online frame output by the depth compaction module and the current online frame into the depth prediction module to obtain the third predicted depth map corresponding to the current online frame; determining the fifth reconstructed image corresponding to the current frame by using the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the third predicted depth map; and using the third predicted depth map and the fifth reconstructed image as a fourth sub-result. Further, a fifth loss function is determined based on the difference between the third predicted depth map and the sparse depth map corresponding to the current online frame, and the difference between the fifth reconstructed image and the current online frame; and the parameters of the pose estimation module and the depth prediction module are adjusted based on the fifth loss function.

[0094] Considering that online model tuning serves as supervision, and to reduce the reliance on dense depth annotations during model training, a self-supervised loss function is used as the optimization objective during both training and model tuning. The fourth and fifth loss functions can also adopt the aforementioned self-supervised loss function form, which will not be elaborated upon here. The tuning processes in the first and second stages can refer to the aforementioned embodiments.

[0095] like Figure 5 As shown, the online adjustment includes a first stage and a second stage. First, based on a coarse dense depth map and relative pose, a self-supervised loss is calculated to optimize the Coarse Net. Then, the optimized Coarse Net is used to predict the final depth, and based on the final predicted depth and relative pose, a self-supervised loss function is calculated to optimize the Depth Net and Pose Net. The Pose Net outputs the relative pose of the camera during the two frames captured. , … Equivalents are the basis for calculating self-supervised loss. , … This is a rough density depth map. , … The image is a predicted depth map. , … , etc. are the parameters of the Coarse Net. , … These are the parameters of the depth completion model.

[0096] In step S410, it is determined whether the online video has ended. If it has, the process ends; otherwise, the process returns to step S402 and the next frame is used as the current frame to restart the process.

[0097] For scenarios where it is not necessary to obtain the depth map corresponding to each frame in real time, the predicted depth map corresponding to each online frame output by the depth completion model can be obtained after step S410 and used as the depth map corresponding to each online frame.

[0098] For scenarios that require real-time acquisition of depth maps for each frame, such as in autonomous driving scenarios, the camera and LiDAR on the autonomous vehicle can obtain one frame of image and the corresponding sparse depth map each time, which can then be input into a depth completion model to obtain a predicted depth map. This predicted depth map can be directly used to control the autonomous vehicle's driving and perform the aforementioned online adjustment process.

[0099] The method described above utilizes online video to adjust the depth completion model online, enabling the model to better adapt to online videos. Even when the content of the online video differs significantly from the video sequence used for training, the depth completion model can still accurately predict the depth information corresponding to the online video, thus improving prediction accuracy. Furthermore, decoupling the inputs for different modalities and dividing the online adjustment process into two stages can further improve the prediction accuracy of the depth completion model.

[0100] In some embodiments, a 3D scene is reconstructed based on the depth map corresponding to each online frame of the online video.

[0101] In some embodiments, three-dimensional targets in each online frame are identified based on the depth map corresponding to each online frame of the online video.

[0102] In some embodiments, the autonomous vehicle is controlled to drive based on the depth map corresponding to each online frame of the online video.

[0103] The depth map output by the depth completion model can be used for various downstream tasks, not limited to the examples given.

[0104] This disclosure also provides a training device, which is described below in conjunction with... Figure 6 Describe it.

[0105] Figure 6 These are structural diagrams of some embodiments of the training device disclosed herein. Figure 6 As shown, the device 60 in this embodiment includes: an acquisition unit 610, a first adjustment unit 620, a second adjustment unit 630, and a cycle triggering unit 640.

[0106] The acquisition unit 610 is used to acquire the current frame, the previous frame and the next frame from the trained video sequence, and to form the current image pair with the current frame and the previous frame, and to form the next image pair with the next frame.

[0107] The first adjustment unit 620 is used to input the current image pair and the sparse depth map corresponding to the current frame into the depth completion model, and adjust the parameters of the depth completion model according to the first result output by the depth completion model.

[0108] In some embodiments, the depth completion model includes a depth compaction module, a pose estimation module, and a depth prediction module. The first result includes a first sub-result and a second sub-result. The first adjustment unit 620 is used to input the current image pair and the sparse depth map corresponding to the current frame into the depth completion model, adjust the parameters of the depth compaction module according to the first sub-result output by the depth completion model, obtain the second sub-result output by the depth completion model again according to the adjusted parameters of the depth compaction module, and adjust the parameters of the pose estimation module and the depth prediction module according to the second sub-result.

[0109] In some embodiments, the first adjustment unit 620 is used to input the current image to the pose estimation module to obtain the output of the relative pose of the camera when the current frame was captured and the previous frame; input the sparse depth map corresponding to the current frame to the depth compaction module to obtain the output of the first dense depth map corresponding to the current frame; and determine the first reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first dense depth map; wherein the first dense depth map and the first reconstructed image are used as the first sub-result.

[0110] In some embodiments, the first adjustment unit 620 is used to input the second dense depth map corresponding to the current frame output by the adjusted depth compaction module and the current frame into the depth prediction module to obtain the first predicted depth map corresponding to the current frame; determine the second reconstructed image corresponding to the current frame based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first predicted depth map; and use the first predicted depth map and the second reconstructed image as the second sub-result.

[0111] In some embodiments, the first adjustment unit 620 is configured to determine a first loss function based on the difference between the first dense depth map and the sparse depth map corresponding to the current frame, and the difference between the first reconstructed image and the current frame; and adjust the parameters of the depth compaction module based on the first loss function.

[0112] In some embodiments, the first adjustment unit 620 is used to determine a second loss function based on the difference between the first predicted depth map and the sparse depth map corresponding to the current frame and the difference between the second reconstructed image and the current frame; and to adjust the parameters of the pose estimation module and the depth prediction module based on the second loss function.

[0113] The second adjustment unit 630 is used to input the next image pair and the sparse depth map corresponding to the next frame into the depth completion model, and adjust the parameters of the depth completion model again according to the second result output by the depth completion model.

[0114] In some embodiments, the second adjustment unit 630 is used to input the next image to the pose estimation module to obtain the output of the next frame and the relative pose of the camera when the current frame was captured; input the sparse depth map corresponding to the next frame to the depth compaction module to obtain the output of the third dense depth map corresponding to the next frame; input the next frame and the third dense depth map to the depth prediction module to obtain the second predicted depth map corresponding to the next frame; and determine the third reconstructed image corresponding to the next frame based on the current frame, the next frame, the relative pose of the camera when the current frame was captured, and the second predicted depth map; wherein the second predicted depth map and the third reconstructed image are used as the second result.

[0115] In some embodiments, the second adjustment unit 630 is used to determine a third loss function based on the difference between the second predicted depth map and the sparse depth map corresponding to the next frame, and the difference between the third reconstructed image and the next frame; and to adjust the parameters of the depth compaction module, the pose estimation module and the depth prediction module based on the third loss function.

[0116] The loop triggering unit 640 is used to trigger the acquisition unit, the first adjustment unit and the second adjustment unit to repeatedly execute the above process until the training of the depth completion model is completed.

[0117] The training devices in the embodiments of this disclosure can be implemented by various computing devices or computer systems, as described below. Figure 7 as well as Figure 8 Describe it.

[0118] Figure 7 These are structural diagrams of some embodiments of the training device disclosed herein. Figure 7 As shown, the apparatus 70 of this embodiment includes a memory 710 and a processor 720 coupled to the memory 710. The processor 720 is configured to execute training methods in any of the embodiments of this disclosure based on instructions stored in the memory 710.

[0119] The memory 710 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, database, and other programs.

[0120] Figure 8 These are structural diagrams of other embodiments of the training device disclosed herein. Figure 8 As shown, the device 80 in this embodiment includes a memory 810 and a processor 820, which are similar to the memory 710 and processor 720, respectively. It may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, and the memory 810 and processor 820 can be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 840 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0121] This disclosure also provides a depth completion device, which is described below in conjunction with... Figure 9 Describe it.

[0122] Figure 9 The diagram shows the structure of some embodiments of the depth completion device disclosed herein. Figure 9 As shown, the device 90 in this embodiment includes: an acquisition unit 910, an adjustment unit 920, and a depth completion unit 930.

[0123] The acquisition unit 910 is used to acquire the current online frame of the online video and the sparse depth map corresponding to the current online frame, and to form the current online image pair with the previous online frame of the current online frame.

[0124] The adjustment unit 920 is used to input the current online image pair and the sparse depth map corresponding to the current online frame into the depth completion model, and adjust the parameters of the depth completion model online according to the third result output by the depth completion model.

[0125] In some embodiments, the depth completion model includes a depth compaction module, a pose estimation module, and a depth prediction module. The third result includes a third sub-result and a fourth sub-result. The adjustment unit 920 is used to input the sparse depth map corresponding to the current online image pair and the current online frame into the depth completion model, adjust the parameters of the depth compaction module according to the third sub-result output by the depth completion model, obtain the fourth sub-result output by the depth completion model again according to the adjusted parameters of the depth compaction module, and adjust the parameters of the pose estimation module and the depth prediction module according to the fourth sub-result.

[0126] In some embodiments, the adjustment unit 920 is used to input the current online image to the pose estimation module to obtain the output of the current online frame and the relative pose of the camera when the previous online frame was captured; input the sparse depth map corresponding to the current online frame to the depth compaction module to obtain the output of the fourth dense depth map corresponding to the current online frame; and use the relative pose of the camera when the previous online frame was captured, the current online frame, and the previous online frame, as well as the fourth dense depth map, to determine the fourth reconstructed image corresponding to the current online frame; wherein the fourth dense depth map and the fourth reconstructed image are used as the third sub-result.

[0127] In some embodiments, the adjustment unit 920 is used to input the fifth dense depth map corresponding to the current online frame output by the adjusted depth compaction module and the current online frame into the depth prediction module to obtain the third predicted depth map corresponding to the current online frame; to determine the fifth reconstructed image corresponding to the current frame by taking the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the third predicted depth map; and to use the third predicted depth map and the fifth reconstructed image as the fourth sub-result.

[0128] In some embodiments, the adjustment unit 920 is used to determine a fourth loss function based on the difference between the fourth dense depth map and the sparse depth map corresponding to the current online frame, and the difference between the fourth reconstructed image and the current online frame; and to adjust the parameters of the depth compaction module based on the fourth loss function.

[0129] In some embodiments, the adjustment unit 920 is used to determine a fifth loss function based on the difference between the third predicted depth map and the sparse depth map corresponding to the current online frame, and the difference between the fifth reconstructed image and the current online frame; and to adjust the parameters of the pose estimation module and the depth prediction module based on the fifth loss function.

[0130] The depth completion unit 930 is used to take the predicted depth map corresponding to the current online frame output by the depth completion model as the depth map corresponding to the current online frame.

[0131] In some embodiments, the device 90 further includes at least one of a control unit 940, a scene reconstruction unit 950, and a target detection unit 960. The control unit 940 is configured to control the autonomous vehicle's movement based on the depth maps corresponding to each online frame of the online video. The scene reconstruction unit 950 is configured to reconstruct a three-dimensional scene based on the depth maps corresponding to each online frame of the online video; the target detection unit 960 is configured to identify three-dimensional targets in each online frame based on the depth maps corresponding to each online frame of the online video.

[0132] The training devices in the embodiments of this disclosure can be implemented by various computing devices or computer systems, as described below. Figure 10 as well as Figure 11 Describe it.

[0133] Figure 10 The diagram shows the structure of some embodiments of the depth completion device disclosed herein. Figure 10 As shown, the apparatus 100 of this embodiment includes a memory 1010 and a processor 1020 coupled to the memory 1010. The processor 1020 is configured to execute the depth completion method in any of the embodiments of this disclosure based on instructions stored in the memory 1010.

[0134] The memory 1010 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0135] Figure 11 Structural diagrams of other embodiments of the depth completion device of this disclosure are shown. Figure 11 As shown, the device 110 of this embodiment includes a memory 1110 and a processor 1120, which are similar to the memory 1010 and processor 1020, respectively. It may also include an input / output interface 1130, a network interface 1140, a storage interface 1150, etc. These interfaces 1130, 1140, 1150, and the memory 1110 and processor 1120 can be connected, for example, via a bus 1160. The input / output interface 1130 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 1140 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 1150 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0136] This disclosure also provides a deep completion system, which will be discussed below. Figure 12 Describe it.

[0137] Figure 12 The diagram shows the structure of some embodiments of the deep autocomplete system disclosed herein. Figure 12 As shown, the system 12 of this embodiment includes: the training device 60 / 70 / 80 of any of the foregoing embodiments, and the depth completion device 90 / 100 / 110.

[0138] This disclosure also provides an unmanned vehicle, which is described below in conjunction with... Figure 13 Describe it.

[0139] Figure 13 This is a structural diagram of some embodiments of the unmanned vehicle disclosed herein. Figure 13 As shown, the unmanned vehicle 13 in this embodiment includes: the depth completion device 90 / 100 / 110 of any of the preceding embodiments.

[0140] In some embodiments, the unmanned vehicle 13 further includes: a camera device 131 for acquiring online video and sending it to the depth completion device 90 / 100 / 110; and a sensor 132 for acquiring sparse depth maps corresponding to each frame in the online video and sending them to the depth completion device 90 / 100 / 110.

[0141] In some embodiments, the unmanned vehicle 13 further includes: the training device 60 / 70 / 80 of any of the foregoing embodiments. The unmanned vehicle may also include conventional components such as a vehicle body, which will not be described in detail here.

[0142] Figure 14 These are structural diagrams of other embodiments of the unmanned vehicle disclosed herein. For example... Figure 14 As shown, the unmanned vehicle 14 in this embodiment includes four parts: an autonomous driving module 141, a chassis module 142, a remote monitoring and propulsion module 143, and a cargo box module 144.

[0143] The autonomous driving module 141 includes an autonomous driving sensor and core processing unit (Orin or Xavier module) assembly 1411, a traffic light recognition camera 1412, front, rear, left, and right surround view cameras 14131, 14132, 14133, and 14134, a multi-line LiDAR 1414, a positioning module 1415 (such as BeiDou, GPS, etc.), and an inertial navigation unit 1416. The cameras and the autonomous driving module can communicate; to improve transmission speed and reduce wiring, a GMSL link communication can be used. The autonomous driving sensor and core processing unit (Orin or Xavier module) assembly 1411 includes a depth completion device as described in any of the embodiments of this disclosure and is configured to execute the depth completion method as described in any of the embodiments of this disclosure. The front, rear, left, and right surround view cameras 14131, 14132, 14133, and 14134 can function as imaging devices to acquire online video and send it to the depth completion device. The multi-line LiDAR 1414 can function as a sensor to acquire sparse depth maps corresponding to each frame in the online video.

[0144] In some embodiments, the autonomous driving module 141 further includes a switch 1417 and front, rear, left, and right blind spot radars 14181, 14182, 14183, and 14184.

[0145] The chassis module 142 mainly includes a battery 1421, a power management device 1422, a chassis controller 1423, a motor driver 1424, and a drive motor 1425. The battery 1421 provides power to the entire autonomous vehicle system. The power management device 1422 converts the battery output into different voltage levels usable by various functional modules and controls power-on and power-off. The chassis controller 1423 receives motion commands from the autonomous driving module and controls the autonomous vehicle's steering, forward movement, reverse movement, braking, etc. Motion commands, for example, are generated by a depth completion device based on the depth map corresponding to each online frame of the online video. The chassis module 142 also includes a main battery 1426.

[0146] The remote monitoring streaming module 143 consists of a front monitoring camera 1431, a rear monitoring camera 1432, a left monitoring camera 1433, a right monitoring camera 1434, and a streaming module 1435. This module transmits the video data collected by the monitoring cameras to the backend server for viewing by backend operators.

[0147] The cargo container module 144 is the cargo-carrying device for the unmanned vehicle, including a delivery container 1441. A display and interaction module 1442 is also provided on the cargo container module 144. The display and interaction module 1442 is used for interaction between the unmanned vehicle and the user, allowing the user to perform operations such as picking up items, storing goods, and purchasing goods. The type of cargo container can be changed according to actual needs. For example, in logistics scenarios, the cargo container can include multiple sub-containers of different sizes, which can be used to load goods for delivery. In retail scenarios, the cargo container can be set as a transparent container so that users can clearly see the products for sale.

[0148] The cargo box module 144 also includes an antenna 1443. The chassis module 142 also includes a wireless communication module 1427. The wireless communication module 1427 communicates with the back-end server through the antenna 1443, enabling remote control of the unmanned vehicle by back-end operators.

[0149] Figure 15 This is a side view showing an unmanned vehicle according to some embodiments of the present disclosure.

[0150] like Figure 15 As shown, the unmanned vehicle includes a display and interaction module 151, a chassis 152, a left-side blind spot radar 153, a right-side blind spot radar 154, a rear-side blind spot radar 155, a lidar 156, a right-side camera 157, and a cargo box 158. The functions of the interaction module 151, chassis 152, left-side blind spot radar 153, right-side blind spot radar 154, rear-side blind spot radar 155, lidar 156, right-side camera 157, and cargo box 158 can be found in [reference needed]. Figure 14 The description in the text will not be repeated here.

[0151] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A training method, comprising: The current frame, the previous frame, and the next frame are obtained from the video sequence used for training. The current frame and the previous frame are combined to form a current image pair, and the next frame and the current frame are combined to form a next image pair. The current image pair and the sparse depth map corresponding to the current frame are input into the depth completion model, wherein the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module; Based on the first sub-result output by the depth completion model, adjust the parameters of the depth compaction module; Based on the adjusted parameters of the depth compaction module, the second sub-result output by the depth completion model is obtained again; Based on the second sub-result, adjust the parameters of the pose estimation module and the depth prediction module; The next image pair and the sparse depth map corresponding to the next frame are input into the depth completion model. Based on the second result output by the depth completion model, the parameters of the depth completion model are adjusted again. Repeat the above process until the training of the deep completion model is complete.

2. The training method of claim 1, wherein, The step of inputting the current image pair and the sparse depth map corresponding to the current frame into the depth completion model includes: The current image is input into the pose estimation module to obtain the relative pose of the camera when the current frame and the previous frame were captured. The sparse depth map corresponding to the current frame is input into the depth compaction module to obtain the first dense depth map corresponding to the current frame. Based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first dense depth map, determine the first reconstructed image corresponding to the current frame; The first dense depth map and the first reconstructed image are used as the first sub-result.

3. The training method of claim 2, wherein, The step of obtaining the second sub-result output by the depth completion model based on the adjusted parameters of the depth compaction module includes: The adjusted second dense depth map corresponding to the current frame output by the depth compaction module and the current frame are input into the depth prediction module to obtain the first predicted depth map corresponding to the current frame. Based on the previous frame, the current frame, the relative pose of the camera when the previous frame was captured, and the first predicted depth map, determine the second reconstructed image corresponding to the current frame; The first predicted depth map and the second reconstructed image are used as the second sub-result.

4. The training method of claim 2, wherein, The step of adjusting the parameters of the depth compaction module based on the first sub-result output by the depth completion model includes: The first loss function is determined based on the difference between the first dense depth map and the sparse depth map corresponding to the current frame, as well as the difference between the first reconstructed image and the current frame. The parameters of the deep compaction module are adjusted according to the first loss function.

5. The training method of claim 3, wherein, The step of adjusting the parameters of the pose estimation module and the depth prediction module based on the second sub-result includes: The second loss function is determined based on the difference between the first predicted depth map and the sparse depth map corresponding to the current frame, and the difference between the second reconstructed image and the current frame; The parameters of the pose estimation module and the depth prediction module are adjusted according to the second loss function.

6. The training method of claim 1, wherein, The step of inputting the next image pair and the sparse depth map corresponding to the next frame into the depth completion model includes: The next image is input into the pose estimation module to obtain the relative pose of the camera when the next frame and the current frame were captured. The sparse depth map corresponding to the next frame is input into the depth compaction module to obtain the third dense depth map corresponding to the next frame. The next frame and the third dense depth map are input into the depth prediction module to obtain the second predicted depth map corresponding to the next frame; Based on the current frame, the next frame, the relative pose of the camera when the current frame was captured, and the second predicted depth map, determine the third reconstructed image corresponding to the next frame; The second predicted depth map and the third reconstructed image are used as the second result.

7. The training method of claim 6, wherein, The step of readjusting the parameters of the depth completion model based on the second result output by the depth completion model includes: The third loss function is determined based on the difference between the second predicted depth map and the sparse depth map corresponding to the next frame, and the difference between the third reconstructed image and the next frame; The parameters of the depth compaction module, the pose estimation module, and the depth prediction module are adjusted according to the third loss function.

8. A depth completion method, comprising: Obtain the current online frame of the online video and the sparse depth map corresponding to the current online frame, and combine the current online frame with the previous online frame to form a current online image pair; The current online image pair and the sparse depth map corresponding to the current online frame are input into the depth completion model, wherein the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module; Based on the third sub-result output by the depth completion model, adjust the parameters of the depth compaction module; Based on the adjusted parameters of the depth compaction module, the fourth sub-result output by the depth completion model is obtained again; Based on the fourth sub-result, adjust the parameters of the pose estimation module and the depth prediction module; The predicted depth map corresponding to the current online frame output by the depth completion model is used as the depth map corresponding to the current online frame.

9. The depth completion method of claim 8, wherein, Inputting the current online image pair and the sparse depth map corresponding to the current online frame into the depth completion model includes: The current online image is input into the pose estimation module to obtain the relative pose of the camera when the current online frame and the previous online frame were captured. Input the sparse depth map corresponding to the current online frame into the depth compaction module to obtain the fourth dense depth map corresponding to the current online frame; The fourth reconstructed image corresponding to the current online frame is determined by combining the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the fourth dense depth map. The fourth dense depth map and the fourth reconstructed image are considered as the third sub-result.

10. The depth completion method of claim 9, wherein, The step of obtaining the fourth sub-result output by the depth completion model based on the adjusted parameters of the depth compaction module includes: The adjusted fifth dense depth map corresponding to the current online frame output by the depth compaction module and the current online frame are input into the depth prediction module to obtain the third predicted depth map corresponding to the current online frame; The fifth reconstructed image corresponding to the current online frame is determined by combining the previous online frame, the current online frame, the relative pose of the camera when the previous online frame was captured, and the third predicted depth map. The third predicted depth map and the fifth reconstructed image are used as the fourth sub-result.

11. The depth completion method of claim 9, wherein, The step of adjusting the parameters of the depth compaction module based on the third sub-result output by the depth completion model includes: The fourth loss function is determined based on the difference between the fourth dense depth map and the sparse depth map corresponding to the current online frame, as well as the difference between the fourth reconstructed image and the current online frame; The parameters of the deep compaction module are adjusted according to the fourth loss function.

12. The depth completion method of claim 10, wherein, The step of adjusting the parameters of the pose estimation module and the depth prediction module based on the fourth sub-result includes: The fifth loss function is determined based on the difference between the third predicted depth map and the sparse depth map corresponding to the current online frame, as well as the difference between the fifth reconstructed image and the current online frame. The parameters of the pose estimation module and the depth prediction module are adjusted according to the fifth loss function.

13. The depth completion method according to claim 8, further comprising at least one of the following: The autonomous vehicle is controlled to move based on the depth map corresponding to each online frame of the online video. Based on the depth map corresponding to each online frame of the online video, a 3D scene is reconstructed; Based on the depth map corresponding to each online frame of the online video, identify the three-dimensional targets in each online frame.

14. A training device, comprising: The acquisition unit is used to acquire the current frame, the previous frame and the next frame from the trained video sequence, and to combine the current frame and the previous frame into a current image pair, and to combine the next frame and the current frame into a next image pair. The first adjustment unit is used to input the current image pair and the sparse depth map corresponding to the current frame into a depth completion model, wherein the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. Based on the first sub-result output by the depth completion model, the parameters of the depth compaction module are adjusted. Based on the adjusted parameters of the depth compaction module, the second sub-result output by the depth completion model is obtained again. Based on the second sub-result, the parameters of the pose estimation module and the depth prediction module are adjusted. The second adjustment unit is used to input the sparse depth map corresponding to the next image pair and the next frame into the depth completion model, and adjust the parameters of the depth completion model again according to the second result output by the depth completion model. The loop triggering unit is used to trigger the acquisition unit, the first adjustment unit and the second adjustment unit to repeatedly execute the above process until the training of the deep completion model is completed.

15. A depth completion device, comprising: The acquisition unit is used to acquire the current online frame of the online video and the sparse depth map corresponding to the current online frame, and to form a current online image pair by combining the current online frame with the previous online frame of the current online frame; An adjustment unit is used to input the current online image pair and the sparse depth map corresponding to the current online frame into a depth completion model, wherein the depth completion model includes: a depth compaction module, a pose estimation module, and a depth prediction module. Based on the third sub-result output by the depth completion model, the parameters of the depth compaction module are adjusted. Based on the adjusted parameters of the depth compaction module, a fourth sub-result output by the depth completion model is obtained again. Based on the fourth sub-result, the parameters of the pose estimation module and the depth prediction module are adjusted. The depth completion unit is used to take the predicted depth map corresponding to the current online frame output by the depth completion model as the depth map corresponding to the current online frame.

16. A training device, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the training method as described in any one of claims 1-7.

17. A depth completion device, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the deep completion method as described in any one of claims 8-13.

18. A deep completion system, comprising: The training apparatus of claim 14 or 16 and the depth completion apparatus of claim 15 or 17.

19. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein, When executed by a processor, the program implements the steps of the method according to any one of claims 1-14.

20. An unmanned vehicle comprising: The depth completion device according to claim 15 or 17.

21. The unmanned vehicle according to claim 20, further comprising: A camera device is used to acquire online video and send it to the depth completion device; A sensor is used to acquire sparse depth maps corresponding to each frame in the online video and send them to the depth completion device.

22. The unmanned vehicle of claim 20, further comprising: The training apparatus according to claim 14 or 16.

Images