Depth map interpolation method, apparatus, computer device, and computer program

Abstract

This application discloses a depth map interpolation method, apparatus, computer device, and storage medium relating to the field of artificial intelligence. The method includes the steps of: aggregating features of a scene graph and a sparse depth map to obtain aggregated features, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph; applying noise processing to the aggregated features; and spreading and interpolating the aggregated features by a depth interpolation network based on a diffusion intensity parameter to obtain depth interpolation features, wherein the depth interpolation network is based on a diffusion model, and the diffusion intensity parameter is used to control the intensity of dediffusion in the depth interpolation process; and restoring an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has a higher degree of completeness of depth information than the sparse depth map. By employing the method of the embodiment of this application, the risk of overfitting in the model training process can be reduced and the robustness of the depth interpolation network in the inference stage can be improved.

Description

Technical Field

[0001] This application claims priority based on a Chinese patent application filed on October 20, 2023, with an application number of 202311375449.1 and an invention title of "Depth Map Completion Method, Apparatus, Computer Device, and Storage Medium", the entire content of which is incorporated herein by reference.

[0002] The embodiments of this application relate to the field of artificial intelligence, and in particular, to a depth map completion method, apparatus, computer device, and storage medium.

Background Art

[0003] In the field of 3D AIGC (Artificial Intelligence Generated Content), depth map completion is one of the important basic tasks. Depth map completion is a process of obtaining a dense depth map by complementing a sparse depth map based on the sparse depth map and the related scene graph, and the dense depth map contains the scene depth information of each pixel in the scene graph.

[0004] In related technologies, usually, a neural network model is used to achieve depth map completion. For example, by using the supervised learning method, a large number of images and depth information in a specific scene are collected and calibrated, and then the neural network model is trained using the calibrated data, and further, other sparse depth maps in that scene are complemented using the trained neural network model.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when training a neural network model with a supervised learning-based method, the phenomenon of overfitting is likely to occur, resulting in a decline in the quality of depth completion.

Means for Solving the Problems

[0006] Embodiments of this application provide a depth map interpolation method, apparatus, computer device, and storage medium. The technical invention is as follows:

[0007] In one embodiment, an embodiment of the present application provides a depth map interpolation method performed by a computer device, the method being A step of aggregating the features of a scene graph and a sparse depth map to obtain an aggregated feature, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph, and the aggregated feature is subjected to noise addition processing. A step of obtaining depth-completed features by diffusing and completing the aggregated features using a depth-completed network based on diffusion intensity parameters, wherein the depth-completed network is based on a diffusion model, and the diffusion intensity parameters are used to control the intensity of backdiffusion in the depth-completed process. The method includes the step of restoring an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has a higher degree of completeness of depth information than the sparse depth map.

[0008] In another embodiment, the embodiments of this application provide a depth map interpolation device, the device is A feature aggregation module that aggregates features of a scene graph and a sparse depth map to obtain aggregated features, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph, and the aggregated features are subjected to noise addition processing. A depth interpolation module that obtains depth interpolation features by dispersing and interpolating the aggregated features using a depth interpolation network based on a diffusion intensity parameter, wherein the depth interpolation network is based on a diffusion model, and the diffusion intensity parameter is used to control the intensity of backdiffusion in the depth interpolation process. An image restoration module that restores an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has a higher degree of depth information completeness than the sparse depth map.

[0009] In yet another embodiment, the embodiment of the present application provides a computer device including a processor and memory, wherein at least one computer command is stored in the memory, and the at least one computer command is loaded and executed by the processor to realize the depth map interpolation method described in the above embodiment.

[0010] In yet another embodiment, the embodiment of the present application provides a computer-readable storage medium in which at least one computer command is stored, and the at least one computer command is loaded and executed by a processor to realize the depth map interpolation method described in the above embodiment.

[0011] In yet another embodiment, an embodiment of the present application provides a computer program product including a computer command, the computer command being stored in a computer-readable storage medium. A processor of a computer device reads the computer command from the computer-readable storage medium, and the processor executes the computer command, thereby causing the computer device to perform the depth map interpolation method described in the above embodiment. [Effects of the Invention]

[0012] In the embodiments of this application, features of the scene graph and sparse depth map are aggregated and noise addition processing is used to obtain aggregated features. Based on the diffusion intensity parameter, the aggregated features are diffused and interpolated by a depth interpolation network to generate depth interpolation features for image reconstruction and obtain a dense depth map. By adopting the method according to the embodiments of this application, the diffusion noise reduction process using a diffusion model is incorporated into the depth interpolation task, reducing the risk of overfitting in the training process of the depth interpolation network, improving the robustness of the depth interpolation network in the inference stage, and thus improving the quality of dense depth map generation.

[0013] To more clearly explain the technical concepts in the embodiments of this application, the drawings used in describing the embodiments are briefly introduced below. Clearly, the drawings described below show only a portion of the embodiments of this application, and those skilled in the art can obtain other drawings based on these without expending any creative effort. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram of the configuration of a computer system according to an exemplary embodiment of this application. [Figure 2] This is a flowchart of a depth map interpolation method according to an exemplary embodiment of this application. [Figure 3] This is a schematic diagram of a depth interpolation network according to an exemplary embodiment of this application. [Figure 4] This is a schematic diagram of the internal structure of the residual layer ResBlock-D according to an exemplary embodiment of this application. [Figure 5] This is a schematic diagram of the internal structure of the residual layer ResBlock-U according to an exemplary embodiment of this application. [Figure 6] This is a schematic diagram of a process in which a diffusion and interpolation of aggregated features is iterated N times using a depth interpolation network, according to an exemplary embodiment of this application. [Figure 7] This is a flowchart for training a depth interpolation network according to an exemplary embodiment of this application. [Figure 8] It is a configuration block diagram of a depth map completion device according to an exemplary embodiment of the present application. [Figure 9] It is a schematic configuration diagram of a computer device according to an exemplary embodiment of the present application.

Mode for Carrying Out the Invention

[0015] In order to make the object, technical solution and advantages of the present application clearer, the embodiments of the present application will be described in more detail below with reference to the drawings.

[0016] In the field of 3D AIGC, depth map completion is one of the important basic tasks. Depth map completion is a process of obtaining a dense depth map by complementing a sparse depth map based on the sparse depth map and the related scene graph, and the dense depth map contains the scene depth information of each pixel in the scene graph.

[0017] In related technologies, usually, a neural network model is used to realize depth map completion. For example, using the method of supervised learning, a large number of images and depth information in a specific scene are collected and calibrated, and then a neural network model is trained using the calibrated data, and further, other sparse depth maps in that scene are complemented using the trained neural network model.

[0018] However, when training a neural network model with a supervised learning-based method, the phenomenon of overlearning is likely to occur, and the quality of depth completion decreases.

[0019] The present application proposes a depth map completion method based on a diffusion model, which can improve the completion quality of depth maps.

[0020] The depth map completion method proposed in the present application can be applied to various application scenes related to depth perception, including but not limited to the following scenes. 1. In the field of 3D modeling, the depth map interpolation method proposed in this application can interpolate depth maps acquired by various depth sensing devices (e.g., LiDAR, depth cameras, etc.) and depth maps calculated by multi-view depth algorithms, thereby improving the completeness of depth sensing and the modeling quality of 3D models. 2. Autonomous driving field. Based on the depth map interpolation method proposed in this application, the scene depth sensed by the in-vehicle device can be interpolated, thereby enhancing the ability of the autonomous driving algorithm to sense the entire surrounding environment and enabling more effective driving decisions. 3. Augmented Reality Field. Based on the depth map interpolation method proposed in this application, the depth estimation results of real scenes can be improved, thereby allowing for a better understanding of the relative relationship between real scenes and virtual objects from the current viewpoint and improving the user experience. 4. 3D printing field. Based on the depth map interpolation method proposed in this application, it is possible to better sense the depth information of the target object, thereby fully capturing the details of the printed object and improving the quality of the printed model.

[0021] In some embodiments, the depth map interpolation method proposed in this application is applicable to monocular depth map interpolation tasks, thereby interpolating sparse depth maps corresponding to monocular scene graphs. Monocular depth map interpolation, also known as monocular depth estimation, involves using a scene graph from one or a single viewpoint to estimate the distance of each pixel point in the scene graph to the source of photography.

[0022] The depth map interpolation method proposed in this application is also applicable to game development scenarios. For example, in AR (Augmented Reality Games) and VR (Virtual Reality Games), to enhance realism, users can scan the real environment using VR or AR devices, and the device can construct a virtual environment in the game based on the environmental scan results. Since constructing the virtual environment requires the use of the environment's depth information, if a VR or AR device can only acquire a sparse depth map based on the environmental scan results, the depth map interpolation method proposed in this application can be used to perform depth map interpolation based on the sparse depth map and the environmental images captured during the scanning process, thereby constructing a virtual environment based on a dense depth map.

[0023] Referring to Figure 1, a block diagram of a computer system according to an exemplary embodiment of this application, the computer system may include a terminal 110 and a server 120. Data communication between the terminal 110 and the server 120 is performed via a communication network. Optionally, the communication network may be either a wired network or a wireless network, and may be at least one of a local area network, a metropolitan area network, and a wide area network.

[0024] Terminal 110 is an electronic device on which an application with a depth map interpolation function is installed. This depth map interpolation function is realized by performing depth interpolation tasks based on a depth interpolation network. This depth map interpolation function is provided as a function of a native application within terminal 110 or as a function of a third-party application, and terminal 110 may be a smartphone, tablet computer, notebook computer, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 1 shows terminal 110 as a desktop computer, but is not limited to this.

[0025] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In embodiments of this application, Server 120 may also be a backend server for an application with depth map interpolation capabilities, and Server 120 stores depth interpolation networks and associated network parameters.

[0026] In some embodiments, data is exchanged between the server and the terminal. Schematically, as shown in Figure 1, the terminal 110 acquires a scene graph and a sparse depth map corresponding to the scene graph, and transmits the scene graph and sparse depth map to the server 120. The server 120 aggregates the features of the sparse depth map and scene graph, obtains aggregated features using noise addition processing, and further, based on diffusion intensity parameters, the server 120 diffuses and interpolates the aggregated features using a depth interpolation network to obtain depth interpolated features. Based on these depth interpolated features, the image is reconstructed to obtain a dense depth map corresponding to the scene graph, and the dense depth map is returned to the terminal 110.

[0027] Referring to Figure 2, which is a flowchart of a depth map interpolation method according to an exemplary embodiment of the present application, this embodiment describes an example of a method applied to a computer device (including terminals and / or servers), the method comprising the following steps.

[0028] Step 201: The features of the scene graph and sparse depth map are aggregated to obtain aggregated features. The sparse depth map is a depth map in which there are gaps in the depth information corresponding to the scene graph, and noise is added to the aggregated features.

[0029] The scene graph is the original image used to calculate the corresponding depth map. In some examples, the scene graph is an RGB (Red, Green, Blue) 3-channel color image.

[0030] Optionally, the scene graph consists of images captured by an image acquisition device (e.g., a camera).

[0031] Optionally, the scene graph is an image obtained through preprocessing or various calculations.

[0032] A sparse depth map is a depth map that has missing depth information corresponding to the scene graph. In other words, a sparse depth map contains depth information for only some of the pixel points.

[0033] For example, a scene graph is an RGB image with resolution (H, W), where H is the pixel height and W is the pixel width. A sparse depth map corresponding to the scene graph is a depth image with resolution (H, W), but the sparse depth map contains depth information for only some of the pixel points in the scene graph; that is, depth information corresponding to some of the pixel points in the scene graph is missing in the sparse depth map.

[0034] Optionally, a sparse depth map is a depth map acquired by a depth-sensing device (e.g., a lidar, depth camera, etc.). Due to physical hardware limitations, factors such as reflections from smooth object surfaces, translucent or transparent objects, dark-colored objects, and objects outside the measurement range may cause some depth information to be lost in the depth map acquired by the depth-sensing device.

[0035] Optionally, the sparse depth map may be a depth map obtained through preprocessing or various calculations, in which some depth information is missing.

[0036] In some embodiments, the computer device encodes the scene graph and the sparse depth map to obtain scene features corresponding to the scene graph and sparse depth features corresponding to the sparse depth map, and further aggregates the scene features and sparse depth features and obtains aggregated features by noise addition processing.

[0037] Optionally, the noise may be obtained by random sampling of Gaussian noise, or it may be other types of noise. Optionally, a computer device may obtain a noise image with resolution (H,W) by sampling Gaussian noise based on the image size of a sparse depth map, and the values ​​of each pixel point in the noise image.

number

number

[0038] In several other embodiments, the computer device may aggregate image features by other means, for example, by first fusing a scene graph and a sparse depth map image, encoding the fused image, and adding noise to obtain aggregated features. The embodiments of this application are not limited to specific methods of feature aggregation.

[0039] In some embodiments, the computer device employs a feature aggregation model, including a CNN (Convolutional Neural Network), and directly obtains the noise-added aggregated features output from the feature aggregation model by inputting a scene graph and a sparse depth map into the feature aggregation model.

[0040] For example, the feature resolution of the aggregated features obtained by feature aggregation processing is:

number

[0041] Step 202: Based on the diffusion intensity parameter, the aggregated features are diffused and complemented by the depth interpolation network to obtain depth interpolation features. The depth interpolation network is based on a diffusion model, and the diffusion intensity parameter is used to control the intensity of backdiffusion in the depth interpolation process.

[0042] Depth interpolation networks are used to perform depth interpolation on aggregated features, and the resulting depth-interpolated features allow for the reconstruction of depth maps that contain more depth information.

[0043] Optionally, the depth completion network may be a depth completion network based on a diffusion model, and may also be abbreviated as a DiffDC network.

[0044] The diffusion model, also known as the denoising diffusion probabilistic model (DDPM), is a model that enables content generation using artificial intelligence. The algorithmic theoretical foundation of the diffusion model is training a Markov chain parameterized by variational inference.

[0045] The training process for a diffusion model involves two stages: the forward diffusion process and the reverse diffusion process.

[0046] A forward-spreading process is a process that specifies an initial data distribution and continuously adds Gaussian noise to that distribution; therefore, a forward-spreading process is a Markov process.

[0047] The dediffusion process is used to sequentially restore noise to the initial data. By predicting and removing the noise added at each step, the dediffusion process gradually restores the image to a noise-free state.

[0048] Optionally, the diffusion intensity parameter is a parameter for controlling the inverse diffusion intensity in the depth interpolation process. That is, the computer device can adjust the degree of noise reduction at each step in the inverse diffusion process by setting different diffusion intensity parameters. Optionally, the diffusion intensity parameter may be a fixed value set in advance before depth interpolation, or it may be a value that is dynamically adjusted based on the progress of depth interpolation in the depth interpolation process, and the embodiments of this application are not limited thereto. Exemplarily, the diffusion intensity parameter

number

number

[0049] In some embodiments, after acquiring aggregated features, the computer device obtains depth-completed features corresponding to a sparse depth map by spreading and completing the aggregated features using a depth completion network based on diffusion intensity parameters.

[0050] Step 203: Reconstruct the image based on depth interpolation features to obtain a dense depth map, and the depth information in this dense depth map has higher depth information completeness than that in a sparse depth map.

[0051] In one possible implementation, a computer device can obtain a dense depth map by using a neural network model to reconstruct an image based on depth-filled features. For example, the computer device can input depth-filled features into a CNN network, which then learns and extracts the depth-filled features to generate a dense depth map.

[0052] Optionally, dense depth maps offer greater completeness of depth information than sparse depth maps; that is, the number of pixel points with depth information in a dense depth map is greater than the number of pixel points with depth information in a sparse depth map.

[0053] For example, a sparse depth map might lack depth information for 1000 pixel points, while a dense depth map might lack depth information for only 500 or 300 pixel points, or the dense depth map might include depth information for each pixel point.

[0054] In some embodiments, the computer device may gradually impart depth information of pixel points in a sparse depth map through multiple rounds of iteration, that is, through multiple rounds of iteration, a dense depth map with higher depth information completeness is gradually obtained, and the more rounds of iteration there are, the higher the depth information completeness of the dense depth map.

[0055] As described above, by aggregating the features of the scene graph and sparse depth map and obtaining aggregated features using noise addition processing, the aggregated features are diffused and complemented by a depth interpolation network based on the diffusion intensity parameter to generate depth interpolation features for image reconstruction and obtain a dense depth map. By adopting the method of the embodiment of this application, the diffusion noise reduction process of the diffusion model is introduced into the depth interpolation task, reducing the risk of overfitting in the training process of the depth interpolation network, improving the robustness of the depth interpolation network in the inference stage, and thus improving the quality of dense depth map generation.

[0056] Regarding the method for obtaining aggregated features, in some embodiments, a computer device can obtain aggregated features by encoding sparse depth map and scene graph features, respectively, and then adding noise to the encoded features. This process includes the following steps.

[0057] Step 1: The sparse depth map features are encoded by the first encoder to obtain the sparse depth features.

[0058] In some embodiments, a computer device can obtain sparse depth features by encoding the sparse depth map features with a first encoder.

[0059] The first encoder may optionally be a CNN-structured encoder, a self-encoder, a Transformer encoder, or any other encoder capable of encoding image features, and the embodiments of this application are not limited thereto.

[0060] Optionally, the first encoder is a CNN-structured encoder, which includes two convolutional neural network layers and one nonlinear activation function. The number of convolutional kernels in the first CNN layer and the second CNN layer may be the same or different. For example, the first CNN layer may contain 32 1x1 convolutional kernels, and the second CNN layer may contain 32 3x3 convolutional kernels.

[0061] Optionally, the image resolution of the sparse depth map is

number

[0062] For example, a computer device has a resolution that is determined by a first encoder.

number

number

number

number

number

number

number

[0063] Step 2: The scene graph features are encoded using a second encoder to obtain the scene features.

[0064] In some embodiments, a computer device can obtain scene features by encoding scene graph features with a second encoder.

[0065] The embodiments of this application are not limited thereto, but the second encoder may optionally be a CNN-structured encoder, a self-encoder, a Transformer encoder, or any other encoder capable of encoding image features.

[0066] Optionally, the second encoder is a CNN-structured encoder, which includes two convolutional micronetwork layers and one nonlinear activation function. The number of convolutional kernels in the first CNN layer and the second CNN layer may be the same or different. For example, the first and second CNN layers may each contain 32 3x3 convolutional kernels.

[0067] Optionally, to improve image data processing efficiency, the computer device may first normalize the pixel values ​​of each pixel point in the scene graph before encoding the scene graph with a second encoder, and then use the second encoder to encode the normalized scene graph features.

[0068] Optionally, the image resolution of the scene graph is:

number

[0069] For example, a computer device has a resolution that is (

number

number

number

number

number

number

number

[0070] Step 3: Scene features and sparse depth features are aggregated, and noise is added to obtain aggregated features, ensuring that the dimensions of the scene features, sparse depth features, and noise are consistent.

[0071] In some embodiments, after obtaining sparse depth features corresponding to a sparse depth map and scene features corresponding to a scene graph, the computer device aggregates the scene features and sparse depth features, and then obtains aggregated features by noise addition processing.

[0072] Note that the dimensions of scene features, sparse depth features, and noise are consistent. Optionally, the resolution and channel dimensions of scene features, sparse depth features, and noise are consistent. For example, the resolution of scene features is

number

number

number

[0073] In some embodiments, the computer device first combines scene features, sparse depth features, and noise, and then obtains aggregated features by convolution.

[0074] Optionally, the computer device first uses sparse depth features

number

number

number

number

number

number

number

number

[0075] As those skilled in the art will understand, the above description of the first encoder, the second encoder, and feature aggregation is merely an example, and encoding and feature aggregation may be performed on the scene graph and sparse depth map by other reasonable feature encoding methods. For example, the 2D 1x1 convolution operation in the first encoder can be replaced with a multilayer perceptron or Transformer structure, and the second encoder can be replaced with any image pre-trained large-scale model, etc. All such variations are, for example, within the scope of protection of this application.

[0076] Optionally, the depth interpolation network includes a downsampling spread subnetwork and an upsampling spread subnetwork. To extract deep information from aggregated features, a computer device can first perform downsampling spread on the aggregated features to obtain downsampling depth features, and then perform upsampling spread on the downsampling depth features to obtain depth interpolation features. This process includes the following steps:

[0077] Step 1: Based on the diffusion intensity parameter, downsampling diffusion is performed on the aggregated features using a downsampling diffusion subnetwork to obtain downsampling depth features.

[0078] In some embodiments, a computer device can obtain downsampled depth features by performing downsampled diffusion on aggregate features using a downsampled diffusion subnetwork based on diffusion intensity parameters. Optionally, the resolution of the aggregate features is

number

number

[0079] Optionally, to improve downsampling accuracy and obtain more effective image features, the computer device may include multiple downsampling layers in the downsampling spread subnetwork to perform stepwise downsampling spread on aggregated features.

[0080] Optionally, a downsampling spread subnetwork includes n downsampling layers, where n is a positive integer.

[0081] For example, when n is 4, the downsampling spread subnetwork performs stepwise downsampling spread on aggregated features using four downsampling layers. The resolution levels are as follows:

number

number

number

number

[0082] Refer to Figure 3, which is a schematic diagram of a depth interpolation network according to an exemplary embodiment of the present application.

[0083] As shown in Figure 3, the depth interpolation network includes a downsampling spread subnetwork 341. The downsampling spread subnetwork 341 contains n downsampling layers (n=4), which are downsampling layers D0, D1, D2, and D3, respectively.

[0084] Regarding the process of obtaining downsampling depth features by performing downsampling spread on aggregate features using a downsampling spread subnetwork, in some embodiments, a computer device can sequentially perform downsampling on aggregate features using n downsampling layers.

[0085] In some embodiments, the computer device obtains a first downsampled feature by performing downsampling diffusion on the aggregated feature using a first downsampling layer based on the diffusion intensity parameter, then obtains the (i+1)th downsampled feature by performing downsampling diffusion on the ith downsampled feature using the (i+1)th downsampling layer based on the diffusion intensity parameter, and finally, the nth downsampled feature output from the nth downsampling layer is used as the downsampling depth feature.

[0086] In general terms, as shown in Figure 3, the computer device encodes the sparse depth map 311 using a first encoder 321 to obtain sparse depth features 331, encodes the scene graph 312 using a second encoder 322 to obtain scene features 332, aggregates the sparse depth features 331 and scene features 332, applies noise processing with noise 313 to obtain aggregated features, and uses these aggregated features as input to the first downsampling layer D0 in the downsampling spread subnetwork 341.

[0087] In some embodiments, the input to the downsampling diffusion subnetwork further includes encoded diffusion intensity parameters. Schematically, as shown in Figure 3, the computer device encodes the diffusion intensity parameter 314 feature to obtain a diffusion intensity feature 334, which is also used as the input to the first downsampling layer D0 in the downsampling diffusion subnetwork 341.

[0088] Diffusion intensity parameters

number

number

number

number

number

number

number

number

number

[0089] Subsequently, a two-layer linear neural network is used to generate high-dimensional diffusion intensity features.

number

number

number

number

number

number

[0090] As those skilled in the art will understand, the above method for encoding the diffusion intensity parameters is merely an example, and the diffusion intensity parameters can also be encoded by other reasonable feature encoding methods. For example, a linear neural network can be replaced with a Transformer structure, or the sine and cosine functions can be replaced with other basis functions (e.g., spherical harmonics), and all such variations are within the scope of this application.

[0091] The process for obtaining downsampling depth features will be explained below, using a downsampling spread subnetwork 341 containing four downsampling layers as an example.

[0092] In some embodiments, the computer device can obtain a first downsampled feature by performing downsampling diffusion on the aggregated feature using a first downsampling layer based on the diffusion intensity parameter.

[0093] For example, a computer device inputs the diffusion intensity feature 334 and aggregated feature (obtained by aggregating the sparse depth feature 331 and scene feature 332 and adding noise 313) into the first downsampling layer D0, and outputs the first downsampling feature from D0.

number

[0094] For example, the first downsampling layer D0 is downsampling by 1x, and the first downsampling feature output from the first downsampling layer D0 is

number

number

[0095] In some embodiments, the computer device performs downsampling diffusion on the i+1th downsampling feature by the i+1th downsampling layer based on the diffusion intensity parameter to obtain the i+1th downsampling feature.

[0096] For example, i = 1, 2, or 3.

[0097] For example, a computer device has a first downsampling feature

number

number

number

[0098] For example, the second downsampling layer D1 is a 2x downsampling, and the second downsampling feature

number

number

[0099] Furthermore, the computer device has a second downsampling feature

number

number

number

[0100] For example, the third downsampling layer D2 is a 2x downsampling, and the third downsampling feature

number

number

[0101] Ultimately, the computer device has a third downsampling feature.

number

number

number

number

[0102] For example, the fourth downsampling layer D3 is a 2x downsampling, and the fourth downsampling feature

number

number

[0103] Step 2: Based on the diffusion intensity parameters, upsampling diffusion is performed on the downsampling depth features using an upsampling diffusion subnetwork to obtain depth-complementary features.

[0104] In some embodiments, a computer device can obtain depth-interpolated features by performing upsampling diffusion on downsampling depth features using an upsampling diffusion subnetwork based on diffusion intensity parameters. Optionally, aggregated features have a resolution of

number

number

number

[0105] Optionally, to improve downsampling accuracy and acquire more effective image features, the computer device may include multiple upsampling layers in the upsampling spread subnetwork to perform stepwise upsampling spread on downsampling depth features.

[0106] Optionally, the number of upsampling layers in an upsampling spread subnetwork may be the same as the number of downsampling layers in a downsampling spread subnetwork; that is, the upsampling layers may also include n upsampling layers.

[0107] For example, when n is 4, the upsampling spread subnetwork performs stepwise upsampling spread on downsampling depth features using four upsampling layers. The resolution levels are as follows:

number

number

number

number

[0108] In general terms, as shown in Figure 3, the depth interpolation network includes an upsampling spread subnetwork 342. The upsampling spread subnetwork 342 includes n upsampling layers (n=4), which are upsampling layers U0, U1, U2, and U3, respectively.

[0109] In some embodiments, the computer device may perform upsampling diffusion on a downsampled depth feature using a first upsampling layer based on a diffusion intensity parameter to obtain a first upsampled feature, and further perform upsampling diffusion on the i+1th upsampled feature using an i+1th upsampling layer based on a diffusion intensity parameter to obtain an i+1th upsampled feature, and finally use the nth upsampled feature output from the nth upsampling layer as the depth-complemented feature.

[0110] In some embodiments, to extract even deeper features, the computer device may place a bottleneck encoding subnetwork between the downsampling and upsampling subnetworks before processing the downsampling depth features using the upsampling spread subnetwork. First, the bottleneck encoding is performed on the downsampling depth features by the bottleneck encoding subnetwork, and then the encoded features are input to the upsampling spread subnetwork, thereby optimizing the sampling process and reducing the risk of overfitting.

[0111] Optionally, the input features of the bottleneck encoding subnetwork have a resolution of

number

number

[0112] In general terms, as shown in Figure 3, the depth interpolation network includes a bottleneck encoding subnetwork mid, which lies between the downsampling spread subnetwork 341 and the upsampling spread subnetwork 342.

[0113] In some embodiments, the computer device outputs a fourth downsampling feature from the fourth downsampling layer D3.

number

number

[0114] The process of obtaining depth-interpolated features will be explained below, using the example of an upsampling spread subnetwork 342 containing four upsampling layers.

[0115] In some embodiments, a computer device can obtain a first upsampled feature by performing upsampling diffusion on a downsampled depth feature using a first upsampling layer based on a diffusion intensity parameter.

[0116] For example, a computer device has a diffusion intensity feature 334 and a downsampling depth feature.

number

number

number

number

[0117] For example, the first upsampling layer U0 is a 2x upsampling, and the first upsampling feature

number

number

[0118] In some embodiments, the computer device may perform upsampling diffusion on the i+1th upsampling feature by an upsampling layer based on the diffusion intensity parameter to obtain the (i+1th)th upsampling feature.

[0119] For example, i = 1, 2, or 3.

[0120] For example, a computer device has a first upsampling feature

number

number

number

[0121] Exemplarily, the second upsampling layer U1 is a 2-fold upsampling, and the second upsampling feature

Number

Number

[0122] Furthermore, the computer device inputs the second upsampling feature

Number

Number

Number

[0123] Exemplarily, the third upsampling layer U2 is a 2-fold upsampling, and the third upsampling feature

Number

Number

[0124] Finally, the computer device

Number

number

number

number

[0125] For example, the fourth upsampling layer U0 is a 1x upsampling layer, and the fourth upsampling feature

number

number

[0126] In some embodiments, computer devices perform depth interpolation features.

number

[0127] Optionally, the image resolution of the dense depth map is

number

[0128] Exemplarily, depth completion features [Number] The process of restoring an image based on 351 to obtain a dense depth map 361 can be realized by a single two-dimensional convolution operation, and the formula is as follows. [Number]

[0129] In some embodiments, in order to fuse feature information of different scales, before upsampling diffusion, the computer device first fuses the i-th upsampling feature and the (n - i)-th downsampling feature to obtain the i-th fused feature, and then, based on the diffusion intensity parameter, performs upsampling diffusion on the i-th fused feature by the (i + 1)-th upsampling layer to obtain the (i + 1)-th upsampling feature.

[0130] Exemplarily, i = 1, 2 or 3.

[0131] For example, the computer device first [Number] the first upsampling feature [Number] and the third downsampling feature [Number] to obtain the first fused feature [Number] and then, based on the diffusion intensity parameter, performs upsampling diffusion on the first fused feature by the second upsampling layer U1 to obtain the second upsampling feature

[0132] Furthermore, the computer device has a second upsampling feature.

number

number

number

number

[0133] Ultimately, the computer device has a third upsampling feature.

number

number

number

number

[0134] In this embodiment, by setting up multiple downsampling layers in the downsampling spread subnetwork to downsample aggregated features layer by layer, and by setting up multiple upsampling layers in the upsampling spread subnetwork to upsample downsampling depth features layer by layer, sampling accuracy is improved and image features can be sufficiently extracted.

[0135] Furthermore, in the upsampling process, by fusing the i-th upsampling feature with the ni-th downsampling feature to obtain the i-th fused feature, and then upsampling the i-th fused feature, the upsampling layer in the upsampling spread subnetwork can learn feature information at different scales, thereby improving the effectiveness of depth interpolation.

[0136] The sampling diffusion process for each downsampling layer in a downsampling diffusion subnetwork and each upsampling layer in an upsampling diffusion subnetwork will be described below with reference to examples.

[0137] In some embodiments, the downsampling layer, upsampling layer, and bottleneck encoding subnetwork mid each include sequentially connected residual blocks (ResBlock), attention layers, and other residual blocks.

[0138] Based on the difference in the sampling operation of the sampling layer (Sample) in the residual layer (ResBlock), the residual layer ResBlock is further divided into three types: ResBlock-D, ResBlock-I, and ResBlock-U. If the Sample operation is downsampling, the corresponding ResBlock is ResBlock-D; if the Sample operation is upsampling, the corresponding ResBlock is ResBlock-U; and if the Sample operation is identity mapping, i.e., no sampling operation is performed, the corresponding ResBlock is ResBlock-I.

[0139] For example, the downsampling layer D0 and the upsampling layer U3 sequentially include a residual layer ResBlock-I, an attention layer, and another residual layer ResBlock-I.

[0140] For example, the downsampling layers D1, D2, and D3 sequentially include a residual layer ResBlock-D, an attention layer, and a residual layer ResBlock-I.

[0141] For example, the upsampling layers U0, U1, and U2 sequentially include a residual layer ResBlock-U, an attention layer, and a residual layer ResBlock-I.

[0142] In the downsampling process, the residual layer ResBlock-D is used to fuse the i-th downsampling feature and the diffusion intensity feature using a skip connection method.

[0143] With respect to the downsampling diffusion process of a single downsampling layer, in some embodiments, a computer device can downsample the i-th downsampling feature to obtain a downsampling intermediate feature, fuse the diffusion intensity feature and the downsampling intermediate feature to generate a downsampling fused feature, and further fuse the downsampling intermediate feature and the downsampling fused feature to generate the (i+1)th downsampling feature.

[0144] Furthermore, the diffusion strength characteristics

number

number

[0145] Referring to Figure 4, which is a schematic diagram of the internal structure of the residual layer ResBlock-D according to an exemplary embodiment of this application.

[0146] Using ResBlock-D as an example, we will explain the process of generating the (i+1)th downsampling feature.

[0147] The sampling layer (Sample) in the residual layer ResBlock-D downsamples the i-th downsampled feature 412 to obtain the downsampled intermediate feature 421(

number

number

number

number

number

number

number

number

number

number

number

number

[0148] In the upsampling process, the residual layer ResBlock-U is used to fuse the i-th upsampling feature and the diffusion intensity feature using a skip connection method.

[0149] With respect to the upsampling diffusion process of a single upsampling layer, in some embodiments, a computer device can upsample the i-th upsampling feature to obtain an upsampling intermediate feature, fuse the diffusion intensity feature and the upsampling intermediate feature to generate an upsampling fused feature, and further fuse the upsampling intermediate feature and the upsampling fused feature to generate the (i+1)th upsampling feature.

[0150] Diffusion strength characteristics

number

number

[0151] Referring to Figure 5, which is a schematic diagram of the internal structure of the residual layer ResBlock-U according to an exemplary embodiment of this application.

[0152] Using ResBlock-U as an example, we will explain the process of generating the (i+1)th upsampling feature.

[0153] In the residual layer ResBlock-U, the sampling layer (Sample) performs an upsampling process on the i-th upsampling feature 512 to obtain the upsampling intermediate feature 521(

number

number

number

number

number

number

number

number

number

number

number

number

[0154] In this embodiment, the i-th upsampling feature (or i-th downsampling feature) and the diffusion intensity feature can be fused using a skip connection method based on the residual layer ResBlock, thereby enabling the diffusion intensity feature to be used to control the intensity of backdiffusion in the depth interpolation process.

[0155] In some embodiments, to obtain a dense depth map with high depth information completeness, the computer device may perform depth interpolation by multiple rounds of iteration.

[0156] Optionally, the diffusion intensity parameter used in each iteration of the N-round iterative depth interpolation process may differ. In some embodiments, the diffusion intensity parameter gradually decreases with increasing iterations, thereby enabling a depth interpolation process that progresses from coarser to finer stages.

[0157] In some embodiments, before depth interpolation, the computer device can first determine a diffusion intensity parameter sequence, which includes N diffusion intensity parameters (where N is a positive integer), and the (k+1)th diffusion intensity parameter is smaller than the kth diffusion intensity parameter. The value of k is an integer from 1 to N-1.

[0158] In some embodiments, the diffusion intensity parameter sequence can be determined randomly based on a (0,1) uniform distribution. For example, a computer device randomly selects N diffusion intensity parameters based on a (0,1) uniform distribution and sorts them in descending order.

number

number

number

number

[0159] In some embodiments, a computer device can obtain depth-completed features by repeating the diffusion and completion of aggregated features N rounds using a depth-completion network based on the diffusion intensity parameters in the diffusion intensity parameter sequence.

[0160] Referring to Figure 6, which is a schematic diagram of a process in which the diffusion and interpolation of aggregated features is iterated N times by a depth interpolation network according to an exemplary embodiment of the present application.

[0161] As shown in Figure 6, for each iteration of the first round, the input to the DiffDC network is the aggregated features of the first round, obtained by aggregating the features of the sparse depth map 611-1 and scene graph 612 of the first round and applying noise addition processing with the noise 613-1 of the first round.

[0162] As an example, the diffusion intensity parameter used in the first round of iterations is 0.95.

[0163] The following explanation uses the k-th round iteration as an example. In the k-th round iteration, the computer device aggregates the features of the scene graph 612 and the sparse depth maps 611-k of the k-th round, and applies noise processing with the noise 613-k of the k-th round to obtain the aggregated features of the k-th round.

[0164] The k-th round of noise is randomly generated based on a Gaussian distribution. For example, the value of each pixel point in the noisy image composed of the k-th round of noise.

number

number

[0165] The sparse depth map of the k+1th round is the dense depth map of the kth round obtained by the depth interpolation process of the kth round; that is, the sparse depth map 611-k of the kth round is the dense depth map of the k-1th round generated by the previous iteration.

[0166] In some embodiments, a computer device diffuses and interpolates the aggregated features of the kth round by a depth interpolation network based on the kth diffusion intensity parameter in the diffusion intensity parameter sequence to obtain the kth round depth interpolation features, and then reconstructs the image based on the kth round depth interpolation features to obtain the kth round dense depth map.

[0167] For example, the diffusion intensity parameter used in the k-th round iteration is 0.60.

[0168] Similarly, in the k+1th round iteration, the computer device aggregates the features of the scene graph 612 and the sparse depth maps 611-k+1 of the k+1th round, and obtains the aggregated features of the k+1th round by noise addition processing with noise 613-k+1 of the k+1th round. The aggregated features of the k+1th round are then diffused and interpolated by a depth interpolation network to obtain the depth-interpolated features of the k+1th round, thereby reconstructing the image based on the depth-interpolated features of the k+1th round to obtain the dense depth map of the k+1th round.

[0169] As an example, the diffusion intensity parameter adopted in the k+1th round iteration is 0.49.

[0170] The sparse depth map 611-k+1 for the k+1th round is the dense depth map for the kth round generated in the kth round iteration.

[0171] In some embodiments, the scene graph is

number

number

number

number

number

number

number

number

number

[0172] In an optional depth interpolation process that is iterated through multiple rounds, the computer device continues until the Nth round of the depth interpolation process is executed, i.e., until the Nth round of dense depth map is obtained as the final dense depth map.

number

number

number

[0173] For example, the diffusion intensity parameter used in the Nth round iteration is 0.11.

[0174] In this embodiment, a depth interpolation network is used to perform N rounds of iterative depth interpolation on aggregated features. In each iteration, the dense depth map generated in the previous iteration is used as input, and simultaneously, a sequentially decreasing diffusion intensity parameter is used in each iteration. This realizes a depth interpolation process that progresses from coarse to fine stages, thereby improving the quality of depth interpolation.

[0175] In some embodiments, to train a depth interpolation network, a computer device collects a sample scene graph, a sample sparse depth map, sample noise, and a sample depth map. After aggregating the features of the sample scene graph and the sample sparse depth map, it obtains a sample aggregation feature by adding sample noise. This sample aggregation feature is input to the depth interpolation network to obtain a sample depth interpolation feature. Based on the sample depth interpolation feature, the image is reconstructed to obtain a sample dense depth map. Using the sample depth map as training data, the interpolation loss is determined based on the difference between the sample dense depth map and the sample depth map. The depth interpolation network can then be trained based on this interpolation loss.

[0176] To improve training quality, in some embodiments, the computer device can also reduce the spreading randomness of the depth interpolation network by introducing a sample guidance map during the training process.

[0177] Referring to Figure 7, which is a flowchart for training a depth interpolation network according to an exemplary embodiment of the present application, the flowchart includes the following steps.

[0178] Step 701: The features of the sample scene graph, sample sparse depth map, and sample noise image are aggregated to obtain the first sample aggregated feature.

[0179] Regarding the generation method for sample noise images, one possible implementation is that a computer device can obtain a sample noise image by adding noise to a sample depth map using Gaussian noise based on the sample diffusion intensity parameter.

[0180] For example, a sample noise image

number

number

number

number

number

[0181] Here,

number

number

number

[0182] In some embodiments, the first sample aggregation feature

number

number

[0183] Here,

number

number

number

[0184] Step 702: Based on the sample diffusion intensity parameters, the first sample aggregation features are diffused and complemented by a depth interpolation network to obtain the first sample depth interpolated features.

[0185] Step 703: Reconstruct the image based on the first sample depth interpolation features to generate the first sample dense depth map.

[0186] For example, a sample noise image

number

number

number

number

number

number

[0187] Step 704: Based on the first sample-dense depth map, determine a sample guidance map to reduce the diffusion randomness of the depth-complementing network.

[0188] Regarding the method for determining the sample guidance map, in one possible implementation, the probability that the sample guidance map is assigned to the first sample density depth map is the first probability, the probability that the sample guidance map is assigned to a tensor whose element values ​​are all zero is the second probability, and the sum of the first and second probabilities is 1.

[0189] For example, both the first and second probabilities are 50%, and the sample guidance map is

number

number

number

number

[0190] Step 705: The features of the sample guidance map, sample scene graph, sample sparse depth map, and sample noise image are aggregated to obtain a second sample aggregate feature.

[0191] In some embodiments, a second sample aggregation feature

number

number

[0192] Here,

number

number

number

number

[0193] Step 706: Based on the sample diffusion intensity parameters, the second sample aggregation features are diffused and complemented by the depth interpolation network to obtain the second sample depth interpolation features.

[0194] Step 707: Reconstruct the image based on the second sample depth interpolation features to generate a second sample-dense depth map.

[0195] In some embodiments, the second sample density depth map can be expressed by the following formula:

number

[0196] Step 708: Determine the interpolation loss based on the difference between the second sample-dense depth map and the sample-depth map.

[0197] For example, the interpolation loss

number

number

number

number

number

number

[0198] Step 709: Train the depth interpolation network based on the interpolation loss.

[0199] For example, a computer device can train a depth interpolation network using gradient descent or other training methods.

[0200] In this embodiment, by determining a second sample density depth map based on a sample guidance map, the diffusion process of the depth interpolation network can be guided, and the diffusion randomness of the depth interpolation network can be reduced. That is, in the current diffusion process, the previous calculation result is used as a reference, so that the current calculation process does not deviate significantly from the previous calculation result, thereby achieving a better training effect.

[0201] Optionally, the depth map interpolation method provided by the embodiments of this application can be applied to various scenes such as 3D modeling scenes, autonomous driving scenes, and 3D printing scenes. Regarding 3D modeling scenes,

[0202] Optionally, in a 3D modeling scene, the depth camera can only acquire a sparse depth map corresponding to the 3D scene. Therefore, to improve the efficiency and accuracy of 3D modeling, the computer device can first generate a dense depth map based on the sparse depth map, and then use the dense depth map in the 3D modeling process.

[0203] In some embodiments, the computer device first obtains a scene graph corresponding to a 3D modeling scene captured by an imaging device and a sparse depth map acquired by a depth camera. Furthermore, feature aggregation is performed on the scene graph and sparse depth map, and aggregated features are obtained using noise addition processing. Based on the diffusion intensity parameter, the aggregated features are diffused and interpolated by a depth interpolation network to obtain depth interpolation features, and the image is reconstructed based on the depth interpolation features to obtain a dense depth map.

[0204] Optionally, to improve the accuracy of depth interpolation, the computer device can determine a diffusion intensity parameter sequence containing N diffusion intensity parameters, where the (k+1)th diffusion intensity parameter is smaller than the kth diffusion intensity parameter. Furthermore, based on the diffusion intensity parameters in the diffusion intensity parameter sequence, the depth interpolation network obtains the depth interpolation features by repeating the diffusion and interpolation of aggregated features N times.

[0205] Regarding the process of generating the k-th round dense depth map, in one possible embodiment, a computer device aggregates the features of the scene graph and the k-th round sparse depth map to obtain the k-th round aggregated features, the k+1-th round sparse depth map is the k-th round dense depth map obtained by the k-th round depth interpolation process, the k-th round aggregated features are processed by the k-th round noise addition, and the k-th round noise is randomly generated based on a Gaussian distribution. Furthermore, the computer device diffuses and interpolates the k-th round aggregated features by a depth interpolation network based on the k-th diffuse intensity parameter in the diffuse intensity parameter sequence to obtain the k-th round depth interpolation features, and then reconstructs the image based on the k-th round depth interpolation features to obtain the k-th round dense depth map.

[0206] Optionally, to fully learn image features and improve depth interpolation efficiency, the depth interpolation network may include downsampling diffusion subnetworks and upsampling diffusion subnetworks. In one possible embodiment, the computer device first performs downsampling diffusion on aggregate features using a downsampling diffusion subnetwork based on diffusion intensity parameters to obtain downsampling depth features, and then performs upsampling diffusion on the downsampling depth features using an upsampling diffusion subnetwork based on diffusion intensity parameters to obtain depth interpolation features.

[0207] Optionally, to improve the accuracy of downsampling, the downsampling diffusion subnetwork includes n downsampling layers, and the upsampling diffusion subnetwork includes n upsampling layers. In one possible embodiment, the computer device performs downsampling diffusion on an aggregated feature using a first downsampling layer based on a diffusion intensity parameter to obtain a first downsampling feature, and then performs downsampling diffusion on the i-th downsampling feature using the (i+1)th downsampling layer based on the diffusion intensity parameter to obtain the (i+1)th downsampling feature, and finally uses the (n)th downsampling feature output from the (n)th downsampling layer as the downsampling depth feature. Furthermore, the computer device performs upsampling diffusion on the downsampling depth feature using a first upsampling layer based on a diffusion intensity parameter to obtain a first upsampling feature, and then performs upsampling diffusion on the i-th upsampling feature using the (i+1)th upsampling layer based on the diffusion intensity parameter to obtain the (i+1)th upsampling feature, and finally uses the (n)th upsampling feature output from the (n)th upsampling layer as the depth-interpolated feature.

[0208] Optionally, with respect to the downsampling diffusion process of a single downsampling layer, in one possible embodiment, a computer device downsamples the i-th downsampling feature to obtain a downsampling intermediate feature, fuses the diffusion intensity feature and the downsampling intermediate feature to generate a downsampling fused feature, the diffusion intensity feature is obtained based on feature extraction of the diffusion intensity parameter, thereby fusing the downsampling intermediate feature and the downsampling fused feature to generate the (i+1)th downsampling feature.

[0209] In one possible embodiment, with respect to an upsampling diffusion process of a single upsampling layer, the computer device upsamples the i-th upsampling feature to obtain an upsampling intermediate feature, fuses the diffusion intensity feature and the upsampling intermediate feature to generate an upsampling fused feature, the diffusion intensity feature is obtained based on feature extraction of the diffusion intensity parameter, thereby fusing the upsampling intermediate feature and the upsampling fused feature to generate the (i+1)th upsampling feature.

[0210] In one possible embodiment, the computer device may fuse the i-th upsampling feature and the ni-th downsampling feature to obtain the i-th fused feature, and then, based on the diffusion intensity parameter, perform upsampling diffusion on the i+1-th fused feature by an i+1-th upsampling layer to obtain the i+1-th upsampling feature.

[0211] In an optional process for obtaining aggregated features, in one possible embodiment, the computer device first encodes the features of a sparse depth map with a first encoder to obtain sparse depth features, and then encodes the features of a scene graph with a second encoder to obtain scene features. The scene features and sparse depth features are then aggregated, and an aggregated feature is obtained by noise addition processing, where the dimensions of the scene features, sparse depth features, and noise are consistent. About autonomous driving scenarios

[0212] Optionally, in autonomous driving scenarios, devices such as LiDAR can only acquire sparse depth maps corresponding to the driving scene. Therefore, to improve the safety of autonomous driving, the computer device can first generate a dense depth map based on the sparse depth map, and then use the dense depth map in the autonomous driving process.

[0213] In some embodiments, the computer device first acquires a road condition map captured by the imaging device and a sparse depth map corresponding to the road condition map. Furthermore, it performs feature aggregation on the road condition map and the sparse depth map, and obtains aggregated features using noise addition processing. Based on the diffusion intensity parameter, the aggregated features are diffused and interpolated by a depth interpolation network to obtain depth interpolation features, and the image is restored based on the depth interpolation features to obtain a dense depth map.

[0214] Optionally, to improve the accuracy of depth interpolation, the computer device can determine a diffusion intensity parameter sequence containing N diffusion intensity parameters, where the (k+1)th diffusion intensity parameter is smaller than the kth diffusion intensity parameter. Furthermore, based on the diffusion intensity parameters in the diffusion intensity parameter sequence, the depth interpolation network obtains the depth interpolation features by repeating the diffusion and interpolation of aggregated features N times.

[0215] Regarding the process of generating the k-th round dense depth map, in one possible embodiment, a computer device aggregates the features of the driving road condition map and the k-th round sparse depth map to obtain the k-th round aggregated features, the k+1-th round sparse depth map is the k-th round dense depth map obtained by the k-th round depth interpolation process, the k-th round aggregated features are processed by the k-th round noise addition, and the k-th round noise is randomly generated based on a Gaussian distribution. Furthermore, the computer device diffuses and interpolates the k-th round aggregated features by a depth interpolation network based on the k-th diffuse intensity parameter in the diffuse intensity parameter sequence to obtain the k-th round depth interpolated features, thereby reconstructing the image based on the k-th round depth interpolated features to obtain the k-th round dense depth map.

[0216] Optionally, to fully learn image features and improve depth interpolation efficiency, the depth interpolation network may include downsampling diffusion subnetworks and upsampling diffusion subnetworks. In one possible embodiment, the computer device first performs downsampling diffusion on aggregate features using a downsampling diffusion subnetwork based on diffusion intensity parameters to obtain downsampling depth features, and then performs upsampling diffusion on the downsampling depth features using an upsampling diffusion subnetwork based on diffusion intensity parameters to obtain depth interpolation features.

[0217] Optionally, to improve downsampling accuracy, the downsampling diffusion subnetwork includes n downsampling layers, and the upsampling diffusion subnetwork includes n upsampling layers. In one possible embodiment, the computer device performs downsampling diffusion on an aggregated feature using a first downsampling layer based on a diffusion intensity parameter to obtain a first downsampling feature, and then performs downsampling diffusion on the i-th downsampling feature using the (i+1)th downsampling layer based on the diffusion intensity parameter to obtain the (i+1)th downsampling feature, and then uses the (n)th downsampling feature output from the (n)th downsampling layer as the downsampling depth feature. Furthermore, the computer device performs upsampling diffusion on the downsampling depth feature using a first upsampling layer based on a diffusion intensity parameter to obtain a first upsampling feature, and then performs upsampling diffusion on the i-th upsampling feature using the (i+1)th upsampling layer based on the diffusion intensity parameter to obtain the (i+1)th upsampling feature, and finally uses the (n)th upsampling feature output from the (n)th upsampling layer as the depth-interpolated feature.

[0218] In one possible embodiment, with respect to a downsampling diffusion process of a single downsampling layer, the computer device downsamples the i-th downsampling feature to obtain a downsampling intermediate feature, fuses the diffusion intensity feature and the downsampling intermediate feature to generate a downsampling fused feature, and the diffusion intensity feature is obtained based on feature extraction of the diffusion intensity parameter, and then fuses the downsampling intermediate feature and the downsampling fused feature to generate the (i+1)th downsampling feature.

[0219] In one possible embodiment, with respect to an upsampling diffusion process of a single upsampling layer, the computer device upsamples the i-th upsampling feature to obtain an upsampling intermediate feature, fuses the diffusion intensity feature and the upsampling intermediate feature to generate an upsampling fused feature, and the diffusion intensity feature is obtained based on feature extraction of the diffusion intensity parameter, and then fuses the upsampling intermediate feature and the upsampling fused feature to generate the (i+1)th upsampling feature.

[0220] In one possible embodiment, the computer device may fuse the i-th upsampling feature and the ni-th downsampling feature to obtain the i-th fused feature, and further perform upsampling diffusion on the i+1-th fused feature by an i+1-th upsampling layer based on the diffusion intensity parameter to obtain the i+1-th upsampling feature.

[0221] In one possible embodiment, regarding the process of obtaining aggregated features, the computer device first encodes the features of a sparse depth map using a first encoder to obtain sparse depth features, and then encodes the features of a driving road condition map using a second encoder to obtain driving scene features. The driving scene features and sparse depth features are then aggregated, and an aggregated feature is obtained by noise addition processing, where the dimensions of the driving scene features, sparse depth features, and noise are consistent.

[0222] Referring to Figure 8, Figure 8 is a block diagram of the configuration of a depth map interpolation device according to an exemplary embodiment of the present application. The device is A feature aggregation module 801 that aggregates features of a scene graph and a sparse depth map to obtain aggregated features, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph, and the aggregated features are subjected to noise addition processing. A depth interpolation module 802 obtains depth interpolation features by dispersing and interpolating the aggregated features using a depth interpolation network based on diffusion intensity parameters, wherein the depth interpolation network is based on a diffusion model, and the diffusion intensity parameters are used to control the intensity of backdiffusion in the depth interpolation process. An image restoration module 803 that restores an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has a higher degree of depth information completeness than the sparse depth map.

[0223] Optionally, the device further: The iterative module for determining a diffusion intensity parameter sequence includes an iterative module in which the diffusion intensity parameter sequence includes N diffusion intensity parameters, and the (k+1)th diffusion intensity parameter is less than the kth diffusion intensity parameter, where N is a positive integer and k is a positive integer.

[0224] Optionally, depth interpolation module 802 is selected. Based on the diffusion intensity parameters in the diffusion intensity parameter sequence, the depth interpolation network is used to obtain the depth interpolation features by repeating the diffusion and interpolation of the aggregated features N times.

[0225] Optionally, the feature aggregation module 801 is selected. The features of the scene graph and the sparse depth map of round k are aggregated to obtain the aggregated features of round k. The sparse depth map of round k+1 is the dense depth map of round k obtained by depth interpolation of round k. The aggregated features of round k are subjected to noise addition processing of round k, and the noise of round k is randomly generated based on a Gaussian distribution.

[0226] Optionally, depth interpolation module 802 is selected. Based on the k diffusion intensity parameter in the diffusion intensity parameter sequence, the depth interpolation network diffuses and interpolates the aggregated features of the k round to obtain the depth interpolated features of the k round.

[0227] Optionally, the image recovery module 803 is selected. The image is reconstructed based on the depth interpolation features of the k rounds to obtain a dense depth map of the k rounds.

[0228] Optionally, the depth interpolation network includes a downsampling spread subnetwork and an upsampling spread subnetwork, and the depth interpolation module 802 is Based on the diffusion intensity parameter, downsampling diffusion is performed on the aggregated feature by the downsampling diffusion subnetwork to obtain downsampling depth features. Based on the diffusion intensity parameter, the upsampling diffusion subnetwork performs upsampling diffusion on the downsampling depth feature to obtain the depth-compensated feature.

[0229] Optionally, the downsampling spread subnetwork includes n downsampling layers, and the upsampling spread subnetwork includes n upsampling layers, where n is a positive integer, and the depth interpolation module 802 is, Based on the diffusion intensity parameter, a first downsampling layer performs downsampling diffusion on the aggregated feature to obtain a first downsampling feature. Based on the diffusion intensity parameter, downsampling diffusion is performed on the i-th downsampling feature by the i+1th downsampling layer to obtain the i+1th downsampling feature, where i is a positive integer. The nth downsampling feature output from the nth downsampling layer is defined as the downsampling depth feature. Based on the diffusion intensity parameter, the first upsampling layer performs upsampling diffusion on the downsampling depth feature to obtain the first upsampling feature. Based on the diffusion intensity parameter, the (i+1)th upsampling layer performs upsampling diffusion on the (i)th upsampling feature to obtain the (i+1)th upsampling feature. The nth upsampling feature output from the nth upsampling layer is defined as the depth interpolation feature.

[0230] Optionally, depth interpolation module 802 is selected. The downsampling features described above are downsampled to obtain downsampling intermediate features. The diffusion intensity features and the downsampling intermediate features are merged to generate downsampling fused features, and the diffusion intensity features are obtained based on the feature extraction of the diffusion intensity parameters. The downsampling intermediate features and the downsampling fusion features are fused to generate the i+1 downsampling features.

[0231] Optionally, depth interpolation module 802 is selected. The upsampling features described above are upsampled to obtain upsampling intermediate features. The diffusion intensity features and the upsampling intermediate features are fused to generate upsampling fused features, and the diffusion intensity features are obtained based on the feature extraction of the diffusion intensity parameters. The upsampling intermediate feature and the upsampling fusion feature are fused to generate the i+1 upsampling feature.

[0232] Optionally, depth interpolation module 802 is selected. The upsampling feature i and the downsampling feature ni are merged to obtain the i-th merged feature. Based on the diffusion intensity parameter, upsampling diffusion is performed on the fusion feature i using the i+1 upsampling layer to obtain the i+1 upsampling feature.

[0233] Optionally, the feature aggregation module 801 is selected. The features of the sparse depth map are encoded by the first encoder to obtain sparse depth features. The features of the scene graph are encoded by a second encoder to obtain scene features. The scene features and sparse depth features are aggregated, and the noise addition process is performed to obtain the aggregated features, wherein the dimensions of the scene features, sparse depth features and noise are the same.

[0234] Optionally, the device further includes a training module, which includes: The features of the sample scene graph, sample sparse depth map, and sample noise image are aggregated to obtain the first sample aggregated feature. Based on the sample diffusion intensity parameter, the depth interpolation network diffuses and interpolates the first sample aggregation feature to obtain the first sample depth interpolation feature. Based on the first sample depth interpolation features, the image is reconstructed to generate a first sample-dense depth map. Based on the first sample-dense depth map, a sample guidance map is determined to reduce the diffusion randomness of the depth-complementing network. The features of the sample guidance map, the sample scene graph, the sample sparse depth map, and the sample noise image are aggregated to obtain a second sample aggregation feature. Based on the sample diffusion intensity parameter, the depth interpolation network diffuses and interpolates the second sample aggregation feature to obtain a second sample depth interpolation feature. Based on the second sample depth interpolation feature, the image is reconstructed to generate a second sample-dense depth map. Based on the difference between the second sample density depth map and the sample depth map, the interpolation loss is determined. The depth interpolation network is trained based on the interpolation loss.

[0235] The probability that the sample guidance map is assigned to the first sample density depth map by choice is the first probability, the probability that the sample guidance map is assigned to a tensor whose element values ​​are all zero is the second probability, and the sum of the first and second probabilities is 1.

[0236] Optionally, the training module is: Based on the sample diffusion intensity parameter, noise is added to the sample depth map using Gaussian noise to obtain the sample noise image.

[0237] Referring to Figure 9, which is a schematic diagram of the configuration of a computer device according to an exemplary embodiment of this application.

[0238] Specifically, the computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including random access memory 902 and read-only memory 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 also includes an input / output (I / O) system 906 for transmitting information between devices within the computer, and a mass storage device 907 for storing an operating system 913, applications 914, and other program modules 915.

[0239] The basic input / output system 906 includes a display 908 for displaying information and an input device 909, such as a mouse or keyboard, for the user to input information. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output control 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output control 910 for receiving and processing input from a plurality of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output control 910 also provides output to a display, printer, or other type of output device.

[0240] The mass storage device 907 is connected to the central processing unit 901 via a mass storage device control (not shown) connected to the system bus 905. The mass storage device 907 and the associated computer-readable media provide non-volatile storage to the computer device 900. In other words, the mass storage device 907 may include a computer-readable media (not shown), such as a hard disk or a drive.

[0241] Without loss of generality, the computer-readable medium may include computer storage mediums and communication mediums. Computer storage mediums include volatile and non-volatile, portable and non-portable mediums implemented by any method or technique for storing information such as computer-readable instructions, data configurations, program modules, or other data. Computer storage mediums include random access memory (RAM), read-only memory (ROM), flash memory, or other solid-state memory, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. Of course, those skilled in the art will understand that the computer storage medium is not limited to those described above. The system memory 904 and mass storage device 907 described above may be collectively referred to as memory.

[0242] Memory stores one or more programs, which are arranged to be executed by one or more central processing units 901, which include instructions for implementing the above method, and the central processing units 901 execute the one or more programs to implement the method provided by each embodiment of the above method.

[0243] According to various embodiments of this application, the computer device 900 can be connected to and operated by a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or it can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 911.

[0244] The memory further includes one or more programs which are stored in the memory and which include steps performed by a computer device in the manner provided by the embodiments of this application.

[0245] Embodiments of this application further provide a computer-readable storage medium in which at least one instruction is stored, and at least one instruction is loaded and executed by a processor, thereby realizing the method of any of the embodiments described above.

[0246] The computer-readable storage medium may optionally include ROM, RAM, solid-state drives (SSDs), or optical discs. RAM may include resistive random-access memory (ReRAM) and dynamic random-access memory (DRAM).

[0247] Embodiments of this application provide a computer program product including a computer command, which is stored in a computer-readable storage medium. The processor of a computer device reads the computer command from the computer-readable storage medium and executes the computer command, thereby causing the computer device to perform the method described in the above embodiment.

[0248] Those skilled in the art will understand that all or some of the steps of the above embodiment can be implemented by hardware, or that the relevant hardware can be instructed to perform them by a program. The program can be stored on a computer-readable storage medium. The storage medium may be a ROM, disk, or optical disk, etc.

[0249] The foregoing are merely selectable embodiments of this application and do not limit it. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should all be included within the scope of protection of this application.

Claims

1. A depth map interpolation method performed by a computer device, A step of aggregating the features of a scene graph and a sparse depth map to obtain an aggregated feature, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph, and the aggregated feature is subjected to noise addition processing. A step of obtaining depth-completed features by diffusing and completing the aggregated features using a depth-completed network based on diffusion intensity parameters, wherein the depth-completed network is based on a diffusion model, and the diffusion intensity parameters are used to control the intensity of backdiffusion in the depth-completed process. A step of restoring an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has a higher degree of completeness of depth information than the sparse depth map. A method that includes this.

2. A step of determining a diffusion intensity parameter sequence, the further step of determining a diffusion intensity parameter sequence comprising N diffusion intensity parameters, wherein the (k+1)th diffusion intensity parameter is less than the kth diffusion intensity parameter, where N is a positive integer and k is a positive integer. The step of obtaining depth-completed features by spreading and completing the aggregated features using a depth-completed network based on diffusion intensity parameters is as follows: The steps include obtaining the depth-completed features by repeating the diffusion and completion of the aggregated features N rounds using the depth-completed network, based on the diffusion intensity parameters in the diffusion intensity parameter sequence, The method according to claim 1, including the method described in claim 1.

3. The above step of aggregating the features of the scene graph and sparse depth map to obtain aggregated features is: A step of aggregating the features of the scene graph and the k-round sparse depth map to obtain the k-round aggregated features, wherein the k+1-round sparse depth map is the k-round dense depth map obtained by the k-round depth interpolation process, the k-round aggregated features are subjected to the k-round noise addition process, and the k-round noise is randomly generated based on a Gaussian distribution. The step of obtaining the depth-completed features by repeating the diffusion and completion of the aggregated features N rounds using the depth-completed network based on the diffusion intensity parameters in the diffusion intensity parameter sequence is as follows: The step includes, based on the diffusion intensity parameter k in the diffusion intensity parameter sequence, the depth interpolation network diffuses and interpolates the aggregated features of the k round to obtain the depth interpolated features of the k round, The step of restoring the image based on the depth interpolation features to obtain a dense depth map is: The method according to claim 2, comprising the step of restoring an image based on the k-round depth interpolation features to obtain a k-round dense depth map.

4. The depth interpolation network includes a downsampling spread subnetwork and an upsampling spread subnetwork. The step of obtaining depth-completed features by spreading and completing the aggregated features using a depth-completed network based on diffusion intensity parameters is as follows: The steps include: obtaining downsampling depth features by performing downsampling diffusion on the aggregated features using the downsampling diffusion subnetwork based on the diffusion intensity parameter; The steps include: obtaining the depth-complementary features by performing upsampling diffusion on the downsampling depth features using the upsampling diffusion subnetwork based on the diffusion intensity parameter; The method according to any one of claims 1 to 3, including the method described in any one of claims 1 to 3.

5. The downsampling spread subnetwork includes n downsampling layers, and the upsampling spread subnetwork includes n upsampling layers, where n is a positive integer. The step of obtaining downsampling depth features by performing downsampling diffusion on the aggregated features using the downsampling diffusion subnetwork based on the diffusion intensity parameter is as follows: The steps include: obtaining a first downsampled feature by performing downsampling diffusion on the aggregated feature using a first downsampling layer based on the diffusion intensity parameter; A step of obtaining the (i+1)th downsampled feature by performing downsampling diffusion on the i-th downsampled feature using the (i+1)th downsampled layer based on the diffusion intensity parameter, wherein i is a positive integer. The steps include: setting the nth downsampling feature output from the nth downsampling layer as the downsampling depth feature; Includes, The step of obtaining the depth-complementary features by performing upsampling diffusion on the downsampling depth features using the upsampling diffusion subnetwork based on the diffusion intensity parameter is as follows: The steps include: obtaining a first upsampling feature by performing upsampling diffusion on the downsampling depth feature using a first upsampling layer based on the diffusion intensity parameter; Based on the diffusion intensity parameter, the (i+1)th upsampling layer performs upsampling diffusion on the (i)th upsampling feature to obtain the (i+1)th upsampling feature. The steps include: using the nth upsampling feature output from the nth upsampling layer as the depth interpolation feature; The method according to claim 4, which includes the following:

6. The step of obtaining the (i+1)th downsampled feature by performing downsampling diffusion on the ith downsampled feature using the (i+1)th downsampled layer based on the diffusion intensity parameter is as follows: The steps include: downsampling the downsampling feature i above to obtain downsampling intermediate features, A step of generating a downsampling fused feature by fusing the diffusion intensity feature and the downsampling intermediate feature, wherein the diffusion intensity feature is obtained based on the feature extraction of the diffusion intensity parameter, The steps include: fusing the downsampling intermediate features and the downsampling fusion features to generate the i+1 downsampling features; The method according to claim 5, including the method described in claim 5.

7. The step of obtaining the (i+1)th upsampling feature by performing upsampling diffusion on the i-th upsampling feature using the (i+1)th upsampling layer based on the diffusion intensity parameter is as follows: The steps include: upsampling the upsampling feature i described above to obtain an upsampling intermediate feature; A step of generating an upsampling fused feature by fusing the diffusion intensity feature and the upsampling intermediate feature, wherein the diffusion intensity feature is obtained based on the feature extraction of the diffusion intensity parameter, The steps include: fusing the upsampling intermediate features and the upsampling fusion features to generate the i+1 upsampling features; The method according to claim 5, including the method described in claim 5.

8. The step of obtaining the (i+1)th upsampling feature by performing upsampling diffusion on the i-th upsampling feature using the (i+1)th upsampling layer based on the diffusion intensity parameter is as follows: The steps include: fusing the i-th upsampling feature and the n-i-th downsampling feature to obtain the i-th fused feature; The steps include: obtaining the i+1 upsampled feature by performing upsampling diffusion on the i fusion feature using the i+1 upsampling layer based on the diffusion intensity parameter; The method according to claim 5, including the method described in claim 5.

9. The step of aggregating the features of the scene graph and sparse depth map to obtain aggregated features is as follows: The steps include: encoding the features of the sparse depth map using a first encoder to obtain sparse depth features; The steps include: encoding the features of the scene graph using a second encoder to obtain scene features; A step of aggregating the scene features and the sparse depth features and obtaining the aggregated features by noise addition processing, wherein the dimensions of the scene features, the sparse depth features and the noise are the same. The method according to any one of claims 1 to 8, including the method described above.

10. The steps include: aggregating the features of the sample scene graph, sample sparse depth map, and sample noise image to obtain a first sample aggregated feature; The steps include: obtaining a first sample depth-interpolated feature by diffusing and interpolating the first sample aggregation feature using the depth interpolation network based on the sample diffusion intensity parameter; The steps include: reconstructing the image based on the first sample depth interpolation features to generate a first sample-dense depth map; The steps include determining a sample guidance map to reduce the diffusion randomness of the depth interpolation network based on the first sample density depth map, The steps include: aggregating the features of the sample guidance map, the sample scene graph, the sample sparse depth map, and the sample noise image to obtain a second sample aggregation feature; The steps include: obtaining a second sample depth-interpolated feature by diffusing and interpolating the second sample aggregation feature using the depth interpolation network based on the sample diffusion intensity parameter; The steps include: restoring the image based on the second sample depth interpolation feature to generate a second sample-dense depth map; The steps include determining the interpolation loss based on the difference between the second sample density depth map and the sample depth map, A step of training the depth interpolation network based on the interpolation loss, The method according to any one of claims 1 to 9, further comprising:

11. The method according to claim 10, wherein the probability that the sample guidance map is assigned to the first sample density depth map is a first probability, the probability that the sample guidance map is assigned to a tensor whose element values ​​are all zero is a second probability, and the sum of the first probability and the second probability is 1.

12. A step of obtaining the sample noise image by adding noise to the sample depth map using Gaussian noise based on the sample diffusion intensity parameter. The method according to claim 10, further comprising:

13. It is a depth map interpolation device, A feature aggregation module that aggregates features of a scene graph and a sparse depth map to obtain aggregated features, wherein the sparse depth map is a depth map with missing depth information corresponding to the scene graph, and the aggregated features are subjected to noise addition processing. A depth interpolation module that obtains depth interpolation features by dispersing and interpolating the aggregated features using a depth interpolation network based on a diffusion intensity parameter, wherein the depth interpolation network is based on a diffusion model, and the diffusion intensity parameter is used to control the intensity of backdiffusion in the depth interpolation procedure. An image restoration module that restores an image based on the depth interpolation features to obtain a dense depth map, wherein the dense depth map has higher depth information completeness than the sparse depth map. A depth map interpolation device that includes [a specific component].

14. A computer device including a processor and memory, A computer device having a memory that stores at least one computer command, the at least one computer command being loaded and executed by the processor to realize the depth map interpolation method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing at least one computer command, wherein the at least one computer command is loaded and executed by a processor to realize the depth map completion method according to any one of claims 1 to 12.

16. A computer program product including a computer command, wherein the computer command is stored in a computer-readable storage medium, and the processor of a computer device reads the computer command from the computer-readable storage medium, and the processor executes the computer command, thereby causing the computer device to execute the depth map interpolation method described in any one of claims 1 to 12.

Images