Depth map acquisition method, device, electronic device and storage medium
By employing an iterative denoising process and a diffusion-guided method in the depth map prediction model, the problems of poor fitting performance and depth map quality in monocular depth completion were solved, thus achieving high-quality depth map generation.
Patent Information
- Application Number
- CN202310369386.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-07
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-04-07
AI Technical Summary
In existing technologies, monocular depth completion methods cannot achieve the expected fitting effect on the augmented training set, and the depth map obtained by interpolating the low-resolution depth center point through spatial propagation completion has poor quality and is accompanied by large edge spikes.
A depth map prediction model is adopted, including a feature extraction layer, a feature aggregation layer, and a monocular-guided denoising layer. The depth map is generated from the random depth distribution through an iterative denoising process, and diffusion guidance is carried out by combining monocular vision conditions and sparse visual depth.
It improves the quality of generated depth maps, effectively avoids low resolution and edge artifacts, and enhances the accuracy of depth prediction.
Smart Images

Figure CN116503460B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle control technology, and in particular to a depth map acquisition method, device, electronic device and storage medium. Background Art
[0002] Monocular depth completion is a computer vision task that involves completing the depth of a scene from a single RGB image, given a sparse depth ground truth. This task has several important applications in areas such as robotics, augmented reality, autonomous driving, and virtual reality. Applications of monocular depth completion include object tracking, scene reconstruction, and 3D modeling, among others.
[0003] Monocular depth completion is a key component of autonomous driving systems because it enables the vehicle to perceive the environment and make decisions based on that perception. By accurately completing the depth of objects in the environment, the system can ensure safe and efficient driving. At the same time, monocular depth prediction does not require radar or multi-view cameras, and has the lowest cost. It has good promotion value for simple tasks.
[0004] Currently, commonly used monocular depth completion methods utilize deep learning models such as convolutional neural networks (CNNs). These models significantly improve the accuracy of depth completion compared to traditional computer vision methods. A popular monocular depth completion method is based on predicting dense depth maps using CNNs. These methods typically involve training a CNN on a large dataset of RGB-D (depth) images and regressing the depth using a pixel-scale depth regressor on top of the features extracted by the CNN to learn the relationship between the RGB image and its corresponding depth map. These methods have achieved impressive results on benchmark datasets such as NYU Depth v2 and KITTI. Another approach to monocular depth completion involves using spatial propagation to sparse depth points and complete the low-resolution depth, and then upsampling the depth to predict the depth using a bilinear interpolation method.
[0005] The existing CNN dense regression model has a one-to-one correspondence between pixels, and the types of image pixel mappings corresponding to the same depth value may vary greatly. This hinders the convergence of the depth prediction of dense regression, and the degree of fitting in the training set, especially in the training set after data augmentation, cannot achieve the expected effect. Moreover, the existing method of interpolating the depth center point to obtain the depth map involving spatial propagation to complete the low resolution cannot achieve the expected effect in the quality of the generated depth map, and is accompanied by relatively large edge glitches. Summary of the invention
[0006] The embodiments of the present application provide a depth map acquisition method, device, electronic device and storage medium to solve the problem that the degree of fitting on the training set after data augmentation in the prior art cannot achieve the expected effect, and the method of obtaining a depth map by interpolating the depth center point of low resolution through spatial propagation completion cannot achieve the expected effect in the quality of the generated depth map, and is accompanied by relatively large edge glitches.
[0007] In order to solve the above technical problems, the embodiments of the present application are implemented as follows:
[0008] In a first aspect, an embodiment of the present application provides a method for acquiring a depth map, the method comprising:
[0009] Obtain the monocular image and sparse depth image corresponding to the target vehicle;
[0010] Inputting the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model comprises: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer;
[0011] Calling the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image;
[0012] Calling the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map;
[0013] The monocularly guided denoising layer is called to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle.
[0014] Optionally, the depth map prediction model further includes: an image segmentation layer and a position embedding layer,
[0015] Before calling the feature extraction layer to extract the monocular image features of multiple scales of the monocular image and the sparse depth image features of the sparse depth image, the method further includes:
[0016] Calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image;
[0017] The position embedding layer is called to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and the position vector is embedded into the segmented image to obtain a segmented image unit.
[0018] Optionally, calling the feature extraction layer to extract monocular image features of multiple scales of the monocular image includes:
[0019] Calling the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit;
[0020] According to the segmented image features and the position vector corresponding to the segmented image unit, monocular image features of multiple scales corresponding to the monocular image are obtained.
[0021] Optionally, the feature aggregation layer includes: a self-attention module and an aggregation module,
[0022] The calling of the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map includes:
[0023] Calling the self-attention module to process the monocular image features of each scale respectively to obtain attention features;
[0024] The aggregation module is called to aggregate the multiple attention features and output the aggregate feature map.
[0025] Optionally, the monocular guided denoising layer comprises: a monocular conditional denoising module and a depth map output module,
[0026] The calling of the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle includes:
[0027] Calling the monocular conditional denoising module to iteratively optimize the depth latent variable according to the aggregated feature map to obtain an optimized depth latent variable;
[0028] The depth map output module is called to process the optimized depth latent variable and the aggregated feature map to obtain the predicted depth map.
[0029] In a second aspect, an embodiment of the present application provides a depth map acquisition device, the device comprising:
[0030] A monocular image acquisition module is used to acquire a monocular image and a sparse depth image corresponding to a target vehicle;
[0031] A monocular image input module, used to input the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer;
[0032] A feature extraction module, used for calling the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image;
[0033] An aggregated feature map acquisition module is used to call the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map;
[0034] The predicted depth map acquisition module is used to call the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain the predicted depth map corresponding to the target vehicle.
[0035] Optionally, the depth map prediction model further includes: an image segmentation layer and a position embedding layer,
[0036] The device also includes:
[0037] A segmented image acquisition module, used for calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image;
[0038] The segmented image unit acquisition module is used to call the position embedding layer to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and embed the position vector into the segmented image to obtain the segmented image unit.
[0039] Optionally, the feature extraction module includes:
[0040] A segmented image feature extraction unit, used to call the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit;
[0041] The monocular image feature acquisition unit is used to obtain monocular image features of multiple scales corresponding to the monocular image according to the segmented image features and the position vector corresponding to the segmented image unit.
[0042] Optionally, the feature aggregation layer includes: a self-attention module and an aggregation module,
[0043] The aggregate feature map acquisition module includes:
[0044] An attention feature acquisition unit, used for calling the self-attention module to process the monocular image features of each scale respectively to obtain attention features;
[0045] The aggregate feature map output unit is used to call the aggregation module to aggregate the multiple attention features and output the aggregate feature map.
[0046] Optionally, the monocular guided denoising layer comprises: a monocular conditional denoising module and a depth map output module,
[0047] The predicted depth map acquisition module includes:
[0048] An optimized latent variable acquisition unit is used to call the monocular conditional denoising module to iteratively optimize the depth latent variable according to the aggregated feature map to obtain an optimized depth latent variable;
[0049] The predicted depth map acquisition unit is used to call the depth map output module to process the optimized depth latent variable and the aggregated feature map to obtain the predicted depth map.
[0050] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0051] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements any of the above-mentioned depth map acquisition methods when executed by the processor.
[0052] In a fourth aspect, an embodiment of the present application provides a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the depth map acquisition methods described above.
[0053] In an embodiment of the present application, a monocular image and a sparse depth image corresponding to the target vehicle are obtained. The monocular image and the sparse depth image are input into a depth map prediction model, and the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular-guided denoising layer. The feature extraction layer is called to extract the monocular image features of multiple scales of the monocular image, and the sparse depth image features of the sparse depth image. The feature aggregation layer is called to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map. The monocular-guided denoising layer is called to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle. The embodiment of the present application redefines the monocular depth estimation problem as an iterative denoising process, generates a depth image from a random depth distribution, and performs diffusion guidance through monocular visual conditions plus sparse visual depth. The diffusion-based generation method can greatly improve the quality of the generated depth map and effectively avoid the problems of low resolution and edge burrs.
[0054] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0056] Figure 1 A flowchart of a method for obtaining a depth map provided in an embodiment of the present application;
[0057] Figure 2 A flowchart of a method for obtaining a segmented image unit provided in an embodiment of the present application;
[0058] Figure 3 A flowchart of a method for acquiring monocular image features provided in an embodiment of the present application;
[0059] Figure 4 A flowchart of a method for outputting an aggregated feature map provided in an embodiment of the present application;
[0060] Figure 5 A flowchart of a method for obtaining a predicted depth map provided in an embodiment of the present application;
[0061] Figure 6 A schematic diagram of a diffusion-noise reduction process provided in an embodiment of the present application;
[0062] Figure 7 A schematic diagram of a model structure of a depth map prediction model provided in an embodiment of the present application;
[0063] Figure 8 A schematic diagram of a conditional noise reduction module provided in an embodiment of the present application;
[0064] Fig. 9 A schematic diagram of the structure of a depth map acquisition device provided in an embodiment of the present application;
[0065] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0067] Reference Figure 1, shows a flowchart of the steps of a depth map acquisition method provided by an embodiment of the present application. Figure 1 As shown, the depth map acquisition method may include: step 101, step 102, step 103, step 104 and step 105.
[0068] Step 101: Obtain a monocular image and a sparse depth image corresponding to a target vehicle.
[0069] The embodiments of the present application can be applied to a scenario in which a monocular image of a vehicle is completed in depth by using a diffusion-denoising model to perform a depth completion task.
[0070] The diffusion-denoising process can be Figure 6 As shown in the figure, the model input is: RGB Image (i.e. monocular RGB image) and Sparse Depth (i.e. sparse depth image). RGB Image and Sparse Depth are aggregated to form a monocular condition (i.e. aggregated feature map) to iteratively optimize the depth latent variable and finally output the corresponding predicted depth map.
[0071] The diffusion model is a type of latent variable model, which is usually used for generative tasks, in which the neural network denoises the image with Gaussian noise by learning to reverse the diffusion process. The diffusion process q(x t |x0), as defined in the formula, the neural network denoises the image with Gaussian noise by learning to reverse the diffusion process.
[0072]
[0073] Iteratively add noise to the desired image distribution x0 and obtain the latent noise sample x after t∈{0,1,...,T} steps t .
[0074] And β s represents the noise variance plan.
[0075] In the denoising process, the neural network μ θ (x t ,t) is trained to reverse x0 by interactively predicting x t-1 As shown below.
[0076] p θ (x t-1 |x t ):=N(x t-1 ;μ θ (x t ,t),σ t 2 I)
[0077] in represents the transfer variance. Sample x0 is transferred from the prior noise x T Starting from this, we iteratively reconstruct x T →x T-Δ →...x0.
[0078] Noise reduction process:
[0079] Given an input image and a sparse depth c, the monocular depth estimation task is usually formulated as p(x|c), where x is the desired depth map.
[0080] In this embodiment, the depth estimation can be restated as a visual condition guided denoising process, which is defined as follows: t Refine it and get the final depth map x0.
[0081] p θ (x t-1 |x t ,c):=N(x t-1 ;μ θ (x t ,t,c),σ t 2 I)
[0082] The model μ θ (x t ,t,c) is trained to transform the deep latent variable x t Thinned to x t-1 .
[0083] In order to speed up the denoising process, this embodiment uses the improved reasoning process in DDIM (Denoising Diffusion Implicit Models), in which Set to 0 to make the prediction output deterministic.
[0084] When performing depth estimation on a monocular image of a target vehicle, a monocular image and a sparse depth image corresponding to the target vehicle may be obtained. In a specific implementation, the monocular image may be an image captured by a monocular camera on the target vehicle, and the sparse depth image may be a scattered image obtained by scanning a radar on the target vehicle.
[0085] After the monocular image and the sparse depth image corresponding to the target vehicle are acquired, step 102 is executed.
[0086] Step 102: Input the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular-guided denoising layer.
[0087] The depth map prediction model refers to a model used to perform depth estimation on a monocular image to predict a corresponding depth map. In this example, the depth map prediction model may include: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer.
[0088] After obtaining the monocular image and the sparse depth image corresponding to the target vehicle, the monocular image and the sparse depth image may be input into the depth map prediction model.
[0089] After the monocular image and the sparse depth image are input into the depth map prediction model, step 103 is performed.
[0090] Step 103: calling the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image.
[0091] After the monocular image and the sparse depth image are input into the depth map prediction model, first, the feature extraction layer can be called to extract the monocular image features of multiple scales of the monocular image and the sparse depth image features of the sparse depth image. Specifically, the number of multiple scales can be at least two, and the monocular image features of the monocular image at multiple scales can be obtained through the feature extraction operation.
[0092] In a specific implementation, before extracting the image features of multiple scales of the monocular image, the monocular image may be segmented to divide the monocular image into multiple visual units. Figure 2 This is described in detail as follows.
[0093] Reference Figure 2 , shows a flow chart of the steps of a method for obtaining a segmented image unit provided by an embodiment of the present application. Figure 2 As shown, the method for acquiring segmented image units may include: step 201 and step 202.
[0094] Step 201: calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image.
[0095] In this embodiment, the depth map prediction model may further include: an image segmentation layer and a position embedding layer.
[0096] After the monocular image is input into the depth map prediction model, the image segmentation layer can be called to segment the monocular image to obtain a segmented image corresponding to the monocular image.
[0097] After the image segmentation layer is called to segment the monocular image and a segmented image corresponding to the monocular image is obtained, step 202 is executed.
[0098] Step 202: calling the position embedding layer to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and embedding the position vector into the segmented image to obtain a segmented image unit.
[0099] After calling the image segmentation layer to segment the monocular image and obtaining the segmented image corresponding to the monocular image, the position embedding layer can be called to generate the position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and the position vector is embedded into the segmented image to obtain a segmented image unit.
[0100] The above process can be Figure 7 As shown, LinearProjectionto Flattened Patches (linear projection layer) and Position Embedding (position embedding layer), image segmentation layer (not shown). Figure 7 As shown in the figure, the input monocular image can be divided into multiple segmented images, which can be projected into visual tokens after Linear Projection to Flattened Patches and adding position embedding vectors through Position Embedding.
[0101] In the specific implementation, at the beginning of the input, a Patch Embedding is done to cut the image into tiles and embed them into the Embedding. In each stage, it consists of Patch Merging and multiple Blocks. The PatchMerging module mainly reduces the image resolution at the beginning of each stage.
[0102] Then, the backbone network (Swin Transformer Backbone) can be used to extract visual features at different scales to preserve the coarse and fine details of the input scene. Figure 3 This is described in detail as follows.
[0103] Reference Figure 3 , shows a flowchart of the steps of a monocular image feature acquisition method provided by an embodiment of the present application. Figure 3 As shown, the monocular image feature acquisition method may include: step 301 and step 302.
[0104] Step 301: calling the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit.
[0105] In this embodiment, after the segmented image unit (ie, visual token) is obtained, the feature extraction layer may be called to extract segmented image features of multiple scales of the segmented image unit.
[0106] After calling the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit, step 302 is performed.
[0107] Step 302: Obtaining monocular image features of multiple scales corresponding to the monocular image according to the segmented image features and the position vector corresponding to the segmented image unit.
[0108] After the feature extraction layer is called to extract segmented image features of multiple scales of the segmented image unit, the monocular image features of multiple scales corresponding to the monocular image can be obtained according to the segmented image features and the position vector corresponding to the segmented image unit.
[0109] After calling the feature extraction layer to extract the monocular image features of multiple scales of the monocular image and the sparse depth image features of the sparse depth image, step 104 is performed.
[0110] Step 104: calling the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map.
[0111] After calling the feature extraction layer to extract the monocular image features of multiple scales of the monocular image and the sparse depth image features of the sparse depth image, the feature aggregation layer can be called to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map. Figure 7 As shown in the figure, Sparse Depth is the sparse depth image feature. Sparse Depth can be aggregated with monocular image features of different scales to obtain aggregate features of different scales. Then, after attention neck, Multi-Scale FPN and Aggregate, the aggregate feature map can be obtained. Figure 4 This is described in detail as follows.
[0112] Reference Figure 4 , shows a flow chart of the steps of a method for outputting an aggregated feature map provided by an embodiment of the present application. Figure 4 As shown, the aggregate feature map output method may include: step 401 and step 402.
[0113] Step 401: Call the self-attention module to process the monocular image features of each scale respectively to obtain attention features.
[0114] In this embodiment, the feature aggregation layer may include: a self-attention module and an aggregation module, such as Figure 7As shown in the figure, the self-attention module is AttentionNeck and the aggregation module is Aggregate.
[0115] After obtaining the monocular image features of multiple scales corresponding to the monocular image, the self-attention module can be called to process the monocular image features of each scale respectively to obtain the attention features. Figure 7 As shown in Figure 1, the monocular image features of each scale can be used to obtain the attention features corresponding to the monocular image features of each scale through AttentionNeck.
[0116] After calling the self-attention module to process the monocular image features of each scale to obtain the attention features, step 402 is executed.
[0117] Step 402: Call the aggregation module to aggregate the multiple attention features and output the aggregate feature map.
[0118] After calling the self-attention module to process the monocular image features of each scale to obtain the attention features, the aggregation module can be called to aggregate multiple attention features to output an aggregate feature map. Figure 7 As shown in the figure, after the monocular image features at each scale can be obtained through AttentionNeck to obtain the attention features corresponding to the monocular image features at each scale, multi-scale detection processing can be performed through Multi-Scale FPN, and then the features of multiple scales can be aggregated through Aggregate, so that the aggregated feature map can be output.
[0119] In this embodiment, hierarchical aggregation and heterogeneous interaction can be used to enhance features between different scales based on multi-scale features. The feature pyramid is used to aggregate features into monocular visual conditions. The sparse depth map is fused with RGB at each layer of the visual pyramid and produces a noise reduction condition.
[0120] The denoising condition can be a feature map after aggregating RGB visual features and sparse depth, with a shape of H / 4×W / 4×c, where H and W are the height and width of the monocular image input, respectively, and c is the channel dimension of the feature.
[0121] In this embodiment, a DiffusionDepth model can be used, which can be applied to most visual backbone networks that can extract multi-scale features. According to extensive experiments, other backbone networks such as ResNet, Efficient and ViT can also achieve competitive performance.
[0122] After calling the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map, step 105 is executed.
[0123] Step 105: Call the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle.
[0124] After the feature aggregation layer is called to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map, the monocular guided denoising layer can be called to denoise the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle. In this embodiment, the depth estimation can be expressed as a denoising process p θ (x t-1 |x t ,c), which improves the prediction accuracy by iteratively optimizing the deep latent variable xt, while being guided by the visual information available in the input image. Figure 5 This is described in detail as follows.
[0125] Reference Figure 5 , shows a flowchart of the steps of a method for obtaining a predicted depth map provided by an embodiment of the present application. Figure 5 As shown, the method for obtaining a predicted depth map may include: step 501 and step 502.
[0126] Step 501: Call the monocular conditional denoising module to iteratively optimize the depth latent variables according to the aggregated feature map to obtain optimized depth latent variables.
[0127] In this embodiment, the monocular guided denoising layer may include: a monocular conditional denoising module and a depth map output module.
[0128] After obtaining the aggregated feature map, the monocular conditional denoising module can be called to iteratively optimize the deep latent variables according to the aggregated feature map to obtain the optimized deep latent variables.
[0129] In the specific implementation, the denoising process is performed through the neural network model μ θ (x t ,t,c), the model perceives the condition c and the current depth latent variable x t , and predict the distribution x t-1 The monocular visual condition c∈R is obtained by multi-scale visual-sparse deep feature aggregation, i.e., the aggregated feature map. This embodiment introduces the monocular condition denoising block (MCDB), such as Figure 8 As shown in Figure 1, this process is implemented. Since depth prediction tasks usually require low-latency inference time for practical applications, this embodiment designs a lightweight denoising head. Condition c is actually a lower-resolution aggregated feature map that has a strong local relationship with the depth latent variable.
[0130] First, a local projection layer can be used to upsample the condition c to the same depth as the latent variable x. t ∈R has the same shape while preserving the local relationship between features. The projected condition directly passes through a CNN block and a self-attention layer, and is combined with the deep latent variable x via an element-wise summation operation. t Fusion. The fused deep latent variables are processed by a common BottleNeck CNN layer and channel-level attention, and residual connections are added. Denoised output x t-1 It is obtained by applying the DDIM mathematical reasoning process to the model output according to the preset diffusion plans β, α.
[0131] The embodiment of the present application can obtain a most suitable depth latent variable by iteratively optimizing the depth latent variable, thereby predicting the most accurate depth map.
[0132] After calling the monocular conditional denoising module to iteratively optimize the depth latent variables according to the aggregated feature map to obtain the optimized depth latent variables, step 502 is executed.
[0133] Step 502: Call the depth map output module to process the optimized depth latent variable and the aggregated feature map to obtain the predicted depth map.
[0134] After calling the monocular conditional denoising module to iteratively optimize the depth latent variables according to the aggregated feature map to obtain the optimized depth latent variables, the depth map output module can be called to process the optimized depth latent variables and the aggregated feature map to obtain the predicted depth map. Figure 7 As shown in Figure 1, the aggregated feature map output by Aggregate performs a denoising process through the MCDB module in Depth LatentSpace, and outputs a predicted depth map through the depth codec.
[0135] The embodiment of the present application can be applied to scenes such as object tracking, scene reconstruction, and 3D modeling by performing a depth completion scheme on a monocular image of a target vehicle. Specifically, depth completion can be used for object tracking to better complete the position and direction of objects in the scene. In scene reconstruction, depth completion can be used to create 3D models of indoor and outdoor environments. In 3D modeling, depth completion can be used to generate depth maps, which are then used to create high-quality 3D models. In particular, monocular depth completion is an important part of an autonomous driving system because it allows the vehicle to perceive the environment and make decisions based on that perception. Applications include: Obstacle detection and avoidance: Monocular depth completion can be used to detect obstacles in the environment and complete their distance from the vehicle. This information can be used to avoid collisions and bypass obstacles. Lane detection and tracking: Monocular depth completion can be used to detect lane markings on the road and complete their position relative to the vehicle. This information can be used to keep the vehicle in its lane and avoid driving into other lanes.
[0136] The depth map acquisition method provided in the embodiment of the present application is obtained by acquiring a monocular image and a sparse depth image corresponding to the target vehicle. The monocular image and the sparse depth image are input into the depth map prediction model, and the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular-guided denoising layer. The feature extraction layer is called to extract the monocular image features of multiple scales of the monocular image, and the sparse depth image features of the sparse depth image. The feature aggregation layer is called to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map. The monocular-guided denoising layer is called to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle. The embodiment of the present application redefines the monocular depth estimation problem as an iterative denoising process, generates a depth image from a random depth distribution, and performs diffusion guidance through monocular visual conditions plus sparse visual depth. The diffusion-based generation method can greatly improve the quality of the generated depth map and effectively avoid the problems of low resolution and edge burrs.
[0137] Reference Fig. 9 , which shows a schematic diagram of the structure of a depth map acquisition device provided in an embodiment of the present application, such as Fig. 9 As shown, the depth map acquisition device 900 may include the following modules:
[0138] The monocular image acquisition module 910 is used to acquire a monocular image and a sparse depth image corresponding to the target vehicle;
[0139] A monocular image input module 920, configured to input the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer;
[0140] A feature extraction module 930, configured to call the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image;
[0141] An aggregated feature map acquisition module 940 is used to call the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map;
[0142] The predicted depth map acquisition module 950 is used to call the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain the predicted depth map corresponding to the target vehicle.
[0143] Optionally, the depth map prediction model further includes: an image segmentation layer and a position embedding layer,
[0144] The device also includes:
[0145] A segmented image acquisition module, used for calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image;
[0146] The segmented image unit acquisition module is used to call the position embedding layer to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and embed the position vector into the segmented image to obtain the segmented image unit.
[0147] Optionally, the feature extraction module includes:
[0148] A segmented image feature extraction unit, used to call the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit;
[0149] The monocular image feature acquisition unit is used to obtain monocular image features of multiple scales corresponding to the monocular image according to the segmented image features and the position vector corresponding to the segmented image unit.
[0150] Optionally, the feature aggregation layer includes: a self-attention module and an aggregation module,
[0151] The aggregate feature map acquisition module includes:
[0152] An attention feature acquisition unit, used for calling the self-attention module to process the monocular image features of each scale respectively to obtain attention features;
[0153] The aggregate feature map output unit is used to call the aggregation module to aggregate the multiple attention features and output the aggregate feature map.
[0154] Optionally, the monocular guided denoising layer comprises: a monocular conditional denoising module and a depth map output module,
[0155] The predicted depth map acquisition module includes:
[0156] An optimized latent variable acquisition unit is used to call the monocular conditional denoising module to iteratively optimize the depth latent variable according to the aggregated feature map to obtain an optimized depth latent variable;
[0157] The predicted depth map acquisition unit is used to call the depth map output module to process the optimized depth latent variable and the aggregated feature map to obtain the predicted depth map.
[0158] The depth map acquisition device provided in the embodiment of the present application obtains a monocular image and a sparse depth image corresponding to the target vehicle. The monocular image and the sparse depth image are input into the depth map prediction model, and the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular-guided denoising layer. The feature extraction layer is called to extract the monocular image features of multiple scales of the monocular image, and the sparse depth image features of the sparse depth image. The feature aggregation layer is called to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map. The monocular-guided denoising layer is called to denoise the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle. The embodiment of the present application redefines the monocular depth estimation problem as an iterative denoising process, generates a depth image from a random depth distribution, and performs diffusion guidance through monocular visual conditions plus sparse visual depth. The diffusion-based generation method can greatly improve the quality of the generated depth map and effectively avoid the problems of low resolution and edge burrs.
[0159] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned depth map acquisition method when executed by the processor.
[0160] Fig.10 FIG. 1 is a schematic diagram showing the structure of an electronic device 1000 according to an embodiment of the present invention. Fig.10 As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 or computer program instructions loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0161] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, a microphone, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0162] The various processes and processing described above may be performed by the processing unit 1001. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the CPU 1001, one or more actions in the method described above may be performed.
[0163] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned depth map acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0164] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0166] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0167] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0169] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or groups can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0172] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0173] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for acquiring a depth map, characterized in that: The method comprises: Obtain the monocular image and sparse depth image corresponding to the target vehicle; Inputting the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model comprises: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer, wherein the monocular guided denoising layer comprises: a monocular conditional denoising module and a depth map output module; Calling the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image; Calling the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map; Calling the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle, including: calling the monocular conditional denoising module to iteratively optimize the depth latent variables according to the aggregated feature map to obtain optimized depth latent variables; the aggregated feature map is obtained by multi-scale visual-sparse depth feature aggregation; calling the depth map output module to process the optimized depth latent variables and the aggregated feature map to obtain the predicted depth map; The denoising process of the monocular conditional denoising module is as follows: the aggregated feature map is upsampled to a size with the same shape as the deep latent variable using a local projection layer, the local relationship between the features is maintained, the projected aggregated feature map is fused with the deep latent variable through an element-level summation operation after passing through a CNN block and a self-attention layer, the fused deep latent variable is processed through a BottleNeck CNN layer and channel-level attention, and after adding a residual connection, DDIM mathematical reasoning is applied to the output result according to a preset diffusion plan to obtain the optimized deep latent variable.
2. The method according to claim 1, characterized in that The depth map prediction model also includes: an image segmentation layer and a position embedding layer, Before calling the feature extraction layer to extract the monocular image features of multiple scales of the monocular image and the sparse depth image features of the sparse depth image, the method further includes: Calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image; The position embedding layer is called to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and the position vector is embedded into the segmented image to obtain a segmented image unit.
3. The method according to claim 2, characterized in that The calling the feature extraction layer to extract the monocular image features of multiple scales of the monocular image includes: Calling the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit; According to the segmented image features and the position vector corresponding to the segmented image unit, monocular image features of multiple scales corresponding to the monocular image are obtained.
4. The method according to claim 1, characterized in that: The feature aggregation layer includes: a self-attention module and an aggregation module, The calling of the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map includes: Calling the self-attention module to process the monocular image features of each scale respectively to obtain attention features; The aggregation module is called to perform aggregation processing on the multiple attention features and output the aggregated feature map.
5. A depth map acquisition device, characterized in that: The device comprises: A monocular image acquisition module is used to acquire a monocular image and a sparse depth image corresponding to a target vehicle; A monocular image input module, used to input the monocular image and the sparse depth image into a depth map prediction model, wherein the depth map prediction model includes: a feature extraction layer, a feature aggregation layer, and a monocular guided denoising layer; A feature extraction module, used to call the feature extraction layer to extract monocular image features of multiple scales of the monocular image and sparse depth image features of the sparse depth image; An aggregated feature map acquisition module is used to call the feature aggregation layer to aggregate the monocular image features of each scale with the sparse depth image features to obtain an aggregated feature map; A predicted depth map acquisition module is used to call the monocular guided denoising layer to perform denoising on the aggregated feature map to obtain a predicted depth map corresponding to the target vehicle, wherein the monocular guided denoising layer includes: a monocular conditional denoising module and a depth map output module, and the aggregated feature map is obtained by multi-scale visual-sparse depth feature aggregation; The predicted depth map acquisition module includes: An optimized latent variable acquisition unit is used to call the monocular conditional denoising module to iteratively optimize the depth latent variable according to the aggregated feature map to obtain an optimized depth latent variable; A predicted depth map acquisition unit, configured to call the depth map output module to process the optimized depth latent variable and the aggregated feature map to obtain the predicted depth map; The denoising process of the monocular conditional denoising module is as follows: the aggregated feature map is upsampled to a size with the same shape as the deep latent variable using a local projection layer, the local relationship between the features is maintained, the projected aggregated feature map is fused with the deep latent variable through an element-level summation operation after passing through a CNN block and a self-attention layer, the fused deep latent variable is processed through a BottleNeck CNN layer and channel-level attention, and after adding a residual connection, DDIM mathematical reasoning is applied to the output result according to a preset diffusion plan to obtain the optimized deep latent variable.
6. The device according to claim 5, characterized in that The depth map prediction model also includes: an image segmentation layer and a position embedding layer, The device also includes: A segmented image acquisition module, used for calling the image segmentation layer to perform segmentation processing on the monocular image to obtain a segmented image corresponding to the monocular image; The segmented image unit acquisition module is used to call the position embedding layer to generate a position vector corresponding to the segmented image according to the position information corresponding to the segmented image, and embed the position vector into the segmented image to obtain a segmented image unit.
7. The device according to claim 6, characterized in that The feature extraction module comprises: A segmented image feature extraction unit, used to call the feature extraction layer to extract segmented image features of multiple scales of the segmented image unit; The monocular image feature acquisition unit is used to obtain monocular image features of multiple scales corresponding to the monocular image according to the segmented image features and the position vector corresponding to the segmented image unit.
8. The device according to claim 5, characterized in that The feature aggregation layer includes: a self-attention module and an aggregation module, The aggregate feature map acquisition module includes: An attention feature acquisition unit, used for calling the self-attention module to process the monocular image features of each scale respectively to obtain attention features; The aggregate feature map output unit is used to call the aggregation module to aggregate the multiple attention features and output the aggregate feature map.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the depth image acquisition method according to any one of claims 1 to 4.
10. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the depth image acquisition method according to any one of claims 1 to 4.
