Depth map super-resolution method based on double-edge consistency constraint

Abstract

The invention relates to a depth map super-resolution method based on double-edge consistency constraint, belongs to the field of image quality enhancement, and solves the problem that the generalization ability is reduced due to the fact that the double-edge consistency constraint is not considered and feature fusion of a color map and a depth map is insufficient in an existing depth map super-resolution method. The method comprises the following steps: firstly, converting a depth map super-resolution problem into a three-task optimization model of double-edge consistency constraint, and secondly, according to an alternating direction multiplier method theory, converting the model into an iterative optimization process of color map edge updating, depth map edge updating, depth map updating and augmented Lagrange multiplier updating. Converting the optimization process into an expansion network, embedding a learnable module into an expansion model of an original pure expression to realize high-throughput transmission so as to enhance the expression capability, and further providing a multi-cross attention compensation fusion module; the module realizes approximation of a near-end operator by aggregating a color map edge, a depth map edge, cross-stage features and auxiliary information.

Description

Technical Field

[0001] The present invention belongs to the field of image quality enhancement, and in particular relates to a depth map super-resolution method based on double-edge consistency constraints. Background Art

[0002] Depth information is one of the most crucial pieces of information for perceiving the 3D world and plays a crucial role in autonomous driving, semantic scene understanding, and 3D reconstruction. Commonly used devices for this information acquisition typically include time-of-flight cameras and structured light scanners. However, due to the limitations of depth sensors, natural conditions, and photosensitive components in actual imaging, depth maps captured by many devices often exhibit low resolution. To address this issue, depth map super-resolution has become an important approach for obtaining high-resolution depth maps. In practical applications, depth image sensors are often equipped with color sensors, allowing for simultaneous acquisition of both color and depth images. Joint depth super-resolution methods leverage the structural consistency between the depth image and the corresponding color image to enhance the edge information in the depth map, thereby improving both the resolution and quality of the depth image. Current depth map super-resolution techniques explicitly or implicitly utilize color edge information to enhance image reconstruction. While RGB images contain object boundaries and internal texture boundaries, depth maps use sparse boundaries to separate objects of varying distances. Therefore, key challenges in color-guided depth map super-resolution techniques are how to leverage color-guided information to enhance degraded depth maps, minimize the impact of texture, and simultaneously fuse color and depth features. Many researchers have investigated how to mitigate the negative effects of color images. For example, depth map super-resolution methods based on optimization models often require the design of complex adaptive weighting strategies to learn complementary information from color and depth map pairs. Some methods implement joint filtering from the perspective of non-convex optimization and weighted least squares, while others optimize depth map reconstruction through explicit inconsistency measurements between color and depth maps.

[0003] Considering that accurate edge reconstruction is crucial for the depth map super-resolution task, some studies have proposed to gradually reconstruct the depth map through an edge refinement network. However, simply using edge detection often leads to inaccurate or discontinuous extracted edges. More importantly, a low-resolution depth map alone is not enough to produce an accurate depth edge map. These methods usually extract the color / depth edge map and the depth map and feed it directly into the reconstruction network. However, this one-way information transfer often leads to blurring and artifacts around discontinuous areas of the depth map. In addition, it is worth noting that many reconstruction methods based on convolutional neural networks always lack interpretability. It can be seen that there is a need to further study depth map super-resolution methods based on interpretable deep learning. Summary of the Invention

[0004] This paper addresses the problems of texture duplication and edge blurring caused by existing depth map super-resolution methods that use color images as static priors and lack dual-edge consistency constraints. This paper proposes a depth map super-resolution method based on dual-edge consistency constraints. Using the alternating direction multiplier algorithm, this method expands the joint optimization model into a dual-edge consistency-constrained depth map super-resolution network with multiple subnetworks that are iteratively updated. Compared to other methods, this method not only has good interpretability but also achieves better performance.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solution: a depth map super-resolution method based on double edge consistency constraint is performed according to the following steps:

[0006] Step S1, constructing a depth map super-resolution dataset;

[0007] Step S2: constructing a depth map super-resolution optimization model based on double-edge consistency constraints;

[0008] Take the color image C and the low-resolution depth image L as input, and obtain the double edge features through the Laplacian operator, that is, the color image edge Ce and the low-resolution depth image edge E1; given the color image, color image edge, low-resolution depth map and low-resolution depth map edge, construct a depth map super-resolution optimization model based on double edge consistency constraints:

[0009]

[0010] Among them, α, β and γ are the trade-off parameters of different items, represents the depth map constraint based on the dual edge structure prior, represents the edge constraint of the depth map based on the structural prior, L is the input low-resolution depth map, E l is the edge map of the low-resolution depth map, H is the desired high-resolution depth map, E h is the corresponding high-resolution depth map edge, m is the upsampling factor, Ce is the color map edge, Ce 0 is the edge of the color image extracted by the learnable Laplacian operator, D d and K d Denote the downsampling operator and blur operator of the depth map, D e and K e They represent the downsampling operator and blur operator of the depth map edge respectively;

[0011] Step S3, solving the constructed depth map super-resolution optimization model;

[0012] Using the alternating direction multiplier method and the proximal gradient descent method, the depth map super-resolution optimization model constructed in the optimization step S2 is solved to obtain the color image edge reconstruction module, depth map edge reconstruction module, depth map reconstruction module and augmented Lagrangian multiplier update module that are iteratively optimized in each stage;

[0013] Step S4: Construct a double-edge consistency constrained depth map super-resolution network

[0014] The solution formula of each stage of the constructed dual-edge consistency constrained deep map optimization model is converted into different sub-networks, and these sub-networks are combined together to construct a dual-edge consistency constrained deep map super-resolution network;

[0015] Step S5: Construct a multi-cross attention compensation fusion module

[0016] In the multi-cross attention compensation fusion, the dual edge features are first input into the cross-modal cross-attention mechanism to achieve the fusion of the dual edge features. Secondly, the cross-stage depth map features are also input into the cross-modal cross-attention mechanism for fusion. The fused edge features and depth map features are input into the cross-content cross-attention mechanism to achieve feature fusion between different contents, thereby obtaining a high-quality reconstructed depth map.

[0017] Step S6, loss function

[0018] The loss function with L1 norm constraint is used to constrain the super-resolution reconstructed image of the depth map, and the L2 norm is used to constrain the edge of the super-resolution reconstructed image of the depth map.

[0019] Furthermore, in step S1, the depth map super-resolution dataset selects 36 pairs of 1312×1072 RGB-D images from the Middlebury RGB-D dataset for network training, and selects 6 pairs of 640×480 RGB-D images for network performance testing; the high-resolution depth image is randomly cropped to obtain a 256×256 true depth map, and the true depth map is downsampled by ×4, ×8, and ×16 respectively through bicubic interpolation to obtain a low-resolution depth map.

[0020] Furthermore, the specific process of step S3 is as follows:

[0021] First, the alternating direction multiplier method is used to decompose formula (1) into the following three sub-problems, and the solutions are iteratively solved by alternating optimization. The optimization models of the three sub-problems are expressed as follows:

[0022]

[0023] Color image edge Ce reconstruction layer: Use the single-step gradient descent method to obtain the explicit solution of Ce, then the solution process of the first sub-problem of formula (2) is:

[0024]

[0025] Among them, δ1 represents a learnable step size parameter, represents the gradient descent operator;

[0026] Depth map edge E h Reconstruction layer: Optimize and update E by solving the second sub-problem of formula (2) h , an auxiliary variable T is introduced into the optimization model for decoupling, and the optimization solution of this subproblem is transformed into:

[0027]

[0028] Using the alternating direction multiplier algorithm to solve formula (4), its augmented Lagrangian function can be expressed as:

[0029]

[0030] Where P is the Lagrange multiplier, ρ>0 is the penalty function, 〈P,TE〉 represents the inner product, and the scaled Lagrange multiplier is used. The alternating direction multiplier method converts formula (5) into {E h ,T,U}, these three sub-optimization problems are expressed as:

[0031]

[0032] Since three variables need to be optimized, they are iteratively updated in an alternating manner to obtain the solution to the subproblem;

[0033] Update E h :First, fix T and U, given Ce and H, use gradient descent to solve this problem, and the solution formula is written as:

[0034]

[0035] Among them, δ2 represents a learnable step size parameter, (·) T Represents a transpose operation;

[0036] Update T: Fixed E h and U, the proximal gradient descent algorithm is used to solve formula (7), and the solution formula can be written as:

[0037]

[0038] in, yes The proximal operator of is the gradient operator, δ3 is the step size, and n is the nth stage;

[0039] Update U: The update of the Lagrange multiplier U can be written as:

[0040]

[0041] Depth map H reconstruction layer: given Ce and E h , the alternating direction multiplier algorithm is used to solve H, and the optimized problem can be written as:

[0042]

[0043] The augmented Lagrangian function of this optimization problem can be written as:

[0044]

[0045] Where Q is the Lagrange multiplier, α>0 is the penalty function, 〈Q,RH〉 represents the inner product, and the scaled Lagrange multiplier is used. The optimization problem of formula (13) is decomposed into the following three sub-problems:

[0046]

[0047]

[0048] Update H: Fix R and V and use the gradient descent method to solve formula (14). The solution formula is written as:

[0049]

[0050] Update R: Fix H and V and use the proximal gradient descent method to solve formula (15). The solution formula is written as:

[0051]

[0052] Update V: Fix H and R, and update the Lagrange multiplier term as:

[0053]

[0054] Furthermore, the specific process of step S4 is as follows:

[0055] Expand formula (3) into the color map boundary reconstruction sub-network CeNet, expand formula (9), formula (10) and formula (11) into the depth map edge reconstruction sub-network ENet, and expand formula (17), formula (18) and formula (19) into the high-fidelity depth map reconstruction sub-network HFNet;

[0056] Color image edge reconstruction subnetwork CeNet: In order to obtain the color image edge feature Ce, formula (3) is expanded into CeNet, that is:

[0057]

[0058] Among them, Res represents the residual block. First, Used to ensure the updated color map edge Ce (n-1) and the initially extracted color map edge Ce (0) Maintain consistency, Used to ensure the edge of the color image Ce (n-1) With the depth map edge E (n-1) consistency between

[0059] Depth map edge reconstruction sub-network ENet: When the color map edge has been updated, use ENet to predict the high-resolution depth map edge. Expand formula (9), formula (10) and formula (11) into ENet.

[0060] E expansion: The expansion of formula (9) can be written as:

[0061]

[0062] in, Represents the mapping relationship between the edge features of the depth map in the previous stage and the edge features of the low-resolution depth map. Conv is a downsampling convolution operation with a scale factor of m, and DConv is a transposed convolution operation with a step size of m. Used to measure the distance between the edge of the color image and the edge of the depth image, is the gradient descent process of E;

[0063] T expansion: Formula (10) can be expanded as follows:

[0064]

[0065] Among them, UNet(·) is used to approximately replace the proximal operator It's about T n Gradient descent formula;

[0066] U expansion: Formula (11) can be expanded as follows:

[0067]

[0068] High-fidelity depth map reconstruction subnetwork HFNet: Formula (17), Formula (18) and Formula (19) are expanded into HFNet:

[0069] H expansion: Formula (17) can be expanded as follows:

[0070]

[0071] in, Represents the mapping relationship between the depth map features of the previous stage and the low-resolution depth map features, is the constraint error compensation term introduced by the alternating direction multiplier algorithm, represents the gradient descent process;

[0072] R expansion: Given H and V, formula (18) can be expanded as:

[0073]

[0074] The multi-cross attention compensation module MAF is used to replace the traditional proximal operator to achieve deep fusion of different modal information in the same scene;

[0075] V expansion: Given H and R, formula (19) can be expressed as:

[0076] V n =V (n-1) +λ(R (n) -H (n) ) (26)

[0077] The proposed dual-edge consistency constrained deep image super-resolution network consists of S stages, each of which contains three sub-networks, namely CeNet, ENet and HFNet.

[0078] Furthermore, in step S5, edge information of different modalities and cross-stage depth map feature information are subjected to multi-cross complementary fusion. The specific fusion operation is expressed as follows:

[0079]

[0080] Among them, H CCAM represents the cross-modal attention fusion mechanism, H CCFM Represents the cross-content attention fusion mechanism.

[0081] Furthermore, the step S6 outputs the super-resolution depth map H SR Compared with the real depth map H GT The distance between them is measured using the L1 norm, and the loss function can be written as

[0082] For the depth map edge prediction model, the L2 norm is used to measure the depth map edge E predicted by the super-resolution model. SR The edge E of the real depth map GTThe distance between them, the loss function expression can be written as

[0083] The total loss function is L total =Loss1+μLoss2, where μ is used to weigh the two loss functions, i is used to represent the i-th pixel, and N represents the total pixels.

[0084] Compared with the prior art, the present invention has the following beneficial effects.

[0085] 1. The present invention uses the alternating direction multiplier method algorithm to expand the joint optimization model into a double-edge consistency constrained deep map super-resolution network with multiple sub-networks alternately and iteratively updated. Compared with other methods, the present invention not only has good interpretability but also has better performance.

[0086] 2. This invention exploits the consistency of the edge structure of the color image and the edge structure of the depth image to obtain a precise edge structure, thereby enabling higher-quality reconstruction of the depth image. Compared with multiple depth image super-resolution methods, the edge prediction method of this invention is more accurate.

[0087] 3. The dual-edge consistency constraint optimization model is expanded into dual-edge guided depth map reconstruction networks CeNet and ENet, and the obtained accurate color image edges and depth map edges are input into the high-fidelity depth map reconstruction network HFNet. A multi-cross attention compensation fusion MAF module is also designed to fuse the dual edge features with the depth map features across modalities and stages, thereby obtaining a higher quality reconstructed depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 Schematic diagram of the method of the double-edge consistency constrained depth map super-resolution network of the present invention.

[0089] Figure 2 Schematic diagram of the color image edge reconstruction network CeNet in the present invention.

[0090] Figure 3 Schematic diagram of the depth map edge reconstruction network ENet in the present invention.

[0091] Figure 4 Schematic diagram of the high-fidelity depth map reconstruction network HFNet in the present invention.

[0092] Figure 5 Schematic diagram of the multi-cross attention compensation module MAF in the present invention.

[0093] Figure 6Comparison results between our method and recent depth map super-resolution methods on an 8× upsampling visualization of the depth map named “Art” on the Middlebury RGB-D dataset. DETAILED DESCRIPTION

[0094] The following will describe the embodiments of the present invention in detail with reference to the accompanying drawings and examples, so as to fully understand and implement the process of how the present invention applies technical means to solve technical problems and achieve technical effects.

[0095] Figure 1 The proposed dual-edge consistency constrained depth map super-resolution network has a total of S iterative stages, each of which contains three sub-networks: the color image edge reconstruction network CeNet, the depth image edge reconstruction network ENet, and the high-fidelity depth map reconstruction network HFNet. The implementation steps of the proposed method are as follows:

[0096] Step S1: Construction of training data and test data sets.

[0097] The depth map super-resolution dataset selects 36 pairs of 1312×1072 RGB-D images from the Middlebury RGB-D dataset for network training, and 6 pairs of 640×480 RGB-D images for network performance testing; the high-resolution depth images are randomly cropped to obtain 256×256 real depth maps, which are then downsampled by ×4, ×8, and ×16 through bicubic interpolation to obtain low-resolution depth maps.

[0098] Step S2: construct a depth map super-resolution optimization model based on double-edge consistency constraints.

[0099] Take the color image C and the low-resolution depth image L as input, and obtain the double edge features through the Laplacian operator, that is, the color image edge Ce and the low-resolution depth image edge E1; given the color image, color image edge, low-resolution depth map and low-resolution depth map edge, construct a depth map super-resolution optimization model based on double edge consistency constraints:

[0100]

[0101] Among them, α, β and γ are the trade-off parameters of different items, Represents the dual edge structure prior constraint of the depth map, Represents the structural prior constraint term on the edge of the depth map, L is the input low-resolution depth map, E l is the edge map of the low-resolution depth map, H is the desired high-resolution depth map, E h is the corresponding high-resolution depth map edge, m is the upsampling factor, Ce is the color map edge, Ce0 is the edge of the color image extracted by the learnable Laplacian operator, D d and K d Denote the downsampling operator and blur operator of the depth map, D e and K e They represent the downsampling operator and blur operator of the depth map edge respectively;

[0102] Step S3: Solve the constructed depth map super-resolution optimization model.

[0103] The alternating direction multiplier method and the proximal gradient descent method are used to solve the depth map super-resolution optimization model constructed in the optimization step S2, and the color image edge reconstruction module, depth map edge reconstruction module, depth map reconstruction module and augmented Lagrangian multiplier update module that are iteratively optimized in each stage are obtained.

[0104] First, the alternating direction multiplier method algorithm can be used to decompose formula (1) into the following three sub-problems, and then iteratively solve them through alternating optimization:

[0105]

[0106] Color image edge reconstruction layer: Use the single-step gradient descent method to solve the explicit solution of Ce, then the solution to the first sub-problem of formula (2) is:

[0107]

[0108] Among them, δ1 represents a learnable step size parameter, represents the gradient descent operator

[0109] Depth map edge E h Reconstruction layer: For the second sub-problem E of formula (2) h The optimization update introduces an auxiliary variable T in the optimization model to decouple E h The optimization problem can be written as:

[0110]

[0111] Using the alternating direction multiplier algorithm to solve formula (4), its augmented Lagrangian function can be expressed as:

[0112]

[0113] Where P is the Lagrange multiplier, ρ>0 is the penalty function, and 〈P,TE〉 represents the inner product. For simplicity, the scaled Lagrange multiplier is used. The alternating direction multiplier method converts formula (5) into the solution of {E h ,T,U} are three sub-problems:

[0114]

[0115] U n =U+ρ(TE h ) (8)

[0116] Here, there are three variables that need to be optimized, and the solution is to iteratively update n times in an alternating manner to obtain the solution to the subproblem.

[0117] Update E h :First, fix T and U, give Ce and H, and use gradient descent to solve formula (6). The expression can be written as:

[0118]

[0119] Among them, δ2 represents a learnable step size parameter, (·) T Represents a transpose operation.

[0120] Update T: Fixed E h and U, the proximal gradient descent algorithm is used to solve formula (7), and the expression can be written as:

[0121]

[0122] in, yes The proximal operator of is the gradient operator, δ3 is the step size, and n is the nth stage.

[0123] Update U: The final update of the Lagrange multiplier U is as follows:

[0124]

[0125] Depth map H reconstruction layer: given Ce, E h , the alternating direction multiplier algorithm is used to solve H, and the optimized problem can be written as:

[0126]

[0127] The corresponding augmented Lagrangian function can be written as:

[0128]

[0129] Where Q is the Lagrange multiplier, α>0 is the penalty function, 〈Q,RH〉 represents the inner product, and the scaled Lagrange multiplier is used. The optimization problem of formula (13) is decomposed into the following three sub-problems:

[0130]

[0131] V=V+λ(RH) (16)

[0132] Update H: Fix R and V and use the gradient descent method to solve formula (14). The solution formula can be written as:

[0133]

[0134] Update R: Fix H and V and still use the proximal gradient descent method to solve formula (15). The solution formula can be written as:

[0135]

[0136] Update V: With H and R fixed, the update of the Lagrange multiplier term can be expressed as:

[0137] V n =V n-1 +λ(R n -H n ) (19).

[0138] Figure 1 A dual-edge consistency constrained deep image super-resolution network is constructed for this invention. Each sub-network in the network has a clear physical meaning and interpretability. Specifically, the dual-edge consistency constrained deep image super-resolution network is designed as S stages, corresponding to S iterations of CeNet, ENet and HFNet. In the initial stage of the network, the input includes H n-1 、R n-1 、V n-1 、E l 、 T n-1 、U n-1 and Ce n-1 , the output of each stage includes H n 、R n 、V n 、 T n 、U n and Ce n In the nth stage of the double-edge consistency constrained depth map super-resolution network, the network generates H n and Used to reconstruct the depth map edge E SR and SR depth map H SR A global residual branch is used to participate in the reconstruction process of depth map super-resolution. The image L has available low-frequency information through the bicubic interpolation method. The network only needs to learn the bicubic linear interpolation image of L and H SRThis can simplify the learning of the network. For ease of understanding, each sub-formula is specifically expanded into the corresponding sub-network.

[0139] Step S4 constructs a double-edge consistency constrained depth map super-resolution network.

[0140] like Figure 2 、 Figure 3 and Figure 4 As shown in Figure 1, the three sub-optimization problems are converted into three sub-networks, namely the color image edge reconstruction subnetwork CeNet, the depth image boundary reconstruction subnetwork ENet, and the high-fidelity depth image reconstruction subnetwork HFNet.

[0141] Color image edge reconstruction network CeNet: In order to obtain the color image edge feature Ce, formula (3) is expanded into CeNet, that is:

[0142]

[0143] Among them, Res represents the residual block, such as Figure 3 (d) shown. Used to ensure the updated color map edge Ce (n-1) and the initially extracted color map edge Ce (0) Maintain consistency, Used to ensure the edge of the color image Ce (n-1) With the depth map edge E (n-1) consistency between them.

[0144] Depth map edge reconstruction subnetwork ENet: When ENet has obtained the color map edge, ENet is designed to predict the high-resolution depth map edge. Formula (9), Formula (10), and Formula (11) are expanded into ENet:

[0145] E expansion: Formula (9) can be expanded as follows:

[0146]

[0147] Represents the mapping relationship between the edge features of the depth map in the previous stage and the edge features of the low-resolution depth map. Conv is a downsampling convolution operation with a scale factor of m, and DConv is a transposed convolution operation with a step size of m. It is used to measure the distance between the edge of the color image and the edge of the depth image, thereby ensuring the consistency between the dual edge features. is the gradient descent process of E.

[0148] T expansion: Expand formula (10) to:

[0149]

[0150] Among them, UNet(·) is used to approximately replace the proximal operator It's about T n The gradient descent formula.

[0151] U expansion: Expand formula (11) to:

[0152]

[0153] High-fidelity depth map reconstruction subnetwork HFNet: When HFNet has obtained the edge of the color map, HFNet is designed to predict the edge of the high-resolution depth map. Formula (17), Formula (18), and Formula (19) are expanded into HFNet:

[0154] H expansion: Expand formula (17) to:

[0155]

[0156] Represents the mapping relationship between the depth map features of the previous stage and the low-resolution depth map features, is the constraint error compensation term introduced by the alternating direction multiplier method algorithm, represents the gradient descent process.

[0157] R expansion: Given H and V, formula (18) can be expanded as:

[0158]

[0159] The multi-cross attention compensation module MAF is used to replace the traditional proximal operator to achieve deep fusion of different modal information in the same scene.

[0160] V expansion: Given H and R, formula (19) can be expanded as:

[0161] V n =V n-1 +λ(R n -H n ) (26)

[0162] The iterative update of the Lagrange multiplier terms of the deep graph features between the primal and dual variables can improve the convergence of the solution while maintaining the consistency between the variables.

[0163] Step S5 constructs a multi-cross attention compensation fusion module.

[0164] like Figure 5As shown in the figure, a multi-cross attention compensation module (MAF) is designed to approximate the proximal operator. First, the dual edge features are input into the cross-modal cross-attention mechanism to achieve dual edge feature fusion. Second, the cross-stage depth map features are input into the cross-modal cross-attention mechanism for fusion. The fused edge features and depth map features are input into the cross-content cross-attention mechanism to achieve feature fusion between different contents, thereby obtaining a high-quality reconstructed depth map.

[0165] Step S6 loss function

[0166] The loss function with L1 norm constraint is used to constrain the super-resolution reconstruction image of the depth map, and the L2 norm is used to constrain the edge loss of the super-resolution reconstruction image of the depth map.

[0167] Network training parameter configuration:

[0168] This paper implements the proposed method using the deep learning framework PyTorch. An NVIDIA GeForce RTX3090 is used for training and testing. The proposed network model is trained with the Adam optimizer, with a learning rate of 1e-4 and optimizer parameters beta1 equal to 0.9 and beta2 equal to 0.999. A total of 200 epochs are used for training, with a batch size of 4.

[0169] Network training:

[0170] The Middlebury RGB-D dataset is used. The present invention outputs a super-resolution depth map H SR The distance between the real depth map is measured using the L1 norm and can be expressed as For the depth map edge prediction model, the L2 norm is used to measure the depth map edge E predicted by the super-resolution model. SR The edge E of the real depth map GT The distance between The total loss function is written as L total = Loss1 + μLoss2, where μ is used to weigh the two loss functions. Use i to represent the i-th pixel and N to represent the total number of pixels.

[0171] 3. Training of Depth Map Super-resolution Method with Double Edge Consistency Constraints

[0172] 1. Comparison of objective indicators of network test results

[0173] As shown in Table 1, the proposed method is objectively compared with other related methods for depth map super-resolution on the Middlebury RGB-D dataset, using metrics such as mean absolute deviation (MAD), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM). The results in Table 1 demonstrate that the proposed method achieves optimal performance across all metrics at all scales, strongly demonstrating the rationality of the network structure designed in this paper.

[0174] 2. Comparison of subjective indicators of network test results

[0175] like Figure 6 As shown in the visual comparison of the 8× upsampling results of the “Art” depth map in the Middlebury RGB-D dataset, by observing the magnified area, it can be seen that the present invention performs excellent in detail reconstruction, can present finer granular details, and can restore structural information that is closer to the real image.

[0176] Table 1

[0177]

[0178] This paper proposes a three-task optimization model with dual-edge consistency constraints. Based on the alternating direction multiplier method (ADMM), the model can be transformed into an iterative sub-optimization process consisting of color edge updates, depth edge updates, depth map updates, and augmented Lagrangian multiplier updates, resulting in an interpretable depth map super-resolution model. This model can be expanded into an interpretable network. In the sub-optimization model, dual-edge consistency is explicitly used to predict depth map edges and color image edges. Furthermore, bidirectional cross-stage information exchange is implemented within the dual-edge sub-optimization model to provide reliable depth edge discontinuity information. Before the depth map and color image are fed into the depth map super-resolution reconstruction network, the predicted depth map edges and color image edges are concatenated as edge priors, which inevitably leads to issues such as edge blurring and artifacts. Therefore, a multi-cross attention compensation fusion (MAF) is introduced to approximate the neighboring operator, thereby better aggregating different features and achieving high-throughput transfer across stages. The proposed method exhibits good interpretability and alleviates issues such as artifact replication caused by color map guidance. A loss function is defined to constrain the network so that the trained network can better achieve depth map super-resolution.

[0179] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A depth map super-resolution method based on double-edge consistency constraints, characterized by: Follow these steps: Step S1, constructing a depth map super-resolution dataset; Step S2: constructing a depth map super-resolution optimization model based on double-edge consistency constraints; Take the color image C and the low-resolution depth image L as input, and obtain the double edge image through the Laplacian operator, namely the color image edge Ce and the low-resolution depth image edge E1; given the color image, color image edge, low-resolution depth map and low-resolution depth map edge, construct a depth map super-resolution optimization model based on double edge consistency constraint: Among them, α, β and γ are the trade-off parameters of different items, represents the depth map constraint based on the dual edge structure prior, represents the edge constraint of the depth map based on the structural prior, L is the input low-resolution depth map, E l is the edge map of the low-resolution depth map, H is the desired high-resolution depth map, E h is the corresponding high-resolution depth map edge, Ce is the color map edge, Ce 0 is the edge of the color image extracted by the learnable Laplacian operator, D d and K d Denote the downsampling operator and blur operator of the depth image, respectively. e and K e Represent the downsampling operator and blur operator of the depth image edge respectively; Step S3, solving the constructed depth map super-resolution optimization model; Using the alternating direction multiplier method and the proximal gradient descent method, the depth map super-resolution optimization model constructed in the optimization step S2 is solved to obtain the color image edge reconstruction module, depth map edge reconstruction module, depth map reconstruction module and augmented Lagrangian multiplier update module that are iteratively optimized in each stage; Step S4: constructing a double-edge consistency constrained depth map super-resolution network; The solution formula of each stage of the constructed dual-edge consistency constrained deep map optimization model is converted into different sub-networks, and these sub-networks are combined together to construct a dual-edge consistency constrained deep map super-resolution network; Step S5: construct a multi-cross attention compensation fusion module; In the multi-cross attention compensation fusion module, the dual edge features are first input into the cross-modal cross-attention mechanism to achieve dual edge feature fusion. Secondly, the cross-stage depth map features are also input into the cross-modal cross-attention mechanism for fusion. Finally, the fused edge features and depth map features are input into the cross-content cross-attention mechanism to achieve fusion between different contents, thereby obtaining a high-quality reconstructed depth map. Step S6, loss function; The loss function with L1 norm constraint is used to constrain the super-resolution reconstructed image of the depth map, and the L2 norm is used to constrain the edge of the super-resolution reconstructed image of the depth map.

2. The depth map super-resolution method based on double edge consistency constraint according to claim 1, characterized in that: In step S1, the depth map super-resolution dataset selects 36 pairs of 1312×1072 RGB-D images from the Middlebury RGB-D dataset for network training, and selects 6 pairs of 640×480 RGB-D images for network performance testing; the high-resolution depth images are randomly cropped to obtain 256×256 true depth maps, and the true depth maps are downsampled by ×4, ×8, and ×16 respectively using the bicubic interpolation method to obtain low-resolution depth maps.

3. The depth map super-resolution method based on double edge consistency constraint according to claim 1, characterized in that: The specific process of step S3 is as follows: First, the alternating direction multiplier method is used to decompose formula (1) into the following three sub-problems, and the solutions are iteratively solved by alternating optimization. The optimization models of the three sub-problems are expressed as follows: Color image edge Ce reconstruction layer: Use the single-step gradient descent method to obtain the explicit solution of Ce, then the solution process of the first sub-problem of formula (2) is: Among them, δ1 represents a learnable step size parameter, represents the gradient descent operator; Depth map edge E h Reconstruction layer: Optimize and update E by solving the second sub-problem of formula (2) h , an auxiliary variable T is introduced into the optimization model for decoupling, and the optimization solution of this subproblem is transformed into: The alternating direction multiplier algorithm is used to solve formula (4), and its augmented Lagrangian function is expressed as: Where P is the Lagrange multiplier, ρ>0 is the penalty function, 〈P,TE〉 represents the inner product, and the scaled Lagrange multiplier is used. The alternating direction multiplier method converts formula (5) into {E h ,T,U}, these three sub-optimization problems are expressed as: HE n =U+ρ(TE h ) (8) Since three variables need to be optimized, the three variables are iteratively updated in an alternating manner, so that the solution to the subproblem can be obtained; Update E h :First, fix T and U, give Ce and H, and use gradient descent to solve formula (6). The solution formula is written as: Among them, δ2 represents a learnable step size parameter, (·) T Represents a transpose operation; Update T: Fixed E h and U, the proximal gradient descent algorithm is used to solve formula (7), and the solution formula is written as: in, yes The proximal operator of is the gradient operator, δ3 is the step size, and n is the nth stage; Update U: The Lagrange multiplier U is updated as follows: Depth map H reconstruction layer: given Ce and E h , the alternating direction multiplier algorithm is used to solve H, and the optimized problem is written as: The augmented Lagrangian function of this optimization problem can be written as: Where Q is the Lagrange multiplier, α>0 is the penalty function, 〈Q,RH〉 represents the inner product, and the scaled Lagrange multiplier is used. The optimization problem of formula (13) is decomposed into the following three sub-problems: V=V+λ(RH) (16) Update H: Fix R and V and use the gradient descent method to solve formula (14). The solution formula is written as: Update R: Fix H and V and use the proximal gradient descent method to solve formula (15). The solution formula is written as: Update V: Fix H and R, and update the Lagrange multiplier term as: V n =V n-1 +λ(R n -H n ) (19)。 4. The depth map super-resolution method based on double edge consistency constraint according to claim 3, characterized in that: The specific process of step S4 is as follows: Expand formula (3) into the color map boundary reconstruction sub-network CeNet, expand formula (9), formula (10) and formula (11) into the depth map edge reconstruction sub-network ENet, and expand formula (17), formula (18) and formula (19) into the high-fidelity depth map reconstruction sub-network HFNet; Color image edge reconstruction subnetwork CeNet: In order to obtain the color image edge feature Ce, formula (3) is expanded into CeNet, that is: Among them, Res represents the residual block. First, Used to ensure the updated color map edge Ce (n-1) and the initially extracted color map edge Ce (0) Maintain consistency, Used to ensure the edge of the color image Ce (n-1) With the depth map edge E (n-1) consistency between Depth map edge reconstruction sub-network ENet: When the color map edge has been updated, use ENet to predict the high-resolution depth map edge. Expand formula (9), formula (10) and formula (11) into ENet. E expansion: The expansion of formula (9) is written as: in, Represents the mapping relationship between the edge features of the depth map in the previous stage and the edge features of the low-resolution depth map. Conv is a downsampling convolution operation with a scale factor of m, and DConv is a transposed convolution operation with a step size of m. Used to measure the distance between the edge of the color image and the edge of the depth image, is the gradient descent process of E; T expansion: The expansion of formula (10) is written as: Among them, UNet(·) is used to approximately replace the proximal operator It's about T n Gradient descent formula; U expansion: The expansion of formula (11) is written as: High-fidelity depth map reconstruction subnetwork HFNet: Formula (17), Formula (18) and Formula (19) are expanded into HFNet: H expansion: Formula (17) is expanded to: in, Represents the mapping relationship between the depth map features of the previous stage and the low-resolution depth map features, is the constraint error compensation term introduced by the alternating direction multiplier algorithm, represents the gradient descent process; R expansion: Given H and V, formula (18) is expanded to: The multi-cross attention compensation module MAF is used to replace the traditional proximal operator to achieve deep fusion of different modal information in the same scene; V expansion: Given H and R, formula (19) can be written as: V n =V (n-1) +λ(R (n) -H (n) ) (26) The proposed dual-edge consistency constrained deep image super-resolution network consists of S stages, each of which contains three sub-networks, namely CeNet, ENet and HFNet.

5. The depth map super-resolution method based on double edge consistency constraint according to claim 4, characterized in that: In step S5, edge information of different modalities and cross-stage depth map feature information are subjected to multi-cross complementary fusion. The specific fusion operation is expressed as follows: Among them, H CCAM represents the cross-modal attention fusion mechanism, H CCFM Represents the cross-content attention fusion mechanism.

6. The depth map super-resolution method based on double edge consistency constraint according to claim 1, characterized in that: The step S6 outputs the super-resolution depth map H SR Compared with the real depth map H GT The distance between them is measured using the L1 norm, and the loss function expression is written as For the depth map edge prediction model, the L2 norm is used to measure the depth map edge E predicted by the super-resolution model. SR The edge E of the real depth map GT The distance between them, the loss function expression is written as The total loss function is L total =Loss1+μLoss2, where μ is used to weigh the two loss functions, i is used to represent the i-th pixel, and N represents the total pixels.

Images