Depth image restoration method based on sparse spike covariance and non-convex regularization
By combining the local covariance of the color image and the gradient field of the depth image with the sparse peak covariance and non-convex regularization depth image restoration method, the problems of edge blurring and structural distortion in depth image restoration are solved, and higher reconstruction accuracy and resolution are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-04-09
- Publication Date
- 2026-06-16
Smart Images
Figure CN122222893A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and computer vision technology, specifically relating to a depth image restoration method based on sparse peak covariance and non-convex regularization. Background Technology
[0002] With the rapid development of technologies such as 3D vision, autonomous driving, virtual reality, and human-computer interaction, depth images, as crucial information representing the geometric structure of a scene, play a vital role in tasks such as target recognition, scene understanding, and spatial localization. Existing depth images are typically acquired using devices such as structured light, time-of-flight, or binocular vision. However, due to limitations in sensor hardware resolution, imaging mechanisms, and environmental noise interference, the acquired depth images generally suffer from low resolution, high noise, blurred edges, and missing details, thus affecting the accuracy and stability of subsequent visual processing.
[0003] To improve the quality of depth images, researchers have proposed numerous depth image restoration and super-resolution reconstruction methods. Existing methods can be broadly categorized as follows: Interpolation-based methods, such as bilinear interpolation and bicubic interpolation, achieve resolution enhancement by spatially interpolating low-resolution depth images. These methods have low computational complexity, but often lead to blurred edges and loss of detail because they do not consider image structural information; Filtering-based methods, such as guided filtering and bilateral filtering, use high-resolution color images as guides to smooth the edges of depth images. While these methods can improve edge quality to some extent, they are prone to structural mismatches in complex texture regions and are sensitive to noise; Optimization model-based methods solve by constructing a joint model of data fidelity terms and prior regularization terms. For example, methods based on total variational regularization can suppress noise but are prone to staircase effects; methods based on sparse representation or low-rank priors can enhance structural representation capabilities but typically rely on nonlocal similarity assumptions and lack stability in complex scenes; Deep learning-based methods achieve depth image super-resolution reconstruction by training neural networks. These methods perform well with large-scale data support, but they suffer from problems such as insufficient model generalization ability, strong dependence on training data, and high computational resource requirements.
[0004] In recent years, guided restoration of low-resolution depth images using high-resolution color images has become a research hotspot. Color images typically possess richer texture and edge information, providing effective structural constraints for depth image restoration. While existing color-guided methods can improve reconstruction results to some extent, most rely solely on local linear relationships or simple smoothing mechanisms, failing to adequately explore the statistical structural characteristics of local regions within the depth image. Consequently, oversmoothing, boundary blurring, and structural distortion are still prone to occur in complex texture or edge regions. Furthermore, traditional methods underutilize local covariance information in depth images, and commonly used gradient regularization terms have limited expressive power for complex structures, making it difficult to simultaneously address noise suppression, edge preservation, and detail restoration. Therefore, a depth image restoration method that integrates color-guided information, local covariance priors, and gradient structural constraints is urgently needed. Summary of the Invention
[0005] The purpose of this invention is to address the problems of edge blurring, structural distortion, and insufficient utilization of local statistical properties in existing depth image restoration methods, and to provide a depth image restoration method based on sparse peak covariance and non-convex regularization. This method, while fully utilizing the structural information of high-resolution color images, introduces a priori local covariance of the depth image and combines it with a non-convex log-regularization term based on the depth image gradient field to construct a unified optimization model. This effectively preserves edge structure and detail information while suppressing noise and artifacts, thereby improving the reconstruction accuracy and spatial resolution of depth images. Specifically, it includes the following steps:
[0006] (1) Input a degraded depth image and its corresponding high-resolution color image Establish an observation model:
[0007]
[0008] in This represents the observed degradation depth image; The dimension is The space of real matrices; This represents the corresponding high-resolution color image; The dimension is The space of real matrices; This represents the high-resolution depth image to be recovered. Denotes the degenerate operator, where For fuzzy operators, For downsampling operators; Indicates additive noise;
[0009] (2) Based on color image Construct a color-guided local neighborhood and calculate each pixel. The corresponding local covariance matrix and adaptive weights And construct a sparse peaked covariance prior regularization term.
[0010]
[0011] in This represents the total number of pixels in the image; This represents the weighting coefficients that are adaptively adjusted based on the color gradient; Represented in pixels Image patch vector centered on the image; Represents the local covariance matrix The inverse matrix; Represents the transpose of a matrix or vector;
[0012] (3) Construct a non-convex log-regularization term based on the gradient field of the depth image
[0013]
[0014] in and These represent the gradient operators for the image in the horizontal and vertical directions, respectively; Representing an image In pixels The depth value at that location; Represents absolute value; To prevent the logarithmic function from having singular stable parameters; This represents the natural logarithm function, which has non-convex properties. It can impose a stronger penalty on smaller gradient values, thus effectively suppressing noise, while imposing a weaker penalty on larger gradient values, which is beneficial for preserving edge structures.
[0015] (4) A unified optimization model is established by combining the data fidelity term, the sparse peaked covariance prior term, and the non-convex log regularization term.
[0016]
[0017] in Represents the variable Minimize; This is a data fidelity item used to constrain the consistency between the restored results and the observed data; express Norm; and These are regularization parameters, which control the influence strength of the covariance prior term and the non-convex log-regularization term, respectively.
[0018] (5) Introducing auxiliary variables The unified optimization model is then iteratively solved using the alternating direction multiplier method.
[0019] (6) When the preset convergence condition is met, output the recovered high-resolution depth image. .
[0020] The innovation of this invention lies in proposing a depth image restoration method based on sparse peaked covariance and non-convex regularization, organically integrating color guidance mechanisms, local statistical covariance modeling, and non-convex gradient regularization constraints into a single optimization framework. Specifically, it utilizes high-resolution color images to construct structure-guided local neighborhood relationships, effectively constraining the structure of depth edges; based on this, it introduces sparse peaked covariance priors to model the statistical correlation of depth images within their neighborhoods, thereby enhancing structural consistency and noise resistance; simultaneously, it employs a non-convex logarithmic regularization term based on the gradient field to characterize the local structural changes in the depth image, applying stronger penalties to smaller gradient values to suppress noise and weaker penalties to larger gradient values to preserve edge and detail information, improving the ability to express complex edges and local details; and by constructing a unified optimization model and using the alternating direction multiplier method for efficient solution, it achieves synergistic optimization of multiple prior information.
[0021] This invention is mainly verified by simulation experiments, and all steps and conclusions have been verified to be correct on MATLAB R2020b. Attached Figure Description
[0022] Figure 1 This is a flowchart of the method of the present invention;
[0023] Figure 2 is a grayscale image of the color image and a ground truth image of the depth image used in the simulation of this invention.
[0024] Figure 3 The depth image used in the simulation of this invention contains Gaussian noise with a standard deviation of 5 and is downsampled by 8 times;
[0025] Figure 4 shows the effects of different methods. Figure 3 The image after restoration and reconstruction;
[0026] Figure 5 shows the effects of different methods. Figure 3 The error between the restored and reconstructed image and the true image. Detailed Implementation
[0027] Reference Figure 1 This invention is a depth image restoration method based on sparse peak covariance and non-convex regularization, and the specific steps are as follows:
[0028] Step 1: Input a degradation depth image with a size of 165×135. and its corresponding high-resolution color image of 1320×1080. An initial high-resolution depth map of 1320×1080 was obtained by bicubic interpolation of the 165×135 resolution depth map. Establish an observation model
[0029] Equation (1)
[0030] The degradation matrix was calculated using the observation model. .
[0031] Step 2: Construct a color-guided local neighborhood for each pixel and set a color similarity threshold. and spatial proximity threshold The values are 2000 and 25 respectively. Then, the local depth covariance matrix and adaptive weights are calculated. ;
[0032] (2a) Define pixels based on color similarity and spatial proximity. local neighborhood
[0033] Equation (2)
[0034] in Representing pixels and In color image The color vector in; Representing pixels and Spatial coordinates;
[0035] (2b) Calculate the mean of the depth image patch vectors in the neighborhood.
[0036] Equation (3)
[0037] Further calculation of the local depth covariance matrix
[0038] Equation (4)
[0039] in Representing the neighborhood The number of inner pixels, For the neighborhood The mean vector of the inner depth image patch vectors;
[0040] (2c) will Set the weight to 10 to control the degree of weight decay, and then calculate the adaptive weights based on the color gradient.
[0041] Equation (5)
[0042] in Represents the color image in pixels gradient at, The scaling parameter is used to control the rate of weight decay.
[0043] Step 3: Construct a non-convex log-regularization term based on the depth image gradient field. Let... For a smaller positive constant used to ensure numerical stability, then we have
[0044] Equation (6)
[0045] Step 4: Combine the data fidelity term, the sparse peaked covariance prior term, and the non-convex logarithmic regularization term to establish a unified optimization model.
[0046] Equation (7)
[0047] in Set it to 0.01. Set it to 0.1.
[0048] Step 5: Iteratively solve equation (7) using the alternating direction multiplier method.
[0049] (5a) To facilitate the solution, auxiliary variables are introduced. Then equation (7) can be rewritten as a constrained optimization problem;
[0050] Equation (8)
[0051] in and They represent auxiliary variables respectively. In pixels The horizontal and vertical components at that location, Indicates "subject to";
[0052] (5b) Using scaling dual variables Constructing the augmented Lagrangian function
[0053] Equation (9)
[0054] (5c) Alternate updates , , :
[0055] fixed ,renew Regarding equation (9) Taking the partial derivative and setting it to 0, we get:
[0056] Equation (10)
[0057] in express transpose; It is the Laplace matrix derived from the sparse peak covariance regularization term; It is a gradient operator The adjoint operator has a negative divergence in the discrete case; solving equation (10) yields... Closed-form solution;
[0058] fixed ,renew ,right The subproblems are:
[0059] Equation (11)
[0060] in Since the logarithmic term in this subproblem is a non-convex function, and and It is difficult to directly obtain an analytical closed-form solution through absolute value and mutual coupling; therefore, an iterative reweighted least squares method is used for the solution. After constructing the reweighting coefficients based on the current iterative value, the two components are updated separately, yielding:
[0061] Equation (12)
[0062] Equation (13)
[0063] in , Represents the image gradient at pixels The component at the location; , Represents Lagrange multipliers In pixels The component at the location; The symbolic function is represented; stable convergence can be achieved by performing 3 to 5 inner iterations on equations (12) and (13).
[0064] Finally, the scaling dual variable is updated according to the standard ADMM update rules. Update:
[0065] Equation (14)
[0066] By iterating through the three sub-steps mentioned above, the stable solution of the optimization problem corresponding to equation (7) can be gradually approximated.
[0067] Step 6, set convergence criteria:
[0068] Equation (15)
[0069] in, The preset convergence threshold has a range of values. If equation (15) is not satisfied, return to step 5 and continue iterating until equation (15) is satisfied or the preset maximum number of iterations (50) is reached. Once the iteration is complete, the reconstructed depth image is output. .
[0070] The effects of this invention can be further illustrated by the following simulation experiments:
[0071] I. Experimental Conditions and Content
[0072] Experimental conditions: The experiment used the Middlebury stereo 2005 benchmark database as the experimental data. The Art images were used as the effect demonstration images, with the color image shown in Figure 2(a) and the ground truth image of the depth image shown in Figure 2(b). Gaussian noise with a standard deviation of 5 was added to the Art depth image, and it was downsampled by 8 times as the degraded depth image. Figure 3 As shown; the experiment uses Peak Signal-to-Noise Ratio (PSNR) and Mean Absolute Difference (MAD) as evaluation metrics. PSNR is defined as:
[0073]
[0074] in and The reference image and the restored image are located at the following positions respectively. Pixel value at that location, MAX represents the image size. MAX is the maximum pixel value of the image; when using an 8-bit grayscale image, MAX is usually set to 255. PSNR is measured in decibels (dB); a higher value indicates higher image quality. Mean Absolute Error (MAD) is defined as:
[0075]
[0076] The smaller the MAD value, the closer the restored image is to the reference image, and the higher the restoration accuracy.
[0077] Experimental content: Under the above conditions, the method of this invention is compared with the classic Bicubic interpolation method, AR method, and the currently leading SPIDM and JLNSR methods in the field of color guided depth image restoration.
[0078] Experiment 1: Using the ground truth depth map in Figure 2, add Gaussian noise with a standard deviation of 5 and downsample by 8 times to obtain a low-resolution depth map. Figure 3 Then, a depth map of the initial size is obtained by bicubic interpolation, and color-guided depth recovery is performed under the same conditions using the method of this invention, the AR method, the JLNSR method, and the SPIDM method.
[0079] The Bicubic method uses bicubic interpolation to upsample low-resolution depth images and restores resolution through local interpolation. The restored result is shown in Figure 4(a), and the corresponding error map is shown in Figure 5(a). The AR method models depth images based on an autoregressive model. It establishes linear regression relationships between pixels in local neighborhoods and uses structural information provided by high-resolution color images to guide the estimation of regression coefficients, thereby reconstructing the depth image. The restored result is shown in Figure 4(b), and the corresponding error map is shown in Figure 5(b). The SPIDM method constructs an optimization model that combines structure preservation and detail enhancement. It constrains the depth image in the gradient domain and uses color images to assist edge preservation. The restored result is shown in Figure 4(c), and the corresponding error map is shown in Figure 5(c). The JLNSR method uses joint low-rank and non-local similarity constraints to group depth image blocks and construct a low-rank model. It also combines color guidance information to achieve structure preservation. The restored result is shown in Figure 4(d), and the corresponding error map is shown in Figure 5(d). The restored result of the method of this invention is shown in Figure 4(e), and the corresponding error map is shown in Figure 5(e).
[0080] The Bicubic method is essentially an interpolation reconstruction that only utilizes the smooth relationships of pixel neighborhoods for upsampling, without incorporating color guidance or depth structure priors. Therefore, it is prone to blurring at depth edges and in detail regions, with errors mainly concentrated at contour transitions. The AR method describes the linear correlation between pixels through local autoregressive modeling and uses color images for guidance, exhibiting better structure recovery capabilities than interpolation methods. However, its linear model has limited adaptability to complex textures, abrupt depth changes, and noise interference, resulting in some error accumulation near edges. The SPIDM method enhances edge consistency through structure-preserving constraints, outperforming traditional interpolation methods in contour recovery, but still involves a trade-off between noise suppression and detail preservation. The JLNSR method improves reconstruction capabilities by utilizing nonlocal similarity and low-rank priors, showing good recovery effects for repetitive structures and overall smooth regions. However, its block matching and grouping accuracy are easily affected under noisy conditions, thus weakening its local detail recovery performance.
[0081] In contrast, the method of this invention, based on color guidance, further introduces a sparse peaked covariance prior to characterize the local statistical structure of the depth image, and combines a non-convex log-regularization term based on the gradient field to constrain edge, contour, and detail variations in the depth image. This non-convex log-regularization term imposes a stronger penalty on smaller gradient values, effectively suppressing noise, while imposing a weaker penalty on larger gradient values, which helps preserve edge structure and detail information in the depth image. Compared to methods that rely solely on interpolation relationships, the method of this invention can more effectively utilize the correspondence between color images and depth structures; compared to methods that rely solely on non-local similarity or structural constraints, the method of this invention achieves a better balance between local structure preservation, detail recovery, and noise suppression. Therefore, as can be seen from the recovery results in Figure 4, the method of this invention exhibits clearer contours in edge regions and more natural structures in texture and local detail regions; as can be seen from the error map in Figure 5, the method of this invention corresponds to fewer high-error regions and the error distribution is more concentrated in a few local areas, indicating that the overall deviation between its recovery results and the reference depth image is smaller, and it has better recovery accuracy and visual quality.
[0082] Table 1 PSNR metrics for different methods in different scenarios
[0083]
[0084] Table 1 presents the PSNR results of different methods in various test scenarios. Overall, the Bicubic method has the lowest PSNR, indicating that simple interpolation is insufficient to effectively recover structural details in depth images. The AR method, based on local modeling, achieves a higher PSNR than Bicubic, demonstrating that color guidance and local regression play a role in improving recovery quality. SPIDM and JLNSR further improve recovery performance; the former reflects the effectiveness of structural constraints, while the latter shows that nonlocal priors have a positive effect on depth reconstruction. In contrast, the method of this invention achieves higher PSNR in most scenarios and has the best overall performance, indicating that by introducing a color guidance mechanism, sparse peaked covariance prior, and a non-convex log-regularization term based on the gradient field, this invention can better preserve structural information and suppress noise while improving reconstruction accuracy, thus exhibiting stronger comprehensive recovery advantages.
[0085] Table 2 MAD index of different methods in different scenarios
[0086]
[0087] Table 2 presents the MAD results for different methods. MAD reflects the absolute deviation between the reconstructed image and the reference image; a smaller value indicates higher reconstruction accuracy. The results in the table show that the Bicubic method has a relatively large MAD, indicating a significant reconstruction error. The AR method shows a lower MAD, indicating that local linear modeling can reduce the error to some extent. SPIDM and JLNSR further improve the reconstruction results, demonstrating the effectiveness of structural constraints and nonlocal priors. In contrast, the method of this invention achieves a smaller MAD in most scenarios, indicating that its reconstruction results are closer to the reference depth map and have higher pixel-level reconstruction accuracy. This shows that, based on color guidance and covariance priors, this invention, by introducing non-convex logarithmic gradient regularization constraints, can more effectively preserve edge and detail information while suppressing noise, thus obtaining more accurate and stable depth image reconstruction results, consistent with the performance in the error map.
[0088] In summary, the method of this invention addresses the challenge of simultaneously achieving noise suppression, edge preservation, and detail reconstruction in depth image restoration. By introducing a sparse peaked covariance prior and a non-convex logarithmic gradient regularization term, it improves the expressive power for complex structures and regions with abrupt depth changes. Experimental results demonstrate that the method exhibits good overall performance in both subjective visual effects and objective evaluation metrics, validating its effectiveness in color-guided depth image restoration.
Claims
1. A depth image restoration method based on sparse peak covariance and non-convex regularization, characterized in that, Includes the following steps: (1) Input a degraded depth image and its corresponding high-resolution color image Establish an observation model: ; in This represents the observed degradation depth image; The dimension is The space of real matrices; This represents the corresponding high-resolution color image; The dimension is The space of real matrices; This represents the high-resolution depth image to be recovered. Denotes the degenerate operator, where For fuzzy operators, For downsampling operators; Indicates additive noise; (2) Based on color image Construct a color-guided local neighborhood and calculate each pixel. The corresponding local covariance matrix and adaptive weights And construct a sparse peaked covariance prior regularization term: ; in This represents the total number of pixels in the image; This represents the weighting coefficients that are adaptively adjusted based on the color gradient; Represented in pixels Image patch vector centered on the image; Represents the local covariance matrix The inverse matrix; Represents the transpose of a matrix or vector; (3) Construct a non-convex log-regularization term based on the gradient field of the depth image: ; in and These represent the gradient operators for the image in the horizontal and vertical directions, respectively; Representing an image In pixels The depth value at that location; Represents absolute value; To prevent the logarithmic function from having singular stable parameters; This represents the natural logarithm function, which has non-convex properties. It can impose a stronger penalty on smaller gradient values, thus effectively suppressing noise, while imposing a weaker penalty on larger gradient values, which is beneficial for preserving edge structures. (4) By combining the data fidelity term, the sparse peaked covariance prior term, and the non-convex log-regularization term, a unified optimization model is established: ; in Represents the variable Minimize; This is a data fidelity item used to constrain the consistency between the restored results and the observed data; express Norm; and These are regularization parameters, which control the influence strength of the covariance prior term and the non-convex log-regularization term, respectively. (5) Introducing auxiliary variables The unified optimization model is then iteratively solved using the alternating direction multiplier method. (6) When the preset convergence condition is met, output the recovered high-resolution depth image. .
2. The depth image restoration method based on sparse peak covariance and non-convex norm according to claim 1, characterized in that, The calculation of the local covariance matrix in step (2) includes: (2a) Define pixels based on color similarity local neighborhood : ; in Representing pixels and In color image The color vector in; Representing pixels and Spatial coordinates; and These are the thresholds for color similarity and spatial proximity, respectively; (2b) Calculate the local depth covariance matrix: ; in Representing the neighborhood The number of pixels within the cell; ; in Representing the neighborhood The mean vector of the inner depth image patch vectors; (2c) Adaptive weight adjustment based on color gradient: ; in Represents the color image in pixels gradient at; The scaling parameter is used to control the rate of weight decay.
3. The depth image restoration method based on sparse peak covariance and non-convex norm according to claim 1, characterized in that, The specific implementation of the alternating direction multiplier method in step (5) includes the following sub-steps: (5a) Introducing auxiliary variables The unified optimization model is rewritten as a constrained optimization problem: ; in and They represent auxiliary variables respectively. In pixels The horizontal and vertical components at that location, Indicates "subject to"; (5b) Using scaling dual variables Construct the augmented Lagrangian function: ; in Let be a Lagrange multiplier matrix, with dimension AND. same; For penalty parameters; Representing a matrix Norm; (5c) Alternately iterate and update variables : fixed and renew ,: Regarding the augmented Lagrange function with respect to Taking the derivative and setting it to 0, we get: ; superscript Indicates the number of iterations. express transpose; It is composed of covariance regularization term The derived Laplacian matrix; It is a gradient operator The adjoint operator; fixed and renew Since the logarithmic function is separable, for each pixel The gradient components are solved independently using the iterative reweighted least squares method: ; ; in , Represents the image gradient at pixels The component at the location; , Represents Lagrange multipliers In pixels The component at the location; Represents a symbolic function; Update Lagrange multipliers 。 4. The depth image restoration method based on sparse peak covariance and non-convex norm according to claim 1, characterized in that, The convergence condition in step (6) is: ; in Indicates the first The depth image estimate obtained in the next iteration is the restored depth image; The preset convergence threshold has a range of values. .