A Deep Completion Method Based on Heterogeneous Non-rigid Constraints

Abstract

This invention belongs to the fields of computer vision, depth completion, and 3D perception, and specifically provides a depth completion method based on heterogeneous non-rigid constraints. It includes the following steps: S1: Construction of a region-aware heterogeneous physical map; S2: Selection of sparse physical anchor points and definition of tracking confidence; S3: Construction of a multi-objective joint energy function; S4: Joint modeling and iterative optimization of scale and translation parameters; S5: Dynamic weight scheduling and result output based on tracking state. This invention is applicable to robot vision, industrial inspection, augmented reality, digital twins, medical imaging, and other monocular vision scenarios with local non-rigid changes, local dynamic perturbations, or only sparse depth observation. By introducing a region-aware heterogeneous physical map, node-level relative elastic weights, node-level relative sticky weights, and a joint optimization mechanism during testing, it achieves highly consistent recovery of dense depth maps under sparse depth constraints.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, depth completion, and 3D perception, specifically a depth completion method based on heterogeneous non-rigid constraints. Background Technology

[0002] Depth completion technology is widely used in fields such as robot vision, industrial inspection, augmented reality, digital twins, medical imaging, and 3D perception of complex scenes. In actual data acquisition, the system often only obtains monocular color images and a small number of sparse depth observations. Therefore, it is necessary to recover the dense depth results corresponding to the current viewpoint under limited geometric priors.

[0003] Existing methods generally fall into three categories: the first is traditional methods based on multi-view geometry or motion reconstruction structures, which typically rely on rigid scene assumptions or strong view coverage; the second is single-image depth estimation and depth completion methods based on convolutional networks, Transformers, or diffusion models, which have strong data-driven capabilities; and the third is methods that incorporate sparse depth, camera intrinsics, or other priors into the learning model to improve depth reconstruction results through test-time optimization or explicit constraints.

[0004] The most similar implementation to this invention is often a depth completion framework based on "image priors + sparse depth constraints". These methods perform well in rigid or near-rigid natural scenes, but in complex, general scenes with local non-rigid deformations, weak textures, specular reflections, occlusion, cross-frame local motion differences, or uneven anchor point distribution, they are prone to problems such as local structural distortion, scale drift, and cross-frame inconsistencies. In particular, existing methods often only utilize pixel-level sparse depth points, failing to fully leverage the local topological relationships, regional attribute differences, and tracking state information generated by front-end tracking or sparse reconstruction processes. Therefore, they still have shortcomings in terms of structural consistency, scale consistency, and overall robustness. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution: a depth completion method based on heterogeneous non-rigid constraints, wherein the input variables include: a color image, a set of sparse depth anchor points of the current frame or an equivalent sparse depth map, a camera intrinsic parameter matrix, and a heterogeneous physical map; the specific method includes the following steps:

[0007] S1: Construction of region-aware heterogeneous physical graph: Using dynamic deformable graphs or equivalent graph structures as carriers, construct a region-aware heterogeneous physical graph, map the region where the node is located to node-level relative elastic weights and node-level relative sticky weights, and further generate edge-level joint weights, thereby explicitly encoding the regional differences into the subsequent depth completion constraints.

[0008] S2: Sparse physical anchor point selection and tracking confidence definition: Project the visible, reprojected, and tracking-stable nodes in the current frame onto the image plane to form a sparse anchor point set for the current frame, and construct a tracking confidence index.

[0009] S3: Multi-objective joint energy function construction: Jointly construct geometric consistency term, viscous smoothness term, elastic recovery term, scale alignment term and boundary constraint term, so that depth prediction is simultaneously constrained in terms of local structure, regional smoothness, prior recovery ability and scale consistency;

[0010] S4: Joint modeling and iterative optimization of scale and translation parameters: The relative depth output by the pre-trained monocular depth model is explicitly converted into a pseudo-metric depth, and the latent variables, scale parameters and translation parameters are jointly updated during the DDIM back-induction process without modifying the diffusion backbone network.

[0011] S5: Dynamic weight scheduling and result output based on tracking status: The weights of each loss term are dynamically adjusted according to the tracking confidence; when the tracking confidence is high, the geometric consistency term and scale alignment term are enhanced; when the tracking confidence is low, the strength of the structured physical constraint is reduced, and the final output is a dense depth map corresponding to the current viewpoint.

[0012] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method of S1 is as follows: the displacement of the graph node at time t is defined as:

[0013]

[0014] in, Represents a node At any moment The displacement vector relative to the reference time; Represents a node At any moment The three-dimensional position; Represents a node The three-dimensional position at the reference time or the initial time;

[0015] To map the region priors to the graph structure, relative elasticity weights and relative stickiness weights are defined for the regions where nodes are located:

[0016]

[0017] in, Represents a node The region label or region number to which the region belongs; Represents a node The relative elasticity weights; Represents a node The relative stickiness weight; and These represent the mapping functions from the region to the relative elastic weight and the relative viscous weight, respectively;

[0018] For the edge The joint elasticity weight and joint stickiness weight are constructed using the attributes of the two endpoints:

[0019]

[0020] in, Representing an edge The joint elasticity weights; Representing an edge The joint stickiness weight; and These represent the relative elastic weights of the nodes at both ends of the edge; and These represent the relative stickiness weights of the nodes at both ends of the edge;

[0021] When a scene experiences local stretching, occlusion, or topological relationship failure, the current length of any edge in the graph is compared with the reference length, and the relative length change rate of that edge is calculated. If the relative length change rate exceeds a preset edge failure determination threshold, the edge is considered to no longer satisfy the common deformation or common motion assumption, and the edge is removed from the current set of valid edges.

[0022]

[0023] in, Represents the Euclidean norm; and Representing time respectively The three-dimensional positions of the nodes at both ends of the edge; and These represent the three-dimensional positions of the nodes corresponding to the reference time; This represents the threshold for determining edge failure.

[0024] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method of S2 is to project the visible, reprojected, and tracked nodes in the current frame onto the image plane to form a sparse anchor point set for the current frame. To reflect the reliability of the current conditions, the tracking confidence level is defined as:

[0025]

[0026] in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; This indicates the number of reference feature points.

[0027] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method of S3 is as follows: Let the depth prediction output by the diffusion model or other generation / optimization module at the current viewpoint be the current predicted depth map, which is pixel-wise... The depth value at a given location is the predicted depth, and the back-projected 3D point is the predicted 3D point. The joint energy is constructed as follows:

[0028]

[0029] in, Represents the joint energy function of the back end; Indicates the geometric consistency term; Indicates the viscous smoothing term; Indicates the elastic recovery item; Indicates the scale alignment item; Represents boundary constraint terms; , , , and These represent the weighting coefficients corresponding to each energy component;

[0030] Geometric consistency constraints are defined as follows:

[0031]

[0032]

[0033] in, Represents the set of graph edges or the set of valid adjacent edges at time t; Representing an edge Geometric consistency weights; and These represent the three-dimensional points obtained by backprojection of the predicted depth; and These represent the corresponding nodes in the front-end graph structure at time [time]. The three-dimensional position; and These represent the pixel positions of the nodes at both ends of the edge after they are projected onto the image plane; and These represent the spatial distance scale parameter and the pixel distance scale parameter, respectively;

[0034] Viscous smoothing constraint is defined as:

[0035]

[0036] in, Represents the image domain; Represents the pixels obtained by propagation of node-level relative stickiness weights. Local smoothing weights at the location; Indicates the predicted depth in pixels gradient at;

[0037] The elastic recovery constraint is defined as:

[0038]

[0039] in, Represents the pixel obtained by propagation of node-level relative elastic weights. Local recovery weights at the location; Represents pixels Predicted depth at the location; Represents pixels The prior depth value at that location;

[0040] Scale alignment constraints are defined as follows:

[0041]

[0042] in, Represents the set of sparse anchor points in the current frame; Indicates the scale parameter of the current frame; Indicates the translation parameters for the current frame; Indicates anchor point Predicted depth at location; Indicates anchor point The provided pseudo-metric depth; Represents a robust kernel function;

[0043] Boundary constraints are defined as:

[0044]

[0045] in, and These represent the lower bound and the upper bound of depth, respectively. Represents pixels Predicted depth at the location.

[0046] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method for jointly modeling the S4 scale and translation parameters is as follows: explicitly modeling the scale parameters. With translation parameters Transform it into a pseudo-metric depth:

[0047]

[0048] in, This represents the relative depth map output by the pre-trained monocular depth model; Indicates the scale parameter Translation parameters The transformed pseudo-metric depth map preferentially adopts a min-max initialization strategy based on the anchor point depth range.

[0049] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method of S4 and iterative optimization is as follows: without modifying the diffusion backbone network, during the DDIM backpropagation process, the latent variable z and the scale parameter are adjusted. Translation parameters For joint optimization, the latent variable update is written as:

[0050]

[0051] in, Let represent the latent variable in the l-th reverse iteration; Represents the updated latent variable; This represents the update step size for the l-th iteration; Represents the joint energy function for latent variables The gradient.

[0052] As a preferred embodiment of the depth completion method based on heterogeneous non-rigid constraints described in this invention, the specific method of step S5 is as follows: based on the aforementioned tracking confidence... The confidence score is calculated using the following formula for dynamic weight scheduling:

[0053]

[0054] in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; Indicates the number of reference feature points; according to The magnitude of the energy component is used to schedule its weight:

[0055] when When the value is high, increase the geometric consistency term. and scale alignment items The weights allow the backend to make fuller use of reliable structured priors;

[0056] when When the range is moderate, a moderately strong structural constraint is adopted to maintain a balance between graph structure traction and graph prior degrees of freedom;

[0057] when When the depth is low, reduce the weights of strong physical constraints such as geometric consistency, elastic recovery, and scale alignment to avoid excessive pull from erroneous or sparse anchor points on depth recovery; after completing all iterative optimizations, output the pseudo-metric depth as the dense depth of the current frame.

[0058]

[0059] in, This indicates the final density depth output for the current frame; This represents a pseudo-metric depth map after scale and translation parameter correction; for video sequences, the above process is repeated for each frame, and the output depth is used for subsequent multi-frame fusion, surface reconstruction, or point cloud post-processing.

[0060] Compared with the prior art, the beneficial effects of the present invention are: the present invention is applicable to robot vision, industrial inspection, augmented reality, digital twins, medical imaging and other monocular vision scenarios with local non-rigid changes, local dynamic disturbances or only sparse depth observation. By introducing heterogeneous physical graphs, node-level relative elastic weights, node-level relative sticky weights and joint optimization mechanisms during testing, the invention achieves high consistency recovery of dense depth graphs under sparse depth constraints. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0062] Figure 1 This is a flowchart of a depth completion method based on heterogeneous non-rigid constraints according to the present invention. Detailed Implementation

[0063] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0064] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0065] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0066] This invention provides a depth completion method based on heterogeneous non-rigid constraints. Using a heterogeneous physical graph as the structural carrier, it incorporates region-dependent relative elasticity and relative viscosity weights into the test-time optimization process. Furthermore, it integrates sparse physical anchors, scale / translation parameters, and tracking confidence into joint constraints, thereby achieving dense depth recovery with consistent scale, consistent structure, and high robustness. Its key technologies consist of six core stages: region-aware heterogeneous physical graph construction, sparse physical anchor selection and tracking confidence modeling, multi-objective joint energy function construction, joint modeling of scale and translation parameters, iterative optimization process, and dynamic weight scheduling based on tracking state.

[0067] For details, please refer to Figure 1 A depth completion method based on heterogeneous non-rigid constraints includes the following steps:

[0068] S1: Construction of Region-Aware Heterogeneous Physical Graph: Using dynamic deformable graphs or equivalent graph structures as carriers, construct a region-aware heterogeneous physical graph, map the region where the node is located to node-level relative elastic weights and node-level relative sticky weights, and further generate edge-level joint weights, thereby explicitly encoding the regional differences into the subsequent depth completion constraints.

[0069] S2, Sparse physical anchor point selection and tracking confidence definition: Project the visible, reprojected, and tracking-stable nodes in the current frame onto the image plane to form a sparse anchor point set for the current frame, and construct a tracking confidence index to adjust the traction strength of structured priors on backend depth optimization.

[0070] S3, Multi-objective joint energy function construction: Jointly construct geometric consistency term, viscous smoothing term, elastic recovery term, scale alignment term and boundary constraint term, so that depth prediction is simultaneously constrained in terms of local structure, regional smoothness, prior recovery ability and scale consistency.

[0071] S4, Joint Modeling and Iterative Optimization of Scale and Translation Parameters: The relative depth output by the pre-trained monocular depth model is explicitly converted into a pseudo-metric depth, and the latent variables, scale parameters and translation parameters are jointly updated during the DDIM backpropagation process without modifying the diffusion backbone network.

[0072] S5: Dynamic weight scheduling and result output based on tracking status: The weights of each loss term are dynamically adjusted according to the tracking confidence; when the tracking confidence is high, the geometric consistency term and scale alignment term are enhanced; when the tracking confidence is low, the strength of the structured physical constraint is reduced, and the final output is a dense depth map corresponding to the current viewpoint.

[0073] The specific steps of its technical solution are as follows:

[0074] The known conditions (i.e., input variables) of this invention include: color image The current frame's sparse depth anchor set Or an equivalent sparse depth map, camera intrinsic matrix Heterogeneous physical diagram Node-level relative elastic weights Node-level relative stickiness weights Prior depth map Tracking status statistics Compared with reference threshold And latent variables of diffusion models or other iterative optimization models. The variable to be determined is the dense depth map. .

[0075] Step 1: Constructing a region-aware heterogeneous physics graph. Define the displacement of a graph node at time t as:

[0076]

[0077] in, Represents a node At any moment The displacement vector relative to the reference time; Represents a node At any moment The three-dimensional position; Represents a node The three-dimensional position at the reference time or the initial time.

[0078] To map the region priors to the graph structure, relative elasticity weights and relative stickiness weights are defined for the regions where nodes are located:

[0079]

[0080] in, Represents a node The region label or region number to which the region belongs; Represents a node The relative elasticity weights; Represents a node The relative stickiness weight; and These represent the mapping functions from the region to the relative elastic weight and the relative viscous weight, respectively.

[0081] For the edge The joint elasticity weight and joint stickiness weight are constructed using the attributes of the two endpoints:

[0082]

[0083] in, Representing an edge The joint elasticity weights; Representing an edge The joint stickiness weight; and These represent the relative elastic weights of the nodes at both ends of the edge; and These represent the relative stickiness weights of the nodes at both ends of the edge.

[0084] When a scene experiences local stretching, occlusion, or topological relationship failure, the current length of any edge in the graph is compared with the reference length, and the relative length change rate of that edge is calculated. If the relative length change rate exceeds a preset edge failure determination threshold, the edge is considered to no longer satisfy the common deformation or common motion assumption, and the edge is removed from the current set of valid edges.

[0085]

[0086] in, Represents the Euclidean norm; and Representing time respectively The three-dimensional positions of the nodes at both ends of the edge; and These represent the three-dimensional positions of the nodes corresponding to the reference time; This represents the threshold for determining edge failure.

[0087] Step 2: Sparse Physical Anchor Point Selection and Tracking Confidence Definition. Nodes in the current frame that are visible, have consistent reprojection, and exhibit stable tracking states are projected onto the image plane to form the sparse anchor point set for the current frame. To reflect the reliability of the current conditions, the tracking confidence level is defined as:

[0088]

[0089] in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; This indicates the number of reference feature points. The larger the value, the more reliable the current front-end constraints are, and the more actively the back-end can utilize the structured priors. The smaller the value, the more the strong constraint traction strength should be reduced in order to retain more degrees of freedom from the image prior.

[0090] Step 3: Multi-objective joint energy function. Let the depth prediction output by the diffusion model or other generation / optimization module at the current viewpoint be the current predicted depth map, which has a pixel... The depth value at a given location is the predicted depth, and the back-projected 3D point is the predicted 3D point. The joint energy is constructed as follows:

[0091]

[0092] in, Represents the joint energy function of the back end; Indicates the geometric consistency term; Indicates the viscous smoothing term; Indicates the elastic recovery item; Indicates the scale alignment item; Represents boundary constraint terms; , , , and These represent the weighting coefficients corresponding to each energy component.

[0093] Geometric consistency constraints are defined as follows:

[0094]

[0095]

[0096] in, Represents the set of graph edges or the set of valid adjacent edges at time t; Representing an edge Geometric consistency weights; and These represent the three-dimensional points obtained by backprojection of the predicted depth; and These represent the corresponding nodes in the front-end graph structure at time [time]. The three-dimensional position. This requirement is that the local structure obtained by dense depth backprojection should maintain an adjacency distance relationship consistent with the structure of the front-end heterogeneous physical map as much as possible, thereby suppressing abrupt changes in local structure. and These represent the pixel positions of the nodes at both ends of the edge after they are projected onto the image plane; and These represent the spatial distance scale parameter and the pixel distance scale parameter, respectively.

[0097] Viscous smoothing constraint is defined as:

[0098]

[0099] in, Represents the image domain; Represents the pixels obtained by propagation of node-level relative stickiness weights. Local smoothing weights at the location; Indicates the predicted depth in pixels The gradient at that location. This term is used to reduce high-frequency jumps caused by weak textures, reflections, and noisy areas.

[0100] The elastic recovery constraint is defined as:

[0101]

[0102] in, Represents the pixel obtained by propagation of node-level relative elastic weights. Local recovery weights at the location; Represents pixels Predicted depth at the location; Represents pixels The prior depth value at that location. This term encourages depth solutions in relatively stable regions to revert to a more reasonable equilibrium state.

[0103] Scale alignment constraints are defined as follows:

[0104]

[0105] in, Represents the set of sparse anchor points in the current frame; Indicates the scale parameter of the current frame; Indicates the translation parameters for the current frame; Indicates anchor point Predicted depth at location; Indicates anchor point The provided pseudo-metric depth; This represents the robust kernel function. This term progressively pulls the predicted depth back to the pseudo-metric space defined by sparse anchor points, and is a key constraint for achieving scale-consistent recovery in this invention.

[0106] Boundary constraints are defined as:

[0107]

[0108] in, and These represent the lower bound and the upper bound of depth, respectively. Represents pixels The predicted depth at that location. This item is used to limit the variation of the predicted depth within a reasonable range.

[0109] Step 4: Joint Modeling of Scale and Translation Parameters. Since pre-trained monocular depth models typically output relative depth maps or affine-invariant depth maps, this invention introduces scale and translation parameters to perform scale and translation transformations on the relative depth map, thereby obtaining a pseudo-metric depth map:

[0110]

[0111] in, This represents the relative depth map output by the pre-trained monocular depth model; Indicates the scale parameter Translation parameters The transformed pseudo-metric depth map. To avoid numerical compression caused by single least-squares fitting when the anchor point distribution is sparse or localized, this invention preferentially adopts a min-max initialization strategy based on the anchor point depth range.

[0112] Step 5: Iterative optimization process. Without modifying the diffusion backbone network, optimize the latent variable z and scale parameter during the DDIM backpropagation process. Translation parameters Perform joint optimization. Latent variable updates can be written as:

[0113]

[0114] in, Let represent the latent variable in the l-th reverse iteration; Represents the updated latent variable; This represents the update step size for the l-th iteration; Represents the joint energy function for latent variables The gradient.

[0115] Step Six: Dynamic Weight Scheduling Based on Tracking State. Calculate the tracking confidence score for the current frame. The formula is:

[0116]

[0117] in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; Indicates the number of reference feature points. The magnitude of the energy component is used to schedule its weight:

[0118] when When the value is high, increase the geometric consistency term. and scale alignment items The weights allow the backend to make fuller use of reliable structured priors;

[0119] when When the range is moderate, a moderately strong structural constraint is adopted to maintain a balance between graph structure traction and graph prior degrees of freedom;

[0120] when When the depth is low, reduce the weights of strong physical constraints such as geometric consistency, elastic recovery, and scale alignment to avoid excessive traction of erroneous or sparse anchor points on depth recovery.

[0121] Step 7: Output Results. After completing all iterative optimizations, output the pseudo-metric depth as the dense depth of the current frame:

[0122]

[0123] in, This indicates the final density depth output for the current frame; This represents a pseudo-metric depth map after scale and translation parameter correction. For video sequences, the above process can be repeated for each frame, and the output depth can be used for subsequent multi-frame fusion, surface reconstruction, or point cloud post-processing. It should be noted that point cloud denoising, surface smoothing, and other steps are only optional end-effector enhancement modules and do not constitute a necessary limitation of this invention.

[0124] The following table compares the dense depth completion errors of this invention in cross-dataset scenarios. The best results in the table are highlighted in bold. The symbol "↓" in the table indicates that the smaller the value of the evaluation metric, the better the performance.

[0125] Table 1 shows the backend dense depth completion results of the method of this invention on the Hamlyn exploration-sequence-20 and SCARED dataset1_sequence2 datasets, and compares them with representative supervised depth completion methods and zero-shot diffusion baselines. The evaluation metrics include mean absolute error (MAE) and root mean square error (RMSE), both of which indicate better performance as the value is smaller.

[0126] Table 1: Comparison of Dense Depth Completion Errors Across Datasets

[0127]

[0128] As shown in Table 1, the method of this invention achieves optimal results on both test sets. On the Hamlynexploration-sequence-20 dataset, the mean absolute error (MAE) decreased from 1.52 (Marigold-DC method) to 1.18, and the root mean square error (RMSE) decreased from 2.94 to 2.31. On the SCARED dataset1_sequence2 dataset, the MAE decreased from 1.89 to 1.45, and the RMSE decreased from 3.47 to 2.76. This demonstrates that by explicitly introducing the heterogeneous physical graph topology, sparse physical anchors, and heterogeneous node attributes into the backend testing optimization, this invention can effectively improve the scale consistency and local geometric consistency of the zero-sample depth completion results.

[0129] Furthermore, the above results also show that the method of the present invention is not only effective under a single dataset or a single scenario, but can maintain a low error level under different endoscopic data distributions, indicating that the method has good cross-dataset generalization ability and engineering practical value.

[0130] In summary, the present invention demonstrates clear technical effectiveness in dense depth completion tasks, achieving higher accuracy, stronger dimensional consistency, and more stable structural recovery results under sparse depth constraints.

[0131] Example:

[0132] This invention provides a depth completion method based on heterogeneous non-rigid constraints. This method is designed for monocular vision scenes with local non-rigid variations, local dynamic perturbations, or only sparse depth observations. The input includes a color image. , current frame sparse anchor point set Or equivalent sparse depth map, camera intrinsic matrix Heterogeneous physical diagram Node-level relative elastic weights Node-level relative stickiness weights Prior depth map Tracking state statistics and latent variables of diffusion models or other iterative optimization models. The output is a dense depth map corresponding to the current viewpoint. .

[0133] In this embodiment, a single-frame color image can be acquired first. Furthermore, the visible nodes and their topological connections in the current frame are obtained through front-end tracking or sparse reconstruction processes, and a heterogeneous physical graph is constructed. For each node, based on its region... Through mapping functions respectively and Generate node-level relative elastic weights Relative stickiness weights at the node level For each edge Joint elastic weights can be further constructed. and joint stickiness weight This allows for the simultaneous constraint of local structural relationships and regional differences in subsequent optimization.

[0134] When a scene experiences local stretching, occlusion, or local topological failure, the relative length change of the edge between the current and reference times can be used to determine whether the edge still satisfies the assumption of common deformation or common motion; when the relative length change exceeds a threshold... If the edge is not found, it should be removed from the graph to reduce the interference of incorrect topological relationships on subsequent depth estimation.

[0135] After obtaining the graph structure, the nodes that are visible in the current frame, have consistent reprojection, and have stable tracking states can be projected onto the image plane to form a sparse anchor point set for the current frame. And based on the number of effective tracking feature points Number of reference feature points Calculate tracking confidence This confidence level is used to characterize the reliability of the structured prior of the current frame, when... When the value is large, structured prior constraints can be enhanced; when When the value is small, the influence of strong constraints is reduced to preserve more degrees of freedom provided by the image prior.

[0136] Subsequently, a multi-objective joint energy function can be established from the current perspective. The energy function is composed of a geometric consistency term. Viscous smoothing item Flexibility recovery item Scale alignment item and boundary constraint terms Together they constitute. Among them, Used to maintain consistency between the local structure obtained by backprojection of the predicted depth and the adjacency distance relationship between the structure of the front-end heterogeneous physical map; Used to suppress high-frequency jumps in areas with weak texture, reflectivity, and noise; This is used to encourage deep solutions in relatively stable regions to regress to a more reasonable equilibrium state; Used to gradually pull the predicted depth back to the pseudo-metric space defined by the anchor point; This is used to limit the prediction depth to always be within a reasonable range.

[0137] Since the output of a pre-trained monocular depth model is usually a relative depth or an affine invariant depth, this invention further introduces a scale parameter. With translation parameters The relative depth map output by the model Convert to pseudo-metric depth map To avoid numerical compression under conditions of sparse or locally distributed anchor points, a min-max initialization strategy based on the anchor point depth range can be preferentially used to initialize the scale and translation parameters.

[0138] During the iterative optimization phase, latent variables can be adjusted during the DDIM backpropagation process without modifying the diffusion backbone network. Scale parameters Translation parameters Perform joint optimization. When updating latent variables, the joint energy function can be used. Gradient correction is applied to z so that the depth solution gradually satisfies heterogeneous non-rigid constraints, scale alignment constraints, and boundary constraints while preserving the prior knowledge of the basic model.

[0139] To further improve stability in complex scenarios, tracking confidence can be used as a basis. Perform dynamic weight scheduling. When At higher levels, enhance structured prior constraints such as geometric consistency and scale alignment; when At a moderate range, a balance is achieved between structural constraints and prior image degrees of freedom; when When the weights are low, reduce the weights of strong physical constraints to avoid excessive traction of depth recovery by erroneous tracking results.

[0140] After completing all iterative optimizations, the pseudo-metric depth map will be... As the final dense depth output of the current frame For video sequences, the above process can be repeated for each frame, and the output depth can be used for subsequent multi-frame fusion, surface reconstruction, or point cloud post-processing. It should be noted that point cloud denoising, surface smoothing, and other steps are only optional end-effector enhancement modules and do not constitute a necessary limitation of this invention.

[0141] Compared with existing depth completion methods that rely solely on pixel-level sparse depth constraints, this invention maps region-aware heterogeneous physical priors to a heterogeneous physical graph and combines this with joint modeling of scale / translation parameters, test-time optimization, and dynamic weight scheduling mechanisms. This allows it to better adapt to local non-rigid changes and local dynamic disturbances in complex general scenarios, and it has significant advantages in geometric consistency, scale consistency, and overall robustness.

[0142] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A depth completion method based on heterogeneous non-rigid constraints, characterized in that, Input variables include: a color image, the current frame's sparse depth anchor set or an equivalent sparse depth map, the camera intrinsic matrix, and a heterogeneous physical map. The specific method includes the following steps: S1: Construction of region-aware heterogeneous physical graph: Using dynamic deformable graphs or equivalent graph structures as carriers, construct a region-aware heterogeneous physical graph, map the region where the node is located to node-level relative elastic weights and node-level relative sticky weights, and further generate edge-level joint weights, thereby explicitly encoding the regional differences into the subsequent depth completion constraints. S2: Sparse physical anchor point selection and tracking confidence definition: Project the visible, reprojected, and tracking-stable nodes in the current frame onto the image plane to form a sparse anchor point set for the current frame, and construct a tracking confidence index. The specific method involves projecting visible, consistent reprojection, and stable tracking nodes in the current frame onto the image plane to form a sparse anchor point set for the current frame. To reflect the reliability of the current conditions, the tracking confidence level is defined as: in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; Indicates the number of reference feature points; S3: Multi-objective joint energy function construction: Jointly construct geometric consistency term, viscous smoothness term, elastic recovery term, scale alignment term and boundary constraint term, so that depth prediction is simultaneously constrained in terms of local structure, regional smoothness, prior recovery ability and scale consistency; S4: Joint modeling and iterative optimization of scale and translation parameters: The relative depth output by the pre-trained monocular depth model is explicitly converted into a pseudo-metric depth, and the latent variables, scale parameters and translation parameters are jointly updated during the DDIM back-induction process without modifying the diffusion backbone network. The specific method for joint modeling of scale and translation parameters is to explicitly model the scale parameters. With translation parameters Transform it into a pseudo-metric depth: in, This represents the relative depth map output by the pre-trained monocular depth model; Indicates the scale parameter Translation parameters Transformed pseudo-metric depth map; The specific method of iterative optimization is to adjust the latent variable z and scale parameter during the DDIM backpropagation process without modifying the diffusion backbone network. Translation parameters For joint optimization, the latent variable update is written as: in, Let represent the latent variable in the l-th reverse iteration; Represents the updated latent variable; This represents the update step size for the l-th iteration; Represents the joint energy function for latent variables The gradient; S5: Dynamic weight scheduling and result output based on tracking status: The weights of each loss term are dynamically adjusted according to the tracking confidence; when the tracking confidence is high, the geometric consistency term and scale alignment term are enhanced; when the tracking confidence is low, the strength of the structured physical constraint is reduced, and the final output is a dense depth map corresponding to the current viewpoint.

2. The depth completion method based on heterogeneous non-rigid constraints according to claim 1, characterized in that, The specific method of S1 is as follows: the displacement of the graph node at time t is defined as: in, Represents a node At any moment The displacement vector relative to the reference time; Represents a node At any moment The three-dimensional position; Represents a node The three-dimensional position at the reference time or the initial time; To map the region priors to the graph structure, relative elasticity weights and relative stickiness weights are defined for the regions where nodes are located: in, Represents a node The region label or region number to which the region belongs; Represents a node The relative elasticity weights; Represents a node The relative stickiness weight; and These represent the mapping functions from the region to the relative elastic weight and the relative viscous weight, respectively; For the edge The joint elasticity weight and joint stickiness weight are constructed using the attributes of the two endpoints: in, Representing an edge The joint elasticity weights; Representing an edge The joint stickiness weight; and These represent the relative elastic weights of the nodes at both ends of the edge; and These represent the relative stickiness weights of the nodes at both ends of the edge; When a scene experiences local stretching, occlusion, or topological relationship failure, the current length of any edge in the graph is compared with the reference length, and the relative length change rate of that edge is calculated. If the relative length change rate exceeds a preset edge failure determination threshold, the edge is considered to no longer satisfy the common deformation or common motion assumption, and the edge is removed from the current set of valid edges. in, Represents the Euclidean norm; and Representing time respectively The three-dimensional positions of the nodes at both ends of the edge; and These represent the three-dimensional positions of the nodes corresponding to the reference time; This represents the threshold for determining edge failure.

3. The depth completion method based on heterogeneous non-rigid constraints according to claim 1, characterized in that, The specific method of S3 is as follows: Let the depth prediction output by the diffusion model or other generation / optimization module at the current viewpoint be the current predicted depth map, which is in pixels... The depth value at a given location is the predicted depth, and the back-projected 3D point is the predicted 3D point. The joint energy is constructed as follows: in, Represents the joint energy function of the back end; Indicates the geometric consistency term; Indicates the viscous smoothing term; Indicates the elastic recovery item; Indicates the scale alignment item; Represents boundary constraint terms; , , , and These represent the weighting coefficients corresponding to each energy component; Geometric consistency constraints are defined as follows: in, Represents the set of graph edges or the set of valid adjacent edges at time t; Representing an edge Geometric consistency weights; and These represent the three-dimensional points obtained by backprojection of the predicted depth; and These represent the corresponding nodes in the front-end graph structure at time [time]. The three-dimensional position; and These represent the pixel positions of the nodes at both ends of the edge after they are projected onto the image plane; and These represent the spatial distance scale parameter and the pixel distance scale parameter, respectively; Viscous smoothing constraint is defined as: in, Represents the image domain; Represents the pixels obtained by propagation of node-level relative stickiness weights. Local smoothing weights at the location; Indicates the predicted depth in pixels gradient at; The elastic recovery constraint is defined as: in, Represents the pixel obtained by propagation of node-level relative elastic weights. Local recovery weights at the location; Represents pixels Predicted depth at the location; Represents pixels The prior depth value at that location; Scale alignment constraints are defined as follows: in, Represents the set of sparse anchor points in the current frame; Indicates the scale parameter of the current frame; Indicates the translation parameters for the current frame; Indicates anchor point Predicted depth at location; Indicates anchor point The provided pseudo-metric depth; Represents a robust kernel function; Boundary constraints are defined as: in, and These represent the lower bound and the upper bound of depth, respectively. Represents pixels Predicted depth at the location.

4. The depth completion method based on heterogeneous non-rigid constraints according to claim 1, characterized in that, The specific method of S5 is based on the aforementioned tracking confidence level. The confidence score is calculated using the following formula for dynamic weight scheduling: in, Indicates the tracking confidence level of the current frame; This indicates the number of valid tracking feature points or valid anchor points in the current frame; Indicates the number of reference feature points; according to The magnitude of the energy component is used to schedule its weight: when When the value is high, increase the geometric consistency term. and scale alignment items The weights allow the backend to make fuller use of reliable structured priors; when When the range is moderate, a moderately strong structural constraint is adopted to maintain a balance between graph structure traction and graph prior degrees of freedom; when When the depth is low, reduce the weights of strong physical constraints such as geometric consistency, elastic recovery, and scale alignment to avoid excessive pull from erroneous or sparse anchor points on depth recovery; after completing all iterative optimizations, output the pseudo-metric depth as the dense depth of the current frame. in, This indicates the final density depth output for the current frame; This represents a pseudo-metric depth map after scale and translation parameter correction; for video sequences, the above process is repeated for each frame, and the output depth is used for subsequent multi-frame fusion, surface reconstruction, or point cloud post-processing.

Images