Pose estimation method based on diffusion model

Abstract

The invention relates to the field of image pose estimation, in particular to a pose estimation method based on a diffusion model. The method comprises the following steps: acquiring a multi-view image, extracting features from the multi-view image, and finding matching point pairs with the same features in the multi-view image; calculating the position of each point in the three-dimensional space, and obtaining the initial pose of the camera; optimizing the initially estimated three-dimensional point coordinates and the initial pose of the camera; a depth map is directly calculated from a multi-view image, and the depth map is converted into a three-dimensional point cloud through back projection; processing the point cloud data, and further extracting three-dimensional features to obtain a three-dimensional feature map; finding matching pairs of corresponding points in the same object or scene in the three-dimensional point cloud; according to the initial estimation result of the three-dimensional point cloud, the three-dimensional point cloud is further aligned through an ICP algorithm, and the three-dimensional point cloud and the pose are optimized; and carrying out weighted average fusion on the optimized poses of the two parts to obtain a final pose. According to the invention, the shielding problem is solved, and the robustness is improved.

Description

Technical Field

[0001] The present invention relates to the field of image pose estimation, and in particular to a pose estimation method based on a diffusion model. Background Art

[0002] Traditional multi-view pose estimation methods are based on SIFT features, using epipolar geometry constraints and triangulation to calculate pose. However, SIFT features do not perform well in the following situations, as shown in the following aspects:

[0003] ① Fewer textures: flat, monochrome surfaces;

[0004] ② Lighting changes: strong light, shadows or dynamic lighting conditions;

[0005] ③ Large changes in viewing angle: extreme angles or partial loss of image content;

[0006] ④Occlusion problem: Feature points are blocked or disappear

[0007] At the same time, in the frame-by-frame calculation process, the final result is too dependent on the initial estimate and there is a large error. Summary of the Invention

[0008] In order to solve the problem of large errors in multi-view pose estimation, the present invention provides a pose estimation method based on a diffusion model, which mainly includes:

[0009] S1: Acquire multi-view images;

[0010] S2: Use the diffusion model to extract features from multi-view images and obtain the diffusion feature map;

[0011] S3: Using the diffusion feature map, find matching point pairs with the same features in the multi-view images;

[0012] S4: Calculate the position of each matching point in three-dimensional space using epipolar geometry constraints and triangulation, obtain the three-dimensional coordinates of each point, and obtain the initial pose of the camera;

[0013] S5: Optimize the initial estimated 3D point coordinates and the initial camera pose;

[0014] S6: Use the depth estimation model to directly calculate the depth map from the multi-view images and convert the depth map into a 3D point cloud through back projection;

[0015] S7: Processing the point cloud data using a three-dimensional diffusion model to further extract three-dimensional features and obtain a three-dimensional feature map;

[0016] S8: Find matching pairs of corresponding points in the same object or scene in the 3D point cloud of the 3D feature map;

[0017] S9: Based on the initial estimation results of the 3D point cloud, the 3D point cloud is further aligned through the ICP algorithm to optimize the 3D point cloud and pose;

[0018] S10: Perform weighted average fusion of the pose optimized by S5 and the pose optimized by S9 to obtain the final pose.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.

[0021] The technical solution provided by the present invention offers the following beneficial effects: By integrating 2D and 3D diffusion models, multi-view geometric constraints, and deep learning techniques, the present invention achieves high-precision reconstruction of camera poses and 3D point clouds from multi-view images. Starting with multi-view image input, the system undergoes diffusion model feature extraction, feature matching, epipolar geometry and triangulation, bundle adjustment, depth estimation, 3D diffusion processing, 3D matching, and ICP optimization, ultimately outputting an accurate pose through dynamic weighted fusion. The diffusion model extracts diffusion features for pose estimation, fully leveraging the semantic information in the scene to avoid the shortcomings of SIFT features. A depth estimation model is simultaneously used to generate a 3D point cloud for registration, addressing occlusion issues and increasing system robustness. Specifically, the diffusion model extends the 2D diffusion model to 3D point cloud processing, utilizing a denoising process to extract high-semantic features, improving the robustness of feature matching and point cloud reconstruction. Multi-source information fusion combines the image constraints of bundle adjustment, the 3D geometric optimization of ICP, and the dense information of depth estimation to achieve high-precision pose estimation through quaternion weighted fusion. End-to-end design: seamless connection from original image to final pose, covering the entire process of feature extraction, geometric calculation and optimization. In complex scenarios such as lighting changes, occlusion or texture loss, the present invention can still output stable pose and point cloud results, thanks to the feature enhancement and multi-stage optimization of the diffusion model. The dynamic weight mechanism and multi-view consistency constraints make it suitable for a variety of environments, without the need to adjust parameters for specific scenarios, and it has strong adaptability. Through the use of sparsity and deep learning acceleration, it balances computational efficiency and accuracy, is suitable for a wide range of applications such as three-dimensional reconstruction and augmented reality, and is highly efficient and versatile. The present invention provides an efficient, accurate and robust solution for the field of pose estimation, with significant practical value and market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0023] Figure 1This is a flow chart of a method for posture estimation based on a diffusion model in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0025] Example 1

[0026] Please refer to Figure 1 , Figure 1 : is a flowchart of a method for posture estimation based on a diffusion model in an embodiment of the present invention, which specifically includes:

[0027] S1: Acquire multi-view images (multiple images of the same scene or object taken from different angles). Multi-view images provide rich spatial information and support subsequent depth estimation, feature extraction, and position estimation.

[0028] S2: Diffusion Model Processing: A diffusion model (DiffusionModel) is used to extract features from multi-view images, enhance image quality, or identify key visual information. Ultimately, diffusion feature maps with high semantic information (such as keypoints, edge information, or local features) are generated. These feature maps are important inputs for feature matching in the subsequent S3. The diffusion model is a generative model that generates data by gradually adding noise to the data space (diffusion process) and learning to remove the noise (backward diffusion process). As an advanced generative modeling method, the diffusion model is used to extract highly semantic, multi-scale feature maps from multi-view images to support feature matching, 3D reconstruction, and camera pose estimation. By simulating the gradual noise addition (forward diffusion) and denoising (backward diffusion) process of the data, the diffusion model effectively captures key visual information in the image and generates feature representations that are robust to illumination changes, viewpoint differences, and noise.

[0029] The theoretical core of the diffusion model is based on a probabilistic generation process, inspired by Markov chains in nonequilibrium thermodynamics. Specifically, the diffusion model defines a forward diffusion process (noise addition) and a backward diffusion process (noise removal) to generate raw data from pure noise.

[0030] (2.1) The key steps of the diffusion model include:

[0031] (2.1.1) Forward diffusion process

[0032] The forward diffusion process is a process of gradually adding Gaussian noise to the input data until the data distribution approaches an isotropic Gaussian distribution. Let the input image be x0∈R H×W×3, where H and W are the height and width of the image respectively, and 3 represents the number of RGB channels. The forward diffusion generates a series of intermediate states {x1, x2, ..., x T The noise addition at each step follows the conditional probability distribution:

[0033] Gradually transform the data x0 into a completely noisy x T :

[0034]

[0035] Where I is the identity matrix, indicating that the noise is isotropic; represents Gaussian distribution; t is the time step, t=0,...,T, represents the degree of diffusion; β t Represents the parameters of a predefined noise schedule, which is used to control the noise added at each step; q(x t ∣x t-1 ) represents Gaussian noise distribution.

[0036] For ease of calculation, the forward process can jump directly from x0 to the state x at any time step t. t Let α t =1-β t , α t =∏ s＝1 α s ,but:

[0037]

[0038] in, Represents all retention rates α from the initial moment to time step t s The cumulative product of x0 in x t The degree of retention in .

[0039] The ultimate goal is to make x T Approximates a Gaussian noise distribution q().

[0040] (2.1.2) Backward diffusion process

[0041] The goal of the reverse diffusion process is to Gradually recover to x0. Its conditional probability distribution is defined as:

[0042]

[0043] Among them, μ θ (x t ,t) and Σ θ (x t,t) are the mean and covariance of the neural network prediction, θ represents the parameters of the neural network; p θ (x t-1 ∣x t ) represents the reverse diffusion process used to sample x t-1 The directional diffusion process is to obtain the conditional probability distribution to predict the image at the previous moment, and the model here is a neural network used to fit the probability distribution, such as a neural network with a U-Net structure.

[0044] According to Bayes' theorem, the true inverse distribution is:

[0045]

[0046] in: represents the true reverse distribution q(x t-1 |x t ,x0) is obtained by the forward noise scheduling parameter β t and cumulative retention rate Calculated.

[0047]

[0048] (2.2) Feature extraction mechanism

[0049] The diffusion model is not only used for image generation, but also extracts feature maps through intermediate representations during back-diffusion, which are used to support cross-view feature matching in pose estimation.

[0050] (2.2.1) U-Net structure and feature extraction

[0051] U-Net is a symmetric encoder-decoder network consisting of:

[0052] Encoder: progressively downsamples through convolution and pooling operations to extract high-level features.

[0053] Decoder: Restore spatial resolution through upsampling and skip connections.

[0054] Skip connections: pass high-level features extracted by the encoder to the decoder, preserving low-level details.

[0055] In U-Net, feature extraction mainly relies on the encoder. Assume that the resolution of the input image x0 is H×W. The encoder consists of L downsampling layers, each of which reduces the resolution by 2×2 maximum pooling while increasing the number of channels.

[0056] (2.2.2) Characteristics of multi-scale features

[0057] Extract feature maps {f1,f2,...,f L}:

[0058] Low-level features (small l): high spatial resolution, capturing local details such as edges and textures.

[0059] High-level features (large l): have low spatial resolution and capture global features such as object outlines and semantic information. These feature maps are generated through the denoising process of the diffusion model and have the following advantages:

[0060] Semantic richness: The denoising process enhances the representation ability of features for key visual elements.

[0061] Robustness: More adaptable to noise, lighting, and viewpoint changes.

[0062] (2.2.3) Feature map generation process

[0063] In the inference phase, given an input image x0:

[0064] 1. Initialization

[0065] 2. Through T-step reverse diffusion, gradually generate x T-1 ,...,x0.

[0066] 3. At each step t, the U-Net encoder extracts the feature map F t =Encoder(x t ,t).

[0067] The final feature map F can be taken from a specific time step (such as t=0) or a weighted combination of features from multiple time steps.

[0068] S3: Feature Matching: Using the diffuse feature map, an algorithm is used to find matching point pairs with identical features in the multi-view images. The algorithm used is SIFT, ORB, or a deep learning-based matching method. The quality of the matching point pairs directly affects subsequent geometric calculations.

[0069] Feature matching is a key step connecting feature extraction from the diffusion model with subsequent geometric computations. Its goal is to use the diffusion feature map generated in step S2 to establish precise correspondences between multi-view images, generating high-quality matching point pairs and providing reliable input for epipolar geometry and triangulation. This section leverages feature description, matching algorithm design, and post-processing optimization to fully exploit the semantic information and spatial consistency of the diffusion feature map, improving matching accuracy and robustness.

[0070] (3.1) The input of feature matching is the diffusion feature map generated in step S2, denoted as F = {F1, F2, ..., F n}, where F i∈R H′×W′×C is the feature map of the i-th view, H′ and W′ are the spatial resolution of the feature map (such as H / 2×W / 2), and C is the number of feature channels (such as 128 or 256). These feature maps have the following properties:

[0071] High semantics: Contains edges, key points, and global context information.

[0072] Multi-scale: Fusion of low-level details and high-level semantics.

[0073] Robustness: It has strong adaptability to changes in lighting, perspective, and noise.

[0074] In addition, the input also includes the original pixel data of the multi-view image I={I1,I2,...,I n}, used to assist in feature positioning and verification.

[0075] (3.2) Feature description and positioning

[0076] The goal of feature point detection is to i Identify significant and repeatable key points (such as corner points and edge points) in the image. The following methods are used for identification:

[0077] Gradient-based detection: Calculate the feature map F i Gradient magnitude at each channel Pixels whose gradient magnitude exceeds the threshold τ are selected as candidate feature points.

[0078] Non-maximum suppression: In a 3×3 or 5×5 neighborhood, the local gradient maximum point is retained and non-significant points are suppressed to ensure the sparsity and distinguishability of feature points.

[0079] Multi-scale expansion: If F i Including multi-scale representation, feature points are detected at different resolution layers and mapped back to the original image coordinates through upsampling.

[0080] After detection, each view generates a set of feature points where p ij =(x ij ,y ij ), represents the two-dimensional coordinates of the j-th feature point of the i-th perspective, m i is the number of feature points.

[0081] For each feature point p ij , from F i Extract the corresponding feature vector as the descriptor. The generation process of the feature descriptor is as follows:

[0082] Dimensionality reduction and normalization: F ij Flattened to a vector The dimension is reduced to a fixed dimension D through principal component analysis (PCA) or a fully connected layer. In this embodiment, D=128.

[0083] Local feature sampling: p ij As the center, in F i Upsample a k×k local area (such as k=5) to obtain the sub-feature map F ij ∈R k×k×C , and perform L2 normalization:

[0084]

[0085] Finally, each feature point p ij Associate a descriptor d ij , forming a feature pair set {(p ij ,d ij )}.

[0086] (3.3) Feature matching algorithm

[0087] The goal of feature matching is to find the feature point set P at different viewpoints. i and P j Find the corresponding relationship between them, that is, get the matching point pair {(p ik ,p jl )}.

[0088] (3.3.1) Preliminary matching based on distance

[0089] The nearest neighbor search method is used to calculate the similarity between descriptors:

[0090] Distance metric: Use Euclidean distance:

[0091] dist(d ik ,d jl )=||d ik -d jl ||2

[0092] Nearest neighbor to next nearest neighbor ratio test: For P i The feature points in ik Find P j The closest descriptor d jl and the next closest descriptor. If: Wherein, ρ is the threshold value. In this embodiment, ρ is set to 0.8, and (p ik ,p jl ) as the initial matching pair.

[0093] Preliminary matching generates a candidate matching set N ij Indicates the number of feature points in the i-th view.

[0094] (3.3.2) Matching Optimization Based on Deep Learning

[0095] To further improve matching accuracy, we combine deep learning-based matching algorithms (such as SuperGlue). This method uses graph neural networks (GNNs) and attention mechanisms to optimize matching:

[0096] Input: Two sets of feature points P i and P j Coordinates and descriptors.

[0097] Self-attention and cross-attention:

[0098] The self-attention layer updates the descriptor of each feature point to enhance its contextual information.

[0099] Cross-Attention Layer Modeling P i and P j The corresponding relationship between them generates a matching probability matrix

[0100] Matching assignment: Normalize S using the Sinkhorn algorithm and output the optimal matching pair.

[0101] (3.4) Matching quality assessment and post-processing

[0102] The quality of matching point pairs directly affects the accuracy of subsequent geometric calculations, so strict screening and optimization are required.

[0103] (3.4.1) Geometric consistency verification

[0104] Verify matching pairs using epipolar geometry constraints:

[0105] Fundamental matrix estimation: From the preliminary matching set M ij Randomly sample 8 point pairs (or more) from , and use the 8-point algorithm to estimate the fundamental matrix F.

[0106] Reprojection error: For each pair of matches (p ik ,p jl )Calculate the distance from a point to the epipolar line:

[0107] e jl =dist(p jl ,Fp ik )

[0108] If e ik +e jl <∈ (e.g., ∈=1 pixel), the matching pair is retained.

[0109] RANSAC optimization: Iteratively optimize F through random sampling consensus (RANSAC), remove outliers, and retain the inlier matching set M ij ′.

[0110] (3.4.2) Symmetry check

[0111] To further reduce false matches, perform two-way matching verification:

[0112] From P i to P j Match p ik →p jl

[0113] From P j to P i Match verification to see if p is obtained jl →p ik

[0114] Only bidirectionally consistent matching pairs are retained.

[0115] (3.4.3) Quantity and distribution optimization

[0116] Quantity control: If the number of matching pairs is insufficient (such as |M ij ′|<Nmin), adjust the detection threshold τ or distance threshold ρ, and regenerate the match.

[0117] Uniform distribution: Through spatial segmentation (such as grid division), ensure that the matching points are evenly distributed in the image to avoid local aggregation.

[0118] (3.5) Output and subsequent steps

[0119] After the above processing, the optimized matching point pair set M′={M ij ′}, where M ij ′={(p ik ,p jl )} represents the matching pair between the i-th and j-th views. Output characteristics:

[0120] High precision: The matching point pairs meet geometric consistency and the error is less than the set threshold.

[0121] High density: The number of matches is sufficient to support subsequent 3D reconstruction.

[0122] These matched point pairs are directly input into the “Epipolar Geometry and Triangulation” part of S4 to calculate the 3D point coordinates and initial camera pose.

[0123] S4: Epipolar Geometry and Triangulation: This is a key step in calculating the 3D coordinates and initial camera pose from the matching point pairs generated in step S3, leveraging the geometric relationships between multi-view images. The matching point pairs are input into the multi-view geometry framework, and the epipolar geometry constraints and triangulation are used to calculate the position of each matching point in 3D space. This results in the 3D coordinates (Position) of each matching point and the initial camera pose. This section establishes the spatial relationship between viewpoints through epipolar geometry constraints and, combined with triangulation principles, reconstructs the 3D structure of the scene, providing the foundation for subsequent bundle adjustment and depth estimation.

[0124] This section will elaborate on four aspects: input preparation, epipolar geometry calculation, triangulation implementation, and result optimization:

[0125] (4.1) Input data

[0126] The input of this step is the optimized matching point pair set M′={M ij '},in represents the matching pair between the i-th and j-th views, p ik =(x ik ,y ik ) and p jl =(x jl ,y jl ) are the two-dimensional image coordinates. In addition, the input also includes:

[0127] Camera intrinsic parameters: intrinsic parameter matrix K for each view i and K j (usually obtained through calibration) in the form of:

[0128]

[0129] where f x ,f y is the focal length, c x ,c y is the optical center coordinate.

[0130] Original image: used to verify the position consistency of matching points.

[0131] (4.2) Epipolar geometry calculation

[0132] Epipolar geometry describes the projective relationships between multi-view images and is a prerequisite for triangulation. This section uses the fundamental matrix F or the essential matrix E, along with the camera's intrinsic and extrinsic parameters and the relationship between the viewpoints, to establish geometric constraints between viewpoints.

[0133] (4.2.1) Fundamental matrix estimation

[0134] Fundamental matrix F ijThe epipolar constraint between the i-th and j-th viewpoints is defined. ik ,p jl ),satisfy

[0135]

[0136] in and are homogeneous coordinates.

[0137] 8-point algorithm:

[0138] From M ij Randomly sample at least 8 matching pairs from ′ and construct a linear system of equations:

[0139] Af=0

[0140] Where A is N ij ×9 matrix, each row is of the form [x jl x ik ,x jl y ik ,x jl ,y jl x ik ,y jl y ik ,y jl ,x ik ,y ik ,1], where f is the 9-element flattened vector of F.

[0141] The null space of A is solved by singular value decomposition (SVD) to obtain the initial F′.

[0142] Enforce rank constraint: perform SVD decomposition on F′, Set the minimum singular value of Σ to 0 and reconstruct

[0143] RANSAC optimization:

[0144] Iteratively sample matching pairs and calculate F ij ;

[0145] For each (p ik ,p jl ), calculate the epipolar error:

[0146]

[0147] If e ik +e jl <∈ (such as ∈=1) is considered an interior point.

[0148] Choose F with the largest number of inliers ij As a final estimate.

[0149] (4.2.2) Essential Matrix Calculation

[0150] If the camera internal parameters are known, the essential matrix E can be further calculated ij :

[0151]

[0152] E ij The relative rotation R and translation t between views (modulo the scale factor) are encoded, satisfying:

[0153] E ij =[t]×R

[0154] Where [t]× is the antisymmetric matrix of the translation vector.

[0155] Decompose E ij :

[0156] To E ij Perform SVD decomposition:

[0157] Construct four possible combinations of rotation matrix R and translation vector t:

[0158]

[0159] in u3 is the third column of U

[0160] Verify through triangulation that the correct solution is chosen with the point in front of both cameras.

[0161] (4.2.3) Initial pose estimation

[0162] From E ij Decomposed R ij and modular scale t ij As the initial relative pose between the i-th and j-th perspectives. Let the pose of the i-th perspective be [I|0], then the pose of the j-th perspective is [R ij ∣t ij ].

[0163] (4.3) Triangulation Implementation

[0164] Triangulation: Based on geometric constraints, the three-dimensional coordinates of each point are calculated by combining the projection positions of point pairs in different perspectives and using matching point pairs and camera poses.

[0165] (4.3.1) Projection model

[0166] Let the matching point pair (p ik ,p jl ) corresponds to the three-dimensional point (homogeneous coordinates). Its projection at the i-th and j-th viewing angles is:

[0167]

[0168] in λ ik and λ jl is the depth factor.

[0169] (4.3.2) Linear triangulation

[0170] Rewrite the projection equation into linear form:

[0171] For perspective i:

[0172] x ik Z k =f x X k +c x Z k ,y ik Z k =f y Y k +c y Z k

[0173] For perspective j:

[0174]

[0175] in

[0176] Construct a system of linear equations:

[0177] A k P k ′=b k

[0178] Among them A k is a 4×3 matrix based on p ik 、p jl 、R ij and t ij Construct. Obtain the least squares solution via SVD.

[0179] The linear method is greatly affected by noise and can be further minimized through nonlinear optimization:

[0180] Objective function:

[0181]

[0182] Iteratively optimize P using the Levenberg-Marquardt algorithm k

[0183] (4.4) Output and Optimization

[0184] (4.4.1) Output results

[0185] 3D point coordinates (Position): Each matching point pair (p ik ,p jl ) corresponds to a three-dimensional point Form a point cloud set P = {P k}.

[0186] Initial camera pose: a set of poses from multiple perspectives (Take a certain perspective as the reference coordinate system).

[0187] (4.4.2) Result verification and optimization

[0188] Reprojection error check:

[0189]

[0190] If err k >δ (e.g. 2 pixels), remove outliers.

[0191] Scale consistency: Due to t ij The module scale is uncertain and subsequent bundle adjustment will further optimize it.

[0192] Output 3D point cloud and the initial pose {[R i ∣t i ]} Directly input the "bundle adjustment" of step S5 to perform global optimization.

[0193] S5: Bundle Adjustment: Bundle adjustment is an optimization step in pose estimation. It aims to improve overall accuracy by jointly adjusting the initial 3D point coordinates and the initial camera pose through global nonlinear optimization. This step uses the epipolar geometry and triangulation results from step S4 to minimize the reprojection error of the 3D points, ensuring that the positions of the 3D points projected back into the multi-view images are consistent with the actual observed positions, thereby optimizing the geometric consistency of the 3D reconstruction.

[0194] (5.1) Input data

[0195] Initial 3D point coordinates: point cloud set output by step S4 in

[0196] Initial camera pose: multi-view pose set

[0197] Matching point pair: M′ obtained in step S3 = {M′ ij} is used to define observation constraints.

[0198] Camera internal parameters: {K i} is kept fixed or used as an optimization variable.

[0199] (5.2) Optimization goal

[0200] The goal of bundle adjustment is to minimize the reprojection error of all 3D points on multi-view images. The optimization function is:

[0201]

[0202] Where: vis(i) represents the index of the 3D point visible from the i-th viewing angle.

[0203] proj(·) is the projection function, which is used to map a 3D point to the image plane.

[0204] K i Indicates the camera internal parameters, [R i |t i ] represents the multi-view pose, P k Represents a 3D point cloud, p ik represents the 3D point of the i-th view, and n represents the number of viewpoints.

[0205] (5.3) Optimization method

[0206] Nonlinear optimization: Using the Levenberg-Marquardt algorithm, P is adjusted by iteration. k and [Ri|ti] converge to a local optimal solution.

[0207] Parameterization: The rotation R is represented using Lie algebra or quaternion to avoid singularities.

[0208] Sparsity utilization: Utilize the sparse structure of the observation matrix to accelerate calculations.

[0209] (5.4) Output results

[0210] The optimized three-dimensional point coordinates: P′={Pk′} have higher accuracy.

[0211] Optimized camera pose: The scale is consistent and the error is minimal.

[0212] S6: Use the depth estimation model to directly predict the depth map from the multi-view image to provide supplementary information for the subsequent 3D point cloud generation and pose optimization. The depth map provides distance information for each pixel, forming depth information on the two-dimensional view. The depth map is converted into a 3D point cloud through back projection. Unlike traditional triangulation methods, this module uses deep learning technology to utilize the semantic and geometric clues of the image to generate pixel-level depth estimation results. The depth map is converted into a 3D point cloud through back projection and fused with the results of steps S4 and S5 to further improve the system accuracy. Specifically:

[0213] (6.1) Input data

[0214] The input of the depth estimation model is a multi-view image set I = {I1, I2, ..., I n},in Represents the RGB image of the i-th view. In addition, the input also includes:

[0215] Camera internal parameters: {K i}Used for back-projection of depth map to 3D point cloud.

[0216] Initial pose: the output of the bundle adjustment in step S5 Can be used as auxiliary input to enhance multi-view consistency.

[0217] Matching point pair: M′ in step S3 = {M′ ij} for supervised training or validation.

[0218] (6.2) Model architecture design

[0219] The depth estimation model uses a deep learning-based encoder-decoder architecture, combined with multi-view consistency constraints, to generate high-precision depth maps. The following is the detailed design of the architecture:

[0220] (6.2.1) Encoder

[0221] The encoder is responsible for extracting multi-scale features from the input image:

[0222] Backbone network: Use pre-trained convolutional neural network (such as ResNet-50 or EfficientNet), input I i , output multi-layer feature map {F i (1) ,F i (2) ,...,F i (L)}

[0223] F i (l) ∈R H / 2l-1×W / 2l-1×Cl , Cl is the number of channels, with the number of layers ll Increase.

[0224] Multi-view fusion: If the initial pose is input, the perspective transformation Projection to The coordinate system is fused into

[0225] (6.2.2) Decoder

[0226] The decoder upsamples the encoder features to the original resolution and generates a depth map:

[0227] Upsampling and skip connection: From deep features F i (L) At the beginning, the corresponding feature F of each layer fusion encoder is gradually upsampled by bilinear interpolation. i (l) (via skip connections).

[0228] Deep regression head: The last layer reduces the feature channel to 1 through 1×1 convolution and outputs the depth prediction

[0229] Activation function: Use Sigmoid or Softplus activation to ensure that the depth value is non-negative:

[0230] D i (xy)=Softplus(D i ′(xy))=ln(1+eD i ′(xy))

[0231] Among them D i (x, y) represents the depth value of pixel (x, y) in the i-th perspective image, D i ′(x,y) represents the depth prediction value of pixel (x,y) in the i-th view image.

[0232] (6.2.3) Multi-view consistency module

[0233] To utilize multi-view information, consistency constraints are added:

[0234] Cost volume construction: Based on the initial pose, construct the depth hypothesis plane and calculate I i and I j The matching cost between D is the number of depth hypotheses.

[0235] Cost aggregation: Aggregate C through 3D convolution or attention mechanism i (d),Enhancing geometric consistency of depth prediction.

[0236] Depth optimization: regress the final depth map D from the aggregated cost volumei .

[0237] (6.3) Depth map generation

[0238] The output of the depth estimation model is a depth map D = {D1, D2, ..., D n}, where D i (x,y) represents the depth value of pixel (x,y) in the i-th perspective image. The generation process is as follows:

[0239] Single-view prediction: the encoder-decoder network directly outputs D i .

[0240] Multi-view fusion: If the consistency module is enabled, multi-view predictions are fused through weighted averaging or median filtering:

[0241] D i (x,y)=median({D i (x,y),Warp(D j (x′,y′),R ij ,t ij )})

[0242] Post-processing: Apply smoothing filtering (such as bilateral filtering) to remove noise and preserve edge details.

[0243] Depth map features:

[0244] Resolution: Same as the input image (H×W)

[0245] Range: The depth value range is determined by the training data (e.g. 0 to 100 meters).

[0246] (6.4) Back-projection to generate 3D point cloud

[0247] The depth map is converted into a 3D point cloud by back-projection:

[0248] Pixel to camera coordinates: For D i (x,y) uses the internal reference K i Calculate 3D coordinates:

[0249]

[0250] in is a point in the camera coordinate system.

[0251] Camera to world coordinates: using pose [R′ i ∣t′ i ]

[0252] P i,xy =R i ′P i,xy ′+ti '

[0253] Point cloud collection: traverse all pixels and generate point clouds

[0254] Multi-view point clouds can be fused into a global point cloud by aligning the coordinate system

[0255] (6.5) Model training and optimization

[0256] (6.5.1) Loss function

[0257] The training goal is to minimize the difference between the predicted depth and the true depth. The loss function includes:

[0258] Depth Loss:

[0259]

[0260] Use L1 loss to ensure robustness, is the true depth, and N is the normalization factor used to average the loss value of pixels so that the losses at different resolutions are comparable.

[0261] Consistency loss:

[0262]

[0263] Total loss:

[0264] L=L depth +λL consist .

[0265] Here, λ is a weight. In this embodiment, λ is 0.5.

[0266] (6.5.2) Training data

[0267] Dataset: Use multi-view depth datasets (such as DTU, Tanks and Temples), which contain RGB images, depth maps, and camera parameters.

[0268] Data enhancement: random cropping, rotation, and brightness adjustment to improve model generalization capabilities.

[0269] (6.5.3) Optimization method

[0270] Optimizer: Adam optimizer, initial learning rate is 10 -4 Decays with epoch.

[0271] Training strategy: batch size 16, training for 50-100 epochs until convergence.

[0272] (6.6) Output and subsequent connection

[0273] Output depth map: Depth map D = {D i} Provides distance information for each pixel, forming depth information on a two-dimensional view.

[0274] Output point cloud: Generated by back-projection. The depth map is converted into a 3D point cloud by back-projection.

[0275] The depth point cloud is input into the "3D diffusion model processing" of step S7 for further denoising or feature extraction.

[0276] S7: 3D Diffusion Model Processing: The 3D diffusion model is a natural extension of the 2D diffusion model in step S2 and is used to process the 3D point cloud generated in step S6. This module uses the 3D diffusion process to denoise and complete the sparse point cloud, further extracting rich 3D features and rich 3D feature maps, such as normal vectors and curvature, to support subsequent matching and pose optimization. Specifically:

[0277] (7.1) Input data

[0278] Input point cloud: generated by back projection in step S6 Each point

[0279] Auxiliary information: Point cloud after optimization in step 5 Or camera pose {[R′ i ∣t′ i ]}.

[0280] (7.2) Processing flow

[0281] The 3D diffusion model follows the theoretical framework of step S2 and extends the denoising process of 2D images to 3D point clouds:

[0282] Forward diffusion: point cloud Gradually add Gaussian noise to generate a noise sequence

[0283] in:

[0284] q(P(t)|P(t-1))=N(P(t);1-βtP(t-1),βtI)

[0285] P(t) represents the point cloud at time step t after adding noise, and P(t-1) represents the point cloud at time step t-1 after adding noise, which is estimated from P(t) using a neural network. βt represents the noise intensity (variance schedule) added at time step t. 1-βtP(t-1) represents the mean term of the Gaussian distribution, which does not completely add noise but retains some original information. As βt increases, the original information retained decreases, and the noise is stronger. βtI represents the covariance matrix, which means that the intensity of the added noise is βt and each dimension is independent (the identity matrix).

[0286] Back diffusion: Train a 3D neural network (such as PointNet or DGCNN) from Stepwise denoising, restoration

[0287] pθ(P(t-1)∣P(t))=N(P(t-1);μθ(P(t),t),Σθ)

[0288] pθ represents the conditional distribution of the reverse process modeling, that is, the distribution learned by the neural network from the prediction; θ represents the parameters of the neural network, which is used to fit the reverse diffusion distribution; μθ(P(t), t) represents the mean function of the neural network output, which is the mean (expectation) predicted by the given input P(t) and the current time step t, P(t-1); Σθ represents the predicted covariance matrix, which controls the randomness during sampling.

[0289] Feature extraction: During the denoising process, the middle layer of the network outputs three-dimensional feature maps, such as normal vectors, curvature, etc.

[0290] (7.3) Output results

[0291] Denoised point cloud: Optimized Remove noise and complete sparse areas.

[0292] 3D feature map: F 3D ={f k}, each point P k , the associated eigenvector f k (e.g. dimension is 64).

[0293] S8: 3D feature matching: using the denoised 3D point cloud generated in step S7 and 3D feature map F 3D ,establishing correspondences between the multi-view point clouds, and finding matching pairs of corresponding points in the same object or scene in the 3D point cloud, further improving the point cloud density and matching accuracy.,This step is a natural extension of the 2D feature matching in step S3, and is,applicable to 3D space.

[0294] (8.1) Input data

[0295] 3D point cloud: Each point is

[0296] 3D feature map: F 3D ={f k}, each point P k Correlation eigenvector f k

[0297] (8.2) Matching process

[0298] Feature description: Directly use f k As a descriptor of 3D points, no additional generation is required.

[0299] Matching algorithm:

[0300] Compute the Euclidean distance between pairs of points:

[0301] dist(f ik ,f jl )=||f ik -f jl ||2

[0302] The nearest neighbor and next nearest neighbor ratio test (as in step 3) is used, and the threshold ρ (taken as 0.8 in this embodiment) is used to generate the preliminary matching pair M 3D ={(P ik ,P jl )}.

[0303] Geometric verification: Utilize the rigid constraints of three-dimensional space to eliminate matching pairs that do not meet transformation consistency.

[0304] (8.3) Output results

[0305] Three-dimensional matching pair: optimized M′ 3D Represents the correspondence between point clouds.

[0306] Dense point cloud: Enhanced by matching pair completion density.

[0307] S9: Based on the initial estimation results of the 3D point cloud, the 3D matching pairs output from step S8 are further aligned with the dense point cloud through the Iterative Closest Point (ICP) algorithm to optimize the 3D point cloud and pose. The accuracy of the 3D structure and pose is improved by iteratively optimizing the rigid transformation between point clouds. This step refines the above matching results and provides high-quality input for the final fusion. The optimization goal is to further adjust the position and pose of the 3D point cloud by minimizing the error between the closest points in the point cloud. Specifically:

[0308] (9.1) Input data

[0309] Initial point cloud: after optimization in step S8 and matching pair M′ 3D ={(P ik ,P jl )}

[0310] Initial transformation: the camera pose {[R′ i ∣t′ i ]} as an initial estimate.

[0311] (9.2) Processing flow

[0312] The ICP algorithm iteratively aligns point clouds through the following steps:

[0313] Closest point search: For source point cloud Every point P in ik , in the target point cloud Find the closest point P jl :

[0314] P ik represents the kth point of the i-th sample in the source point cloud, P jl Represents Pi k In the target point cloud The nearest neighbor point found in is the lth point of the jth sample; P represents the target point cloud A candidate point in , that is, the point that participates in minimizing the distance.

[0315] Transformation estimation: Based on corresponding point pairs, calculate the rigid transformation [R,t]

[0316]

[0317] Solve using SVD decomposition.

[0318] Point Cloud Update: Apply Transformation Update

[0319]

[0320] Iteration: Repeat the above steps until the error converges or the maximum number of iterations is reached.

[0321] (9.3) Output results

[0322] Accurate point cloud: adjusted The alignment error between points is minimal.

[0323] Optimized pose: updated transformation parameters [R′,t′].

[0324] S10: Through the weighted fusion method, the output poses of the two parts are fused, and the multi-source pose estimation results are integrated to generate a high-precision final camera pose. Finally, all the estimation results are weighted averaged to obtain a high-precision final pose. This method combines the output of the bundle adjustment (BA) algorithm in step S5 and the ICP (Iterative Closest Point) algorithm in step S9, and uses dynamic weighting of confidence to significantly improve the accuracy and robustness of pose estimation. It is particularly suitable for complex scenes such as lighting changes and occlusions. The following is the detailed design of this part. Specifically:

[0325] (10.1) Input data

[0326] The fusion module receives the following inputs:

[0327] Bundle adjustment pose: The pose estimate obtained in step S5 is represented as {[R′ i,BA ∣t′ i,BA ]}, where R′ i,BA is the rotation matrix, t′ i,BA is the translation vector.

[0328] ICP optimized pose: The pose estimate obtained in step S9 is expressed as {[R′ i,ICP ∣t′ i,ICP ]}.

[0329] Confidence metric: optional input, such as the reprojection error err for bundle adjustment BA and ICP point cloud alignment error err ICP , used for subsequent weight calculation.

[0330] (10.2) Fusion Strategy

[0331] An innovative weighted average method is used to fuse bundle adjustment and ICP pose estimation, which is divided into two parts: rotation fusion and translation fusion:

[0332] Rotation Fusion:

[0333] Rotation matrix R′ i,BA and R′ i,ICP Convert to quaternion representation q i,BA and q i,ICP , to avoid nonlinear problems in the rotation matrix space.

[0334] Compute the weighted average quaternion:

[0335]

[0336] where w BA and w ICP is the weight, normalized to ensure the result is a unit quaternion; qi,final Represents a weighted average quaternion.

[0337] q i,final Convert back to the rotation matrix R i,final .

[0338] Translational Fusion:

[0339] Directly perform weighted averaging on the translation vectors:

[0340] t i,final =w BA t i,BA′ +w ICP t i,ICP′

[0341] Among them, t i,final Represents the final translation vector, t i,BA′ Represents the translation vector after BA, t i,ICP′ Represents the translation vector after ICP.

[0342] The weights and rotations are fused consistently to ensure the overall pose coordination.

[0343] Weight calculation

[0344] The weights are dynamically generated based on the error metrics of each method:

[0345]

[0346] The smaller the error, the greater the weight, so that the more accurate estimation result dominates the fusion. If there is no explicit error metric, the weight can be preset according to the scene characteristics (for example, bundle adjustment has a higher weight in texture-rich scenes, and ICP has a higher weight when the geometric structure is clear).

[0347] (10.3) Output results

[0348] The final output after fusion is:

[0349] Final camera pose: Represents a series of high-precision and robust pose estimation results.

[0350] Example 2

[0351] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0352] Example 3

[0353] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.

[0354] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A posture estimation method based on a diffusion model, characterized in that: include: S1: Acquire multi-view images; S2: Use the diffusion model to extract features from multi-view images and obtain the diffusion feature map; S3: Using the diffusion feature map, find matching point pairs with the same features in the multi-view images; S4: Calculate the position of each matching point in three-dimensional space using epipolar geometry constraints and triangulation, obtain the three-dimensional coordinates of each point, and obtain the initial pose of the camera; S5: performing bundle adjustment optimization on the initially estimated three-dimensional point coordinates and the initial camera pose to obtain optimized three-dimensional point coordinates and camera pose; S6: Based on the optimized 3D point coordinates and camera pose, a depth estimation model is used to directly calculate the depth map from the multi-view images, and the depth map is converted into a 3D point cloud through back projection; S7: Process the depth point cloud data using a 3D diffusion model, remove noise and complete the sparse point cloud through a 3D diffusion process, further extract 3D features, and obtain a 3D feature map; S8: Using 3D point clouds and 3D feature maps, we establish correspondences between multi-view point clouds to obtain 3D matching pairs and dense point clouds. S9: Based on the initial estimation results of the 3D point cloud, the 3D point cloud is further aligned through the ICP algorithm, and the 3D point cloud and pose are optimized to obtain the accurate point cloud and optimized pose; S10: Perform weighted average fusion of the pose optimized by S5 and the pose optimized by S9 to obtain the final camera pose.

2. A method for posture estimation based on a diffusion model according to claim 1, characterized in that: In S2, the diffusion feature map includes key points, edge information or local features in the image.

3. A method for posture estimation based on a diffusion model according to claim 1, characterized in that: In S2, the process of using the diffusion model is as follows: (1) Forward diffusion process Gradually transform the initial data x0 into a completely noisy x T : Where I is the identity matrix, indicating that the noise is isotropic; represents Gaussian distribution; t is the time step, t=0,...,T, represents the degree of diffusion; β t is the noise parameter associated with time step t, used to control the noise added at each step; q(x t ∣x t-1 ) represents Gaussian noise distribution, x t represents the data at time step t, x t-1 Represents the data at time step t-1; (2) Reverse diffusion process The reverse diffusion process is used to T Restore to original data x0: Among them, μ θ (x t ,t) and Σ θ (x t ,t) are the mean and covariance of the neural network prediction, θ represents the parameters of the neural network; p θ (x t-1 ∣x t ) represents the reverse diffusion process used to sample x t-1 Gaussian distribution.

4. A method for posture estimation based on a diffusion model according to claim 1, characterized in that: S3 specifically: S3.1: According to the diffusion feature map F = {F1, F2, ..., F n }, perform feature point detection to obtain a set of feature points generated for each perspective Among them, F i ∈R H′×W′×C is the feature map of the i-th view, i = 1, 2, ..., n, H′ and W′ are the spatial resolutions of the feature map, and C is the number of feature channels; p ij =(x ij ,y ij ), represents the two-dimensional coordinates of the j-th feature point of the i-th perspective, m i is the number of feature points; S3.2: For each feature point p ij , from F i Extract the corresponding feature vector as the descriptor, and each feature point p ij Associate a descriptor d ij , forming a feature pair set {(p ij ,d ij )}; S3.3: Feature point set P at different viewpoints i and P j Find the corresponding relationship between them, that is, get the matching point pair {(p ik ,p jl )}, through the preliminary matching based on distance and the matching optimization based on deep learning, the optimized matching point pair set M′={M ij ′}, where M ij ′={(p ik ,p jl )} represents the matching pair between the i-th and j-th views.

5. The method for posture estimation based on a diffusion model according to claim 1, wherein: S4 is specifically: Use the camera's internal and external parameters and the relationship between viewing angles to define geometric constraints between viewing angles; According to geometric constraints, combined with the projection positions of point pairs in different perspectives, the three-dimensional coordinates of each point are calculated by matching point pairs and camera poses.

6. A method for posture estimation based on a diffusion model according to claim 1, characterized in that: In S5, the goal of bundle adjustment optimization is to minimize the reprojection error of all 3D points on multi-view images. The optimization function is: Where: vis(i) represents the index of the 3D point visible at the i-th viewing angle, proj(·) is the projection function used to map the 3D point to the image plane, K i Indicates the camera internal parameters, [R i |t i ] represents the multi-view pose, P k Represents a 3D point cloud, p ik represents the 3D point of the i-th view, and n represents the number of viewpoints.

7. A method for posture estimation based on a diffusion model according to claim 1, characterized in that: In S6, the depth estimation model adopts an encoding-decoding architecture based on deep learning, including an encoder, a backbone network, and a decoder. The encoder is responsible for extracting multi-scale features from the input image. The backbone network uses a pre-trained convolutional neural network to output multi-layer feature maps. The decoder is used to upsample the encoder features to the original resolution to generate a depth map for each perspective.

8. The method for posture estimation based on a diffusion model according to claim 1, wherein: In S10, a weighted average method is used to fuse the pose estimation after bundle adjustment optimization with the ICP algorithm, which is divided into two parts: rotation fusion and translation fusion: Rotation Fusion: The rotation matrix R′ i,BA and R′ i,ICP Convert to quaternion representation q i,BA and q i,ICP , to avoid nonlinear problems in the rotation matrix space; Compute the weighted average quaternion: Among them, w BA and w ICP is the weight, q i,final represents the weighted average quaternion; Translational Fusion: Directly perform weighted averaging on the translation vectors: t i,final =w BA t i,BA′ +w ICP t i,ICP′ Among them, t i,final Represents the final translation vector, t i,BA′ Represents the translation vector after BA, t i,ICP′ Represents the translation vector after ICP.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the pose estimation method based on the diffusion model according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the program is executed by a processor, the steps of the posture estimation method based on the diffusion model as described in claims 1 to 7 are implemented.

Images