Camera pose and 3D Gaussian point cloud joint optimization method and system

Abstract

The invention provides a camera pose and 3D Gaussian point cloud joint optimization method and system, and belongs to the technical field of computer vision, and the method comprises the steps: obtaining a to-be-processed image; processing the obtained image by using a pre-trained optimization model to obtain an optimized image; the optimization model comprises a feature extraction module, a matching module, a generation module and an optimization module; the feature extraction module is used for feature extraction; the matching module is used for carrying out feature point matching; the generation module is used for screening inner points through RANSAC and calculating a basic matrix to generate an initial point cloud; and the optimization module is used for performing nonlinear optimization on the initial pose and the point cloud, and outputting camera parameters and sparse Gaussian point cloud. According to the invention, synchronous high-precision estimation of the camera pose and the 3D Gaussian point cloud is realized; dependence on a pre-calculation pose or a pre-training model is abandoned, and the pose and the point cloud are directly combined and recovered from a disordered image; a large-view-angle jump scene is processed through a progressive optimization strategy, and the stability of the pose and the point cloud under sparse matching is ensured.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for joint optimization of camera pose and 3D Gaussian point cloud. Background Technology

[0002] Traditional Novel View Synthesis (NVS) methods rely on external camera pose estimation tools (such as COLMAP), leading to significant technical bottlenecks in the workflow. These tools must run independently outside the 3D reconstruction process, requiring substantial computational resources (e.g., COLMAP takes several hours to complete pose estimation in large-scale scenes). Furthermore, the separate processing of pose estimation and 3D reconstruction causes error propagation problems: initial pose errors directly affect the optimization quality of subsequent Gaussian point cloud parameters, especially in low-texture or dynamic scenes, where accumulated errors result in blurred rendering or distorted geometry. In addition, while existing 3D Gaussian methods that do not require pose input attempt to jointly optimize pose and point clouds, their design is limited by the assumption of temporal continuity between adjacent frames, requiring the input image sequence to have small viewpoint variations and failing to handle scenes with sparse feature distributions. For example, in scenes with only sparse corner points or repetitive textures, optimization strategies relying solely on photometric errors between adjacent frames will fail due to insufficient matching points, leading to pose drift or point cloud collapse.

[0003] Novel perspective synthesis techniques have made significant progress in recent years, driven by 3D Gaussian Splatting (3DGS). However, mainstream approaches still heavily rely on camera poses provided by traditional SfM tools such as COLMAP. For example, the original 3DGS paper required users to obtain precise camera parameters in advance using COLMAP, and then use these as fixed inputs to optimize the position, covariance, opacity, and spherical harmonics of the Gaussian point cloud. This separate workflow leads to two key drawbacks: first, the accuracy of pose estimation directly affects the final rendering quality; and second, the computational time of COLMAP becomes a bottleneck in the workflow. To eliminate the dependence on external poses, researchers have proposed several unsupervised optimization schemes, the most representative of which are BARF based on Neural Radiation Field (NeRF) and CFGS based on 3DGS.

[0004] BARF jointly optimizes camera pose and radiation field parameters through inverse rendering, but its iteration speed is slow (training a single scene can take several hours) and it is sensitive to the initial pose. CFGS, on the other hand, introduced the idea of ​​joint optimization into the 3D Gaussian framework for the first time. Its core idea is to alternately update Gaussian parameters and camera pose: in each iteration, the pose is first fixed and standard 3DGS optimization is performed, then the Gaussian point cloud is fixed and the pose is optimized using the photometric loss of adjacent frames. However, CFGS has a fundamental limitation: its pose optimization only considers pixel-level differences between temporally adjacent frames, assuming that changes in adjacent viewpoints are small and the image sequence is continuous. When the input is an unordered image or there are large viewpoint jumps (such as discrete panoramic images taken around an object), it is difficult to establish feature matching relationships between adjacent frames, leading to pose optimization failure. In addition, CFGS does not introduce multi-view geometric constraints, which can easily lead to error accumulation in sparse feature scenes due to insufficient matching points.

[0005] Another type of approach (such as ZeroGS and InstantSplat) attempts to initialize poses using pre-trained models. For example, ZeroGS uses the DUSt3R model to generate initial poses and then refines the point cloud using 3DGS. However, these methods rely on large-scale pre-trained data and have limited generalization ability to unseen scenes, failing to achieve truly end-to-end optimization. In summary, the core contradiction of existing technologies lies in the fact that methods relying on external tools or pre-trained models sacrifice flexibility and computational efficiency, while unsupervised approaches are limited by the locality of optimization constraints, making it difficult to achieve global consistency.

[0006] The shortcomings of existing technologies can be summarized in three aspects: 1. Computational efficiency: COLMAP-based solutions require independent operation of the pose estimation process, and their computational complexity increases superlinearly with the scene size. Taking the Tanks & Temples dataset as an example, COLMAP takes more than 24 hours to complete pose calculation in a medium-sized scene (e.g., 10,000 images), while 3DGS rendering optimization only takes tens of minutes, resulting in an imbalance in overall process efficiency. 2. Optimization consistency: The separate process leads to inconsistencies between the optimization objectives of pose and point cloud. COLMAP's pose estimation is based on minimizing the reprojection error of sparse feature points, while 3DGS's optimization objective is to minimize the pixel difference between the rendered image and the real image. The mismatch between the two objective functions means that the pose error cannot be corrected during the point cloud optimization stage, especially in low-texture areas, where point cloud parameters may converge to a local optimum to compensate for the pose deviation, ultimately leading to geometric distortion. 3. Scene adaptability: Existing unsupervised methods (such as CFGS) are limited to local time windows for optimization constraints, and their pose updates rely solely on the photometric differences between adjacent frames, lacking global geometric consistency. For example, in a 360-degree surround view scene, the viewpoint difference between the first and last frames may exceed 60 degrees. In this case, the photometric loss between adjacent frames cannot provide an effective pose update gradient, causing the optimization process to stall. Furthermore, these methods have stringent requirements for the continuity of image input and cannot handle non-serialized or partially overlapping image sets. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for joint optimization of camera pose and 3D Gaussian point cloud. The technical problems to be solved include: how to construct a joint optimization framework that does not require pre-calibration input, and how to achieve high-precision synchronous optimization of camera pose and 3D Gaussian point cloud by fusing geometric constraints and photometric consistency, while breaking through the limitations of existing methods on continuous image sequences and large field of view changes.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for joint optimization of camera pose and 3D Gaussian point cloud, comprising:

[0010] Obtain the image to be processed;

[0011] The acquired image is processed using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module uses RANSAC to filter interior points and calculate the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

[0012] As a further limitation of the first aspect of the present invention, the fundamental matrix F satisfies the following epipolar geometric constraint relationship: Where (x,x') represents the homogeneous coordinates of a pair of matched feature points in the image; F is a (3×3) fundamental matrix that describes the epipolar geometric relationship between corresponding pixels;

[0013] For camera pose P i With 3D point X j We perform nonlinear joint optimization with the objective function:

[0014]

[0015] Wherein, π(P) i ,X j ) represents the three-dimensional point X j Through camera parameter P i The pixel coordinates after projection onto the image plane, x ij ρ(·) represents the observed pixel position; it is a function to suppress the influence of outliers. A robust kernel function is used to suppress the negative impact of large errors. P represents the camera pose parameters. i X represents the pose of the i-th camera. j X represents the homogeneous coordinates of the j-th 3D scene point in the world coordinate system. ij This represents the corresponding pixel coordinates actually observed on the i-th image. The entire objective function represents the joint optimization of the poses of all cameras, P. i and three-dimensional point coordinates X j This minimizes the overall reprojection error.

[0016] As a further limitation of the first aspect of the present invention, a joint optimization module is embedded in the 3D Gaussian sputtering framework to alternately perform the following operations:

[0017] Gaussian point cloud update: fix the current camera pose {P iThe loss between the synthesized image and the real image is calculated using a differentiable renderer, and Gaussian properties, including position, covariance, opacity, and spherical harmonic coefficients, are optimized through backpropagation. During the Gaussian point cloud update process, the scene is rendered forward using a differentiable renderer, and the synthesized color C of each pixel is achieved through forward alpha blending.

[0018] LK3D pose optimization: With a fixed Gaussian point cloud, a global optimization target is constructed based on multi-view projection error; the Jacobian matrix of residuals with respect to pose parameters is calculated using the chain rule, and the pose is iteratively updated using the Gauss-Newton method; to maintain the orthogonality of the rotation matrix, the pose is decomposed into Euler angles and translation vectors, and then the rotation matrix is ​​updated; among these, the camera pose is optimized by local image photometric consistency.

[0019] As a further limitation of the first aspect of the present invention, the composite color C of each pixel is achieved through forward alpha blending, and its calculation formula is as follows:

[0020]

[0021] Among them, c i Let α represent the color of the i-th Gaussian point. i Its opacity, N is the set of all Gaussian points within the view frustum of this pixel, α j This represents the opacity of the j-th Gaussian point. This represents the cumulative product of (1 - opacity) of all Gaussian points preceding the i-th Gaussian point. Physically, this means that if a Gaussian point has higher opacity, it will occlude subsequent Gaussian points. For a detailed explanation, please refer to the image compositing technique alpha blending.

[0022] As a further limitation of the first aspect of the present invention, the rendering loss function is designed as a weighted combination of pixel-level reconstruction error and structural similarity loss, specifically in the form of: L render =(1-λ)L1+λL D-SSIM ;

[0023] Where L1 represents the absolute error between pixels, L D-SSIM For the structural similarity loss based on local windows, λ is the weighting coefficient.

[0024] As a further limitation of the first aspect of the present invention, the camera pose is optimized by local image photometric consistency, for the k-th frame image and the three-dimensional Gaussian point g j The correspondence is given by the projection error defined as:

[0025] Among them, c(g) j ) represents the color of the Gaussian point, Ik It is the k-th frame image, π(·) is the projection function, and P k =[R k |t k The image represents the camera pose. To minimize the projection error, the Gauss-Newton method is used to iteratively optimize the pose, and the incremental update formula is as follows:

[0026]

[0027] Where H is the Hessian matrix, and W(·) is the transformation function for the projection of 3D points onto the image. Represents the image gradient. Let g be the Jacobian matrix of the projection pair pose. In the Gaussian Splatting-based rendering method, g represents a discrete 3D scene point whose geometric properties (position, scale, rotation) and appearance (color, transparency) are modeled in the form of a Gaussian distribution. j The physical meaning is the j-th Gaussian point, which has a spatial position x(g j ) and color c(g) j A Gaussian point (c(g)) is the smallest rendering unit in a 3D scene, used to represent object surfaces or feature points. G represents the set of all 3D Gaussian points involved in the current camera pose optimization. Range: Includes all Gaussian points visible in the current optimization frame (frame k). j ) is a three-dimensional Gaussian point g j Color value.I k (·): The sampled value of the k-th frame at the specified pixel position.

[0028]

[0029] Finally, pose update is achieved by P←P⊕ΔP, and this is performed alternately with Gaussian point cloud optimization to gradually converge to the optimal solution.

[0030] Secondly, the present invention provides a joint optimization system for camera pose and 3D Gaussian point cloud, comprising:

[0031] The acquisition module is used to acquire the image to be processed;

[0032] The processing module is used to process the acquired image using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module uses RANSAC to filter interior points and calculate the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

[0033] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the joint optimization method of camera pose and 3D Gaussian point cloud as described in the first aspect.

[0034] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the camera pose and 3D Gaussian point cloud joint optimization method as described in the first aspect.

[0035] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the camera pose and 3D Gaussian point cloud joint optimization method as described in the first aspect.

[0036] The beneficial effects of this invention are as follows: It constructs an adaptive joint optimization framework to achieve synchronous high-precision estimation of camera pose and 3D Gaussian point cloud; it eliminates the dependence on pre-calculated pose or pre-trained model and directly recovers pose and point cloud from unordered image; it combines photometric consistency, multi-view geometric constraints and point cloud spatial continuity to construct a global optimization objective; and it uses a progressive optimization strategy to handle large-view jump scenes, ensuring the stability of pose and point cloud under sparse matching.

[0037] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of the joint optimization method for camera pose and 3D Gaussian point cloud according to an embodiment of the present invention. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0042] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0043] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0044] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0045] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0046] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0047] The existing best-in-class technology, CFGS, suffers from the following shortcomings: Local optimization constraints: It relies solely on the photometric loss of adjacent frames, making it unable to handle scenes with large viewpoint jumps. Error accumulation: Pose errors in adjacent frames propagate through time windows, causing point cloud distortion. Computational redundancy: It requires processing continuous image sequences, making parallel optimization impossible. Therefore, this invention provides a joint optimization method for camera pose and 3D Gaussian point clouds. This invention achieves significant improvements through the following innovations: 1. Global geometric consistency: The LK3D algorithm utilizes multi-view projection errors to ensure global viewpoint consistency; 3. Scene universality: It can handle unordered image inputs and does not require any pre-trained depth models or pre-processed poses; 4. Hardware compatibility: Offline optimization can be performed with a single graphics card (RTX 3090), without requiring real-time data stream input. Application expansion: This solution is applicable to fields such as autonomous driving (dynamic environment reconstruction), VR / AR (rapid modeling of large scenes), and cultural heritage protection (non-continuous image scanning), solving the core pain points of traditional methods such as reliance on pre-calibration equipment, long computation time, and poor scene adaptability.

[0048] Example 1

[0049] In this embodiment 1, a joint optimization system for camera pose and 3D Gaussian point cloud is first provided, including: an acquisition module for acquiring an image to be processed; a processing module for processing the acquired image using a pre-trained optimization model to obtain an optimized image; wherein, the optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module; wherein, the feature extraction module is used to extract features using SIFT; the matching module is used to perform feature point matching; the generation module is used to filter interior points using RANSAC and calculate the fundamental matrix to generate an initial point cloud; the optimization module is used to perform nonlinear optimization on the initial pose and point cloud, and output camera parameters and sparse Gaussian point cloud.

[0050] In this embodiment, the above-described system is used to implement a joint optimization method for camera pose and 3D Gaussian point cloud, including: acquiring an image to be processed; processing the acquired image using a pre-trained optimization model to obtain an optimized image; wherein, the optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module; wherein, the feature extraction module is used to extract features using SIFT; the matching module is used to perform feature point matching; the generation module is used to filter interior points using RANSAC and calculate the fundamental matrix to generate an initial point cloud; the optimization module is used to perform nonlinear optimization on the initial pose and point cloud, and output camera parameters and sparse Gaussian point cloud.

[0051] The fundamental matrix F satisfies the following epipolar geometric constraints: Where (x,x') represents the homogeneous coordinates of a pair of matched feature points in the image; F is a (3×3) fundamental matrix that describes the epipolar geometric relationship between corresponding pixels;

[0052] For camera pose P i With 3D point X j We perform nonlinear joint optimization with the objective function:

[0053]

[0054] Wherein, π(P) i ,X j ) represents the three-dimensional point X j Through camera parameter P i The pixel coordinates after projection onto the image plane, x ij ρ(·) represents the observed pixel position; it is a function to suppress the influence of outliers. A robust kernel function is used to suppress the negative impact of large errors. P represents the camera pose parameters. i X represents the pose of the i-th camera. j Let x represent the homogeneous coordinates of the j-th 3D scene point in the world coordinate system. ij This represents the corresponding pixel coordinates actually observed on the i-th image. The entire objective function represents the joint optimization of the poses of all cameras, P. i and three-dimensional point coordinates X j This minimizes the overall reprojection error.

[0055] Embed a joint optimization module within the 3D Gaussian sputtering framework and perform the following operations alternately:

[0056] Gaussian point cloud update: fix the current camera pose {P i The loss between the synthesized image and the real image is calculated using a differentiable renderer, and Gaussian properties, including position, covariance, opacity, and spherical harmonic coefficients, are optimized through backpropagation. During the Gaussian point cloud update process, the scene is rendered forward using a differentiable renderer, and the synthesized color C of each pixel is achieved through forward alpha blending.

[0057] LK3D pose optimization: With a fixed Gaussian point cloud, a global optimization target is constructed based on multi-view projection error; the Jacobian matrix of residuals with respect to pose parameters is calculated using the chain rule, and the pose is iteratively updated using the Gauss-Newton method; to maintain the orthogonality of the rotation matrix, the pose is decomposed into Euler angles and translation vectors, and then the rotation matrix is ​​updated; among these, the camera pose is optimized by local image photometric consistency.

[0058] The composite color C of each pixel is achieved through forward alpha blending, and its calculation formula is as follows:

[0059]

[0060] Among them, c i Let α represent the color of the i-th Gaussian point. i Its opacity, N is the set of all Gaussian points within the view frustum of this pixel, α j This represents the opacity of the j-th Gaussian point. This represents the cumulative product of (1 - opacity) of all Gaussian points preceding the i-th Gaussian point. Physically, this means that if a Gaussian point has higher opacity, it will occlude subsequent Gaussian points. For a detailed explanation, please refer to the image compositing technique alpha blending.

[0061] The rendering loss function is designed as a weighted combination of pixel-level reconstruction error and structural similarity loss, specifically in the form: L render =(1-λ)L1+λL D-SSIM ;

[0062] Where L1 represents the absolute error between pixels, L D-SSIM For the structural similarity loss based on local windows, λ is the weighting coefficient.

[0063] Camera pose is optimized by using local image photometric consistency, for the k-th frame image and the 3D Gaussian point g j The correspondence is given by the projection error defined as:

[0064] Among them, c(g) j ) represents the color of the Gaussian point, I k It is the k-th frame image, π(·) is the projection function, and P k =[R k |t k The image represents the camera pose. To minimize the projection error, the Gauss-Newton method is used to iteratively optimize the pose, and the incremental update formula is as follows:

[0065]

[0066] Where H is the Hessian matrix, and W(·) is the transformation function for the projection of 3D points onto the image. Represents the image gradient. is the Jacobian matrix of the projection pair pose; in the Gaussian Splatting-based rendering method, g represents a discrete 3D scene point whose geometric properties (position, scale, rotation) and appearance (color, transparency) are modeled in the form of a Gaussian distribution. jThe physical meaning is the j-th Gaussian point, which has a spatial position x(g j ) and color c(g) j A Gaussian point (c(g)) is the smallest rendering unit in a 3D scene, used to represent object surfaces or feature points. G represents the set of all 3D Gaussian points involved in the current camera pose optimization. Range: Includes all Gaussian points visible in the current optimization frame (frame k). j ) is a three-dimensional Gaussian point g j Color value.I k (·): The sampled value of the k-th frame at the specified pixel position.

[0067]

[0068] Finally, pose update is achieved by P←P⊕ΔP, and this is performed alternately with Gaussian point cloud optimization to gradually converge to the optimal solution.

[0069] Example 2

[0070] like Figure 1 As shown, this embodiment provides a method for joint optimization of camera pose and 3D Gaussian point cloud. It includes the following steps:

[0071] The system initialization steps include: inputting the original image sequence and initializing parameters (camera intrinsics, initial pose estimation, point cloud generation threshold). The processing steps include: feature extraction from the input images (keypoint detection and description), establishing epipolar geometric constraints based on feature matching to calculate the fundamental matrix, and performing BA optimization (bundling adjustment).

[0072] min∑‖∏(P i ,X j )-x ij || 2

[0073] Finally, the initial camera pose and sparse 3D point cloud are output.

[0074] Iterative optimization main loop (i = i + 1) steps: The loop condition is: executed when the model has not converged; the processing steps include: camera view sampling, i.e., selecting the optimal view V based on the principle of maximizing information entropy; generating a new view I' using a fast rasterizer; projection loss calculation: comparing the generated view I' with the real image I, calculating the photometric error (RGB difference), and calculating the geometric error (projection point position deviation); model parameter optimization: updating parameters through backpropagation of the optimizer. Iteration count: i = i + 1.

[0075] Periodic pose fine-tuning (j = j + 1) steps: Triggering condition: every 100 iterations; Processing steps include: pose error calculation: analyzing the deviation between the current pose and the observation, LK3D pose optimization: Dynamic control mechanism: dynamically adjust weights based on point cloud confidence. Pose parameter update: Output the optimized rotation matrix R and translation vector t, iteration count: j = j + 1.

[0076] Convergence Judgment and Output Steps: The convergence condition is: the rate of change of projected loss < threshold ε. Pose parameter fluctuation range <∈:‖P new -P old ||<10 -5 Output: Optimized camera pose sequence (R,t), 3D Gaussian point cloud model (including position: color: Transparency: α j ∈[0,1], covariance: ).

[0077] Specifically, in this embodiment, the detailed process steps of the joint optimization method for camera pose and 3D Gaussian point cloud are described as follows.

[0078] S1: Sparse Initialization and Parameter Configuration

[0079] To address the issues of unstable matching and large initialization errors in traditional SfM in low-texture or dynamic scenes, this embodiment improves the initialization process to make features more stable and point clouds more reliable, thereby enhancing the convergence and geometric accuracy of subsequent optimization. By adjusting the Gaussian kernel radius and matching threshold for different scenes, the feature response in weakly textured regions and under motion blur is effectively enhanced, thus providing a good initial solution for joint optimization.

[0080] The method described in this embodiment generates an initial camera pose and a sparse 3D Gaussian point cloud using an improved SfM method. Specifically, it includes: feature extraction and matching: SIFT feature extraction; fundamental matrix estimation: RANSAC is used to filter interior points and calculate the fundamental matrix to generate the initial point cloud; and bundle adjustment optimization: nonlinear optimization is performed on the initial pose and point cloud to output camera parameters P. (0) ={R0,t0} and sparse Gaussian point cloud G (0) .

[0081] The SIFT feature extraction formula is as follows:

[0082] L(x,y,σ)=G(x,y,σ)*I(x,y).

[0083] Where: L represents the scale-space image, and G represents the Gaussian kernel function. I represents the input image, and * represents the convolution operation. X and y are the pixel coordinates in the image, representing their two-dimensional spatial positions on the image plane. σ (sigma) represents the scale parameter, which is the standard deviation of the Gaussian kernel and is used to control the scale of the Gaussian blur. In SIFT, different σ values ​​correspond to different scale spatial levels, used to detect feature points at different scales. The equations represent the epipolar geometric constraints, where x' and x' are a pair of matching feature points represented in homogeneous coordinates (3-dimensional vectors, in the form of...). ).

[0084] The fundamental matrix F should satisfy the following epipolar geometric constraints: (x,x') represents the homogeneous coordinates of a pair of matched feature points in the image; F is a (3×3) fundamental matrix that describes the epipolar geometric relationship between corresponding pixels.

[0085] For low-texture scenes, the weak texture response is enhanced by increasing the Gaussian kernel radius. For dynamic scenes, rotation invariance under motion blur is enhanced.

[0086] During the Bundle Adjustment phase, the camera pose P is adjusted. i With 3D point X j We perform nonlinear joint optimization with the objective function:

[0087] Wherein, π(P) i ,X j ) represents the three-dimensional point X j Through camera parameter P i The pixel coordinates after projection onto the image plane, x ij For the observed corresponding pixel position, ρ(·) is a function to suppress the influence of outliers. A robust kernel function is used to suppress the negative impact of large errors. P represents the camera pose parameters. i X represents the pose of the i-th camera. j X represents the homogeneous coordinates of the j-th 3D scene point in the world coordinate system. ij This represents the corresponding pixel coordinates actually observed on the i-th image. The entire objective function represents the joint optimization of the poses of all cameras, P. i and three-dimensional point coordinates X j This minimizes the overall reprojection error.

[0088] S2: Alternating Optimization Framework

[0089] To address issues such as large viewpoint jumps and disordered image sequences, an alternating optimization strategy of Gaussian point cloud and pose is adopted. Gaussian point properties are optimized through rendering loss, while pose is optimized through multi-view projection residuals. Both approaches converge under mutual supervision, avoiding drift and blurring caused by inconsistencies between point cloud and pose targets in traditional methods, while ensuring optimization stability and global consistency. Specifically, a joint optimization module is embedded within the 3D Gaussian Sputtering (3DGS) framework, alternately performing the following operations:

[0090] S2.1: Gaussian point cloud update

[0091] Fix the current camera pose {P i The L1+D of the synthetic image and the real image is calculated using a differentiable renderer. SSIM Loss, backpropagation optimization of Gaussian properties (position x) j , covariance ∑ j Opacity α j spherical harmonic coefficient SH j Specifically, during the Gaussian point cloud update process, a differentiable renderer is used to perform forward rendering of the scene. The composite color C of each pixel is achieved through forward alpha blending, and its calculation formula is as follows:

[0092] Among them, c i Let α represent the color of the i-th Gaussian point. i Let L be the opacity, and N be the set of all Gaussian points within the view frustum of that pixel. The forward cumulative product ensures the correct order of opacity stacking. The rendering loss function is designed as a weighted combination of pixel-level reconstruction error and structural similarity loss, specifically in the form: L render =(1-λ)L1+λL D-SSIM .

[0093] Where L1 represents the absolute error between pixels, L D-SSIM The structural similarity loss is based on local windows, with λ = 0.2 being the weighting coefficient of D-SSIM. This loss is used through backpropagation to optimize the parameters of the Gaussian points, including their position, covariance matrix, opacity, and color spherical harmonics, thereby improving the fidelity and detail reproduction of the final synthesized image.

[0094] S2.2: LK3D Pose Optimization

[0095] With a fixed Gaussian point cloud, a global optimization objective is constructed based on multi-view projection errors:

[0096] l g (k) =c(g j )-I k (π(P k ·gj )).

[0097] The Jacobian matrix J of the residual with respect to the pose parameters is calculated using the chain rule. Pk The Gauss-Newton method is used to iteratively update the pose. To maintain the orthogonality of the rotation matrix, the pose is decomposed into Euler angles θ = (α, β, γ) and translation vector t, and then the rotation matrix is ​​updated.

[0098] Specifically, in this embodiment, camera pose is optimized through local image photometric consistency. For the k-th frame image and the 3D Gaussian point g... j The correspondence is given by the projection error defined as:

[0099]

[0100] Among them, c(g) j ) represents the color of the Gaussian point, I k It is the k-th frame image, π(·) is the projection function, and P k =[R k |t k The image represents the camera pose. To minimize this error, the Gauss-Newton method is used for iterative pose optimization, with the incremental update formula as follows:

[0101]

[0102] Where H is the Hessian matrix, and W(·) is the transformation function for the projection of 3D points onto the image. Represents the image gradient. Let be the Jacobian matrix of the projection pair's pose. The pose is ultimately updated via P←P⊕ΔP, and this is performed alternately with Gaussian point cloud optimization, gradually converging to the optimal solution.

[0103] S3: Dynamic Convergence Control

[0104] To address the issues of low computational efficiency and susceptibility to oscillations during long-term optimization, an adaptive frequency scheduling mechanism and an early stopping criterion are introduced. The pose optimization frequency is adjusted at different iteration stages, and the system automatically terminates when the pose changes tend to stabilize, avoiding unnecessary computation and improving training efficiency and system stability. This is particularly suitable for large-scene reconstruction tasks with many frames and large motion amplitudes.

[0105] Iteration frequency scheduling: For the first 5000 iterations, pose optimization is performed once every 50 Gaussian updates; from 5000 to 15000 iterations, it is performed once every 100 updates; pose optimization stops after 15000 iterations.

[0106] Early stopping mechanism: If the pose update amount ||ΔP|| in 10 consecutive iterations k ||<10 -5prematurely terminate pose optimization.

[0107] S4: High-quality rendering and output

[0108] Based on the optimized Gaussian point cloud and accurate pose, a new perspective image is generated through the 3DGS differentiable renderer, and a 3D reconstruction model is output.

[0109] In summary, the core steps of the method described in this embodiment include: sparse initialization: generating initial pose and point cloud through improved SfM, breaking through the limitations of traditional COLMAP; alternating optimization framework: jointly optimizing Gaussian point cloud parameters and camera pose, combining geometric constraints and photometric consistency; dynamic convergence control: balancing optimization efficiency and accuracy through frequency scheduling and early stopping mechanisms; accurate rendering output: generating high-quality new perspective images based on the globally consistent optimization results. Key technologies include: LK3D pose optimization algorithm: extending optical flow to 3D space, constructing a global optimization objective through multi-view projection errors; geometric-photometric joint optimization: fusing geometric regularization terms and photometric loss within the 3DGS framework, simultaneously optimizing point cloud and pose; during the joint optimization process, an iterative alternating optimization strategy is adopted. In the t-th iteration, the camera pose P is first fixed. (t) Optimize the Gaussian point cloud G, and then update the point cloud G. (t+1) Based on this, the camera pose P is optimized in reverse, and the Euler angles are parameterized: the rotation matrix is ​​decomposed into Euler angles to simplify the Jacobian calculation and ensure orthogonality; dynamic optimization strategy: adaptive iteration frequency scheduling and early stopping mechanism to prevent overfitting and improve computational efficiency.

[0110] The entire joint optimization process can be represented as follows:

[0111]

[0112] in, This represents the photometric consistency loss function between the rendered image and the real image.

[0113] In the pose optimization stage, a global projection error minimization objective is further constructed based on the LK3D model, in the form of:

[0114]

[0115] Where K is the number of image frames, M is the number of 3D Gaussian points, and c(g j ) represents the color of the j-th Gaussian point, I k Let P be the image of the k-th frame, π(·) be the projection function, and P be the image of the k-th frame. k This represents the camera pose for the corresponding frame. By minimizing this error function, the spatial alignment accuracy between the camera pose and the point cloud can be effectively improved.

[0116] Example 3

[0117] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, these computer instructions implement the camera pose and 3D Gaussian point cloud joint optimization method described above. The method includes:

[0118] The process involves acquiring an image to be processed, then processing the acquired image using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module filters interior points using RANSAC and calculates the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

[0119] Example 4

[0120] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute the camera pose and 3D Gaussian point cloud joint optimization method as described above, the method including:

[0121] The process involves acquiring an image to be processed, then processing the acquired image using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module filters interior points using RANSAC and calculates the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

[0122] Example 5

[0123] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the camera pose and 3D Gaussian point cloud joint optimization method as described above. The method includes:

[0124] The process involves acquiring an image to be processed, then processing the acquired image using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module filters interior points using RANSAC and calculates the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0129] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A method for joint optimization of camera pose and 3D Gaussian point cloud, characterized in that, include: Obtain the image to be processed; The acquired image is processed using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module uses RANSAC to filter interior points and calculate the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

2. The method for joint optimization of camera pose and 3D Gaussian point cloud according to claim 1, characterized in that, The fundamental matrix F satisfies the following epipolar geometric constraints: Where (x,x') represents the homogeneous coordinates of a pair of matched feature points in the image; F is a (3×3) fundamental matrix that describes the epipolar geometric relationship between corresponding pixels; For camera pose P i With 3D point X j We perform nonlinear joint optimization with the objective function as follows: Wherein, π(P) i ,X j ) represents the three-dimensional point X j Using camera pose parameters P i The pixel coordinates after projection onto the image plane, x ij The observed pixel position; ρ(·) represents the function to suppress the influence of outliers; P represents the camera pose parameters, P i X represents the pose of the i-th camera; j x represents the homogeneous coordinates of the j-th 3D scene point in the world coordinate system; ij This represents the corresponding pixel coordinates actually observed on the i-th image.

3. The method for joint optimization of camera pose and 3D Gaussian point cloud according to claim 1, characterized in that, Embed a joint optimization module within the 3D Gaussian sputtering framework and perform the following operations alternately: Gaussian point cloud update: fix the current camera pose {P i The loss between the synthesized image and the real image is calculated using a differentiable renderer, and Gaussian properties, including position, covariance, opacity, and spherical harmonic coefficients, are optimized through backpropagation. During the Gaussian point cloud update process, the scene is rendered forward using a differentiable renderer, and the synthesized color C of each pixel is achieved through forward alphablending. LK3D pose optimization: With a fixed Gaussian point cloud, a global optimization target is constructed based on multi-view projection error; the Jacobian matrix of residuals with respect to pose parameters is calculated using the chain rule, and the pose is iteratively updated using the Gauss-Newton method; to maintain the orthogonality of the rotation matrix, the pose is decomposed into Euler angles and translation vectors, and then the rotation matrix is ​​updated; among these, the camera pose is optimized by local image photometric consistency.

4. The method for joint optimization of camera pose and 3D Gaussian point cloud according to claim 3, characterized in that, The composite color C of each pixel is achieved through forward alpha blending, and its calculation formula is as follows: Among them, c i Let α represent the color of the i-th Gaussian point. i Its opacity, N is the set of all Gaussian points within the view frustum of this pixel, α j This represents the opacity of the j-th Gaussian point. This means multiplying the results of all Gaussian points up to the i-th Gaussian point by 1 minus the opacity.

5. The method for joint optimization of camera pose and 3D Gaussian point cloud according to claim 4, characterized in that, The rendering loss function is designed as a weighted combination of pixel-level reconstruction error and structural similarity loss, specifically in the form: L render =(1-λ)L1+λL D-SSIM ; Where L1 represents the absolute error between pixels, L D-SSIM For the structural similarity loss based on local windows, λ is the weighting coefficient.

6. The method for joint optimization of camera pose and 3D Gaussian point cloud according to claim 5, characterized in that, Camera pose is optimized by using local image photometric consistency, for the k-th frame image and the 3D Gaussian point g j The correspondence is given by the projection error defined as: Among them, c(g) j ) represents the color of the Gaussian point, I k It is the k-th frame image, π(·) is the projection function, and P k =[R k |t k The image represents the camera pose. To minimize the projection error, the Gauss-Newton method is used to iteratively optimize the pose, and the incremental update formula is as follows: Where H is the Hessian matrix, and W(·) is the transformation function for the projection of 3D points onto the image. Represents the image gradient. Let be the Jacobian matrix of the projection pair pose; g represents a discrete 3D scene point, G represents the set of all 3D Gaussian points participating in the current camera pose optimization, c(g) represents the color of the 3D Gaussian point g, and I(·) represents the sampled value at the specified pixel position. Finally passed The pose update is performed alternately with Gaussian point cloud optimization, gradually converging to the optimal solution.

7. A joint optimization system for camera pose and 3D Gaussian point cloud, characterized in that, include: The acquisition module is used to acquire the image to be processed; The processing module is used to process the acquired image using a pre-trained optimization model to obtain an optimized image. The optimization model includes a feature extraction module, a matching module, a generation module, and an optimization module. The feature extraction module uses SIFT for feature extraction; the matching module performs feature point matching; the generation module uses RANSAC to filter interior points and calculate the fundamental matrix to generate an initial point cloud; and the optimization module performs nonlinear optimization on the initial pose and point cloud, outputting camera parameters and a sparse Gaussian point cloud.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the joint optimization method for camera pose and 3D Gaussian point cloud as described in any one of claims 1-6.

9. A computer device, characterized in that, The system includes a memory and a processor, which communicate with each other. The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the camera pose and 3D Gaussian point cloud joint optimization method as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the camera pose and 3D Gaussian point cloud joint optimization method as described in any one of claims 1-6.

Images