Three-dimensional Gaussian spattering reconstruction method and related equipment

Abstract

The invention provides a three-dimensional Gaussian splash reconstruction method and related equipment, and the method comprises the steps: carrying out the grouping processing of an obtained image sequence, obtaining a plurality of image pairs, enabling the image sequence to comprise a plurality of images with uncalibrated attitudes, and enabling any one of the plurality of image pairs to comprise two images in the image sequence; obtaining a global point cloud and a camera attitude based on the paired point diagrams determined by the plurality of image pairs and a confidence map associated with the paired point diagrams; and performing iterative training on parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera attitude to obtain a three-dimensional Gaussian model, the initialized three-dimensional Gaussian ellipsoid being obtained by performing initialization processing on a preset three-dimensional Gaussian ellipsoid according to the global point cloud, and the three-dimensional Gaussian model being used for completing a new view angle synthesis task. The robustness of point cloud initialization is improved, it is ensured that the number of point diagrams is kept high in speed and memory no matter under the dense or sparse condition, and high-quality three-dimensional reconstruction and new view angle synthesis are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional Gaussian splash reconstruction method and related equipment. Background Art

[0002] Novel View Synthesis (NVS) is an important technology in the field of computer vision and is widely used in many fields such as augmented reality (AR), virtual reality (VR), and robot navigation. In recent years, NVS methods based on Neural Radiance Fields (NeRF) and Three-Dimensional Gaussian (3DGS) have attracted much attention.

[0003] The NeRF method uses a multi-layer perceptron (MLP) network to implicitly model three-dimensional objects, and uses volume rendering technology to generate three-dimensional models from known images and synthesize new perspectives. However, NeRF has problems such as long training time, slow rendering speed, and difficult editing of implicit modeling. The 3DGS method explicitly models three-dimensional scenes based on a three-dimensional Gaussian ellipsoid, and projects the three-dimensional Gaussian onto a two-dimensional plane through a rasterization method. It can generate high-quality rendering results in real time and is superior to NeRF in terms of speed and editability.

[0004] At present, most 3DGS-based NVS methods rely on accurate camera pose information and effective point cloud initialization, which is usually obtained with the help of structured light or camera localization and mapping (COLMAP) tools. However, in sparse view, textureless and weak texture scenes, COLMAP often cannot obtain enough feature points, resulting in initialization failure, which in turn affects the reconstruction quality. Summary of the invention

[0005] The embodiment of the present invention provides a three-dimensional Gaussian splash reconstruction method and related equipment to solve the problem of poor reconstruction quality of the existing NVS method based on 3DGS.

[0006] To solve the above technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a three-dimensional Gaussian splash reconstruction method, the method comprising:

[0008] Performing group processing on the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence;

[0009] Acquire a global point cloud and a camera pose based on paired point graphs determined from the plurality of image pairs and confidence graphs associated with the paired point graphs;

[0010] The parameters of the initialized three-dimensional Gaussian ellipsoid are iteratively trained based on the camera posture to obtain a three-dimensional Gaussian model. The initialized three-dimensional Gaussian ellipsoid is obtained by initializing the preset three-dimensional Gaussian ellipsoid according to the global point cloud. The three-dimensional Gaussian model is used to complete the new perspective synthesis task.

[0011] Optionally, the image sequence includes a first number of images without calibrated poses;

[0012] The acquired image sequence is grouped to obtain a plurality of image pairs, including:

[0013] According to the first number of images with uncalibrated postures, the preset overlap rate and the single-group preset number, the first number of images with uncalibrated postures are grouped to obtain a second number of image groups, wherein the number of images in each image group in the second number of image groups is the single-group preset number, the overlap rate of adjacent image groups in the second number of image groups is the preset overlap rate, and the sequence of images in the second number of image groups is the same as the sequence of images in the image sequence;

[0014] The first image in the first image group is paired with other images other than the first image in the first image group in sequence to obtain an image pair corresponding to the first image group, wherein the first image group is any group in the second number of image groups.

[0015] Optionally, the acquiring a global point cloud and a camera pose based on paired point graphs determined based on the multiple image pairs and confidence graphs associated with the paired point graphs comprises:

[0016] Inputting the plurality of image pairs into a pre-trained deep learning model to obtain a paired point map corresponding to each image pair in the plurality of image pairs and a confidence map associated with the paired point map;

[0017] Solve the minimum value of the objective function to determine the global point cloud and camera posture. The goal of the objective function is to determine the optimal transformation matrix, camera projection matrix and scaling factor. The paired point map is used to provide scene construction information, and the confidence map is used to measure the credibility of each point in the corresponding point map.

[0018] The objective function is expressed as follows:

[0019]

[0020] J* is the objective function, J is the transformation matrix, P is the camera projection matrix, σ is the scaling factor, E is a set of image pairs, e is any image pair in E, v is a point in e, i is the pixel position, H is the image height, W is the image width, are corresponding points in the point diagram, for After the camera projection matrix P e Project and multiply by the scaling factor σ e After that, the corresponding points are obtained by transformation matrix J. for Confidence in the confidence map.

[0021] Optionally, the iterative training of parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model includes:

[0022] Initializing the parameters of the preset three-dimensional Gaussian ellipsoid according to the global point cloud to obtain the parameters of the initialized three-dimensional Gaussian ellipsoid, wherein the parameters include a center position, a spherical harmonic function coefficient, a rotation factor, a scaling factor, and opacity;

[0023] After each scene training is completed, a densification process is performed. The scene training is to use N images to train the center position, the spherical harmonic function coefficients, the rotation factor, the scaling factor and the opacity. The N images are images included in the image sequence, and each of the N images is marked with the acquired camera posture. The densification process is to perform splitting and cloning by calculating whether the average gradient size of the Gaussian point in the observable view exceeds the threshold. The splitting is used to split a Gaussian point into multiple new Gaussian points, and the cloning is used to copy the existing Gaussian points. The number of densification processes is M, M is the average number of iterations of each image in the N images, and the total number of iterative training is N×M, and N and M are both positive integers.

[0024] Optionally, after w iterations of training, the opacity is reset once, where w is proportional to N.

[0025] Optionally, the process of obtaining the second number of image groups is as follows:

[0026]

[0027] The number of the uncalibrated posture images is n, the preset overlap rate is r, the single group preset number is t, and the number of the image groups is k.

[0028] In a second aspect, an embodiment of the present invention provides a three-dimensional Gaussian splash reconstruction device, the device comprising:

[0029] A processing module, used for grouping and processing the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence;

[0030] An acquisition module, configured to acquire a global point cloud and a camera pose based on paired point graphs determined by the plurality of image pairs and confidence graphs associated with the paired point graphs;

[0031] A training module is used to iteratively train the parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model. The initialized three-dimensional Gaussian ellipsoid is obtained by initializing the preset three-dimensional Gaussian ellipsoid according to the global point cloud. The three-dimensional Gaussian model is used to complete the new perspective synthesis task.

[0032] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to the first aspect.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0038] In the embodiment of the present invention, a progressive group matching strategy is used to construct image pairs, and then paired point graphs are determined based on the grouped image pairs, and global optimization is performed to obtain global point clouds and camera poses. The robustness of point cloud initialization is improved, long-term memory and global optimization effects are achieved, and the number of point graphs is ensured to be efficient in speed and memory regardless of dense or sparse conditions; iterative training is performed in the process of determining the three-dimensional Gaussian model, so that the three-dimensional Gaussian model can gradually improve the accuracy of scene representation, adapt to the needs of different scenes, and achieve high-quality three-dimensional reconstruction and new perspective synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 is a flow chart of a three-dimensional Gaussian splash reconstruction method provided by an embodiment of the present invention;

[0041] Figure 2 It is a structural schematic diagram of a three-dimensional Gaussian splash reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 , Figure 1 is a flow chart of a three-dimensional Gaussian splash reconstruction method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0044] Step 101, grouping the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence;

[0045] The image sequence may be images of a target scene taken from different angles using a camera, and these images may cover at least a partial area of ​​the target scene. Compared with the multi-view images and corresponding camera poses collected using tools such as COLMAP in the related art, a three-dimensional Gaussian splash reconstruction method provided in this embodiment can better take into account different scene sizes. The image sequence is progressively grouped and progressively transferred backwards through step 101 to achieve long-term memory and global optimization effects, ensuring that the number of point images remains efficient in speed and memory regardless of whether it is dense or sparse.

[0046] Step 102: acquiring a global point cloud and a camera pose based on paired point graphs determined by the plurality of image pairs and confidence graphs associated with the paired point graphs;

[0047] Camera pose is crucial for accurately fusing images from different perspectives into three-dimensional space. In order to improve the robustness of point cloud initialization, MASt3R based on Transformer architecture can be introduced, which can generate point clouds and camera poses from a series of uncalibrated pose images. Considering that related point graph-based methods, such as InstantSplat, rely on fully connected graphs when constructing point graph pairs. This approach has significant disadvantages. A large number of point graph pairs will consume memory resources sharply. This problem is particularly prominent in dense view scenes, making such methods difficult to apply. In addition, constructing a large number of point graph pairs will also take a lot of time, seriously reducing initialization efficiency. Based on this, a three-dimensional Gaussian splash reconstruction method provided in this embodiment, after progressively grouping the image sequence through step 101, further through step 102 based on the progressively grouped image pairs, determines the paired point graphs and the confidence graphs associated with the paired point graphs, which can not only significantly reduce the number of point graph pairs, effectively reduce memory pressure, but also ensure the establishment of effective point graph associations. Through this progressively grouped paired point map construction strategy, even when faced with a large number of input images with uncalibrated poses, they can be processed efficiently, thereby achieving faster acquisition of global point clouds and camera poses, laying a solid foundation for subsequent 3D reconstruction work.

[0048] Step 103, iteratively train the parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model, wherein the initialized three-dimensional Gaussian ellipsoid is obtained by initializing a preset three-dimensional Gaussian ellipsoid according to the global point cloud, and the three-dimensional Gaussian model is used to complete the new perspective synthesis task.

[0049] The preset three-dimensional Gaussian ellipsoid is initialized according to the global point cloud to obtain an initialized three-dimensional Gaussian ellipsoid with parameters such as initial position, covariance matrix, color and opacity. These initial parameters are the starting point for subsequent iterative training. Then, the iterative training cycle is entered. In each iteration, a series of operations are performed based on the current three-dimensional Gaussian ellipsoid parameters and camera posture to update these parameters, and gradually optimize the three-dimensional Gaussian ellipsoid's ability to represent the scene. After each iteration, it is checked whether the preset termination condition is met. The termination condition can be set to reach a certain number of iterations, or when a certain indicator for measuring the training effect (such as the value of the loss function) converges to a range that meets the requirements. If the termination condition is not met, continue to the next iteration; if it is met, the iterative training is terminated to obtain the final three-dimensional Gaussian model.

[0050] In the 3D Gaussian splash reconstruction method provided in this embodiment, the number of iterations can be adaptively adjusted according to the sparsity and complexity of the scene in step 103. For sparse scenes, accurate 3D reconstruction can be obtained by reducing the number of iterations; and for larger or complex scenes, the high accuracy of the final result can be ensured by increasing the number of training rounds.

[0051] In the embodiment of the present invention, a progressive group matching strategy is used to construct image pairs, and then paired point graphs are determined based on the grouped image pairs, and global optimization is performed to obtain global point clouds and camera poses. The robustness of point cloud initialization is improved, long-term memory and global optimization effects are achieved, and the number of point graphs is ensured to be efficient in speed and memory regardless of dense or sparse conditions; iterative training is performed in the process of determining the three-dimensional Gaussian model, so that the three-dimensional Gaussian model can gradually improve the accuracy of scene representation, adapt to the needs of different scenes, and achieve high-quality three-dimensional reconstruction and new perspective synthesis.

[0052] Optionally, the image sequence includes a first number of images without calibrated poses;

[0053] The step 101, grouping the acquired image sequence to obtain a plurality of image pairs, includes:

[0054] According to the first number of images with uncalibrated postures, the preset overlap rate and the single-group preset number, the first number of images with uncalibrated postures are grouped to obtain a second number of image groups, wherein the number of images in each image group in the second number of image groups is the single-group preset number, the overlap rate of adjacent image groups in the second number of image groups is the preset overlap rate, and the sequence of images in the second number of image groups is the same as the sequence of images in the image sequence;

[0055] The first image in the first image group is paired with other images other than the first image in the first image group in sequence to obtain an image pair corresponding to the first image group, wherein the first image group is any group in the second number of image groups.

[0056] In this embodiment, the purpose of grouping is to effectively utilize the information between images and control the consumption of computing resources when processing the input image sequence. After the grouping is completed, an image pair is constructed for each group for subsequent point map generation and posture estimation.

[0057] In the grouping process, by reasonably dividing the first number of uncalibrated posture images in the image sequence, setting the number of views of each group (single group preset number) and the overlap rate between adjacent groups (preset overlap rate), the entire image sequence is divided into multiple groups (a second number of image groups). Exemplarily, the number of uncalibrated posture images is set to n, the preset overlap rate is r, and the single group preset number is t, then the number of image groups k can be determined by the following formula:

[0058]

[0059] For example, assuming there are 100 uncalibrated pose images (n is 100), the number of views per group (i.e., the preset number of a single group) t is set to 10, and the overlap rate r is 0.2, then the number of image groupings k can be determined by calculation to be 13. This means that the 100 images are divided into 13 groups, each with approximately 10 images (due to overlap, the actual number of views per group may be slightly different).

[0060] In this way, correlation and continuity are established between different groups, so that in the subsequent point map generation and posture estimation process, the information transfer between adjacent groups can be fully utilized to improve the accuracy and robustness of initialization.

[0061] In the process of constructing image pairs, taking the first image group as an example, the first image in the first image group is paired with other images in sequence to obtain the image pair corresponding to the first image group. In other words, the view with index 0 (i.e., the first image in the first image group) can be selected as the reference view, and then all non-reference views in the first image group (i.e., other images other than the first image) are matched with the reference view. This construction method can ensure that the information association between the reference view and other views is fully utilized in each group, providing a rich data foundation for the subsequent generation of accurate point maps. It should be understood that the first image group is any group in the second number of image groups, and the same can be applied to other groups.

[0062] For example, an image pair can be represented as:

[0063]

[0064] Where p is the group number, G is the group set, q is the non-reference view number, and i and j are point map numbers.

[0065] This representation clearly defines the source of the image pair, that is, which reference view in which group is matched with other non-reference views. By constructing image pairs in this way, it is possible to reduce the number of unnecessary image pairs while ensuring the validity of the point graph association, avoiding the high memory consumption and computational overhead caused by constructing a complete graph, which is especially suitable for dense view scenes. For example, in an image group with 5 images (t=5), image 0 is the reference view, then the image pairs corresponding to the image group can be constructed, including: (0,1), (0,2), (0,3) and (0,4). Similarly, the image pairs corresponding to other image groups can be determined. These image pairs can be used for subsequent point graph generation and pose estimation.

[0066] In sparse scenes, the progressive grouping matching strategy can still generate high-quality initial feature points. Due to the rationality of the grouping and image pair construction method, even when there are relatively few feature points, it is possible to fully utilize the limited feature information for camera pose estimation through information transfer within and between groups. For example, in some sparse scenes with no texture or weak texture, traditional methods may not be able to accurately estimate the camera pose due to insufficient feature points, but this strategy can mine more effective information from limited features of different perspectives through progressive grouping, ensuring the accuracy of camera pose estimation, thereby avoiding initialization failures caused by insufficient feature points.

[0067] For dense scenes, the advantages of the progressive grouping matching strategy are more obvious. On the one hand, by setting the grouping and overlapping intervals, the number of point-image pairs is effectively reduced, avoiding the huge memory consumption and computational overhead when constructing the complete graph. On the other hand, the progressive grouping method ensures that the early frames can be transmitted to the later frames in a smooth manner, reducing the problem of inaccurate camera pose caused by the large difference between the front and back perspectives. For example, in an urban street scene with a large number of perspectives, if the traditional complete graph is used to construct point-image pairs, it may not be processed due to insufficient memory. However, the use of the progressive grouping matching strategy can accurately complete the point cloud initialization and camera pose estimation within a reasonable range of memory and computing resources, providing a good foundation for subsequent three-dimensional Gaussian training.

[0068] Optionally, the step 102 of acquiring a global point cloud and a camera pose based on paired point graphs determined by the plurality of image pairs and confidence graphs associated with the paired point graphs includes:

[0069] Inputting the plurality of image pairs into a pre-trained deep learning model to obtain a paired point map corresponding to each image pair in the plurality of image pairs and a confidence map associated with the paired point map;

[0070] Solve the minimum value of the objective function to determine the global point cloud and camera posture. The goal of the objective function is to determine the optimal transformation matrix, camera projection matrix and scaling factor. The paired point map is used to provide scene construction information, and the confidence map is used to measure the credibility of each point in the corresponding point map.

[0071] The objective function is expressed as follows:

[0072]

[0073] J* is the objective function, J is the transformation matrix, P is the camera projection matrix, σ is the scaling factor, E is a set of image pairs, e is any image pair in E, v is a point in e, i is the pixel position, H is the image height, W is the image width, are corresponding points in the point diagram, for After the camera projection matrix P e Project and multiply by the scaling factor σ e After that, the corresponding points are obtained by transformation matrix J. for Confidence in the confidence map.

[0074] In this embodiment, MASt3R based on Transformer architecture can be introduced to generate point clouds and camera postures from a series of uncalibrated posture images. Multiple image pairs obtained based on a series of uncalibrated posture images are used as inputs to the MASt3R model. These images contain visual information of the scene, but since the postures are not calibrated, it is difficult to directly extract useful information from them. The MASt3R model is designed to process this raw image data and mine the potential features and geometric relationships therein. Among them, before inputting the MASt3R model, some preprocessing operations, such as normalization, are performed on the image to ensure that the image data is within a suitable numerical range to improve the processing efficiency and accuracy of the model. For example, the pixel values ​​of the image are normalized to a specific interval, such as [0,1] or [-1,1].

[0075] The MASt3R model is based on the Transformer architecture, which is good at processing sequence data and capturing long-distance dependencies. For the input image sequence, the Transformer model extracts features from each part of the image through its multi-layer self-attention mechanism. The self-attention mechanism can automatically focus on the correlation between different regions in the image, thereby effectively learning the global and local feature representations of the image. For example, in a scene image containing multiple objects, the model can use the self-attention mechanism to focus on the relative position relationship, shape features, and texture information between objects. After being processed by the self-attention mechanism, the features of the image are encoded into a series of vector representations that contain rich image information. The MASt3R model further integrates and processes these feature vectors, and converts low-level image features into high-level semantic features through the Transformer's Feed-Forward Neural Network Layers. These semantic features can better describe the geometric structure and visual appearance of the scene.

[0076] The process of point cloud generation can be described as follows: The MASt3R model predicts point clouds. Specifically, the model estimates the position distribution of three-dimensional points in the scene based on the feature information in the image. This process involves complex geometric reasoning and feature mapping. The model generates point clouds by back-projecting the feature information on the image plane into three-dimensional space, combined with the understanding of the scene structure. For example, for an object contour feature in an image, the model can predict the approximate shape and position of the object in three-dimensional space based on its position, shape and relationship with other features in the image, thereby generating the corresponding point cloud part. The generated initial point cloud may have certain errors or inaccuracies, and the MASt3R model will further optimize it. By comprehensively analyzing the feature information under multiple image perspectives, the model adjusts the position and distribution of points in the point cloud to make it more consistent with the real geometric structure of the scene. For example, for the feature points of the same object observed from multiple perspectives, the model will optimize its position in three-dimensional space according to the projection relationship under different perspectives to improve the accuracy and integrity of the point cloud.

[0077] The process of camera pose estimation can be described as follows: While generating the point cloud, the MASt3R model also estimates the camera pose. The model calculates the relative pose change of the camera when taking these images by analyzing the feature correspondence between different images. This process is based on feature matching and geometric constraint solving. For example, by finding the corresponding positions of the same feature points in different images on the image plane, the relative rotation and translation relationship between the cameras is calculated using geometric methods such as triangulation principle. Based on the relative pose estimation results, the MASt3R model further performs global pose optimization. Since relative pose estimation may have cumulative errors, the accuracy of camera pose estimation can be improved by globally optimizing the pose relationship in the entire image sequence. This process may involve minimizing a certain error function, for example, minimizing the reprojection error, that is, projecting the 3D point cloud back to the image plane, calculating the error between the projection point and the feature point of the original image, and adjusting the camera pose parameters through the optimization algorithm to minimize the error.

[0078] Among them, the point cloud is represented by a set of coordinates of three-dimensional points, describing the geometric structure of the scene; the camera posture is represented by rotation matrices and translation vectors, describing the position and direction of each camera when shooting the scene.

[0079] The global point cloud and camera posture are determined by solving the minimum value of the objective function, where the expression of the objective function is as follows:

[0080]

[0081] In the MASt3R model, the initial paired point maps and their associated confidence maps are generated by extracting and processing the features of the input image. It is a preliminary point cloud representation obtained by back-projecting image features into three-dimensional space. The confidence information of these points is provided, which is used to give different weights to points with different confidence levels in the subsequent optimization process. For example, points generated in areas with obvious features and easy matching in the image may have higher confidence levels in the confidence map, which will have a greater impact on the overall results during the optimization process.

[0082] The camera projection matrix P is directly related to the camera posture, which reflects the camera's intrinsic parameters (such as focal length, etc.) and extrinsic parameters (position and direction). During the optimization process, by adjusting P and the transformation matrix J, the points in the point map can be aligned in a unified coordinate system after projection and transformation. This process is actually estimating the camera's posture. For example, when the camera shoots a scene from different angles, the position changes of the feature points in the image in different images are related to the changes in the camera's posture. By minimizing the objective function, the appropriate P and J are found to determine the camera's posture at different shooting positions.

[0083] Among them, in order to find P and σ corresponding to J* that minimizes the objective function, an optimization algorithm such as gradient descent or its variants can be used. During the optimization process, the values ​​of the parameters are gradually adjusted according to the gradient of the objective function with respect to these parameters. For example, the gradient of the objective function with respect to J is calculated, and then J is updated in the opposite direction of the gradient, and this process is repeated until convergence. As the optimization process proceeds, the points in the point map are gradually aligned in a unified coordinate system through the transformation matrix J, and these aligned points constitute a global point cloud. Confidence in the confidence map In this process, a weighting effect is played, so that more reliable points contribute more to the global point cloud, thereby improving the accuracy of the global point cloud. At the same time, the optimized camera projection matrix P and transformation matrix J determine the camera pose. The accuracy of the camera pose is crucial for subsequent 3D Gaussian training because it provides the geometric relationship and observation angle information of the scene, so that the 3D Gaussian model can accurately reconstruct the scene and perform new perspective synthesis based on this information.

[0084] In this way, through the optimization process of the objective function, the MASt3R model can effectively align the point map generated based on image features in a unified coordinate system, thereby obtaining the global point cloud and accurate camera pose, providing key initialization data for the entire three-dimensional Gaussian splash reconstruction technology.

[0085] Optionally, the iterative training of parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model includes:

[0086] Initializing the parameters of the preset three-dimensional Gaussian ellipsoid according to the global point cloud to obtain the parameters of the initialized three-dimensional Gaussian ellipsoid, wherein the parameters include a center position, a spherical harmonic function coefficient, a rotation factor, a scaling factor, and opacity;

[0087] After each scene training is completed, a densification process is performed. The scene training is to use N images to train the center position, the spherical harmonic function coefficients, the rotation factor, the scaling factor and the opacity. The N images are images included in the image sequence, and each of the N images is marked with the acquired camera posture. The densification process is to perform splitting and cloning by calculating whether the average gradient size of the Gaussian point in the observable view exceeds the threshold. The splitting is used to split a Gaussian point into multiple new Gaussian points, and the cloning is used to copy the existing Gaussian points. The number of densification processes is M, M is the average number of iterations of each image in the N images, and the total number of iterative training is N×M, and N and M are both positive integers.

[0088] In this embodiment, the parameters of the preset three-dimensional Gaussian ellipsoid are initialized according to the global point cloud to obtain the parameters of the initialized three-dimensional Gaussian ellipsoid, the parameters including the center position μ∈R 3 , Spherical Harmonic (SH) function coefficients c∈R k , rotation factor r∈R 4 , scaling factor s∈R 3 and opacity α∈R, each Gaussian consists of a mean vector μ and a covariance matrix Σ. Its functional form is as follows:

[0089]

[0090] To ensure the positive semidefiniteness of the covariance matrix Σ during training, it is decomposed into Σ = RSS T R T , these Gaussian distributions are rendered as images through a differentiable renderer, and each pixel C pix The color is calculated via alpha blending as follows:

[0091]

[0092] Then, by calculating the photometric loss between the rendered image and the real image, the back propagation algorithm is used to optimize the parameters of the three-dimensional Gaussian ellipsoid. For example, if the color of a certain area of ​​the rendered image is significantly different from the real image, the photometric loss will be large. Through back propagation, the parameters of the Gaussian points related to the area will be adjusted so that the color of the rendered image in this area is closer to the real image. At the same time, the camera posture information also plays an important role in this process. It affects the changes in the appearance of the scene observed from different perspectives, thereby guiding the model to adjust the parameters of the Gaussian points so that the model can accurately present the real appearance of the scene from different perspectives.

[0093] In addition, during the iterative training process, given N images, the total number of training iterations can be set to N×M, where M is the average number of iterations of each image in the N images. This setting method fully considers the differences between different scenes. In sparse scenes, due to the relatively small number of features, it may not require too many iterations to achieve a good reconstruction effect, so the value of M can be appropriately reduced; while in complex or dense scenes, the scene information is rich, and more iterations are required to fully learn and optimize, and the value of M will increase accordingly. For example, for a simple indoor scene (sparse scene), if there are 30 images, M can be set to 3, and the total number of training iterations is 90; and for a complex urban street scene (dense scene), in the case of 30 images, M can be set to 5, and the total number of training iterations is 150. This adaptive adjustment strategy effectively avoids the problems caused by the fixed number of iterations in related technologies. When facing different scenes, the fixed number of iterations is prone to overfitting (too many iterations in simple scenes, causing the model to over-learn unnecessary information such as noise) or underfitting (insufficient iterations in complex scenes, unable to accurately learn scene features). In this embodiment, by adjusting the number of iterations according to the characteristics of the scene, it is ensured that the model can be properly trained in different scenes, thereby improving the generalization ability and reconstruction quality of the model.

[0094] In addition, after each scene training is completed, a densification process is performed. The densification operation is performed after each round of scene training. This is because the significant change in gradient is most obvious after completing scene training in different areas. During the training process, the three-dimensional Gaussian model continuously adjusts its parameters by learning scene information. After completing a round of training for all N images, the model has a staged understanding of the scene. At this time, the reconstruction of different areas will be clearly reflected in the gradient. For example, if a region has not been fully represented in the current training state, the Gaussian points in the region will show a large change when calculating the gradient, which suggests that a densification operation needs to be performed in the region to promote the growth of Gaussian points in areas where the initial points are insufficient and improve the accuracy of scene representation.

[0095] Among them, the densification process can be performed by calculating whether the average gradient size of the Gaussian points in the observable view exceeds the threshold to perform splitting and cloning. When it is determined that densification is required, the splitting operation is performed by calculating whether the average gradient size of the Gaussian points in the observable view exceeds the threshold. If the average gradient of the area where a Gaussian point is located exceeds the threshold, it means that the area needs a more refined representation. At this time, the Gaussian point will be split into multiple new Gaussian points. For example, a Gaussian point may be split into two or more Gaussian points with different parameters (such as center position, scaling factor, etc.) in space according to the gradient distribution around it. Assuming that a Gaussian point representing the edge of an object has a large gradient change around it, it may be split into two Gaussian points, which extend to the inside and outside of the object respectively to better fit the detailed changes of the edge of the object. The cloning operation is to copy existing Gaussian points in specific cases. When there are some structurally similar or repetitive areas in the scene, cloning Gaussian points can quickly increase the representation accuracy of these areas while reducing the waste of computing resources (compared to regenerating new Gaussian points). The cloned Gaussian points may be fine-tuned according to their position in the scene and the surrounding environment to better adapt to the local scene characteristics. For example, in a wall scene with repeated textures, the Gaussian points corresponding to the learned texture parts are cloned and fine-tuned according to conditions such as lighting at different locations on the wall, which can efficiently reconstruct the texture details of the entire wall.

[0096] Optionally, after w iterations of training, the opacity is reset once, where w is proportional to N.

[0097] In this embodiment, a hyperparameter w can be introduced, and the opacity is reset after w iterations of training. Opacity affects the model's representation of the occlusion relationship and visibility of objects in the scene in a three-dimensional Gaussian model. During the training process, as the number of iterations increases, the opacity may gradually deviate from the optimal value, causing the model to deviate in terms of the occlusion effect in the scene. By resetting the opacity periodically, the model can readjust the representation of the scene to avoid problems caused by unreasonable changes in opacity, such as overfitting or redundant calculations. For example, in a complex scene, there are multiple objects that occlude each other. After a certain number of rounds of training, the opacity may not accurately represent the occlusion relationship. At this time, resetting the opacity can allow the model to relearn and optimize the representation of the occlusion effect and improve the accuracy of scene reconstruction.

[0098] Reasonable setting of the hyperparameter w helps to further improve computational efficiency and reconstruction quality during the global training process. It works in conjunction with other training strategies such as densification operations to dynamically adjust the model training process according to the complexity of the scene. In scenes of different complexities, the value of w can be adjusted to balance the utilization of computing resources and the improvement of reconstruction effects. For example, in a relatively simple scene, the value of w can be appropriately increased to reduce the frequency of opacity reset to speed up training; while in complex scenes, the value of w can be appropriately reduced. In this way, during the global training process, the densification operation can be dynamically enabled according to the complexity of the scene, and the transparency can be reset in a timely manner to avoid overfitting or redundant calculations caused by dense perspectives, further improving computational efficiency and reconstruction quality.

[0099] See also Figure 2 , Figure 2 is a schematic diagram of the structure of a three-dimensional Gaussian splash reconstruction device provided by an embodiment of the present invention, such as Figure 2 As shown, the three-dimensional Gaussian splash reconstruction device 200 includes:

[0100] A processing module 201 is used to group the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence;

[0101] An acquisition module 202, configured to acquire a global point cloud and a camera pose based on paired point graphs determined by the plurality of image pairs and confidence graphs associated with the paired point graphs;

[0102] The training module 203 is used to iteratively train the parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model. The initialized three-dimensional Gaussian ellipsoid is obtained by initializing the preset three-dimensional Gaussian ellipsoid according to the global point cloud. The three-dimensional Gaussian model is used to complete the new perspective synthesis task.

[0103] Optionally, the image sequence includes a first number of images without calibrated poses;

[0104] The processing module 201 is specifically used for:

[0105] According to the first number of images with uncalibrated postures, the preset overlap rate and the single-group preset number, the first number of images with uncalibrated postures are grouped to obtain a second number of image groups, wherein the number of images in each image group in the second number of image groups is the single-group preset number, the overlap rate of adjacent image groups in the second number of image groups is the preset overlap rate, and the sequence of images in the second number of image groups is the same as the sequence of images in the image sequence;

[0106] The first image in the first image group is paired with other images other than the first image in the first image group in sequence to obtain an image pair corresponding to the first image group, wherein the first image group is any group in the second number of image groups.

[0107] Optionally, the acquisition module 202 is specifically used for:

[0108] Inputting the plurality of image pairs into a pre-trained deep learning model to obtain a paired point map corresponding to each image pair in the plurality of image pairs and a confidence map associated with the paired point map;

[0109] Solve the minimum value of the objective function to determine the global point cloud and camera posture. The goal of the objective function is to determine the optimal transformation matrix, camera projection matrix and scaling factor. The paired point map is used to provide scene construction information, and the confidence map is used to measure the credibility of each point in the corresponding point map.

[0110] The objective function is expressed as follows:

[0111]

[0112] J* is the objective function, J is the transformation matrix, P is the camera projection matrix, σ is the scaling factor, E is a set of image pairs, e is any image pair in E, v is a point in e, i is the pixel position, H is the image height, W is the image width, are corresponding points in the point diagram, for After the camera projection matrix P e Project and multiply by the scaling factor σ e After that, the corresponding points are obtained by transformation matrix J. for Confidence in the confidence map.

[0113] Optionally, the training module 203 is specifically used for:

[0114] Initializing the parameters of the preset three-dimensional Gaussian ellipsoid according to the global point cloud to obtain the parameters of the initialized three-dimensional Gaussian ellipsoid, wherein the parameters include a center position, a spherical harmonic function coefficient, a rotation factor, a scaling factor, and opacity;

[0115] After each scene training is completed, a densification process is performed. The scene training is to use N images to train the center position, the spherical harmonic function coefficients, the rotation factor, the scaling factor and the opacity. The N images are images included in the image sequence, and each of the N images is marked with the acquired camera posture. The densification process is to perform splitting and cloning by calculating whether the average gradient size of the Gaussian point in the observable view exceeds the threshold. The splitting is used to split a Gaussian point into multiple new Gaussian points, and the cloning is used to copy the existing Gaussian points. The number of densification processes is M, M is the average number of iterations of each image in the N images, and the total number of iterative training is N×M, and N and M are both positive integers.

[0116] Optionally, after w iterations of training, the opacity is reset once, where w is proportional to N.

[0117] Optionally, the process of obtaining the second number of image groups is as follows:

[0118]

[0119] The number of the uncalibrated posture images is n, the preset overlap rate is r, the single group preset number is t, and the number of the image groups is k.

[0120] The three-dimensional Gaussian splash reconstruction device 200 can implement each process of each embodiment of the above-mentioned three-dimensional Gaussian splash reconstruction method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0121] An embodiment of the present invention further provides an electronic device, including:

[0122] at least one processor; and

[0123] a memory communicatively connected to the at least one processor; wherein,

[0124] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the various processes of the above-mentioned three-dimensional Gaussian splash reconstruction method embodiment and achieve the same technical effect. To avoid repetition, they are not repeated here.

[0125] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned three-dimensional Gaussian splash reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned three-dimensional Gaussian splash reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0127] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A three-dimensional Gaussian splash reconstruction method, characterized in that: The method comprises: Performing group processing on the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence; Acquire a global point cloud and a camera pose based on paired point graphs determined from the plurality of image pairs and confidence graphs associated with the paired point graphs; The parameters of the initialized three-dimensional Gaussian ellipsoid are iteratively trained based on the camera posture to obtain a three-dimensional Gaussian model. The initialized three-dimensional Gaussian ellipsoid is obtained by initializing the preset three-dimensional Gaussian ellipsoid according to the global point cloud. The three-dimensional Gaussian model is used to complete the new perspective synthesis task.

2. The method according to claim 1, characterized in that The image sequence includes a first number of uncalibrated pose images; The acquired image sequence is grouped to obtain a plurality of image pairs, including: According to the first number of images with uncalibrated postures, the preset overlap rate and the single-group preset number, the first number of images with uncalibrated postures are grouped to obtain a second number of image groups, wherein the number of images in each image group in the second number of image groups is the single-group preset number, the overlap rate of adjacent image groups in the second number of image groups is the preset overlap rate, and the sequence of images in the second number of image groups is the same as the sequence of images in the image sequence; The first image in the first image group is paired with other images other than the first image in the first image group in sequence to obtain an image pair corresponding to the first image group, wherein the first image group is any group in the second number of image groups.

3. The method according to claim 1, characterized in that The step of obtaining a global point cloud and a camera pose based on paired point graphs determined based on the plurality of image pairs and confidence graphs associated with the paired point graphs comprises: Inputting the plurality of image pairs into a pre-trained deep learning model to obtain a paired point map corresponding to each image pair in the plurality of image pairs and a confidence map associated with the paired point map; Solve the minimum value of the objective function to determine the global point cloud and camera posture. The goal of the objective function is to determine the optimal transformation matrix, camera projection matrix and scaling factor. The paired point map is used to provide scene construction information, and the confidence map is used to measure the credibility of each point in the corresponding point map. The objective function is expressed as follows: J* is the objective function, J is the transformation matrix, P is the camera projection matrix, σ is the scaling factor, E is a set of image pairs, e is any image pair in E, v is a point in e, i is the pixel position, H is the image height, W is the image width, X i v,e is the corresponding point in the point diagram, J i v For X i v,e After the camera projection matrix P e Project and multiply by the scaling factor σ e After that, the corresponding points obtained by transformation matrix J, C i v,e For X i v,e Confidence in the confidence map.

4. The method according to claim 1, characterized in that: The iterative training of the parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model includes: Initializing the parameters of the preset three-dimensional Gaussian ellipsoid according to the global point cloud to obtain the parameters of the initialized three-dimensional Gaussian ellipsoid, wherein the parameters include a center position, a spherical harmonic function coefficient, a rotation factor, a scaling factor, and opacity; After each scene training is completed, a densification process is performed. The scene training is to use N images to train the center position, the spherical harmonic function coefficients, the rotation factor, the scaling factor and the opacity. The N images are images included in the image sequence, and each of the N images is marked with the acquired camera posture. The densification process is to perform splitting and cloning by calculating whether the average gradient size of the Gaussian point in the observable view exceeds the threshold. The splitting is used to split a Gaussian point into multiple new Gaussian points, and the cloning is used to copy the existing Gaussian points. The number of densification processes is M, M is the average number of iterations of each image in the N images, and the total number of iterative training is N×M, and N and M are both positive integers.

5. The method according to claim 4, characterized in that After w iterations of training, the opacity is reset, where w is proportional to N.

6. The method according to claim 2, characterized in that The process of obtaining the second number of image groups is as follows: The number of the uncalibrated posture images is n, the preset overlap rate is r, the single group preset number is t, and the number of the image groups is k.

7. A three-dimensional Gaussian splash reconstruction device, characterized in that: The device comprises: A processing module, used for grouping and processing the acquired image sequence to obtain a plurality of image pairs, wherein the image sequence includes a plurality of images without calibrated postures, and any image pair in the plurality of image pairs includes two images in the image sequence; An acquisition module, configured to acquire a global point cloud and a camera pose based on paired point graphs determined by the plurality of image pairs and confidence graphs associated with the paired point graphs; A training module is used to iteratively train the parameters of the initialized three-dimensional Gaussian ellipsoid based on the camera posture to obtain a three-dimensional Gaussian model. The initialized three-dimensional Gaussian ellipsoid is obtained by initializing the preset three-dimensional Gaussian ellipsoid according to the global point cloud. The three-dimensional Gaussian model is used to complete the new perspective synthesis task.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so as to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images