Three-dimensional gaussian splatting optimization method for unposed input

Abstract

Disclosed in the present invention is a three-dimensional Gaussian splatting optimization method for an unposed input. The method comprises: for an input image, predicting a ray bundle distribution by using a ray prediction model, to obtain distribution features in the form of ray bundles; on the basis of the distribution features in the form of ray bundles, calculating a camera pose; for the ray bundle distribution, performing sampling on the basis of the volumetric density of light rays, to obtain an initial spatial distribution of a three-dimensional Gaussian point cloud focused on a visual center area; for the input image, obtaining a visible shell by means of view frustum projection and object-mask computation; on the basis of the initial spatial distribution of the three-dimensional Gaussian point cloud and the visible shell, performing three-dimensional Gaussian splatting scene training, to obtain a three-dimensional scene reconstruction model satisfying a preset loss function criterion, the loss function comprising a training regularization term for a camera pose parameter. The present invention provides important scene initialization information for three-dimensional Gaussian splatting training, and significantly improves the quality and the richness of detail of the final three-dimensional structure.

Description

A three-dimensional Gaussian splatting optimization method without pose input TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to a three-dimensional Gaussian splatting optimization method without pose input. BACKGROUND

[0002] Novel view synthesis refers to rendering a picture corresponding to a target pose given source images and source poses and the target pose, which usually involves three-dimensional understanding of the scene. Novel view synthesis has wide applications in 3D reconstruction, AR / VR, etc. In recent years, deep learning, especially convolutional neural networks, neural radiance fields, and three-dimensional Gaussian splatting models, have played a key role in novel view synthesis. They can learn complex scene representations and lighting models to generate more realistic images. For example, three-dimensional Gaussian splatting (3DGaussian Splatting) uses a series of Gaussian functions (usually 3D Gaussian distributions) to represent objects in the scene. The scene is decomposed into many Gaussian ellipsoids, each with its own center position, color, direction, and size. During rendering, these ellipsoids are rasterized into pixels, and the color value of each pixel is determined through an integration or sampling process, resulting in the final image. Compared with traditional volume rendering or mesh-based rendering, three-dimensional Gaussian splatting can achieve real-time rendering speed.

[0003] With the rapid development of three-dimensional Gaussian splatting technology, novel view synthesis technology has made significant breakthroughs in rendering quality, efficiency, and interactivity, effectively promoting the widespread expansion of visual industry applications. These applications cover digital humans, autonomous driving scene simulation, three-dimensional content generation, unmanned aerial vehicle rapid mapping, large-scale scene reconstruction, and three-dimensional language fields. Typically, such commercial applications require input of multi-view scene images, and then perform scene reconstruction on high-performance server side. Although the training efficiency of novel view synthesis applications has been significantly improved, the quality of the reconstruction results is limited due to its reliance on sparse point clouds estimated by the Structure-from-Motion technology during camera calibration. Structure-from-Motion is a method for automatically recovering the three-dimensional structure of a scene and the internal and external parameters of a camera from multiple images or video sequences. It usually estimates the depth of the scene and the trajectory of the camera based on geometric constraints and motion models. The workflow of Structure-from-Motion includes feature detection, feature matching, relative pose estimation, triangulation, global optimization, and dense reconstruction. The main challenges faced by Structure-from-Motion include handling occlusions, lighting changes, low-texture areas, and image noise, which can cause feature matching errors and inaccurate reconstruction. This dependence on the matching efficiency and accuracy of Structure-from-Motion often means that users need to wait for a long time for service-oriented applications.

[0004] In addition, supporting multi-view pictures captured by arbitrary devices in arbitrary ways is crucial for new view synthesis applications, especially for application scenarios that aim to popularize services on common mobile devices. Camera pose estimation methods based on motion recovery structure face severe challenges in the face of sparse photo input with insufficiently labeled pose data. In environments with few views, there may be a lack of sufficient matching textures, which directly affects the accuracy of camera pose estimation.

[0005] To overcome the low efficiency of the motion recovery structure method in the three-dimensional Gaussian sputtering initialization stage and the inaccuracy of camera pose estimation, there are currently two main improvement strategies:

[0006] 1) Adopt a more accurate and robust camera pose estimation method. The core of this direction is to use a deep neural network based on a visual Transformer architecture to improve the accuracy and robustness of camera pose estimation. The Transformer model has strong sequence modeling capabilities and parallel computing advantages, and it performs well in handling complex spatial relationships and long-distance dependencies. Specifically, by introducing a self-attention mechanism, the Transformer can effectively capture the relationships between elements in the input sequence, thereby achieving accurate estimation of camera pose without explicit geometric constraints. In addition, the training process of the deep neural network can automatically learn rich feature representations, further enhancing the adaptability to occlusion, lighting changes, and dynamic scenes, and improving the reliability of the estimation results.

[0007] 2) Camera pose as part of the iterative training parameters. This strategy involves incorporating camera pose parameters into the iterative training process, treating them as learnable variables rather than fixed and unchanging prior knowledge. The core idea of this method is to use gradient descent algorithms to minimize loss functions such as re-projection error or photometric error to optimize camera pose parameters. During training, the model will continuously adjust the camera pose until it converges to the optimal estimate.

[0008] Through analysis, the existing technology mainly has the following defects:

[0009] 1) Motion recovery structure technology can recover the camera's motion trajectory and the three-dimensional structure of the scene from a series of images. In this process, not only can the camera's pose be inferred, but also the scene point cloud generated is a key basis for scene initialization in three-dimensional Gaussian sputtering training, which has an important impact on the final training effect of the scene. Although the method of using a deep neural network can improve the accuracy of camera pose prediction under sparse views, if it directly replaces the motion recovery structure method of three-dimensional Gaussian sputtering, it will inevitably lead to a loss of accuracy in scene initialization, and thus the reconstruction result will appear blurred.

[0010] 2) The combination of camera pose parameters and iterative training of three-dimensional Gaussian sputtering scenes, although theoretically capable of providing a more accurate model optimization approach, due to the deep integration of the two, often leads to an exceptionally complex model optimization process in actual operation. This over-coupling not only makes it difficult for the model to achieve a global optimal solution, but also can cause training instability and other problems, limiting the further improvement of the model performance. More critically, the inclusion of camera pose parameters in the training process means that each iteration requires recalculations and adjustments, significantly increasing the number of iterations and extending the overall training time, posing a serious challenge to computing resources and time efficiency.

[0011] In summary, in the field of new view synthesis technology, camera pose information is an indispensable part of scene training. Existing technologies either presuppose known camera pose information or rely on motion recovery structure algorithms applied from dense views to estimate the camera's position. This process is accompanied by high time costs, especially due to the time-consuming nature of comprehensive matching calculations. In addition, in the face of occluded viewing angles or highly repetitive texture structures in sparse scenes, motion recovery structure algorithms often struggle to accurately recover camera poses, leading to inaccurate positioning. Furthermore, the point cloud data generated by the motion recovery structure, while serving as the starting point for neural field training to construct a scene model, is limited in quality by the uncertainty of the above-mentioned pose estimation.

[0012] SUMMARY

[0013] The purpose of the present application is to overcome the above-mentioned deficiencies of the prior art and provide a three-dimensional Gaussian sputtering optimization method without pose input. The method comprises the following steps:

[0014] For the input image, a light ray prediction model is used to predict the light ray bundle distribution, obtaining distribution features in the form of light ray bundles, including the momentum of the light ray, the direction of the light ray, and the volume density of the light ray;

[0015] Based on the distribution features in the form of light ray bundles, the camera pose is calculated;

[0016] For the light ray bundle distribution, sampling is performed based on the volume density of the light ray to obtain an initial spatial distribution of three-dimensional Gaussian point clouds focused on the visual center region;

[0017] For the input image, a visible shell is obtained through view cone projection and object mask calculation, which reflects the structural information of the scene objects;

[0018] Based on the initial spatial distribution of the three-dimensional Gaussian point clouds and the visible shell, a three-dimensional Gaussian sputtering scene training is performed to obtain a three-dimensional scene reconstruction model that meets the set loss function standard, wherein the loss function includes a training regularization term for camera pose parameters.

[0019] Compared with the prior art, the advantage of the present application is that a three-dimensional Gaussian sputtering scene initialization scheme based on camera light ray prediction is proposed. The scheme deeply integrates the camera pose estimation method with the scene reconstruction process, and constructs an end-to-end optimization framework. Through the integrated design, the camera pose and the scene structure are optimized jointly, so that more accurate and efficient model training is realized. On the basis of ensuring the accuracy of camera pose estimation, the present application accelerates the initialization process of training scene, and improves the accuracy of synthesizing new views by using the three-dimensional Gaussian sputtering method under the condition of sparse no-pose view.

[0020] Other features of the present application, and their advantages, will become apparent from the following detailed description of illustrative embodiments of the present application, when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0022] Fig. 1 is a schematic diagram of the overall process of the three-dimensional Gaussian sputtering optimization method for no-pose input according to one embodiment of the present application;

[0023] Fig. 2 is a flowchart of the three-dimensional Gaussian sputtering optimization method for no-pose input according to one embodiment of the present application;

[0024] Fig. 3 is a schematic diagram of a density light ray prediction model according to one embodiment of the present application;

[0025] Fig. 4 is a schematic diagram of density-based light ray sampling according to one embodiment of the present application;

[0026] Fig. 5 is a schematic diagram of initializing a three-dimensional scene by combining visual hull and light beam density information according to one embodiment of the present application. DETAILED DESCRIPTION

[0027] Various illustrative embodiments of the present application will now be described in detail with reference to the accompanying figures. It should be noted that the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not limiting to the scope of the present application, unless otherwise specifically stated.

[0028] The following description of at least one illustrative embodiment is merely exemplary in nature and is in no way intended to limit the present application or its application or uses.

[0029] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0030] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0031] It should be noted that like numerals and letters refer to like items throughout the drawings, and once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0032] Three-dimensional Gaussian Splatting as an important innovation in the field of novel view synthesis has the potential to reshape the digital content production pipeline. The present invention can effectively expand the generalization capability and hardware scenarios of novel view synthesis technology by solving the scene training problem under sparse pose-free views, thereby promoting the development of visual applications such as digital humans, autonomous driving scene simulation, three-dimensional content generation, unmanned aerial vehicle rapid mapping, large-scale scene reconstruction, and three-dimensional language field, and facilitating the popularization of novel view synthesis applications to common mobile devices.

[0033] Overall, referring to FIG. 1, the provided three-dimensional Gaussian Splatting optimization method for pose-free input mainly includes four parts: camera density ray prediction, three-dimensional Gaussian scene initialization (three-dimensional Gaussian initial training scene), visual shell redundancy elimination, and camera pose regularization.

[0034] Camera density ray prediction is derived from camera ray prediction. The core idea of camera ray prediction is that neural network learning usually benefits from over-parameterized distributed representation, so instead of using compact camera feature representation such as intrinsic and extrinsic matrix, a set of ray bundles is used to represent the possible orientation direction of the camera. The ray bundle converges to the camera center point from the pixel point of the image, and is represented by Plucker coordinates. The task of the neural network is to recover the position and direction of the ray bundle from a set of images as much as possible, and then according to the convergence position of the ray bundle, for example, using least squares optimization to solve the intrinsic and extrinsic properties of the camera. The camera density ray prediction of the present invention supports ray information with density to provide prior information for scene initialization, which will be described in detail later.

[0035] Three-dimensional Gaussian scene initialization initially initializes three-dimensional Gaussian points with sparse point clouds of scene structures derived by motion recovery structure. In order to form a more accurate pose estimation method and form an end-to-end training process, the present invention provides prior information for scene initialization by means of over-parameterized information of camera density rays, thereby integrating camera pose estimation and initialization of three-dimensional Gaussian scene. Three-dimensional Gaussian points are sampled on rays according to density relationship, rather than using conventional uniform random sampling.

[0036] In order to realize the visual shell redundancy elimination, the depth estimation information of the image is obtained from the input multi-view picture, the visual shell of the scene is constructed, and then the wrong initialization point cloud position outside the shell is eliminated. In the training process of the three-dimensional Gaussian scene, the visual shell provides the rough structure information of the scene object, which is used to eliminate the unreasonable Gaussian point cloud distribution, thereby suppressing the generation of visual floating objects and maintaining the consistency of the scene.

[0037] In the training process of the three-dimensional Gaussian scene, the training regular term of the camera pose parameter is added, and the camera pose parameter is adjusted to realize the scene reconstruction effect with higher precision.

[0038] In the present application, in the initialization process of the three-dimensional Gaussian scene, facing the sparse picture (or image) input without pose annotation, first, the depth information and pose information are estimated for joint method optimization. Specifically, referring to FIG. 2, the provided three-dimensional Gaussian sputtering optimization method for input without pose includes the following steps:

[0039] In step S210, for the input image, the light ray prediction model is used to predict the light ray bundle distribution, and the distribution characteristics in the form of light ray bundle are obtained, including the momentum of the light ray, the direction of the light ray and the body density of the light ray.

[0040] Camera density light ray prediction is a proxy task of camera pose estimation, which outputs a large number of light ray distribution parameters. The camera light ray method only provides the function of recovering the camera pose from multiple pictures, and does not have the function of deriving the initialization point cloud like the motion recovery structure algorithm. In the embodiment of the present application, the light ray features in the Plucker coordinates are improved to light ray features with density by improving the constructed light ray distribution representation. Based on the characteristics of three-dimensional Gaussian sputtering body rendering, the light ray with body density information is predicted, which provides a basis for sampling the initial three-dimensional Gaussian training scene from the light ray bundle.

[0041] FIG. 3 is a light ray prediction model with density, which is constructed based on a visual Transformer model, wherein the pose parameter λ of the camera is encoded as a series of light rays p i represents any point on the i-th light ray, d i represents the direction vector of the i-th light ray. The position information of each light ray in space is represented by Plucker coordinates in the prediction model. In order to realize the initialization of the three-dimensional Gaussian scene, in an embodiment, a key parameter of light ray density ρ is introduced. Therefore, each light ray can be specifically represented as an extended form of l i =[d i ;m i ;ρ i ], wherein d i and m i are the direction and momentum features of the light ray, respectively, and the momentum feature is represented by mi = p i × d i calculated. p i is the improved spatial density information carried by the light rays. Specifically, the input image I is processed by block processing to form a sequence S, and the sequence with position encoding φ(·) is input into the visual Transformer model V to predict the light ray direction d, momentum m and density p corresponding to the pixel block, which is represented as:

[0042] where

[0043] wherein, represents the predicted light beam distribution parameters, including a series of parameter information of light rays. i represents the image block position, y i represents the image block input information, φ(x i ) represents the position encoding information corresponding to the image block, and N represents the number of image blocks. In the following, φ(x is used to refer to the light beam distribution, unless otherwise indicated according to the context.

[0044] In summary, the light prediction model can obtain the distribution characteristics in the form of light beams, including the momentum of light rays, the direction of light rays and the density of light rays, etc.

[0045] In step S220, the camera pose is calculated based on the distribution characteristics in the form of light beams.

[0046] The camera pose estimation is based on the distribution position of a large number of light rays, and the camera pose associated with multiple perspectives is calculated from the least square optimization. For example, for the light beams The camera light prediction method solves the internal and external parameters λ of the camera by the least square method, which includes specific pose parameters θ.

[0047] Since the input of the three-dimensional Gaussian sputtering technology is usually multiple pictures of multiple perspectives, the camera pose calculation considers the joint optimal solution of the light beams of multiple pictures of the same perspective, and benefits from the distributed representation in the form of light beams. These light beams can be combined into a light beam for least square optimization calculation.

[0048] In one embodiment, in order to further improve the accuracy of pose prediction and cope with the prediction uncertainty caused by the lack of perspective in sparse views, the predicted light ray direction and momentum can be further adjusted. For example, by means of the diffusion model, the light rays are added with noise and denoised, so as to obtain more rich light beam distribution characteristics, and then obtain more accurate light prediction under sparse views.

[0049] ​In summary, in step S220, a proxy task for predicting rays is used to recover the camera's pose information based on the camera ray method. Considering the application characteristics of volume rendering technology in 3D Gaussian scene modeling, the prediction task is improved by adding the key dimension of "density," enabling the output of density information related to the initial scene while acquiring the camera pose. This density information carried by rays essentially represents the reverse operation of the volume rendering process, that is, deriving the scene structure from the density information. The Gaussian point cloud of the scene is initialized using the density information, providing a foundation for training a high-precision 3D Gaussian scene.

[0050] Step S230: Based on the volume density of the light beams, sample the distribution of the light beams to obtain the initial spatial distribution of the three-dimensional Gaussian point cloud focused on the visual center region.

[0051] In this step, the initialization of the 3D Gaussian scene is achieved within the camera ray prediction framework with density, that is, the distribution of Gaussians in 3D space is initialized based on ray density sampling.

[0052] The process begins with a proxy task, namely camera light prediction, which acquires a large number of light parameters. Each ray of light i In the Plück coordinate system, the direction vector d i and momentum vector m i Definition. By analyzing these ray parameters, their positions can be reconstructed in 3D space, and the volume density ρ of the rays can be used to reveal the point cloud distribution characteristics of the scene. This strategy utilizes the surrogate task of regression prediction of ray bundles, which is employed by camera ray pose estimation methods, to capture prior information closely related to camera pose by analyzing the distribution characteristics of ray bundles in scene space.

[0053] Specifically, starting from the image pixel, along the light path passing through the center of the camera. Samples (M is the number of ray paths) are extracted and used as the starting point cloud positions for the 3D Gaussian sputtering scene reconstruction. Each ray l i Both map from three-dimensional world space to two-dimensional image space By sampling the light beams, the initial point cloud layout concentrated in the visual center region can be obtained from the projection trajectory.

[0054] Figure 4 is a schematic diagram of the initial spatial distribution for constructing a 3D Gaussian point cloud based on density-based ray sampling, specifically including the following steps:

[0055] Step S41, convert the ray parameters in the Plücker coordinate system Mapping back to the actual positions in three-dimensional space yields the distribution of light rays in three-dimensional space.

[0056] Step S42: Initialize a set of voxel meshes in three-dimensional space. And according to certain rules, the light It is divided into multiple segments. Each voxel mesh... It may contain multiple rays. Each ray of light also carries its density distribution along its path. k is the voxel grid index, and i is the ray index that intersects with the voxel grid. Let L be the independent variable, representing the grid width. This represents the ray beam that intersects with the voxel grid.

[0057] Step 43: Based on this information, initialize a 3D Gaussian sphere at the center of the voxel grid. The size of the Gaussian point is based on the integral of multiple ray densities within the voxel grid, thus forming the initial spatial distribution of the 3D Gaussian point cloud. According to the ray density information, in each voxel grid v... k Initialize a three-dimensional Gaussian distribution point g at the center position. k Its size s is determined based on the integral density of light within that voxel. This allows us to construct the initial spatial distribution of a 3D Gaussian point cloud. K represents the number of Gaussian kernels in the scene.

[0058] These Gaussian points will be further adjusted in subsequent scene density control training. Multiple small Gaussian points will be combined into larger Gaussian points through deletion and gradient updates. During the subsequent scene density control training phase, these Gaussian points will undergo further optimization and adjustment, with smaller Gaussian points g... k Through gradient descent or merging operations, the scene evolves into larger Gaussian points, ultimately forming a more refined 3D scene representation. This process not only enhances the scene's geometry but also improves the accuracy of scene density prediction.

[0059] In summary, this invention uses Transformer to predict ray density distribution for initializing scene point clouds, and samples initial 3D Gaussian points through voxel mesh integration to form the initial spatial distribution of the 3D Gaussian point cloud. This approach reduces the number of iterations of the density control algorithm during subsequent scene training and improves the accuracy of rendering results.

[0060] Step S240: Obtain a visual shell that reflects the structural information of objects in the scene through view frustum projection and object mask calculation.

[0061] To further enhance the quality and robustness of three-dimensional scene training, in an embodiment, a visible shell is introduced, such as constructing a three-dimensional visual shell based on two-dimensional depth segmentation information to eliminate redundancy. The visible shell combines the structural prior knowledge of objects in the input image. For example, the visible shell is calculated through frustum projection and object mask. Compared with the sparse point cloud generated simply by sampling, the visible shell can better retain the structural consistency required for multi-view reconstruction and has the ability to exclude invalid point cloud data that may lead to unreasonable Gaussian distribution. FIG. 5 is a schematic diagram of initializing a three-dimensional scene with a visible shell combined with ray beam density information, which constitutes a scene density grid suitable for initialization.

[0062] It is worth mentioning that the calculation cost of the visible shell is relatively low, which only needs to be generated from a small number of key mask views with the help of the segmentation model, so it can be easily integrated into the initialization process to achieve an efficient and accurate three-dimensional scene reconstruction starting condition.

[0063] In an embodiment, during the three-dimensional scene reconstruction process, the three-dimensional grid constructed in combination with the visible shell includes the following steps:

[0064] Step S51, extracting depth information and segmentation mask of the two-dimensional picture from the semantic segmentation model Where I i represents i input images, D i and M i correspond to its depth map and segmentation mask respectively, and N is the number of images.

[0065] Step S52, obtaining the camera intrinsic parameter matrix K and the extrinsic parameter matrix E of the picture to determine the conversion relationship between the camera coordinate system and the world coordinate system.

[0066] Step S53, projecting the initialized scene voxel grid onto the two-dimensional plane through the camera parameter matrices K and E to obtain the corresponding two-dimensional projection grid

[0067] Step S54, filtering the projection of non-reconstruction object content using the object mask M i to filter the two-dimensional projection grid and exclude the content of non-reconstruction target objects, thereby forming a three-dimensional grid filtered by the visible shell in the three-dimensional space This process ensures the accuracy and effectiveness of three-dimensional reconstruction while reducing the complexity of subsequent processing.

[0068] In summary, the redundant point cloud elimination technology based on visual shell can effectively accelerate the model training speed while significantly suppressing the emergence of unreasonable visual elements such as visual floaters. The visual shell technology significantly improves the efficiency and accuracy of the three-dimensional scene initialization stage, not only reducing unnecessary computational burden, but also laying a solid foundation for subsequent Gaussian point cloud optimization and scene density control, ensuring the clarity and coherence of the reconstruction results,

[0069] In step S250, based on the initial spatial distribution of the three-dimensional Gaussian point cloud and the visual shell, a three-dimensional Gaussian scene training is performed with the loss function minimization as the optimization objective, and a three-dimensional scene reconstruction model is obtained, wherein the loss function includes a training regularization term of the camera pose parameters.

[0070] In this step, the three-dimensional scene training is initialized based on the initial spatial distribution of the three-dimensional Gaussian point cloud and the visual shell until the set loss function standard is met. During the training process, the loss function includes a training regularization term of the camera pose parameters, and by adjusting the pose parameters during the three-dimensional Gaussian scene training, a higher-precision scene reconstruction effect can be achieved.

[0071] When processing scenes with multiple perspectives, the correction of camera extrinsic parameters becomes particularly important due to the possible scale differences between different perspectives. In addition, due to the inconsistency of illumination between perspectives or slight motion blur caused by handheld devices during shooting, the final correction effect may not be satisfactory. Therefore, the present application simultaneously optimizes the camera extrinsic parameters and the three-dimensional model during the scene training process. In this process, the camera parameters and Gaussian attributes jointly participate in the adjustment to achieve more precise control. In addition, the present application imposes a certain regularization condition λ · ‖E-E0‖ on the camera parameters. Wherein, E0 is the camera extrinsic parameter matrix of the initial pose predicted by the camera ray method, which is also the global camera pose used for scene training, E represents the camera extrinsic parameter matrix of the current perspective, and λ is the coefficient of the regularization term. In order to prevent the optimized camera pose from deviating too much, thereby optimizing the alignment quality in the entire three-dimensional Gaussian scene training process.

[0072] In one embodiment, the loss function is set as:

[0073] The loss function represents the optimization process of the three-dimensional Gaussian scene and the camera pose parameters. Wherein, represents the optimized three-dimensional Gaussian scene, E j* represents the optimized camera extrinsic parameter matrix under the current perspective j, is the initial three-dimensional Gaussian scene to be optimized in the training process, E j represents the camera extrinsic parameter matrix to be optimized under the current perspective j. represents the three-dimensional Gaussian scene to be optimized rendering the 2D image under the pose parameter corresponding to the view angle j. represents the real input image under the view angle j, and N represents the number of input images corresponding to the view angle j. For the regularization term, lambda is the coefficient of the regularization term, represents the initial camera extrinsic matrix associated with the view angle j, and V represents the view angle classification of the input image.

[0074] During the training process, the effective addition of the regularization term for the camera pose optimizes the real pose of the camera through the neural network. Since the camera ray method provides an initial term for the camera pose parameter, only a small amount of constraints and training costs are required to make the estimation of the camera pose more accurate and robust, so as to obtain more accurate scene reconstruction results from sparse views.

[0075] After the training is completed, the model of the three-dimensional scene reconstruction can be obtained, and then applied to new view synthesis. The model can synthesize multi-view new views for input images without pose.

[0076] In summary, compared with the prior art, the present application has the following advantages:

[0077] 1) The present application proposes a scene initialization strategy for sparse input without pose, aiming at the emerging three-dimensional Gaussian sputtering method in the field of new view synthesis technology. The strategy can efficiently realize accurate estimation of camera pose and joint initialization of the scene. The present application combines the advantages of the camera ray method, ensures the efficiency and accuracy of camera pose estimation in the motion recovery structure, and provides important initialization scene information for three-dimensional Gaussian sputtering training, which significantly improves the quality and detail richness of the final three-dimensional structure.

[0078] 2) The present application provides an end-to-end implementation scheme for the three-dimensional Gaussian sputtering technology for sparse input without pose. Under the premise of ensuring the quality and consistency of scene synthesis, the efficiency of three-dimensional Gaussian sputtering scene initialization is greatly improved. By integrating the camera ray method into the three-dimensional Gaussian sputtering training process, the non-well-posed problem of camera pose prediction caused by insufficient information and texture repetition under sparse views is solved. The estimation accuracy and efficiency of the camera ray method under sparse views are superior to the current motion recovery structure method.

[0079] 3) The present application makes full use of the proxy task of light prediction, increases the dimension of light density, and realizes the prediction of three-dimensional space structure density information. The camera pose prediction and three-dimensional Gaussian scene initialization process are combined to initialize the three-dimensional Gaussian point cloud, which ensures the accuracy and rendering quality of the training scene.

[0080] 4) The application obtains the depth information of the picture by means of the visual depth segmentation model, projects the two-dimensional scene into a three-dimensional visual shell, thereby constraining the emergence of redundant point clouds in the training process, reducing unreasonable visual floating objects, and reducing the computational burden of training.

[0081] 5) The application combines the training method of camera pose optimization, further corrects the inconsistency of camera pose in multi-view pictures, thereby improving the accuracy of camera pose estimation under sparse view. At the same time, the regular term is used to constrain the excessive deviation of the initial camera pose prediction value, and the update is terminated in advance to avoid introducing excessive computational cost, which can optimize the alignment quality in the whole three-dimensional Gaussian sputtering scene training process with less computational amount.

[0082] 6) After multiple simulation tests, using the application, whether for input images with or without pose, multi-view pictures meeting the expected requirements can be synthesized, improving the computational efficiency while improving the quality of the synthesized images.

[0083] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the application.

[0084] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanism that encodes instructions on a punch card or in a punch hole, and any suitable combination of the foregoing. The computer readable storage medium used here is not to be interpreted as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (for example, an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0085] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0086] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0087] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0088] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all. The computer readable storage medium can also have instructions stored thereon or therein which may

[0089] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0090] The flow diagrams and block diagrams in the accompanying drawings show archi tectures, functional and operational architectures of possible implementations of systems, methods and computer program products according to the present application. In this regard, each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of instructions which comprise one or more executable instructions for implementing the specified logical functions (acts). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in some cases, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0091] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art, without departing from the scope and spirit of the described embodiments. The selection of terms to be used in the description is intended to best explain the principles of the embodiments, the practical application, or technical improvement over the prior art, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the application is defined by the claims appended hereto.

Claims

1. A method for three-dimensional Gaussian Splatting optimization without pose input, comprising the following steps: For an input image, predicting a light beam distribution using a light ray prediction model, obtaining distribution features in the form of light beams, including momentum of light rays, direction of light rays and volume density of light rays; Based on the distribution features in the form of light beams, calculating camera pose; For the light beam distribution, sampling based on the volume density of light rays, obtaining an initial spatial distribution of three-dimensional Gaussian point cloud focusing on the visual center area; For the input image, obtaining a visible shell through view frustum projection and object mask calculation, the visible shell reflecting the structural information of scene objects; Based on the initial spatial distribution of three-dimensional Gaussian point cloud and the visible shell, performing three-dimensional Gaussian Splatting scene training to obtain a three-dimensional scene reconstruction model meeting the set loss function standard, wherein the loss function contains a training regularization term of camera pose parameters.

2. The method of claim 1, wherein, The step of predicting a light beam distribution using a light ray prediction model comprises: After the input image I is processed by blocking, an image block sequence S is formed; The sequence with position encoding is input into the light ray prediction model V to predict the direction d, momentum m and body density p of the light ray, denoted as: wherein is the predicted light ray bundle distribution, the pose parameters λ of the camera are encoded as light ray bundles Each light ray is represented by its extended Plücker coordinates vector l = [d; m; p], d and m being respectively the direction and momentum features of the light ray, p being the spatial volume density information carried by the light ray, the light beam Each light ray l i The spatial position information is defined by a direction vector d i and a momentum vector m i in the Poincare coordinate system, x i denotes the image block position, y i denotes the image block input information, φ(x i ) denotes the position encoding information corresponding to the image block, and N denotes the number of image blocks.

3. The method of claim 1, wherein, The camera pose is calculated using the least squares method, for multiple pictures with the same view angle, based on the distribution representation in the form of light beams, the light rays are combined into a light beam, and then the internal and external parameters of the camera, including the pose parameters, are solved by the least squares method.

4. The method of claim 1, wherein, The initial spatial distribution of three-dimensional Gaussian point cloud focusing on the visual center area is obtained according to the following steps: For the light ray bundle distribution, starting from an image pixel, along the light ray path through the camera center Extracting samples to obtain an initial spatial distribution of three-dimensional Gaussian point clouds focusing on the central region of vision as the starting point cloud position of three-dimensional Gaussian sputtering scene reconstruction, where M is the number of extracted light paths, P j represents the jth light path.

5. The method of claim 2, wherein, The initial spatial distribution of three-dimensional Gaussian point cloud focusing on the visual center area is obtained according to the following steps Distributing light beams in pruik coordinates mapping back to the actual positions in the three-dimensional space, resulting in a distribution of the light rays in the three-dimensional space Initializing a set of voxel grids in three-dimensional space and divide the light rays into multiple segments according to the set rules, wherein each voxel grid A plurality of light rays Each ray carries its density distribution along the path where k is the voxel grid index, i is the ray index intersecting the voxel grid, z is the independent variable, and L represents the grid width, The light beams intersecting with the voxel grid are represented; A three-dimensional Gaussian spherical point is initialized at the center of the voxel grid, the size of the Gaussian point is based on the integration of the density of multiple light rays in the voxel grid, forming the initial spatial distribution of three-dimensional Gaussian point cloud; According to the light ray body density information, a three-dimensional Gaussian distribution point g k is initialized at the center position of each voxel grid v k , and the size s of the three-dimensional Gaussian distribution point g obtaining an initial spatial distribution of the three-dimensional Gaussian point cloud Where K represents the number of Gaussian kernels in the scene.

6. The method of claim 5, wherein, In the three-dimensional Gaussian Splatting scene training process, a three-dimensional grid is constructed according to the following steps: The depth information and segmentation mask of the two-dimensional image are extracted from the semantic segmentation model, denoted as where I i represents i-th input image, D i is a depth map of I i , M i is a segmentation mask of I i ; Obtain the camera internal parameter matrix K and the external parameter matrix E of the image to determine the conversion relationship between the camera coordinate system and the world coordinate system; initializing a scene voxel grid The corresponding two-dimensional projection grid is obtained by projecting into a two-dimensional plane through the camera intrinsic parameter matrix K and the extrinsic parameter matrix E Utilizing object mask M i To a two-dimensional projection grid filtering, excluding the contents of non-reconstruction target objects, forming a filtered three-dimensional grid in three-dimensional space 7. The method of claim 6, wherein, The loss function is set as: wherein is the optimized three-dimensional Gaussian scene, E j* denotes the optimized camera extrinsic matrix of the current view j, is the initial three-dimensional Gaussian scene to be optimized in the training process, E j represents the current camera extrinsic matrix to be optimized under the view angle j, representing a three-dimensional Gaussian scene to be optimized rendering a 2D image from the j-th view angle under the corresponding pose parameter, represents the real input image at the i-th under the view j, N represents the number of input images corresponding to the view j, λ is the coefficient of the regularization term, E j represents the camera extrinsic matrix of the current view j, V represents the view classification of the input image.

8. The method of claim 1, wherein, Further comprising: For the actually collected image, the corresponding multi-view target image is obtained using the trained three-dimensional scene reconstruction model.

9. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8. The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images