Three-dimensional Gaussian sputtering optimization method for pose-free input
Through the light prediction model and cone projection technology, the problems of inaccurate camera pose estimation under sparse poseless view in the three-dimensional Gaussian sputtering method are solved, and more efficient and accurate three-dimensional scene reconstruction is achieved. .
Patent Information
- Application Number
- CN202410997744.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-22
AI Technical Summary
In the new view synthesis, especially in the three-dimensional Gaussian sputtering method, the prior art faces the problems of inaccurate camera pose estimation under sparse pose-free view and low scene initialization efficiency, resulting in blurred reconstruction results and large computing resources consumption.
The light prediction model is used to predict the light beam distribution, calculate the camera pose, and obtain the visual shell through the visual cone projection and object mask. The end-to-end optimization framework is combined with the three-dimensional Gaussian point cloud to achieve the combination of the camera pose and scene structure. optimization.
The accuracy and efficiency of the three-dimensional Gaussian sputtering method in sparse pose-free view is improved, the accuracy of scene training and the optimization of computing resources is ensured, and the popularization of new view synthesis technology on mobile devices.
Smart Images

Figure CN119006678A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to a three-dimensional Gaussian sputtering optimization method for non-pose input. Background Art
[0002] New view synthesis refers to rendering and generating images corresponding to the target pose given a source image, source pose, and target pose, which usually involves a three-dimensional understanding of the scene. New view synthesis has a wide range of applications in 3D reconstruction, AR / VR, and other fields. In recent years, deep learning, especially models such as convolutional neural networks, neural radiance fields, and 3D Gaussian splatting, has played a key role in new view synthesis. They can learn complex scene representations and lighting models to generate more realistic images. For example, 3D Gaussian Splatting uses a series of Gaussian functions (usually 3D Gaussian distributions) to represent objects in the scene. The scene is decomposed into many Gaussian ellipsoids, each with its own center position, color, orientation, and size. When rendering, these ellipsoids are rasterized into pixels, and the color value of each pixel is determined by an integration or sampling process to produce the final image. Compared with traditional volume rendering or grid-based rendering, 3D Gaussian splatting can achieve real-time rendering speed.
[0003] With the rapid development of 3D Gaussian sputtering technology, new view synthesis technology has made significant breakthroughs in rendering quality, efficiency and interactivity, which has strongly promoted the wide expansion of visual industry applications. These applications cover digital humans, autonomous driving scene simulation, 3D content generation, drone rapid mapping, large-scale scene reconstruction, and 3D language fields. Usually, such commercial applications require input of multi-view scene images and then perform scene reconstruction on a high-performance server. Although the training efficiency of new view synthesis applications has been significantly improved, the quality of reconstruction results is limited because its construction basis relies on the sparse point cloud estimated by the Structure-from-Motion technology during the camera calibration stage. Motion recovery structure is a method that automatically recovers the 3D structure of the scene and the internal and external parameters of the camera from multiple images or video sequences. It usually estimates the depth of the scene and the trajectory of the camera based on geometric constraints and motion models. The workflow of motion recovery structure includes feature detection, feature matching, relative pose estimation, triangulation, global optimization, and dense reconstruction. The main challenges faced by motion restoration structure at present include dealing with occlusion, illumination changes, low texture areas and image noise, which can lead to feature matching errors and inaccurate reconstruction. This reliance on the efficiency and accuracy of motion restoration structure matching often means that users need to endure long waits for service-oriented applications.
[0004] In addition, supporting multi-view images captured by any device in any way is crucial for new view synthesis applications, especially for application scenarios that popularize services on common mobile devices. Faced with sparse photo inputs without sufficient annotated pose data, the camera pose estimation method based on motion recovery structure faces severe challenges. In environments with sparse viewpoints, there may be a lack of sufficient matching textures, which directly affects the accuracy of camera pose estimation.
[0005] In order to overcome the problems of low efficiency of the motion recovery structure method in the 3D Gaussian sputtering initialization stage and inaccurate camera pose estimation, there are currently two main improvement strategies:
[0006] 1) Adopt more accurate and robust camera pose estimation methods. The core of this direction is to use deep neural networks based on the visual Transformer architecture to improve the accuracy and robustness of camera pose estimation. The Transformer model, with its powerful sequence modeling capabilities and parallel computing advantages, has demonstrated excellent performance in processing complex spatial relationships and long-distance dependencies. Specifically, by introducing the self-attention mechanism, the Transformer can effectively capture the relationship between elements in the input sequence, thereby achieving accurate estimation of the camera pose without explicit geometric constraints. In addition, the training process of the deep neural network can automatically learn rich feature representations, further enhancing the adaptability to occlusion, lighting changes, and dynamic scenes, and improving the reliability of the estimation results.
[0007] 2) Use the camera pose as part of the iterative training parameters. This strategy is to incorporate the camera pose parameters into the iterative training process and treat them as learnable variables rather than fixed prior knowledge. The core idea of this method is to use the gradient descent algorithm to optimize the camera pose parameters by minimizing loss functions such as reprojection error or photometric error. During the training process, the model will continuously adjust the camera pose until it converges to the best estimate.
[0008] After analysis, the existing technology mainly has the following defects:
[0009] 1) Structure from motion technology can recover the camera's motion trajectory and the 3D structure of the scene from a series of images. In this process, not only can the camera's position be inferred, but the scene point cloud it generates is the key basis for scene initialization in 3D Gaussian sputtering training, and has an important impact on the final scene training effect. Although the use of deep neural network methods can improve the accuracy of camera pose prediction under sparse views, if the structure from motion method is directly replaced by 3D Gaussian sputtering, it will inevitably lead to a loss of accuracy in scene initialization, resulting in blurred reconstruction results.
[0010] 2) Combining the camera pose parameters with iterative training of the 3D Gaussian sputtering scene can theoretically provide a more accurate model optimization approach, but due to the deep integration of the two, the model optimization process often becomes extremely complicated in actual operation. This over-coupling not only makes it difficult for the model to reach the global optimal solution, but may also cause problems such as training instability, limiting the further improvement of model performance. More importantly, incorporating the camera pose parameters into the training process means that each iteration needs to be recalculated and adjusted, which will significantly increase the number of iterative training and extend the overall training time, posing severe challenges to the consumption of computing resources and time efficiency.
[0011] In summary, in the field of new view synthesis technology, camera pose information is an indispensable part of scene training. The existing technology either presupposes that the camera pose information is known, or relies on the application of motion recovery structure algorithm from dense views to infer the position of the camera. This process is accompanied by a high time cost, especially due to the time-consuming comprehensive matching calculation. In addition, in the face of occlusion of observation perspective or highly repetitive texture structure in sparse scenes, the motion recovery structure algorithm often finds it difficult to accurately restore the camera pose, resulting in inaccurate positioning. In addition, although the point cloud data initially generated by the motion recovery structure is used as the starting point for neural field training to construct a scene model, its quality is limited by the uncertainty of the above-mentioned pose estimation. Summary of the invention
[0012] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a three-dimensional Gaussian sputtering optimization method for no posture input. The method comprises the following steps:
[0013] For the input image, the light prediction model is used to predict the light bundle distribution, and the distribution characteristics of the light bundle form are obtained, including the momentum of the light, the direction of the light and the volume density of the light;
[0014] Calculating the camera pose based on the distribution characteristics of the light beam form;
[0015] For the light beam distribution, sampling is performed based on the volume density of the light to obtain an initial spatial distribution of a three-dimensional Gaussian point cloud focused on a visual center area;
[0016] For the input image, a visual shell is obtained through frustum projection and object mask calculation, where the visual shell reflects the structural information of the scene object;
[0017] Based on the initial spatial distribution of the three-dimensional Gaussian point cloud and the visible shell, a three-dimensional Gaussian sputtering scene training is performed to obtain a three-dimensional scene reconstruction model that meets the set loss function standard, wherein the loss function includes a training regularization term of the camera pose parameters.
[0018] Compared with the prior art, the advantage of the present invention is that it proposes a 3D Gaussian sputtering scene initialization scheme based on camera light prediction. The scheme deeply integrates the camera pose estimation method with the scene reconstruction process to build an end-to-end optimization framework. This integrated design achieves more accurate and efficient model training by jointly optimizing the camera pose and scene structure. On the basis of ensuring the accuracy of camera pose estimation, the present invention accelerates the initialization process of the training scene and improves the accuracy of synthesizing new views using the 3D Gaussian sputtering method under sparse pose-free views.
[0019] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0021] Figure 1 is a schematic diagram of the overall process of a three-dimensional Gaussian sputtering optimization method without pose input according to an embodiment of the present invention;
[0022] Figure 2 is a flow chart of a three-dimensional Gaussian sputtering optimization method for no pose input according to an embodiment of the present invention;
[0023] Figure 3 is a schematic diagram of a density ray prediction model according to an embodiment of the present invention;
[0024] Figure 4 is a schematic diagram of density-based light sampling according to an embodiment of the present invention;
[0025] Figure 5 It is a schematic diagram of initializing a three-dimensional scene with a visual shell in combination with light beam density information according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0029] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0030] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0031] As an important innovation in the field of new view synthesis, 3D Gaussian sputtering technology has the potential to reshape the digital content production process. By solving the scene training problem under sparse pose-free views, the present invention can effectively expand the general capabilities and hardware scenarios of new view synthesis technology, thereby promoting the development of visual applications such as digital humans, autonomous driving scene simulation, 3D content generation, drone rapid mapping, large-scale scene reconstruction, and 3D language fields, and is conducive to promoting the popularization of new view synthesis applications to common mobile devices.
[0032] In general, see Figure 1 As shown, the provided 3D Gaussian sputtering optimization method for pose-free input mainly includes four parts: camera density light prediction, 3D Gaussian scene initialization (3D Gaussian initial training scene), visual shell elimination and camera pose regularization.
[0033] Camera density ray prediction is improved from camera ray prediction. The core idea of camera ray prediction is that neural network learning usually benefits from over-parameterized distributed representations. Therefore, instead of using compact camera feature representations such as intrinsic and extrinsic matrix, a set of light beams are used to represent the possible orientation direction of the camera. The light beams start from the pixels of the image, converge to the center point of the camera, and are represented by Plücker coordinates. The task of the neural network is to restore the position and direction of the light beam from a set of images as much as possible, so as to solve the intrinsic and extrinsic properties of the camera according to the convergence position of the light beam, for example, using the least squares optimization method. The camera density ray prediction of the present invention supports light information with density to provide prior information for scene initialization, which will be introduced in detail later.
[0034] The 3D Gaussian scene initialization is initially performed by using the sparse point cloud of the scene structure derived from the motion recovery structure to initialize the 3D Gaussian points. In order to form a more accurate pose estimation method and form an end-to-end training process, the present invention uses the over-parameterized information of the camera density ray to provide a priori for scene initialization, thereby integrating the camera pose estimation and the initialization of the 3D Gaussian scene. The 3D Gaussian points are sampled on the ray according to the density relationship, rather than using conventional uniform random sampling.
[0035] In order to achieve visual shell redundancy, the depth estimation information of the image is obtained from the input multi-view images, and the visual shell of the scene is constructed, thereby eliminating the erroneous initialization point cloud positions outside the shell. During the training process of the 3D Gaussian scene, the visual shell provides rough structural information of the scene objects, which is used to eliminate unreasonable Gaussian point cloud distribution, thereby suppressing the generation of visual floating objects and maintaining the consistency of the scene.
[0036] During the 3D Gaussian scene training process, the training regularization term of the camera pose parameters is added and the camera pose parameters are adjusted to achieve a higher precision scene reconstruction effect.
[0037] In the present invention, in the initialization process of the three-dimensional Gaussian scene, facing the sparse picture (or image) input without pose annotation, its depth information and pose information are first estimated to perform joint method optimization. Specifically, see Figure 2 As shown, the provided three-dimensional Gaussian sputtering optimization method for no pose input includes the following steps:
[0038] Step S210, for the input image, using the light prediction model to predict the light bundle distribution, and obtain the distribution characteristics of the light bundle form, including the momentum of the light, the direction of the light and the volume density of the light.
[0039] Camera density ray prediction is a proxy task for camera pose estimation, which will output a large number of distribution parameters of rays. The camera ray method only provides the function of recovering the camera pose from multiple pictures, and does not have the function of exporting the initial point cloud like the motion recovery structure algorithm. In an embodiment of the present invention, the constructed distributed representation of light is improved, and the light characteristics under the Plücker coordinates are transformed into light characteristics with density. Based on the characteristics of three-dimensional Gaussian sputtering volume rendering, light with volume density information is predicted, which provides a basis for sampling the initial three-dimensional Gaussian training scene from the light beam.
[0040] Figure 3 It is a light prediction model with density, built on the visual Transformer model, where the camera pose parameter λ is encoded as a series of rays p i represents any point on the i-th ray, d i represents the direction vector of the i-th light ray. The position information of each light ray in space is represented by Plücker coordinates in the prediction model. In order to realize the initialization of the three-dimensional Gaussian scene, in one embodiment, the key parameter of light density ρ is introduced. Therefore, each light ray can be specifically represented as l i =[d i ;m i ρ i ] is an expanded form, where d i and m iThey are the direction and momentum characteristics of the light, and the momentum characteristic is represented by m i =p i ×d i Calculated. i is the spatial volume density information carried by the improved light. Specifically, the input image I is processed into a sequence S after block processing, and the sequence with position encoding φ(·) is input into the visual Transformer model V to predict the light direction d, momentum m and density ρ corresponding to the pixel block, expressed as:
[0041]
[0042] in, Represents the predicted light beam distribution parameters, including a series of light parameter information. i Indicates the image block position, y i Represents the image block input information, φ(x i ) represents the position coding information corresponding to the image block, and N represents the number of image blocks. replace Used to refer to the light beam distribution unless the context indicates otherwise.
[0043] In summary, the light prediction model can be used to obtain the distribution characteristics of the light beam form, including the momentum of the light, the direction of the light, and the volume density of the light.
[0044] Step S220, calculating the camera pose based on the distribution characteristics of the light beam form.
[0045] The camera pose estimation is based on the distribution of a large number of rays, and the camera pose associated with multiple perspectives is calculated from the least squares optimization. For example, for the ray bundle formed by the input image I The camera light prediction method uses the least squares method to solve the camera's internal and external parameters λ, which includes the specific posture parameters θ.
[0046] Since the input of 3D Gaussian sputtering technology is usually multiple images from multiple perspectives, the camera pose calculation considers the joint optimal solution of the light bundles of multiple images with the same perspective. Thanks to the distributed representation in the form of light bundles, these rays can be combined into a light bundle for least squares optimization calculations.
[0047] In one embodiment, in order to further improve the accuracy of pose prediction and cope with the prediction uncertainty caused by the lack of perspective in sparse views, the predicted light direction and momentum can be further adjusted. For example, by using a diffusion model, the light is denoised and added to obtain richer light beam distribution features, thereby obtaining more accurate light prediction in sparse views.
[0048] In summary, in step S220, based on the camera ray method, a proxy task of predicting rays is used to restore the camera's pose information. Taking into account the application characteristics of volume rendering technology in three-dimensional Gaussian scene modeling, the prediction task is improved, and the key dimension of "density" is added to achieve the output of density information related to the initial scene while obtaining the camera pose. The density information carried by this ray essentially represents the reverse operation of the volume rendering process, that is, the scene structure is reversely deduced through density information. The Gaussian point cloud of the scene is initialized through density information, which provides a basis for training high-precision three-dimensional Gaussian scenes.
[0049] Step S230 , sampling is performed based on the volume density of the light beam distribution to obtain an initial spatial distribution of a three-dimensional Gaussian point cloud focused on the visual center area.
[0050] In this step, the initialization of the three-dimensional Gaussian scene is achieved under the camera light prediction framework with density, that is, the distribution of the Gaussian in the three-dimensional space is initialized based on light density sampling.
[0051] The process starts with a proxy task, which is camera ray prediction to obtain a large number of ray parameters Each ray l i In the Plücker coordinate system, the direction vector d i and momentum vector m i Definition. By analyzing these light parameters, their positions can be reconstructed in three-dimensional space, and the volume density ρ of the light can be used to reveal the point cloud distribution characteristics of the scene. This strategy uses the proxy task of regressing and predicting light bundles adopted by the camera-based pose estimation method to capture prior information closely related to the camera pose by analyzing the distribution characteristics of the light bundles in the scene space.
[0052] Specifically, starting from the image pixel, along the ray path passing through the center of the camera Samples are extracted (M is the number of ray paths) and used as the starting point cloud position for the reconstruction of the 3D Gaussian sputtering scene. i All mapped from the three-dimensional world space To the two-dimensional image space By sampling the light beam, we can get the initial point cloud layout gathered in the visual center area.
[0053] Figure 4 This is a schematic diagram of the initial spatial distribution of a 3D Gaussian point cloud constructed by sampling light based on density, which specifically includes the following steps:
[0054] Step S41: transform the light parameters in the Plücker coordinate system Map back to the actual position in three-dimensional space to obtain the distribution of light in three-dimensional space
[0055] Step S42, initializing a set of voxel grids in three-dimensional space And according to certain rules, the light Divided into multiple segments. Each voxel grid May contain multiple rays Each ray also carries its density distribution along the path {ρ i (z)} z∈[0,L] , k is the voxel grid index, i is the ray index that intersects the voxel grid, z is the independent variable, L represents the grid width, Represents a bundle of rays that intersect a voxel grid.
[0056] Step 43, based on this information, a three-dimensional Gaussian sphere point is initialized at the center of the voxel grid. The size of the Gaussian point is based on the integration of multiple light density in the voxel grid, thereby forming the initial spatial distribution of the three-dimensional Gaussian point cloud. k Initialize a three-dimensional Gaussian distribution point g at the center position k , whose size s is determined by the integrated density of light in the voxel Thus constructing the initial spatial distribution of the three-dimensional Gaussian point cloud K represents the number of Gaussian kernels in the scene.
[0057] These Gaussian points will be further adjusted in the subsequent scene density control training, and multiple small Gaussian points will be deleted and gradient updated to form larger Gaussian points. In the subsequent scene density control training stage, these Gaussian points will undergo further optimization and adjustment, and the small Gaussian points g k Through gradient descent or merging operations, it evolves into larger Gaussian points, ultimately forming a more refined 3D scene representation. This process not only strengthens the geometric structure of the scene, but also improves the accuracy of scene density prediction.
[0058] In summary, the present invention predicts the light density distribution based on Transformer, which is used to initialize the scene point cloud, and samples the initial three-dimensional Gaussian points by voxel grid integration to form the initial spatial distribution of the three-dimensional Gaussian point cloud. This method reduces the number of iterations of the density control algorithm in the subsequent scene training process and improves the accuracy of the rendering results.
[0059] Step S240, obtaining a visible shell reflecting the structural information of the scene object through frustum projection and object mask calculation.
[0060] In order to further improve the quality and robustness of 3D scene training, in one embodiment, a visual shell is introduced, such as building a 3D visual shell based on 2D depth segmentation information to eliminate redundancy. The visual shell combines the structural prior knowledge of objects in the input image. For example, the visual shell is calculated by frustum projection and object mask. Compared with the sparse point cloud generated by sampling alone, the visual shell can better preserve the structural consistency required for multi-view reconstruction and has the ability to eliminate invalid point cloud data that may lead to unreasonable Gaussian distribution. Figure 5 It is a schematic diagram of initializing a 3D scene using the visible shell in conjunction with the light bundle density information, forming a scene density grid suitable for initialization.
[0061] It is worth mentioning that the computational cost of the visible shell is relatively low. It only needs to be generated from a small number of key mask views with the help of the segmentation model. Therefore, it can be easily integrated into the initialization process to achieve efficient and accurate starting conditions for 3D scene reconstruction.
[0062] In one embodiment, during the reconstruction of a three-dimensional scene, constructing a three-dimensional mesh in combination with a visual shell includes the following steps:
[0063] Step S51: extract the depth information and segmentation mask of the two-dimensional image from the semantic segmentation model Among them I i represents i input images, D i and M i correspond to their depth map and segmentation mask respectively, and N is the number of images.
[0064] Step S52, obtaining the camera intrinsic parameter matrix K and extrinsic parameter matrix E of the image to determine the conversion relationship between the camera coordinate system and the world coordinate system.
[0065] Step S53: Initialize the scene voxel grid By projecting the camera parameter matrices K and E onto the two-dimensional plane, we get the corresponding two-dimensional projection grid.
[0066] Step S54, filtering the projection of non-reconstructed object content, using the object mask M i Projection grid Filter and exclude the content of non-reconstructed target objects, so as to form a three-dimensional mesh filtered by the visible shell in three-dimensional space. This process ensures the accuracy and effectiveness of 3D reconstruction while reducing the complexity of subsequent processing.
[0067] In summary, the redundant point cloud elimination technology based on the visual shell can effectively accelerate the model training speed while significantly suppressing the appearance of unreasonable visual elements, such as visual floating objects. The visual shell technology significantly improves the efficiency and accuracy of the 3D scene initialization stage, not only reducing unnecessary computational burden, but also laying a solid foundation for subsequent Gaussian point cloud optimization and scene density control, ensuring the clarity and coherence of the reconstruction results.
[0068] Step S250, taking minimization of the set loss function as the optimization goal, performing three-dimensional Gaussian sputtering scene training based on the initial spatial distribution and visible shell of the three-dimensional Gaussian point cloud to obtain a three-dimensional scene reconstruction model, wherein the loss function includes a training regularization term of the camera pose parameters.
[0069] In this step, the 3D scene training is initialized based on the initial spatial distribution and visible shell of the 3D Gaussian point cloud until the set loss function standard is met. During the training process, the loss function contains the training regularization term of the camera pose parameters. By adjusting the pose parameters during the 3D Gaussian scene training process, a higher-precision scene reconstruction effect can be achieved.
[0070] When dealing with scenes from multiple perspectives, the calibration of camera extrinsics becomes particularly important because of the possible scale differences between different perspectives. In addition, due to the inconsistency of illumination between perspectives or the slight motion blur caused by handheld device shooting, the final calibration effect will be poor. Therefore, the present invention optimizes the camera's extrinsics and three-dimensional model simultaneously during the scene training process. In this process, the camera parameters and Gaussian properties are jointly adjusted to achieve more refined control. In addition, the present invention imposes a certain regularization condition λ·‖E-E0‖ on the camera parameters. Among them, E0 is the camera extrinsic parameter matrix of the initial pose predicted by the camera ray method, and is also the global camera pose used for scene training. E represents the camera extrinsic parameter matrix of the current perspective, and λ is the coefficient of the regularization term. In order to prevent the optimized camera pose from being too offset, the alignment quality during the entire three-dimensional Gaussian sputtering scene training process is optimized.
[0071] In one embodiment, the loss function is set to:
[0072]
[0073] This loss function represents the optimization process of the three-dimensional Gaussian scene and camera pose parameters. represents the optimized three-dimensional Gaussian scene, E j* represents the optimized camera extrinsic matrix at the current viewing angle j, is the initial three-dimensional Gaussian scene to be optimized during training, E j Represents the camera extrinsic parameter matrix to be optimized at the current viewing angle j. Represents the three-dimensional Gaussian scene to be optimized The process of rendering into a 2D image under the corresponding pose parameters of view j. represents the real input image of the i-th image under the view angle j, and N represents the number of input images corresponding to the view angle j. For the regularization term, λ is the coefficient of the regularization term, represents the initial camera extrinsic parameter matrix associated with view j, and V represents the view classification of the input image.
[0074] During the training process, regularization terms for the camera pose are effectively added to optimize the camera's true pose through a neural network. Since the camera ray method provides the initial terms for the camera pose parameters, only a small amount of constraints and training costs are required to make the camera pose estimation more accurate and robust, thereby obtaining more accurate scene reconstruction results from sparse views.
[0075] After training is completed, a model for 3D scene reconstruction can be obtained, which can then be applied to new view synthesis. The model can synthesize new views from multiple perspectives for pose-free input images.
[0076] In summary, compared with the prior art, the present invention has the following advantages:
[0077] 1) Aiming at the emerging 3D Gaussian sputtering method in the field of new view synthesis technology, this paper proposes a scene initialization strategy for sparse pose-free input, which can efficiently realize the accurate estimation of camera pose and the joint initialization of the scene. This paper combines the advantages of the camera ray method, and on the basis of ensuring the efficiency and accuracy of camera pose estimation in motion recovery structure, provides crucial initialization scene information for 3D Gaussian sputtering training, significantly improving the quality and detail richness of the final 3D structure.
[0078] 2) The present invention provides an end-to-end implementation of 3D Gaussian sputtering technology for sparse pose-free input. Under the premise of ensuring the quality and consistency of scene synthesis, the efficiency of 3D Gaussian sputtering scene initialization is greatly improved. By integrating the camera ray method into the 3D Gaussian sputtering training process, the problem of ill-posed camera pose prediction caused by insufficient information and repeated textures in sparse views is solved. The estimation accuracy and efficiency of the camera ray method in sparse views are better than the current motion recovery structure method.
[0079] 3) The present invention makes full use of the proxy task of light prediction, increases the dimension of light density, and realizes the prediction of three-dimensional spatial structure density information. The camera pose prediction and three-dimensional Gaussian scene initialization process are combined to initialize the three-dimensional Gaussian point cloud, ensuring the accuracy and rendering quality of the training scene.
[0080] 4) The present invention obtains the depth information of the image with the help of the visual depth segmentation model and reprojects the two-dimensional scene into a three-dimensional visible shell, thereby constraining the appearance of redundant point clouds during the training process, reducing unreasonable visual floaters, and alleviating the computational burden of training.
[0081] 5) The present invention combines the training method of camera pose optimization to further correct the inconsistency of camera pose in multi-view images, thereby improving the accuracy of camera pose estimation under sparse views. At the same time, the regularization term is used to constrain the excessive deviation of the initial camera pose prediction value, and the update is terminated early to avoid introducing too much computational cost, which can optimize the alignment quality of the entire 3D Gaussian sputtering scene training process with less computational effort.
[0082] 6) After multiple simulation tests, the present invention can synthesize multi-view images that meet expectations, regardless of whether the input images are posed or not, thereby improving the quality of the synthesized images while increasing the computational efficiency.
[0083] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0084] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0085] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0086] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0087] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0088] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0090] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0091] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. A three-dimensional Gaussian sputtering optimization method for no pose input, comprising the following steps: For the input image, the light prediction model is used to predict the light bundle distribution, and the distribution characteristics of the light bundle form are obtained, including the momentum of the light, the direction of the light and the volume density of the light; Calculating the camera pose based on the distribution characteristics of the light beam form; For the light beam distribution, sampling is performed based on the volume density of the light to obtain an initial spatial distribution of a three-dimensional Gaussian point cloud focused on a visual center area; For the input image, a visual shell is obtained through frustum projection and object mask calculation, where the visual shell reflects the structural information of the scene object; Based on the initial spatial distribution of the three-dimensional Gaussian point cloud and the visible shell, a three-dimensional Gaussian sputtering scene training is performed to obtain a three-dimensional scene reconstruction model that meets the set loss function standard, wherein the loss function includes a training regularization term of the camera pose parameters.
2. The method according to claim 1, characterized in that The method of using the light prediction model to predict the light beam distribution includes: After the input image I is processed into blocks, an image block sequence S is formed; The sequence with position encoding is input into the light prediction model V to predict the direction d, momentum m and volume density ρ of the light, expressed as: in, is the predicted light bundle distribution, and the camera pose parameter λ is encoded as a light bundle Each ray is represented by the vector l = [d; m; ρ] after the expansion of its Plücker coordinates, where d and m are the direction and momentum characteristics of the ray, respectively, and ρ is the spatial volume density information carried by the ray. Each ray l i The spatial position information is represented by the direction vector d in the Plücker coordinate system. i and momentum vector m i Definition, x i Indicates the image block position, y i Represents the image block input information, φ(x i ) represents the position coding information corresponding to the image block, and N represents the number of image blocks.
3. The method according to claim 1, characterized in that The camera pose is calculated using the least squares method. For multiple pictures with the same viewing angle, the light beams are combined into one light beam based on the distribution representation in the form of light beams, and then the internal and external parameters of the camera, including the pose parameters, are solved using the least squares method.
4. The method according to claim 1, characterized in that The initial spatial distribution of the three-dimensional Gaussian point cloud focused on the visual center area is obtained according to the following steps: For the ray bundle distribution, starting from the image pixel, along the ray path passing through the center of the camera Extract samples to obtain the initial spatial distribution of the three-dimensional Gaussian point cloud focused on the visual center area as the starting point cloud position for the three-dimensional Gaussian sputtering scene reconstruction, where M is the number of extracted light paths, P j represents the j-th ray path.
5. The method according to claim 2, characterized in that: The initial spatial distribution of the three-dimensional Gaussian point cloud focused on the visual center area is obtained according to the following steps: The light beam distribution in the Plücker coordinate system Map back to the actual position in three-dimensional space to obtain the distribution of light in three-dimensional space Initialize a set of voxel grids in 3D space The light is divided into multiple segments according to the set rules, where each voxel grid Contains multiple rays Each ray carries its density distribution along its path Where k is the voxel grid index, i is the ray index that intersects the voxel grid, is the independent variable, L represents the grid width, represents a bundle of rays that intersect a voxel grid; Initialize a 3D Gaussian sphere point at the center of the voxel grid. The size of the Gaussian point is based on the integration of multiple light density in the voxel grid to form the initial spatial distribution of the 3D Gaussian point cloud. According to the light volume density information, in each voxel grid v k Initialize a three-dimensional Gaussian distribution point g at the center position k , whose size s is determined by the integrated density of light in the voxel Get the initial spatial distribution of constructing a three-dimensional Gaussian point cloud Where K represents the number of Gaussian kernels in the scene.
6. The method according to claim 5, characterized in that During the three-dimensional Gaussian sputtering scene training process, a three-dimensional grid is constructed according to the following steps: The depth information and segmentation mask of the two-dimensional image are extracted from the semantic segmentation model, which is expressed as Among them I i represents i input images, D i isI i The depth map of M i isI i Segmentation mask; Obtain the camera intrinsic parameter matrix K and extrinsic parameter matrix E of the image to determine the conversion relationship between the camera coordinate system and the world coordinate system; The scene voxel grid that will be initialized By projecting the camera's intrinsic parameter matrix K and extrinsic parameter matrix E onto a two-dimensional plane, we get the corresponding two-dimensional projection grid. Using object mask M i Projecting the 2D grid Filtering is performed to exclude the content of non-reconstructed target objects, and a filtered three-dimensional grid is formed in the three-dimensional space.
7. The method according to claim 6, characterized in that The loss function is set to: in, is the optimized three-dimensional Gaussian scene, E j* represents the optimized camera extrinsic matrix of the current viewing angle j, is the initial three-dimensional Gaussian scene to be optimized during the training process, E j represents the current camera extrinsic matrix to be optimized under viewing angle j, Represents the three-dimensional Gaussian scene to be optimized The process of rendering into a 2D image under the corresponding pose parameters of the viewing angle j, represents the real input image of the i-th image under the perspective j, N represents the number of input images corresponding to the perspective j, λ is the coefficient of the regularization term, E j represents the camera extrinsic parameter matrix of the current view j, and V represents the view classification of the input image.
8. The method according to claim 1, characterized in that Also includes: For the actually collected images, the trained three-dimensional scene reconstruction model is used to obtain corresponding multi-view target images.
9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Cited By
Three-dimensional scene reconstruction method based on multi-modal regularization and temperature smoothing constraint
CN119169208A
Three-dimensional scene reconstruction method based on multimodal regularization and temperature smoothness constraint
CN119169208B
New viewpoint image synthesis method based on camera pose light field coding
CN119273813A
High-quality large-scale scene reconstruction method based on 3D Gaussian sputtering
CN119942014A
Scene three-dimensional reconstruction method based on camera and laser radar fusion and related device
CN120088403A
