A method for synthesizing a new view of the moon surface based on a Gaussian radiation field

Abstract

This invention relates to a method for synthesizing novel lunar perspectives based on Gaussian radiation fields, comprising: acquiring internal and external parameters of a lunar surface image and sparse point clouds of the scene; completing the point cloud based on the geometric relationships between perspectives and the characteristics of the lunar surface to construct a series of three-dimensional Gaussian distributions; projecting the three-dimensional mean of the established series of three-dimensional Gaussian distributions onto a two-dimensional pixel plane to convert them into two-dimensional Gaussian distributions, and calculating the contribution weight of the two-dimensional Gaussian distributions to each pixel; weighted mixing of all Gaussian distributions within the field of view using the contribution weights; executing an adaptive Gaussian clipping strategy based on the optimization process; and rendering the imaging result under an arbitrary camera pose based on the optimization result. This invention enables the synthesis of novel perspectives in 360° lunar scenes, enhances the modeling capability of Gaussian radiation fields under sparse perspectives, and improves the image rendering quality under novel perspectives.

Description

Technical Field

[0001] This invention relates to the field of novel perspective synthesis, and more specifically to a method for synthesizing novel lunar perspectives based on Gaussian radiation fields. Background Technology

[0002] Novel perspective compositing is an advanced technique combining computer graphics and computer vision that generates new perspective images from a limited set of image inputs. This technology allows us to virtually observe scenes or objects from angles never actually captured, providing users with a richer visual experience. Compared to traditional image compositing techniques such as copying, cropping, and stitching, novel perspective compositing aims to create more realistic and visually distortion-free images, which is particularly important in virtual reality (VR), augmented reality (AR), and other computer vision applications.

[0003] Novel perspective synthesis techniques employ complex algorithms and models, including Generative Adversarial Networks (GANs), conditional generative models, and spatial transformer networks. These models enable the system to learn the transformation rules between input images and effectively infer new perspectives, significantly increasing the diversity and naturalness of synthesized images. Despite significant progress in novel perspective synthesis, challenges remain in handling large-scale scenes, sparse perspectives, or low-resolution images. Solving these problems is crucial for the further application and development of the technology. Future improvements and optimizations will further drive its application in game development, filmmaking, and virtual and augmented reality, enhancing user experience and scene realism.

[0004] The shortcomings of existing technologies for the synthesis of new perspectives for lunar scenes are mainly reflected in the following aspects: (1) The scale of lunar scenes varies greatly, making it difficult to optimize continuous parameters that conform to the real physical world; (2) The training perspectives are sparse, and the synthesized new perspectives often contain holes; (3) Existing new perspective synthesis algorithms often generate floating objects in space, and the rendered images contain artifacts. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a new perspective synthesis method for the lunar surface based on Gaussian radiation field. On the basis of limited known perspectives, it improves the quality of new perspective synthesis of the model, while reducing the number of parameters of Gaussian radiation field, so as to achieve high-quality image rendering under arbitrary perspective.

[0006] The technical solution of this invention is a method for synthesizing a new lunar surface perspective based on a Gaussian radiation field, comprising the following steps:

[0007] Step 1: Use motion reconstruction structure to obtain the internal parameters, external parameters and sparse point cloud of the lunar surface image. Based on the geometric relationship between viewpoints and the characteristics of the lunar surface, the point cloud is completed and a series of three-dimensional Gaussian distributions are constructed in the spatial position of the point cloud.

[0008] Step 2: For the series of three-dimensional Gaussian distributions established in Step 1, project their three-dimensional mean onto the two-dimensional pixel plane to transform the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution. Based on the difference between the mean of the two-dimensional Gaussian distribution and the pixel coordinates, calculate the contribution weight of the two-dimensional Gaussian distribution to each pixel.

[0009] Step 3: Using the contribution weight of each two-dimensional Gaussian distribution to the pixel color obtained in Step 2, weighted mix of all Gaussian distributions in the field of view is performed to render the detailed attributes of each two-dimensional Gaussian distribution.

[0010] Step 4: Based on the optimization process in Step 3, implement an adaptive Gaussian pruning strategy: every predetermined number of training sessions, remove the 1% of Gaussian distributions with the largest variance and the 1% with the smallest variance from the current set of two-dimensional Gaussian distributions.

[0011] Step 5: Based on the optimization results in Step 4, given an arbitrary camera pose, render the imaging result under that camera pose.

[0012] The advantages of this invention compared to the prior art are as follows:

[0013] (1) This invention uses a series of three-dimensional Gaussian distributions to characterize the lunar surface scene, increases the quality of the initial point cloud through multi-view images, and avoids the influence of extreme scale Gaussian on optimization through an adaptive Gaussian cropping strategy, thereby effectively improving the synthesis quality of new view images.

[0014] (2) This invention takes into account the color, geometry and frequency of Gaussian distribution, and combines a coarse-to-fine optimization strategy to increase the geometric consistency of the scene.

[0015] In summary, the method employed in this invention is simple in principle and can achieve the goal of synthesizing high-quality new perspectives in lunar scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for synthesizing a new lunar surface perspective based on a Gaussian radiation field, according to the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0018] like Figure 1 As shown, the specific implementation steps of the lunar surface novel perspective synthesis method based on Gaussian radiation field of the present invention are as follows:

[0019] Step 1: Use Structure from Motion (SFM) to obtain the internal parameters, external parameters, and sparse point cloud of the lunar surface image. Based on the geometric relationship between viewpoints and the characteristics of the lunar surface, the point cloud is completed, and a series of three-dimensional Gaussian distributions are constructed in the spatial location of the point cloud.

[0020] Using the structure-of-motion (SOG) algorithm, we recover the internal and external parameters of lunar surface images, as well as the sparse point cloud of the scene:

[0021] ,

[0022] in, It's the camera's internal parameters. It is the extrinsic parameter of the i-th camera. This is the i-th lunar image. It is a sparse point cloud of the scene. The camera's intrinsic and extrinsic parameters determine the imaging plane for subsequent rendering.

[0023] The process of completing point clouds based on the geometric relationships between viewpoints and the characteristics of the lunar surface is as follows:

[0024] Considering the relatively flat nature of the lunar surface, the least squares method can be applied to fit an ideal geometric plane based on sparse point cloud data of the scene:

[0025] (1)

[0026] in, These represent the normal vector and offset of the ideal geometric plane, respectively. This represents the least squares method.

[0027] Furthermore, on the fitted ideal geometric plane, according to 1024 A 1024 resolution mesh is used to more accurately handle the correspondence between the sparse point cloud of the scene and the plane. By projecting the sparse point cloud of the scene onto this meshed plane, mesh vertices not covered by the point cloud can be systematically collected and treated as points to be filled in. These vertices mark the missing parts of the point cloud that need further completion. For each mesh vertex that needs completion, it is mapped across all images (above). The pixel colors corresponding to the projection positions in the image are queried and summarized. The summarized result is a set of pixel colors in each image. If a vertex is visible in multiple images (visibility is determined by whether the vertex's projected position extends beyond the image boundaries), the color of the vertex is determined by averaging the colors of the corresponding pixels in these images. This step ensures that information obtained from multiple perspectives is fully utilized, thereby improving the accuracy of color reconstruction and the naturalness of the visual effect.

[0028] The process of constructing a series of 3D Gaussian distributions in the spatial location of a point cloud is as follows: Each vertex of the completed mesh is randomly offset along the direction of the vertical plane. Using the position of each point as the mean, a series of 3D Gaussian distributions are constructed, and then... Assign an initial color to each three-dimensional Gaussian distribution.

[0029] Step 2: For the series of three-dimensional Gaussian distributions established in Step 1, project their three-dimensional means onto the two-dimensional pixel plane, transforming the three-dimensional Gaussian distributions into two-dimensional Gaussian distributions. Based on the difference between the mean of the two-dimensional Gaussian distribution and the pixel coordinates, calculate the influence factor of the two-dimensional Gaussian distribution on each pixel.

[0030] (2)

[0031] in, Let be the influence factor of the k-th Gaussian distribution on the pixel. The superscript 2D indicates that the influence factor is calculated in a two-dimensional plane. Let be the mean of the k-th Gaussian distribution, and T denote the matrix transpose operation. Let be the covariance matrix of the k-th Gaussian distribution. For pixel coordinates, It is the identity matrix. The expansion coefficient is used to adjust the width of the Gaussian distribution.

[0032] For any pixel, after identifying all 2D Gaussian distributions that can influence it (an elliptical region described by a 2D covariance matrix that includes the pixel location can be considered as influencing the pixel), all 2D Gaussian distributions are sorted according to their depth. Then, the volume rendering formula is used to calculate the contribution weight of each 2D Gaussian distribution to the pixel color:

[0033] (3)

[0034] in, The weights of the k-th Gaussian distribution are... Let k be the influence factor of the k-th Gaussian distribution on the pixel. Let be the volume density of the k-th Gaussian distribution, where k is the index after sorting the two-dimensional Gaussians, and j is the index of a certain Gaussian distribution preceding the k-th two-dimensional Gaussian distribution.

[0035] Step 3: Assess the contribution weights of each two-dimensional Gaussian distribution obtained in Step 2 to the pixel color. This process involves weighted mixing of all Gaussian distributions within the field of view to render the detailed properties of each two-dimensional Gaussian distribution. This step enables refined image processing and visual optimization.

[0036] The volume rendering equation is used to obtain the color and depth of each pixel in the image, and the Gaussian distribution property parameters are optimized throughout the scene. This is because in image rendering, the color of each Gaussian distribution and its depth value in the camera coordinate system directly affect the final pixel color and depth rendering.

[0037] The color and corresponding depth value of each pixel are rendered using the following formulas, where formulas (4) and (5) are weighted mixtures of the Gaussian distribution as described above:

[0038] (4)

[0039] (5)

[0040] Let be the contribution weight of the k-th two-dimensional Gaussian distribution to the final pixel color. The color of the k-th Gaussian distribution. Let z be the z-coordinate of the k-th Gaussian distribution mean in the camera coordinate system. The pixel color to be rendered. Where N is the pixel depth to be rendered, and N is the total number of pixels affected.

[0041] The process of optimizing the attribute parameters of a Gaussian distribution is described below. According to an embodiment of the present invention, the attribute parameters include color, geometry, and frequency. During the optimization process, the true color of the pixel is used to supervise the color field, and the depth predicted by a monocular depth estimation model is used to supervise the rendered depth.

[0042] First, the process of optimizing color is described. The color loss function is calculated using the difference between the pixel's true color and the rendered color, based on the following formula:

[0043] (6)

[0044] in, It is a color loss function, where r is a pixel in the image, and R is the set of all pixels in the image. The true color of the pixel. Where N is the pixel color to be rendered, and N is the total number of pixels in the current image. For training images, To render the image, This is a function for calculating the structural similarity of images. This is the weighting coefficient (which can be set to 0.1).

[0045] This step involves not only direct color differences but also improving the visual quality of the rendered image through image structure similarity loss. The process of optimizing geometry is described below:

[0046] Monocular depth estimation models such as DepthAnying are used to predict the depth maps corresponding to the training images. By normalizing the predicted depth and the rendered depth, the impact of scale differences on the loss calculation can be reduced.

[0047] (7)

[0048] in, It is a depth loss function, where r is a pixel in the image. To predict depth for the model, For rendering depth, mean and std are the operations for calculating the mean and variance, respectively, patch is an image patch randomly cropped from the image, and M is the number of image patches.

[0049] Adding smoothing constraints to the depth map promotes the continuity of depth values ​​in non-edge regions, thereby enhancing the geometric consistency of the scene:

[0050] (8)

[0051] in, It is a smoothing loss function that encourages depths outside the edge regions to approach continuous smoothness. yes Adjacent depth values, It is a function that determines whether two depths are not on the edge, and adj is the operation of selecting adjacent pixels.

[0052] The rendered and training images are transformed from the spatial domain to the frequency domain using Fourier transform (Equation 9) to obtain the amplitude and phase angle of the image in the frequency domain. A bandpass filter is then used to process the frequency domain results (Equation 10), initially optimizing the low-frequency components of the image and then gradually adding the high-frequency components. This coarse-to-fine strategy helps to progressively optimize the image details, and the loss is calculated based on the difference in amplitude and phase angle after filtering (Equation 11). This helps to accurately adjust the frequency domain properties of the image, thereby better matching the visual features of the real image.

[0053] (9)

[0054] (10)

[0055] (11)

[0056] in, This is the result of converting the image to the frequency domain. H is the height of the image, W is the width of the image, x and y are pixel coordinates, I is the training image, and u and v are the coordinates in the frequency domain. It is a bandpass filter. This is the result after filtering. It is the frequency loss function. These represent the amplitude and phase angle of the real image, respectively. These represent the amplitude and phase angle of the rendered image, respectively.

[0057] Step 4: Building upon the optimization process in Step 3, implement an adaptive Gaussian pruning strategy: Every predetermined number of training iterations, remove the 1% with the largest variance and the 1% with the smallest variance from the current set of 2D Gaussian distributions. Here, variance refers to the variance of the Gaussian distribution. The meaning of "every predetermined number of training iterations" is explained below: In some examples, the model requires tens of thousands of training iterations, and steps 2 and 3 are executed in each round. During these tens of thousands of training iterations, step 4 is executed once every predetermined number of training iterations. This addresses the problem of an excessive number of Gaussian distributions generated by the algorithm in unbounded environments, effectively controlling and optimizing the quantity and quality of Gaussian distributions. The predetermined number could be, for example, 500.

[0058] Gaussian distributions with large variance are typically observed from more viewpoints, and because they participate in gradient optimization across multiple pixels, their average gradient at each location tends to be smaller. In such cases, these Gaussian distributions struggle to reach the splitting threshold, easily forming floating objects during optimization, leading to artifacts and occlusion. Gaussian distributions with small variance, on the other hand, cover a very small spatial region, equivalent to an extremely high signal frequency. According to the Nyquist sampling theorem, to accurately recover the original signal, the required sampling frequency must be greater than twice the signal frequency. Since image resolution limits the spatial sampling frequency, excessively high signal frequencies can lead to undersampling, preventing the recovery of the original signal and potentially causing overfitting.

[0059] By eliminating Gaussian distributions with excessively large or small variances, unnecessary computational resource consumption can be effectively reduced, while avoiding rendering quality issues caused by Gaussian distribution density mismatch.

[0060] Step 5: Based on the optimization results of Step 4, given an arbitrary camera pose, render the image result under that camera pose. This rendering process uses the intrinsic and extrinsic parameters extracted in Step 1. After completing the entire optimization process, given an arbitrary viewing angle (camera pose), use the light cone of the viewing angle to filter out all visible Gaussian distributions in the scene. Then, use rasterization technology to convert these Gaussian distributions into specific pixel representations to obtain a high-quality image under that viewing angle. During rasterization, depth information for each pixel is also generated, forming a depth map.

[0061] In complex 3D scenes, to generate images from any viewing angle, the first step is to filter out all visible Gaussian distributions in the scene using the light cone properties of the viewing angle. These Gaussian distributions represent key visual elements of the scene, defining the appearance and position of objects through their geometric and optical properties. This filtering process ensures that only those parts visible from the current viewpoint are processed, thereby improving rendering efficiency and reducing computational resource consumption. Subsequently, through rasterization, these selected Gaussian distributions are transformed into specific pixels, forming the final image from the viewing angle. Furthermore, a synchronously generated depth map provides distance information from each pixel to the camera.

[0062] Contents not described in detail in this specification are common knowledge to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications will be readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the inventive concept are protected.

Claims

1. A method for synthesizing a new view of the moon surface based on Gaussian radiation field, characterized in that, include: Step 1: Use motion reconstruction structure to obtain the internal parameters, external parameters and sparse point cloud of the lunar surface image. Based on the geometric relationship between viewpoints and the characteristics of the lunar surface, the point cloud is completed and a series of three-dimensional Gaussian distributions are constructed in the spatial position of the point cloud. Step 2: For the series of three-dimensional Gaussian distributions established in Step 1, project their three-dimensional mean onto the two-dimensional pixel plane to transform the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution. Based on the difference between the mean of the two-dimensional Gaussian distribution and the pixel coordinates, calculate the contribution weight of the two-dimensional Gaussian distribution to each pixel. Step 3: Using the contribution weight of each two-dimensional Gaussian distribution to the pixel color obtained in Step 2, weighted mix of all Gaussian distributions in the field of view is performed to render the detailed attributes of each two-dimensional Gaussian distribution. Step 4: Based on the optimization process in Step 3, implement an adaptive Gaussian pruning strategy: every predetermined number of training sessions, remove the 1% of Gaussian distributions with the largest variance and the 1% with the smallest variance from the current set of two-dimensional Gaussian distributions. Step 5: Based on the optimization results in Step 4, given an arbitrary camera pose, render the imaging result under that camera pose. Step 1, the process of acquiring the intrinsic parameters, extrinsic parameters, and sparse point cloud of the lunar surface image using motion reconstruction structures, includes: ， wherein, is the camera intrinsic, is the extrinsic of the ith camera, is the ith image of the moon surface, is the sparse point cloud of the scene, SFM is the structure from motion algorithm; The process of completing point clouds based on the geometric relationships between viewpoints and the characteristics of the lunar surface includes: The least squares method is used to fit an ideal geometric plane based on sparse point cloud data of the scene: （1） wherein, respectively represent a normal vector of an ideal geometric plane and an offset, represents a least squares method, On the fitted ideal geometric plane, according to 1024 A 1024 resolution is used to divide the scene into a grid. The sparse point cloud of the scene is projected onto this gridded plane, and the grid vertices that are not covered by the point cloud are collected and regarded as the point cloud locations to be filled. For each grid vertex that needs to be completed, the pixel color corresponding to its projection position in all lunar images is queried and summarized, and the summary result is V; if a vertex is visible in multiple images, the color of the corresponding pixel in these images is taken to determine the color of the vertex. The process of constructing a series of 3D Gaussian distributions on the point cloud spatial location is as follows: Each vertex of the completed mesh is randomly offset along the direction of the vertical plane. Using the position of each point as the mean, a series of 3D Gaussian distributions are constructed, and the results are summarized. Assign an initial color to each three-dimensional Gaussian distribution.

2. The method for synthesizing new lunar perspectives based on Gaussian radiation fields according to claim 1, characterized in that, Step 2: For the series of three-dimensional Gaussian distributions established in Step 1, project their three-dimensional means onto the two-dimensional pixel plane, transforming the three-dimensional Gaussian distributions into two-dimensional Gaussian distributions. Based on the difference between the mean of the two-dimensional Gaussian distribution and the pixel coordinates, calculate the contribution weight of the two-dimensional Gaussian distribution to each pixel, including: （2） in, Let be the influence factor of the k-th Gaussian distribution on the pixel. The superscript 2D indicates that the influence factor is calculated in a two-dimensional plane. Let be the mean of the k-th Gaussian distribution, and T denote the matrix transpose operation. Let be the covariance matrix of the k-th Gaussian distribution. For pixel coordinates, It is the identity matrix. The expansion coefficient is used to adjust the width of the Gaussian distribution; For any pixel, after identifying all the two-dimensional Gaussian distributions that can influence it, all the two-dimensional Gaussian distributions are sorted according to their depth. Then, the contribution weight of each two-dimensional Gaussian distribution to the pixel color is calculated using the volume rendering formula: （3） in, The weights of the k-th Gaussian distribution are... Let k be the influence factor of the k-th Gaussian distribution on the pixel. Let be the volume density of the k-th Gaussian distribution, where k is the index after sorting the two-dimensional Gaussians, and j is the index of a certain Gaussian distribution preceding the k-th two-dimensional Gaussian distribution.

3. The method for synthesizing new lunar perspectives based on Gaussian radiation fields according to claim 2, characterized in that, Step 3: For the contribution weight of each 2D Gaussian distribution to pixel color obtained in Step 2, perform a weighted blending of all Gaussian distributions within the field of view to render the detailed attributes of each 2D Gaussian distribution, including: The color and corresponding depth value of each pixel are rendered using the following formula: （4） （5） Let be the contribution weight of the k-th two-dimensional Gaussian distribution to the final pixel color. The color of the k-th Gaussian distribution. Let z be the z-coordinate of the k-th Gaussian distribution mean in the camera coordinate system. The pixel color to be rendered. Where N is the pixel depth to be rendered, and N is the total number of pixels affected by that pixel. The color loss function is calculated using the difference between the true color and the rendered color of a pixel, as shown in the following formula: （6） in, It is a color loss function, where r is a pixel in the image, and R is the set of all pixels in the image. The true color of the pixel. Where N is the pixel color to be rendered, and N is the total number of pixels in the current image. For training images, To render the image, This is a function for calculating the structural similarity of images. These are the weighting coefficients; The depth map corresponding to the training image is predicted using a monocular depth estimation model: （7） in, It is a depth loss function, where r is a pixel in the image. To predict depth for the model, For rendering depth, mean and std are the operations for calculating the mean and variance, respectively; patch is an image patch randomly cropped from the image; and M is the number of image patches. The rendered and real images are transformed from the spatial domain to the frequency domain using Fourier transform to obtain the amplitude and phase angle of the image in the frequency domain. A bandpass filter is then used to process the frequency domain results, and the loss is calculated based on the difference in amplitude and phase angle after filtering. （9） （10） （11） in, This is the result of converting the image to the frequency domain. H is the height of the image, W is the width of the image, x and y are pixel coordinates, I is the training image, and u and v are the coordinates in the frequency domain. It is a bandpass filter. This is the result after filtering. It is the frequency loss function. These represent the amplitude and phase angle of the real image, respectively. These represent the amplitude and phase angle of the rendered image, respectively.

Images