A Novel Perspective Synthesis Method and System Based on Generative Adversarial Strategy and Gaussian Sputtering

Abstract

This application discloses a novel perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering. The method includes: acquiring a scene image from a reconstructed scene perspective; obtaining an iterative intermediate rendered image from the scene image based on a pre-constructed Gaussian sputtering model; constructing a discriminator for the generative adversarial model, using the Gaussian sputtering model as the generator of the generative adversarial model, and training the Gaussian sputtering model based on the discriminator and the generator, using the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model; acquiring the target camera pose and inputting the target camera pose into the optimized Gaussian sputtering model to obtain a target scene image. This application can reduce visual artifacts in the rendering results and better fit the details in the scene, enhancing the rendering effect of scene details.

Description

Technical Field

[0001] This application relates to the field of novel perspective synthesis technology, and in particular to a novel perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering. Background Technology

[0002] 3D scene modeling is a crucial requirement in fields such as virtual reality, augmented reality, and autonomous driving. With the rapid development of these fields, higher demands are being placed on the efficiency and accuracy of 3D scene modeling. New perspective synthesis technology is one of the core issues in 3D scene modeling. Its goal is to use 2D images from different perspectives and their camera parameter information as input, model the scene's geometry, appearance, lighting, and other attributes, synthesize a 3D scene, and then render the new perspective image. Traditional view synthesis techniques mainly employ three scene representation methods: meshes, point clouds, and voxels. Each has its advantages and disadvantages, but also shares some common limitations: mesh and point cloud representations typically require large amounts of data and computational resources, while voxel representation cannot handle detailed and complex scenes well.

[0003] Neural Radiance Fields (NeRF) is a novel technique in computer graphics and 3D scene reconstruction that can synthesize realistic scenes from sparse sets of input viewpoints, pioneering a new paradigm for viewpoint synthesis tasks. However, this comes at a cost: NeRF methods are computationally intensive, typically requiring long training times and significant resources for rendering, especially for high-resolution outputs. Therefore, despite its remarkable achievements in image synthesis, the high computational cost and resource requirements limit its widespread adoption in practical applications.

[0004] 3D Gaussian Splatting (3D GS) uses Gaussian point clouds to represent scenes and achieves high-speed rendering through fast rasterization. 3D Gaussian Splatting reconstructs the scene in 3D space using Gaussian primitives, representing the scene as a superposition of three-dimensional Gaussian functions, thus significantly improving rendering efficiency and reducing computational costs. Compared to neural radiation fields, 3D Gaussian Splatting can generate new perspective images faster while maintaining rendering quality, making it highly promising for real-time applications and large-scale scene reconstruction.

[0005] However, while 3D Gaussian sputtering is currently advantageous for representing complex geometries, it produces unexpected visual artifacts. In the composite results, flocculent floating objects are generated near the viewpoint, affecting the rendering effect.

[0006] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0007] The main purpose of this application is to provide a new perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering, which aims to solve the problem that visual artifacts are easily generated in the reconstruction of 3D Gaussian sputtering scenes in the prior art, resulting in poor rendering effect.

[0008] The first aspect of this application provides a novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering. The method includes the following steps: acquiring a scene image from a reconstructed scene perspective; obtaining an iterative intermediate rendered image based on the scene image using a pre-built Gaussian sputtering model; constructing a discriminator for the generative adversarial model, using the Gaussian sputtering model as the generator of the generative adversarial model, and training the Gaussian sputtering model based on the discriminator and generator using the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model; acquiring the target camera pose and inputting the target camera pose into the optimized Gaussian sputtering model to obtain a target scene image.

[0009] Optionally, in one embodiment of this application, obtaining the iterative intermediate rendering image based on the scene image specifically includes: performing camera calibration on the scene image to obtain a camera pose; inputting the scene image and the camera pose into the Gaussian sputtering model to obtain an initial rendering image; calculating the mean absolute error and structural similarity error between the initial rendering image and the scene image; performing gradient backpropagation based on the mean absolute error and the structural similarity error to obtain an optimized Gaussian sputtering model; and iteratively processing the initial rendering image based on the optimized Gaussian sputtering model to obtain the iterative intermediate rendering image.

[0010] Optionally, in one embodiment of this application, the step of training the Gaussian sputtering model based on the discriminator and the generator, according to the scene image and the intermediate rendered image, to obtain an optimized Gaussian sputtering model specifically includes: inputting the scene image and the intermediate rendered image into the discriminator to obtain a discrimination result; optimizing the Gaussian sputtering model according to the discrimination result to obtain the generative adversarial loss of the generative adversarial model; calculating the scene depth information corresponding to the scene image, calculating the rendering depth information corresponding to the intermediate rendered image, and obtaining the depth constraint loss according to the scene depth information and the rendering depth information; obtaining the total loss function of the Gaussian sputtering model according to the generative adversarial loss and the depth constraint loss; and updating the Gaussian sputtering model based on the total loss function to obtain the optimized Gaussian sputtering model.

[0011] Optionally, in one embodiment of this application, the discriminator includes a first fully connected layer, a multilayer perceptron hybrid layer, a global pooling layer, and a second fully connected layer; the step of inputting the scene image and the intermediate rendered image into the discriminator to obtain a discrimination result specifically includes: splitting the scene image into multiple image blocks; the first fully connected layer processing the multiple image blocks to obtain embedded features; the multilayer perceptron hybrid layer obtaining image features based on the embedded features; and the global pooling layer and the second fully connected layer discriminating the image features to obtain a discrimination result.

[0012] Optionally, in one embodiment of this application, the loss function of the generative adversarial model is expressed as:

[0013]

[0014] in, This indicates the generation of adversarial loss. It is the symbol for mathematical expectation, x ~ P r Represents the true data distribution, x ~ P g The generator represents the data distribution; D(x) represents the discriminator's output for input x. Indicates about the sample The gradient of , ||·||2 represents the Euclidean norm, and λ is a hyperparameter; Indicates the distribution of real data P r The expected value obtained after the data point x obtained from the sampling is passed through the discriminator D; P represents the distribution of rendering data generated from generator G. g The expected value obtained after the data point x obtained from the sampling is passed through the discriminator D.

[0015] Optionally, in one embodiment of this application, the total loss function of the optimized Gaussian sputtering model is expressed as:

[0016]

[0017] in, For the total loss, λ is the mean absolute error, and 1-λ1 is the weight of the mean absolute error; λ1 represents the structural similarity error, and λ1 is the weight of the structural similarity error. The generator loss is used to generate the adversarial loss, and λ2 is the full weight of the adversarial loss. λ3 represents the depth constraint loss, and λ3 represents the weight of the depth constraint loss.

[0018] Optionally, in one embodiment of this application, updating the Gaussian sputtering model based on the total loss function to obtain the optimized Gaussian sputtering model specifically includes: assigning weights to each loss according to the total loss function to obtain the current weights of each loss; obtaining the current losses corresponding to the intermediate rendered image based on the Gaussian sputtering model and the current weights; updating the Gaussian sputtering model based on the total loss function and the current losses, and repeating the above steps of calculating the current losses based on the updated Gaussian sputtering model and continuing to update the Gaussian sputtering model based on the current losses, until the updated Gaussian sputtering model meets the preset requirements, then the last updated Gaussian sputtering model is taken as the optimized Gaussian sputtering model.

[0019] A second aspect of this application also provides a novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, wherein the novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes:

[0020] The image acquisition module is used to acquire scene images from the perspective of the reconstructed scene.

[0021] The image rendering module is used to obtain an iterative intermediate rendered image based on the scene image, according to a pre-built Gaussian sputtering model.

[0022] The model training module is used to construct a discriminator for the generative adversarial model. The Gaussian sputtering model is used as the generator of the generative adversarial model, and the Gaussian sputtering model is trained based on the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model.

[0023] The target image output module is used to acquire the target camera pose and input the target camera pose into the optimized Gaussian sputtering model to obtain the target scene image.

[0024] A third aspect of this application also provides a terminal, wherein the terminal includes: a memory, a processor, and a new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering stored in the memory and executable on the processor, wherein when the new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering is executed by the processor, it implements the steps of the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described above.

[0025] A fourth aspect of this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a novel perspective synthesis program based on a generative adversarial strategy and Gaussian sputtering, and the novel perspective synthesis program based on a generative adversarial strategy and Gaussian sputtering, when executed by a processor, implements the steps of the novel perspective synthesis method based on a generative adversarial strategy and Gaussian sputtering as described above.

[0026] Beneficial effects: This application provides a novel perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering. In this method, the parameters of the original Gaussian sputtering model are gradually optimized through generative adversarial strategy, making the generated rendered image more realistic, reducing visual artifacts in the rendering result, and better fitting the details in the scene, thereby enhancing the rendering effect of scene details. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a preferred embodiment of the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering in this application;

[0029] Figure 2 This is a flowchart of the Gaussian sputtering model in a preferred embodiment of the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering in this application;

[0030] Figure 3 This is a flowchart illustrating the specific implementation steps of the entire execution process in a preferred embodiment of the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering in this application.

[0031] Figure 4 This is a discriminator network structure diagram in a preferred embodiment of the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering in this application;

[0032] Figure 5 This is a comparison of rendering effects in a preferred embodiment of the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering in this application;

[0033] Figure 6 This is a structural diagram of a preferred embodiment of the novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering in this application;

[0034] Figure 7 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0035] Explanation of reference numerals in the attached figures:

[0036] 100. Image acquisition module; 200. Image rendering module; 300. Model training module; 400. Target image output module. Detailed Implementation

[0037] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.

[0038] Currently, there are still some problems with 3D Gaussian sputtering technology that require further research and solutions: Although 3D Gaussian sputtering is beneficial for representing complex geometries, it can produce unexpected visual artifacts; in the composite results, flocculent floating objects are generated near the viewpoint, affecting the rendering effect; and there is excessive blurring of details in the scene, such as specular highlight areas and solid color areas with textures.

[0039] The following description, with reference to the accompanying drawings, describes a novel perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering, according to embodiments of this application. Addressing the problem mentioned above in related technologies where 3D Gaussian sputtering scene reconstruction easily produces visual artifacts, resulting in poor rendering effects, this application provides a novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering. In this method, the parameters of the original Gaussian sputtering model are progressively optimized through a generative adversarial strategy, making the generated rendered image more realistic, reducing visual artifacts in the rendering result, and better fitting the details in the scene, thus enhancing the rendering effect of scene details. This solves the technical problem in related technologies where 3D Gaussian sputtering scene reconstruction easily produces visual artifacts, resulting in poor rendering effects.

[0040] This application achieves higher quality new perspectives while maintaining a simple engineering implementation. First, images or videos containing various perspectives of the scene to be reconstructed are acquired and calibrated using a structure-of-motion (SOMO) algorithm to obtain camera parameters, determine camera pose, and generate sparse point clouds, constructing the dataset required for training. Then, a monocular depth estimation algorithm is used to calculate the depth information of the images in the dataset. Next, a discriminator network for a generative adversarial model (GAP) is constructed using a multilayer perceptron and a convolutional neural network. Then, the sparse point cloud is used as the initial point cloud input to a Gaussian sputtering model, along with images from each perspective and their camera poses, and the model is trained. The Gaussian sputtering model serves as the generator for the GAP, outputting the optical and depth information of the rendered images from each perspective during training. Finally, the scene images from each perspective and the output of the Gaussian sputtering model are compared. The Gaussian sputtering model parameters are optimized by minimizing the mean absolute error and structural similarity error of the optical information of the rendered viewpoint images. Then, the optical information of the viewpoint images from each viewpoint in the dataset, along with the corresponding viewpoint image optical information output by the Gaussian sputtering model, is input into a discriminator for training. The discriminator feeds back the quality of the rendering results to the Gaussian sputtering model, which acts as the generator, minimizing the generative adversarial loss and improving the optimization process of the Gaussian sputtering model parameters. Next, the Pearson correlation coefficient between the depth of the scene image at each viewpoint and the depth information of the corresponding viewpoint image output by the Gaussian sputtering model is calculated to minimize the depth constraint loss. Finally, after training, an arbitrary camera pose is input into the constructed Gaussian sputtering model to render the scene image at the corresponding viewpoint. The generative adversarial strategy designed in this application can improve the original Gaussian sputtering model, reduce visual artifacts in the rendering results, better fit the details in the scene, and enhance the rendering effect of scene details. The depth constraint used in this application can reduce the generation of flocculent objects in the scene, and using correlation as weak supervision can enhance the stability of model training and reduce the impact of depth estimation errors on model training.

[0041] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0042] The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering described in the preferred embodiment of this application, such as Figure 1 As shown, the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering includes the following steps:

[0043] In step S101, a scene image from the perspective of the reconstructed scene is acquired.

[0044] Specifically, the camera pose and intrinsic parameter matrix of the input scene image are calibrated using the structure-of-motion (SOG) algorithm to generate a sparse point cloud of the scene. This includes: capturing images of the scene from various perspectives from different angles, ensuring uniform lighting during capture. If the original video file is in video format, keyframes are extracted and converted to image format before further processing. The SOG algorithm is used to calibrate the acquired images, obtaining the camera's intrinsic and extrinsic parameters for each image, determining the camera position and pose, reconstructing the sparse point cloud of the scene and the camera trajectory, and generating a sparse 3D point cloud.

[0045] Understandably, the process involves: capturing scene images from different angles to ensure uniform lighting; if video is being captured, keyframes are extracted and converted to image format. Camera calibration involves using the Structure for Motion Restoration (SfM) algorithm to calibrate the acquired images; obtaining the intrinsic (e.g., focal length, optical center) and extrinsic (e.g., camera position, pose) parameters for each image; and reconstructing the sparse point cloud and camera trajectory of the scene.

[0046] In step S102, based on the pre-built Gaussian sputtering model, an iterated intermediate rendering image is obtained from the scene image.

[0047] In one possible implementation, the scene image is calibrated to obtain the camera pose; the scene image and the camera pose are input into the Gaussian sputtering model to obtain an initial rendered image; the mean absolute error and structural similarity error between the initial rendered image and the scene image are calculated; gradient backpropagation is performed based on the mean absolute error and the structural similarity error to obtain an optimized Gaussian sputtering model; based on the optimized Gaussian sputtering model, the initial rendered image is iteratively processed to obtain an iterative intermediate rendered image.

[0048] Specifically, such as Figure 2 As shown, a sparse point cloud is used as the initial point cloud input to the Gaussian sputtering model, along with images from various viewpoints and their camera poses, and the model is trained. Then, the Gaussian sputtering model is used as the generator for a generative adversarial model, outputting the optical information (rendered image) and depth information (rendered depth image) of each viewpoint image rendered after each iteration during training.

[0049] It is understandable that Gaussian sputtering model training involves using sparse point clouds as initial point clouds as input to the Gaussian sputtering model, inputting images from various viewpoints and their camera poses for training, and outputting the optical and depth information of the rendered images from each viewpoint after each iteration during the training process.

[0050] Specifically, such as Figure 2As shown, the Gaussian sputtering model calculates and minimizes the mean absolute error and structural similarity error of the optical information between the scene images from each viewpoint and the corresponding viewpoint images rendered by the Gaussian sputtering model. Then, gradient backpropagation is used to optimize the parameters of the Gaussian sputtering model.

[0051] Understandably, the loss function is minimized by calculating the mean absolute error (L1 loss) and structural similarity error (DSSIM loss) between the Gaussian sputtering model output rendered image and the real scene image, and then optimizing the parameters of the Gaussian sputtering model using gradient backpropagation.

[0052] In step S103, a discriminator for the generative adversarial model is constructed, and the Gaussian sputtering model is used as the generator of the generative adversarial model. Based on the discriminator and the generator, the Gaussian sputtering model is trained according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model.

[0053] In one possible implementation, the scene image and the intermediate rendered image are input into the discriminator to obtain a discrimination result; the Gaussian sputtering model is optimized based on the discrimination result to obtain the generative adversarial loss of the generative adversarial model; the scene depth information corresponding to the scene image is calculated, the rendering depth information corresponding to the intermediate rendered image is calculated, and the depth constraint loss is obtained based on the scene depth information and the rendering depth information; the total loss function of the Gaussian sputtering model is obtained based on the generative adversarial loss and the depth constraint loss; the Gaussian sputtering model is updated based on the total loss function to obtain the optimized Gaussian sputtering model.

[0054] Specifically, a monocular depth estimation algorithm is used to calculate the depth information of the images in the dataset. The input is the optical information of the scene images from each viewpoint (the input scene image), the distance of each pixel in the image relative to the camera is estimated, and the depth information of the scene images from each angle is output.

[0055] In one possible implementation, the discriminator includes a first fully connected layer, a multilayer perceptron hybrid layer, a global pooling layer, and a second fully connected layer. The scene image is split into multiple image patches; the first fully connected layer processes the multiple image patches to obtain embedded features; the multilayer perceptron hybrid layer obtains image features based on the embedded features; the global pooling layer and the second fully connected layer discriminate the image features to obtain a discrimination result.

[0056] Specifically, a discriminator network for a generative adversarial model is constructed using multilayer perceptrons and convolutional neural networks. For example... Figure 4As shown, the specific steps are as follows: First, the input image is split into multiple patches. Then, a fully connected network is used to process all patches and convert them into embedded features. Next, a multilayer perceptron hybrid layer is constructed by cross-using token mixing layers and channel mixing layers to extract input image features. Finally, a global average pooling layer and a fully connected layer are used to discriminate the image.

[0057] In this embodiment of the application, the loss function of the generative adversarial model is:

[0058]

[0059] In this formula, the meanings of each symbol are as follows:

[0060] Represents the distance loss function; It is the symbol for mathematical expectation, representing the operation of calculating the average of a random variable; x ~ P r Represents the true data distribution; x ~ P g D(x) represents the data distribution generated by the generator; D(x) represents the output of the discriminator for the input x, which is usually a probability value indicating whether the input comes from the true data distribution. Indicates about the sample The gradient of the vector; ||·||2 represents the Euclidean norm, which is the square root of the sum of the squares of the elements of the vector; λ is a hyperparameter used to adjust the balance in adversarial training; the whole formula describes the loss function in GAN, which measures the quality of the generative model by comparing the probability distribution of the discriminator on the real data and the generated data. At the same time, the third term in the formula is a constraint on the smoothness of the discriminator's gradient, ensuring that the gradient of the discriminator will not be too large or too small, thus avoiding the problems of gradient vanishing or exploding.

[0061] Part One Indicates the distribution of real data P r The expected value obtained after the data point x obtained from the sampling is processed by the discriminator D. The goal of this part is to make the discriminator classify the real data as "true" as much as possible; the second part P represents the distribution of rendering data generated from generator G. g The expected value obtained after the data point x obtained from the mid-sample is processed by the discriminator D. The goal of this part is to make the discriminator classify the rendered data generated by the generator as "fake" as much as possible. Part 3 This is an important constraint introduced in WGAN-GP, called the "gradient penalty". Here, λ is a weight constant used to control the degree of influence of this part. The goal of this part is to limit the size of the discriminator's weight matrix so that it does not exceed a certain threshold, thus avoiding gradient vanishing or exploding.

[0062] Understandably, the discriminator network is constructed as follows: a discriminator for a generative adversarial model (GAN) is built using a multilayer perceptron (MLP) and a convolutional neural network (CNN); the input image is split into multiple patches, and a fully connected network is used to process the patches and transform them into embedded features; an MLP hybrid layer is constructed by cross-using token mixing layers and channel mixing layers to extract input image features; and a global average pooling layer and a fully connected layer are used to discriminate the image.

[0063] Understandably, the discriminator loss function is as follows: the discriminator needs to maximize the difference between the real data and the generated data; the loss function consists of two parts: the difference between the expected value of the real data after passing through the discriminator and the expected value of the generated data after passing through the discriminator; in order to ensure the Lipschitz continuity of the loss function, a gradient penalty term is also required.

[0064] In one possible implementation, the weights of each loss are assigned according to the total loss function to obtain the current weights of each loss; based on the Gaussian sputtering model and the current weights, the current losses corresponding to the intermediate rendered image are obtained; based on the total loss function, the Gaussian sputtering model is updated according to the current losses, and the above steps of calculating the current losses based on the updated Gaussian sputtering model and updating the Gaussian sputtering model according to the current losses are repeated until the updated Gaussian sputtering model meets the preset requirements, and then the last updated Gaussian sputtering model is taken as the optimized Gaussian sputtering model.

[0065] In this application, a generative adversarial strategy (model) is used to improve the parameter optimization process of a Gaussian sputtering model. A discriminator module is added to the original Gaussian sputtering model, which is then used as a generator to optimize the Gaussian sputtering model parameters using a generative adversarial strategy.

[0066] Specifically, such as Figure 3 As shown, the optical information of the images from each viewpoint in the dataset and the corresponding viewpoint images output by the Gaussian sputtering model are input into the discriminator for training. The discriminator maximizes the generative adversarial model loss and outputs the discrimination result for the input image. The Gaussian sputtering model improves its parameter optimization based on the discriminator's output, minimizing the generative adversarial model loss. It only includes the discriminator and real data samples, and does not affect the generator gradient, so it can be ignored. Its generative adversarial loss is:

[0067]

[0068] Understandably, GAN training involves: using a Gaussian sputtering model as the generator of the GAN; inputting images from various viewpoints in the dataset and the rendered images output by the Gaussian sputtering model into the discriminator for training; the discriminator maximizing the GAN loss and outputting the discrimination result for the input image; and the Gaussian sputtering model improving its parameters based on the discriminator result to minimize the GAN loss.

[0069] The discriminator needs to maximize this loss, that is, to distinguish the difference between the original image and the image generated by the Gaussian sputtering model. The discriminator loss needs to maximize L. W That is, to maximize:

[0070]

[0071] Since gradient descent is used, the discriminator loss needs to be minimized. That is, minimize:

[0072]

[0073] The generator loss needs to be minimized. Therefore, it is:

[0074]

[0075] because It only includes the discriminator and real data samples, and does not affect the generator gradient, so it can be ignored. Therefore, its loss is:

[0076]

[0077] To ensure the Lipschitz continuity of the loss function (meaning the rate of change of a function between any two points in its domain does not exceed a constant), a gradient penalty term needs to be added to the discriminator network loss; simultaneously, to ensure the Lipschitz continuity of the loss function, a gradient penalty term needs to be added to the discriminator network loss.

[0078]

[0079] Understandably, the depth constraint loss involves calculating the Pearson correlation coefficient between the depth of the scene image at each viewpoint and the rendered depth image output by the Gaussian sputtering model; using depth constraints to reduce the generation of flocculent floating objects and optimize the reconstructed scene structure. The total loss function combines L1 loss, DSSIM loss, GAN loss, and depth constraint loss; weights (λ_1, λ_2, λ_3) are assigned to each loss term to balance their impact on the optimization process.

[0080] Specifically, the Pearson correlation coefficient between the scene image depth at each viewpoint and the corresponding viewpoint image depth information output by the Gaussian sputtering model is calculated using the following formula:

[0081]

[0082] Among them, D r To render a depth image, D g The image represents the scene depth. The Gaussian sputtering model needs to minimize the depth constraint loss. Using depth constraints can reduce the generation of flocculent floating objects and optimize the reconstructed scene structure. Therefore, the overall loss function of the improved Gaussian sputtering model in this invention is:

[0083]

[0084] For the total loss function, The average absolute error of the optical information of the scene images from each viewpoint and the corresponding viewpoint images output and rendered by the Gaussian sputtering model is given by 1-λ1 as its weight. The structural similarity error between the scene images from each viewpoint and the corresponding viewpoint images output and rendered by the Gaussian sputtering model is given by λ1 as its weight. The generator loss is used to generate the adversarial loss, with λ2 as its full weight. The depth-constrained loss is represented by λ2, which is its weight.

[0085] In this application, depth constraints are used to reduce the generation of flocculent floating objects. By pre-calculating the depth information of the scene and comparing it with the depth information output by the Gaussian sputtering model during training, the depth of the 3D Gaussian primitives is corrected to make their positions more accurate and reduce the erroneous generation of flocculent floating objects.

[0086] In step S104, the target camera pose is obtained and input into the optimized Gaussian sputtering model to obtain the target scene image.

[0087] Specifically, after training, any camera pose is input into the constructed Gaussian sputtering model, and the scene image at the corresponding viewpoint is rendered and output, which is the new viewpoint image (i.e. the target scene image).

[0088] It should be noted that in another embodiment of this application, other types of neural networks (such as convolutional neural networks, Transformers, etc.) may be used as the discriminator, or the loss function of the generative adversarial model may be modified. In another embodiment, different formulas may be used to calculate the depth constraints, such as other correlation coefficients.

[0089] This application tested the effectiveness of the method of this invention and the original 3D Gaussian sputtering algorithm on the Mip-NeRF360 dataset (unbounded anti-aliased neural radiation field dataset), the Tanks & Templates dataset (tank and template dataset), and the Deep Blending dataset (deep blending dataset). Rendering examples are shown below. Figure 5 As shown in Table 1, the metrics are as follows. The comparison metrics are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), with higher values ​​indicating higher rendering quality; and Learning Perceptual Patch Similarity (LPIPS), with lower values ​​indicating higher rendering quality.

[0090] Table 1: Corresponding Indicators of the Original 3D Gaussian Sputtering Algorithm and the Method of this Application

[0091]

[0092]

[0093] This application demonstrates superior image quality across multiple datasets, outperforming the original 3D Gaussian sputtering method.

[0094] Next, referring to the accompanying drawings, a novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, according to an embodiment of this application, is described.

[0095] Figure 6 This is a structural diagram of a novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, according to an embodiment of this application.

[0096] like Figure 6 As shown, the novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes: an image acquisition module 100, an image rendering module 200, a model training module 300, and a target image output module 400.

[0097] Specifically, the image acquisition module 100 is used to acquire scene images from the perspective of the reconstructed scene;

[0098] Image rendering module 200 is used to obtain an iterative intermediate rendering image based on the scene image according to a pre-built Gaussian sputtering model;

[0099] The model training module 300 is used to construct a discriminator for the generative adversarial model, using the Gaussian sputtering model as the generator of the generative adversarial model, and training the Gaussian sputtering model based on the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model.

[0100] The target image output module 400 is used to acquire the target camera pose and input the target camera pose into the optimized Gaussian sputtering model to obtain the target scene image.

[0101] Figure 7 A structural diagram of a terminal provided in an embodiment of this application. The terminal may include:

[0102] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0103] When the processor 502 executes the program, it implements the novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering provided in the above embodiments.

[0104] Furthermore, the terminal also includes:

[0105] Communication interface 503 is used for communication between memory 501 and processor 502.

[0106] The memory 501 is used to store computer programs that can run on the processor 502.

[0107] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0108] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0109] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0110] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0111] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering.

[0112] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the features described in this application. Figure 1 The corresponding embodiments provide a novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0115] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0117] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0118] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0121] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering, characterized in that, The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering includes: Acquire scene images from the perspective of the reconstructed scene; Based on a pre-built Gaussian sputtering model, an iterative intermediate rendering image is obtained from the scene image; A discriminator for a generative adversarial model is constructed, and the Gaussian sputtering model is used as the generator of the generative adversarial model. Based on the discriminator and the generator, the Gaussian sputtering model is trained according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model. The target camera pose is obtained and input into the optimized Gaussian sputtering model to obtain the target scene image.

2. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 1, characterized in that, The step of obtaining the iterative intermediate rendering image based on the scene image specifically includes: Camera calibration is performed on the scene image to obtain the camera pose; The scene image and the camera pose are input into the Gaussian sputtering model to obtain the initial rendered image; Calculate the mean absolute error and structural similarity error between the initial rendered image and the scene image; Gradient backpropagation is performed based on the mean absolute error and the structural similarity error to obtain an optimized Gaussian sputtering model. Based on the optimized Gaussian sputtering model, the initial rendered image is iteratively processed to obtain an intermediate rendered image after iteration.

3. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 1, characterized in that, The step of training the Gaussian sputtering model based on the discriminator and generator, using the scene image and the intermediate rendered image, to obtain an optimized Gaussian sputtering model specifically includes: The scene image and the intermediate rendered image are input into the discriminator to obtain the discrimination result; The Gaussian sputtering model is optimized based on the discrimination results to obtain the generative adversarial loss of the generative adversarial model; Calculate the scene depth information corresponding to the scene image, calculate the rendering depth information corresponding to the intermediate rendering image, and obtain the depth constraint loss based on the scene depth information and the rendering depth information; Based on the generative adversarial loss and the depth constraint loss, the total loss function of the Gaussian sputtering model is obtained; The Gaussian sputtering model is updated based on the total loss function to obtain the optimized Gaussian sputtering model.

4. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that, The discriminator includes a first fully connected layer, a multilayer perceptron hybrid layer, a global pooling layer, and a second fully connected layer. The step of inputting the scene image and the intermediate rendered image into the discriminator to obtain the discrimination result specifically includes: The scene image is split into multiple image blocks; The first fully connected layer processes multiple image patches to obtain embedded features; The multilayer perceptron hybrid layer obtains image features based on the embedded features; The global pooling layer and the second fully connected layer discriminate the image features to obtain the discrimination result.

5. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that, The loss function of the generative adversarial model is expressed as: in, This indicates the generation of adversarial loss. It is the symbol for mathematical expectation, x ~ P r Represents the true data distribution, x ~ P g d(x) represents the data distribution generated by the generator; d(x) represents the output of the discriminator for the input x. Indicates about the sample The gradient of , ||·||2 represents the Euclidean norm, and λ is a hyperparameter; Indicates the distribution of real data P r The expected value obtained after the data point x obtained from the sampling is passed through the discriminator D; P represents the distribution of rendering data generated from generator G. g The expected value obtained after the data point x obtained from the sampling is passed through the discriminator D.

6. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that, The total loss function of the optimized Gaussian sputtering model is expressed as: in, For the total loss, λ is the mean absolute error, and 1-λ1 is the weight of the mean absolute error; λ1 represents the structural similarity error, and λ1 is the weight of the structural similarity error. The generator loss is used to generate the adversarial loss, and λ2 is the full weight of the adversarial loss. λ3 represents the depth constraint loss, and λ3 represents the weight of the depth constraint loss.

7. The novel perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 6, characterized in that, The step of updating the Gaussian sputtering model based on the total loss function to obtain the optimized Gaussian sputtering model specifically includes: The weights of each loss are assigned according to the total loss function to obtain the current weights of each loss. Based on the Gaussian sputtering model and the current weights, the current losses corresponding to the intermediate rendered image are obtained; Based on the total loss function, the Gaussian sputtering model is updated according to the current losses, and the above steps of calculating the current losses based on the updated Gaussian sputtering model and updating the Gaussian sputtering model according to the current losses are repeated until the updated Gaussian sputtering model meets the preset requirements. Then, the last updated Gaussian sputtering model is taken as the optimized Gaussian sputtering model.

8. A novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, characterized in that, The novel perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes: The image acquisition module is used to acquire scene images from the perspective of the reconstructed scene. The image rendering module is used to obtain an iterative intermediate rendered image based on the scene image, according to a pre-built Gaussian sputtering model. The model training module is used to construct a discriminator for the generative adversarial model. The Gaussian sputtering model is used as the generator of the generative adversarial model. Based on the discriminator and the generator, the Gaussian sputtering model is trained according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model. The target image output module is used to acquire the target camera pose and input the target camera pose into the optimized Gaussian sputtering model to obtain the target scene image.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering stored in the memory and executable on the processor. When the new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering is executed by the processor, it implements the steps of the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a novel perspective synthesis program based on a generative adversarial strategy and Gaussian sputtering, which, when executed by a processor, implements the steps of the novel perspective synthesis method based on a generative adversarial strategy and Gaussian sputtering as described in any one of claims 1-7.

Images