New view angle synthesis method and system based on generative adversarial strategy and Gaussian sputtering

Abstract

The invention discloses a new view angle synthesis method and system based on a generative adversarial strategy and Gaussian sputtering. The method comprises the following steps: acquiring a scene image under a reconstructed scene view angle; based on a pre-constructed Gaussian sputtering model, obtaining an iterated intermediate rendering image according to the scene image; constructing a discriminator of a generative adversarial model, taking the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model according to the scene image and the intermediate rendering image based on the discriminator and the generator to obtain an optimized Gaussian sputtering model; and obtaining a target camera pose, and inputting the target camera pose to the optimized Gaussian sputtering model to obtain a target scene image. Visual artifacts in the rendering result can be reduced, detail parts in the scene can be better fitted, and the rendering effect of scene details is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of new perspective synthesis, and in particular to a new perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering. Background Art

[0002] Three-dimensional scene modeling is an important requirement in the fields of virtual reality, augmented reality, and autonomous driving. With the rapid development of these fields, higher requirements are placed on the efficiency and accuracy of three-dimensional scene modeling. New view synthesis technology is one of the core issues of three-dimensional scene modeling. Its goal is to use two-dimensional images of different perspectives and their camera parameter information as input, synthesize a three-dimensional scene by modeling the scene's geometry, appearance, lighting and other properties, and then render a new view image. Traditional view synthesis technology mainly uses three scene representation methods: grid, point cloud and voxel. They each have their own advantages and disadvantages, and also have some common limitations: grid and point cloud representation methods usually require a lot of data and computing resources, while voxel representation methods cannot handle details and complex scenes well.

[0003] Neural Radiance Fields (NeRF) is a new technology in computer graphics and 3D scene reconstruction that can synthesize realistic scenes from a sparse set of input perspectives, creating a new paradigm for perspective synthesis tasks. But this also comes with a price: Neural Radiance Fields-like methods are computationally intensive and usually require long training times and a lot of resources for rendering, especially for high-resolution outputs. Therefore, although Neural Radiance Fields has achieved remarkable success in image synthesis, its high computational cost and resource requirements limit its popularity in practical applications.

[0004] The 3D Gaussian Splatting (3D GS) method uses Gaussian point clouds to express scenes and achieves high-speed rendering through fast rasterization. 3D Gaussian Splatting uses Gaussian primitives to reconstruct scenes in 3D space, representing the scene as a superposition of a set of 3D Gaussian functions, which significantly improves rendering efficiency and reduces computational costs. Compared with neural radiation fields, 3D Gaussian Splatting can generate new perspective images faster while ensuring rendering quality, which makes it have great potential in real-time applications and large-scale scene reconstruction.

[0005] Then, although the current three-dimensional Gaussian sputtering is conducive to representing complex geometries, it will produce visual artifacts beyond expectations. In the synthesis results, floc-like floating objects will be generated near the viewing angle, affecting the rendering effect.

[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0007] The main purpose of this application is to provide a new perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering, aiming to solve the problem in the prior art that three-dimensional Gaussian sputtering scene reconstruction is prone to produce visual artifacts, resulting in poor rendering effect.

[0008] A first aspect of an embodiment of the present application provides a new perspective synthesis method based on a generative adversarial strategy and Gaussian sputtering, and the new perspective synthesis method based on a generative adversarial strategy and Gaussian sputtering includes the following steps: obtaining a scene image under the perspective of a reconstructed scene; based on a pre-constructed Gaussian sputtering model, obtaining an iterated intermediate rendering image according to the scene image; constructing a discriminator of the generative adversarial model, using the Gaussian sputtering model as a generator of the generative adversarial model, and based on the discriminator and the generator, training the Gaussian sputtering model according to the scene image and the intermediate rendering image to obtain an optimized Gaussian sputtering model; obtaining a target camera pose, and inputting the target camera pose into the optimized Gaussian sputtering model to obtain a target scene image.

[0009] Optionally, in one embodiment of the present application, obtaining an iterative intermediate rendered image based on the scene image specifically includes: performing camera calibration on the scene image to obtain a camera pose; inputting the scene image and the camera pose into the Gaussian sputtering model to obtain an initial rendered image; calculating a mean absolute value error and a structural similarity error between the initial rendered image and the scene image; performing gradient back propagation based on the mean absolute value error and the structural similarity error to obtain an optimized Gaussian sputtering model; and based on the optimized Gaussian sputtering model, performing iterative processing based on the initial rendered image to obtain an iterative intermediate rendered image.

[0010] Optionally, in one embodiment of the present application, the Gaussian sputtering model is trained based on the discriminator and the generator according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model, which specifically includes: inputting the scene image and the intermediate rendered image into the discriminator to obtain a discrimination result; optimizing the Gaussian sputtering model according to the discrimination result to obtain a generative adversarial loss of the generative adversarial model; calculating the scene depth information corresponding to the scene image, calculating the rendering depth information corresponding to the intermediate rendered image, and obtaining a depth constraint loss according to the scene depth information and the rendering depth information; obtaining a total loss function of the Gaussian sputtering model according to the generative adversarial loss and the depth constraint loss; and updating the Gaussian sputtering model based on the total loss function to obtain an optimized Gaussian sputtering model.

[0011] Optionally, in one embodiment of the present application, the discriminator includes a first fully connected layer, a multi-layer perceptron hybrid layer, a global pooling layer and a second fully connected layer; the scene image and the intermediate rendered image are input into the discriminator to obtain a discrimination result, which specifically includes: splitting the scene image to obtain multiple image blocks; the first fully connected layer processes the multiple image blocks to obtain embedded features; the multi-layer perceptron hybrid layer obtains image features based on the embedded features; the global pooling layer and the second fully connected layer discriminate the image features to obtain a discrimination result.

[0012] Optionally, in one embodiment of the present application, the loss function of the generative adversarial model is expressed as:

[0013]

[0014] in, represents the generative adversarial loss, is the symbol of mathematical expectation, x~P r Represents the real data distribution, x~P g represents the data distribution generated by the generator; D(x) represents the output of the discriminator for the input x; Indicates about the sample The gradient of , ||·||2 represents the Euclidean norm, and λ is a hyperparameter; Represents the distribution of real data P r The expected value of the data point x sampled from the discriminator D; Denotes the rendering data distribution P generated from the generator G g The expected value of the data point x sampled from the discriminator D.

[0015] Optionally, in one embodiment of the present application, the total loss function of the optimized Gaussian sputtering model is expressed as:

[0016]

[0017] in, is the total loss, is the mean absolute error, 1-λ1 is the weight of the mean absolute error; is the structural similarity error, λ1 is the weight of the structural similarity error; is the generator loss of the generated adversarial loss, λ2 is the full weight of the generated adversarial loss; is the depth constraint loss, and λ3 is the weight of the depth constraint loss.

[0018] Optionally, in one embodiment of the present application, the Gaussian sputtering model is updated based on the total loss function to obtain the optimized Gaussian sputtering model, which specifically includes: weighting each loss according to the total loss function to obtain the current weight of each loss; obtaining the current losses corresponding to the intermediate rendered image based on the Gaussian sputtering model and the current weight; updating the Gaussian sputtering model according to the current losses based on the total loss function, and repeating the above steps of calculating the current losses based on the updated Gaussian sputtering model, and continuing to update the Gaussian sputtering model according to the current losses, until the updated Gaussian sputtering model meets the preset requirements, and then using the last updated Gaussian sputtering model as the optimized Gaussian sputtering model.

[0019] The second aspect of the embodiment of the present application further provides a new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, wherein the new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes:

[0020] An image acquisition module, used to obtain a scene image from a reconstructed scene perspective;

[0021] An image rendering module, used for obtaining an iterative intermediate rendering image according to the scene image based on a pre-built Gaussian sputtering model;

[0022] A model training module, used for constructing a discriminator of a generative adversarial model, taking the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model;

[0023] The target image output module is used to obtain the target camera posture and input the target camera posture into the optimized Gaussian sputtering model to obtain the target scene image.

[0024] The third aspect of an embodiment of the present application also provides a terminal, wherein the terminal includes: a memory, a processor, and a new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering stored in the memory and executable on the processor, wherein the new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering implements the steps of the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described above when executed by the processor.

[0025] The fourth aspect of an embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a new perspective synthesis program based on a generative adversarial strategy and Gaussian sputtering, and when the new perspective synthesis program based on a generative adversarial strategy and Gaussian sputtering is executed by a processor, the steps of the new perspective synthesis method based on a generative adversarial strategy and Gaussian sputtering as described above are implemented.

[0026] Beneficial effects: The present application provides a new perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering. In this method, the parameters of the original Gaussian sputtering model are gradually optimized through the generative adversarial strategy, so that the generated rendering image is more realistic, the visual artifacts in the rendering results are reduced, and the details in the scene are better fitted, thereby enhancing the rendering effect of the scene details. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of a preferred embodiment of the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0029] Figure 2 It is a Gaussian sputtering model flow chart in a preferred embodiment of the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0030] Figure 3 It is a flowchart of specific implementation steps of the entire execution process in a preferred embodiment of the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0031] Figure 4 This is a diagram of the discriminator network structure in a preferred embodiment of the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0032] Figure 5 This is a rendering effect comparison diagram of a preferred embodiment of the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0033] Figure 6 It is a structural diagram of a preferred embodiment of the new perspective synthesis system based on the generative adversarial strategy and Gaussian sputtering of the present application;

[0034] Figure 7 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0035] Description of reference numerals:

[0036] 100, image acquisition module; 200, image rendering module; 300, model training module; 400, target image output module. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and effect of the present application clearer and more specific, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. The described embodiments are only possible technical implementations of the present application, not all possible implementations. Based on the embodiments in the present application, those skilled in the art can completely combine the embodiments of the present application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of the present application.

[0038] At present, there are still some problems with 3D Gaussian sputtering technology that need further research and resolution: Although 3D Gaussian sputtering is conducive to representing complex geometries, it will produce unexpected visual artifacts; in the synthesis results, floc-like floating objects will be generated close to the viewing angle, affecting the rendering effect; there is excessive blurring of the details of the scene, such as mirror highlight areas and solid color areas with textures.

[0039] The following describes a new perspective synthesis method and system based on generative adversarial strategy and Gaussian sputtering according to an embodiment of the present application with reference to the accompanying drawings. In view of the problem that the reconstruction of the three-dimensional Gaussian sputtering scene in the above-mentioned related technology is prone to produce visual artifacts, resulting in poor rendering effects, the present application provides a new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering. In this method, the parameters of the original Gaussian sputtering model are gradually optimized through the generative adversarial strategy, so that the generated rendering image is more realistic, the visual artifacts in the rendering results are reduced, and the details in the scene are better fitted, thereby enhancing the rendering effect of the scene details. Thus, the technical problem that the reconstruction of the three-dimensional Gaussian sputtering scene in the related technology is prone to produce visual artifacts, resulting in poor rendering effects, is solved.

[0040] This application can obtain higher quality of new perspectives, while the engineering implementation is simple. First, collect images or videos containing each perspective of the scene to be reconstructed, and calibrate them through the motion recovery structure algorithm, obtain camera parameters, determine the camera pose, and generate sparse point clouds to construct the data set required for training; then, use a monocular depth estimation algorithm to calculate the depth information of the images in the data set; then, use a multi-layer perceptron and a convolutional neural network to build a discriminator network for the generative adversarial model; then, input the sparse point cloud as the initial point cloud into the Gaussian sputtering model, input images of each perspective and their camera poses and train them, use the Gaussian sputtering model as the generator of the generative adversarial model, and output the optical information and depth information of the rendered images of each perspective during the training process; then, calculate the scene images of each perspective and the output of the Gaussian sputtering model The average absolute value error and structural similarity error of the rendered corresponding perspective image optical information are minimized, and the parameters of the Gaussian sputtering model are optimized; then, the optical information of each perspective image of the data set and the corresponding perspective image output by the Gaussian sputtering model are input into the discriminator and trained, and the discriminator feeds back the quality of the rendering result to the Gaussian sputtering model as a generator, minimizes the generation adversarial loss, and improves the optimization process of the Gaussian sputtering model parameters; then, the Pearson correlation coefficient of the depth of each perspective scene image and the depth information of the corresponding perspective image output by the Gaussian sputtering model is calculated to minimize the depth constraint loss; finally, after the training is completed, any camera pose is input into the constructed Gaussian sputtering model, and the scene image under the corresponding perspective is rendered and output. The generative adversarial strategy designed in this application can improve the original Gaussian sputtering model, reduce the visual artifacts in the rendering results, better fit the details in the scene, and enhance the rendering effect of the scene details. The depth constraint used in this application can reduce the generation of flocs in the scene, and the use of correlation as weak supervision can enhance the stability of model training and reduce the impact of depth estimation errors on model training.

[0041] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0042] The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering described in the preferred embodiment of the present application is as follows: Figure 1 As shown, the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering includes the following steps:

[0043] In step S101, a scene image under a reconstructed scene viewing angle is obtained.

[0044] Specifically, the camera shooting posture and camera intrinsic parameter matrix of the input scene image are calibrated through the motion recovery algorithm and a sparse point cloud of the scene is generated. Specifically, it includes: shooting images of each viewpoint of the scene from different angles to ensure uniform light during shooting. If the original file is in video format, it needs to be converted into image format by extracting key frames before subsequent processing; the camera is calibrated on the acquired images using the motion structure recovery algorithm to obtain the camera intrinsic and extrinsic parameters of each image, determine the camera position and posture, reconstruct the sparse point cloud and camera trajectory of the scene, and generate a sparse three-dimensional point cloud.

[0045] It can be understood that the scene image is taken: each view of the scene is taken from different angles to ensure uniform lighting; if the video is taken, the key frames are extracted and converted into image format. Camera calibration: the camera is calibrated on the acquired images using the motion recovery structure (SfM) algorithm; the camera intrinsic parameters (such as focal length, optical center, etc.) and extrinsic parameters (such as camera position, posture, etc.) of each image are obtained; the sparse point cloud and camera trajectory of the scene are reconstructed.

[0046] In step S102, based on the pre-constructed Gaussian sputtering model, an iterative intermediate rendering image is obtained according to the scene image.

[0047] In a possible implementation, the scene image is calibrated with a camera to obtain a camera pose; the scene image and the camera pose are input into the Gaussian sputtering model to obtain an initial rendered image; the mean absolute value error and the structural similarity error between the initial rendered image and the scene image are calculated; gradient back propagation is performed based on the mean absolute value error and the structural similarity error to obtain an optimized Gaussian sputtering model; based on the optimized Gaussian sputtering model, iterative processing is performed based on the initial rendered image to obtain an intermediate rendered image after iteration.

[0048] Specifically, Figure 2 As shown in the figure, the sparse point cloud is used as the initial point cloud to input the Gaussian sputtering model, and the images of each view and their camera poses are input and trained. Then the Gaussian sputtering model is used as the generator of the generative adversarial model, and the optical information (rendered image) and depth information (rendered depth image) of each view image rendered after each round of iteration are output during the training process.

[0049] It can be understood that the Gaussian sputtering model training: the sparse point cloud is input as the initial point cloud into the Gaussian sputtering model, and the images of each perspective and their camera poses are input for training. During the training process, the optical information and depth information of the images of each perspective rendered after each round of iteration are output.

[0050] Specifically, Figure 2As shown, the Gaussian sputtering model calculates and minimizes the average absolute value error and structural similarity error of the optical information of each perspective scene image and the corresponding perspective image rendered by the Gaussian sputtering model output, and then the gradient is back-propagated to optimize the Gaussian sputtering model parameters.

[0051] It can be understood that the loss function is minimized: the mean absolute error (L1 loss) and structural similarity error (DSSIM loss) between the rendered image output by the Gaussian sputtering model and the real scene image are calculated, and the parameters of the Gaussian sputtering model are optimized using gradient back propagation.

[0052] In step S103, a discriminator of a generative adversarial model is constructed, and the Gaussian sputtering model is used as a generator of the generative adversarial model. Based on the discriminator and the generator, the Gaussian sputtering model is trained according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model.

[0053] In a possible implementation, the scene image and the intermediate rendered image are input into the discriminator to obtain a discrimination result; the Gaussian sputtering model is optimized according to the discrimination result to obtain a generative adversarial loss of the generative adversarial model; the scene depth information corresponding to the scene image is calculated, the rendering depth information corresponding to the intermediate rendered image is calculated, and the depth constraint loss is obtained according to the scene depth information and the rendering depth information; the total loss function of the Gaussian sputtering model is obtained according to the generative adversarial loss and the depth constraint loss; the Gaussian sputtering model is updated based on the total loss function to obtain the optimized Gaussian sputtering model.

[0054] Specifically, a monocular depth estimation algorithm is used to calculate the depth information of the images in the dataset. The optical information of the scene images at each perspective is input (input scene image), the distance of each pixel in the image relative to the camera is estimated, and the depth information of the scene images at each angle is output.

[0055] In a possible implementation, the discriminator includes a first fully connected layer, a multi-layer perceptron hybrid layer, a global pooling layer, and a second fully connected layer. The scene image is split to obtain multiple image blocks; the first fully connected layer processes the multiple image blocks to obtain embedded features; the multi-layer perceptron hybrid layer obtains image features based on the embedded features; the global pooling layer and the second fully connected layer discriminate the image features to obtain a discrimination result.

[0056] Specifically, a multi-layer perceptron and a convolutional neural network are used to construct a discriminator network for generating adversarial models. Figure 4As shown in the figure, the specific steps are as follows: first, the input image is split into multiple patches, and then a fully connected network is used to process all patches and convert them into embedded features. Then, the token mixing layer and the channel mixing layer are cross-used to construct a multi-layer perceptron mixing layer to extract the input image features, and finally the global average pooling layer and the fully connected layer are used to discriminate the image.

[0057] In the embodiment of the present application, the loss function of the generated adversarial model is:

[0058]

[0059] In this formula, the meanings of the symbols are as follows:

[0060] represents the distance loss function; It is the symbol of mathematical expectation, which means to calculate the average value of a random variable; x~P r Represents the real data distribution; x~P g Represents the data distribution generated by the generator; D(x) represents the output of the discriminator for the input x, which is usually a probability value indicating whether the input comes from the real data distribution; Indicates about the sample The gradient of the vector; ||·||2 represents the Euclidean norm, which is the Euclidean norm, and calculates the square root of the sum of the squares of the elements of the vector; λ is a hyperparameter used to adjust the balance in adversarial training; the whole formula describes the loss function in GAN, which measures the quality of the generative model by comparing the difference between the probability distribution of the discriminator for real data and the generated data. At the same time, the third term in the formula is a constraint on the smoothness of the discriminator gradient, which ensures that the discriminator gradient will not be too large or too small, thereby avoiding the problem of gradient disappearance or explosion.

[0061] Part I Represents the distribution of real data P r The expected value of the data point x sampled in the discriminator D. The goal of this part is to make the discriminator classify the real data as "true" as much as possible; the second part Denotes the rendering data distribution P generated from the generator G g The expected value of the sampled data point x after passing through the discriminator D. The goal of this part is to let the discriminator classify the rendered data generated by the generator as "fake" as much as possible. Part III It is an important constraint introduced in WGAN-GP, called "gradient penalty". Here λ is a weight constant used to control the influence of this part. The goal of this part is to limit the size of the discriminator's weight matrix so that it does not exceed a certain threshold, so as to avoid the gradient disappearance or explosion.

[0062] It can be understood that the discriminator network is constructed by: using multi-layer perceptron (MLP) and convolutional neural network (CNN) to build the discriminator of the generative adversarial model GAN; splitting the input image into multiple patches (image blocks), using a fully connected network to process the patches and convert them into embedded features; cross-using token mixing layers and channel mixing layers to build MLP mixing layers to extract input image features; using global average pooling layers and fully connected layers to discriminate images.

[0063] It can be understood that the discriminator loss function: the discriminator needs to maximize the difference between the real data and the generated data; the loss function consists of two parts: the difference between the expected value of the real data passing through the discriminator and the expected value of the generated data passing through the discriminator; in order to ensure the Lipschitz continuity of the loss function, a gradient penalty term must be added.

[0064] In one possible implementation, weights of various losses are allocated according to the total loss function to obtain current weights of various losses; current losses corresponding to the intermediate rendered image are obtained based on the Gaussian sputtering model and the current weights; based on the total loss function, the Gaussian sputtering model is updated according to the current losses, and the above steps of calculating the current losses based on the updated Gaussian sputtering model and continuing to update the Gaussian sputtering model according to the current losses are repeated until the updated Gaussian sputtering model meets preset requirements, and the Gaussian sputtering model after the latest update is used as the optimized Gaussian sputtering model.

[0065] In this application, a generative adversarial strategy (model) is used to improve the Gaussian sputtering model parameter optimization process. A discriminator module is added to the original Gaussian sputtering model, the original Gaussian sputtering model is used as a generator, and the Gaussian sputtering model parameter optimization is performed using a generative adversarial strategy.

[0066] Specifically, Figure 3 As shown in the figure, the optical information of each view image of the dataset and the corresponding view image output by the Gaussian sputtering model are input into the discriminator and trained. The discriminator maximizes the loss of the generated adversarial model and outputs the discrimination result for the input image. The Gaussian sputtering model improves the parameter optimization according to the result output by the discriminator and minimizes the loss of the generated adversarial model. It only contains the discriminator and real data samples, which has no effect on the generator gradient and can be ignored. Its generation adversarial loss is:

[0067]

[0068] It can be understood that GAN training: the Gaussian sputtering model is used as the generator of GAN; the images of each perspective of the dataset and the rendered images output by the Gaussian sputtering model are input into the discriminator for training; the discriminator maximizes the GAN loss and outputs the discrimination result of the input image; the Gaussian sputtering model improves the parameters according to the discriminator result and minimizes the GAN loss.

[0069] The discriminator needs to maximize this loss, that is, to distinguish the difference between the original image and the image generated by the Gaussian sputtering model. The discriminator loss needs to maximize L W , that is, maximize:

[0070]

[0071] Since the gradient descent method is used, the discriminator loss needs to be minimized That is, minimize:

[0072]

[0073] The generator loss needs to be minimized So:

[0074]

[0075] because It only contains the discriminator and real data samples, which has no effect on the generator gradient and can be ignored, so its loss is:

[0076]

[0077] In order to ensure the Lipschitz continuity of the loss function (the rate of change of a function between any two points in the domain of definition will not exceed a constant), the discriminator network loss needs to add a gradient penalty term; at the same time, in order to ensure the Lipschitz continuity of the loss function, the discriminator network loss needs to add a gradient penalty term:

[0078]

[0079] It can be understood that the depth constraint loss: calculates the Pearson correlation coefficient between the depth of the scene image at each perspective and the rendered depth image output by the Gaussian sputtering model; uses the depth constraint to reduce the generation of flocculent floating objects and optimizes the reconstruction of the scene structure. Total loss function: combines L1 loss, DSSIM loss, GAN loss and depth constraint loss to form a total loss function; assigns weights (λ_1, λ_2, λ_3) to each loss to balance their influence in the optimization process.

[0080] Specifically, the Pearson correlation coefficient between the depth of each viewing angle scene image and the depth information of the corresponding viewing angle image output by the Gaussian sputtering model is calculated, and the calculation formula is:

[0081]

[0082] Among them, D r To render the depth image, D g is the scene depth image. The Gaussian sputtering model needs to minimize the depth constraint loss. The use of depth constraint can reduce the generation of flocculent floating objects and optimize the reconstructed scene structure. Therefore, the total loss function of the improved Gaussian sputtering model of the present invention is:

[0083]

[0084] is the total loss function, is the mean absolute value error of the optical information of the scene image at each viewing angle and the corresponding viewing angle image rendered by the Gaussian sputtering model output, and 1-λ1 is its weight; is the structural similarity error between the optical information of the scene image at each viewing angle and the corresponding viewing angle image rendered by the Gaussian sputtering model output, and λ1 is its weight; is the generator loss for generating adversarial loss, and λ2 is its full weight; is the depth constraint loss and λ2 is its weight.

[0085] In this application, depth constraints are used to reduce the generation of flocculent floating objects. By pre-calculating the depth information of the scene and comparing it with the depth information output by the Gaussian sputtering model during the training process, the depth of the three-dimensional Gaussian primitive is corrected to make its position more accurate and reduce the erroneous generation of flocculent floating objects.

[0086] In step S104, the target camera posture is acquired, and the target camera posture is input into the optimized Gaussian sputtering model to obtain a target scene image.

[0087] Specifically, after the training is completed, any camera pose is input into the constructed Gaussian sputtering model, and the scene image under the corresponding perspective is rendered and output, that is, the new perspective image (that is, the target scene image).

[0088] It should be noted that in another embodiment of the present application, other types of neural networks (such as convolutional neural networks, Transformer, etc.) can also be used as discriminators, or the loss function of the generative adversarial model can be modified. In another embodiment, different formulas can also be used to calculate the depth constraint, such as other correlation coefficients.

[0089] This application tests the effect of the method of the present invention and the original three-dimensional Gaussian sputtering algorithm on the Mip-NeRF360 dataset (unbounded anti-aliased neural radiation field dataset), Tanks&Temples dataset (tank and template dataset), and Deep Blending dataset (deep blending dataset). The rendering effect examples are as follows: Figure 5 The indicators are shown in Table 1. The comparison indicators are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), the larger the value, the higher the rendering quality; Learning Perceptual Image Patch Similarity (LPIPS), the smaller the value, the higher the rendering quality.

[0090] Table 1: Corresponding index table of the original three-dimensional Gaussian sputtering algorithm and the method of this application

[0091]

[0092]

[0093] This application can show good image quality on multiple data sets, which is better than the original three-dimensional Gaussian sputtering method.

[0094] Secondly, a new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0095] Figure 6 This is a structural diagram of a new perspective synthesis system based on a generative adversarial strategy and Gaussian sputtering according to an embodiment of the present application.

[0096] like Figure 6 As shown, the new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes: an image acquisition module 100, an image rendering module 200, a model training module 300 and a target image output module 400.

[0097] Specifically, the image acquisition module 100 is used to obtain a scene image under the perspective of the reconstructed scene;

[0098] An image rendering module 200 is used to obtain an iterative intermediate rendering image according to the scene image based on a pre-built Gaussian sputtering model;

[0099] A model training module 300, for constructing a discriminator of a generative adversarial model, using the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model;

[0100] The target image output module 400 is used to obtain the target camera posture and input the target camera posture into the optimized Gaussian sputtering model to obtain the target scene image.

[0101] Figure 7 This is a structural diagram of a terminal provided in an embodiment of the present application. The terminal may include:

[0102] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0103] When the processor 502 executes the program, the new perspective synthesis method based on the generative adversarial strategy and Gaussian sputtering provided in the above embodiment is implemented.

[0104] Furthermore, the terminal further includes:

[0105] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0106] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0107] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0108] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EIS) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0110] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0111] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described above.

[0112] One embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the present application Figure 1 Any of the corresponding embodiments provides a new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering.

[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in a computer memory.

[0117] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0121] It should be understood that the application of the present application is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to this application.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering, characterized in that: The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering includes: Obtaining a scene image from a reconstructed scene perspective; Based on a pre-built Gaussian sputtering model, obtaining an iterative intermediate rendering image according to the scene image; Constructing a discriminator of a generative adversarial model, using the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model based on the discriminator and the generator according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model; The target camera pose is obtained, and the target camera pose is input into the optimized Gaussian sputtering model to obtain a target scene image.

2. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 1, characterized in that: The step of obtaining an iterative intermediate rendering image according to the scene image specifically includes: Performing camera calibration on the scene image to obtain a camera pose; Inputting the scene image and the camera pose into the Gaussian sputtering model to obtain an initial rendered image; Calculating the mean absolute value error and the structural similarity error between the initial rendered image and the scene image; Gradient back propagation is performed according to the mean absolute value error and the structural similarity error to obtain an optimized Gaussian sputtering model. Based on the optimized Gaussian sputtering model, iterative processing is performed according to the initial rendered image to obtain an iterated intermediate rendered image.

3. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 1, characterized in that: The step of training the Gaussian sputtering model based on the discriminator and the generator according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model specifically includes: Inputting the scene image and the intermediate rendering image into the discriminator to obtain a discrimination result; Optimizing the Gaussian sputtering model according to the discrimination result to obtain a generative adversarial loss of the generative adversarial model; Calculating scene depth information corresponding to the scene image, calculating rendering depth information corresponding to the intermediate rendered image, and obtaining a depth constraint loss according to the scene depth information and the rendering depth information; Obtaining a total loss function of the Gaussian sputtering model according to the generative adversarial loss and the depth constraint loss; The Gaussian sputtering model is updated based on the total loss function to obtain the optimized Gaussian sputtering model.

4. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that: The discriminator includes a first fully connected layer, a multi-layer perceptron hybrid layer, a global pooling layer, and a second fully connected layer; The step of inputting the scene image and the intermediate rendered image into the discriminator to obtain a discrimination result specifically includes: Splitting the scene image to obtain multiple image blocks; The first fully connected layer processes the plurality of image blocks to obtain embedded features; The multi-layer perceptron mixing layer obtains image features according to the embedded features; The global pooling layer and the second fully connected layer discriminate the image features to obtain a discrimination result.

5. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that: The loss function of the generative adversarial model is expressed as: in, represents the generative adversarial loss, is the symbol of mathematical expectation, x~P r Represents the real data distribution, x~P g represents the data distribution generated by the generator; D(x) represents the output of the discriminator for the input x; Indicates about the sample The gradient of , ∥·∥2 represents the Euclidean norm, and λ is a hyperparameter; Represents the distribution of real data P r The expected value of the data point x sampled from the discriminator D; Denotes the rendering data distribution P generated from the generator G g The expected value of the data point x sampled from the discriminator D.

6. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 3, characterized in that: The total loss function of the optimized Gaussian sputtering model is expressed as: in, is the total loss, is the mean absolute error, 1-λ1 is the weight of the mean absolute error; is the structural similarity error, λ1 is the weight of the structural similarity error; is the generator loss of the generated adversarial loss, λ2 is the full weight of the generated adversarial loss; is the depth constraint loss, and λ3 is the weight of the depth constraint loss.

7. The new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering according to claim 6, characterized in that: The updating of the Gaussian sputtering model based on the total loss function to obtain the optimized Gaussian sputtering model specifically includes: Allocate weights of various losses according to the total loss function to obtain current weights of various losses; Based on the Gaussian sputtering model and the current weight, obtaining each current loss corresponding to the intermediate rendered image; Based on the total loss function, the Gaussian sputtering model is updated according to the current losses, and the above steps of calculating the current losses based on the updated Gaussian sputtering model and continuing to update the Gaussian sputtering model according to the current losses are repeated until the updated Gaussian sputtering model meets the preset requirements, and the last updated Gaussian sputtering model is used as the optimized Gaussian sputtering model.

8. A new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering, characterized in that: The new perspective synthesis system based on generative adversarial strategy and Gaussian sputtering includes: An image acquisition module, used to obtain a scene image from a reconstructed scene perspective; An image rendering module, used for obtaining an iterative intermediate rendering image according to the scene image based on a pre-built Gaussian sputtering model; A model training module, used for constructing a discriminator of a generative adversarial model, taking the Gaussian sputtering model as a generator of the generative adversarial model, and training the Gaussian sputtering model based on the discriminator and the generator according to the scene image and the intermediate rendered image to obtain an optimized Gaussian sputtering model; The target image output module is used to obtain the target camera posture and input the target camera posture into the optimized Gaussian sputtering model to obtain the target scene image.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering stored in the memory and executable on the processor. When the new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering is executed by the processor, the steps of the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering. When the new perspective synthesis program based on generative adversarial strategy and Gaussian sputtering is executed by a processor, the steps of the new perspective synthesis method based on generative adversarial strategy and Gaussian sputtering as described in any one of claims 1-7 are implemented.

Images