New viewpoint synthesis method based on video diffusion model and feature embedding guidance

Abstract

The invention discloses a new viewpoint synthesis method based on a video diffusion model and feature embedding guidance, which belongs to the technical field of new viewpoint synthesis, and comprises the following steps: constructing a multi-scene viewpoint data set comprising images of multiple viewpoints in multiple scenes and camera poses, and dividing the multi-scene viewpoint data set into a training set and a test set, preprocessing viewpoint data in the training data set in the previous step, carrying out size adjustment and linear normalization on viewpoint images, carrying out one-dimensional processing on a camera pose to obtain a camera track, and carrying out camera pose light field coding on the camera pose to obtain a camera pose light field; and inputting the obtained viewpoint image into an encoder part of the VAE auto-encoder, and generating potential image features of the input viewpoint, so that the three-dimensional consistency and viewpoint accuracy of a video diffusion model generation result are enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of novel view synthesis, and particularly relates to a novel view synthesis method based on a video diffusion model and feature embedding guidance. BACKGROUND

[0002] The novel view synthesis task is an important task in the field of computer vision, aiming to generate images under a new view based on a single or multiple known view images through algorithmic means. Novel view synthesis has wide application prospects and research significance in multiple fields, such as scene roaming, virtual reality, three-dimensional object and scene reconstruction, etc.

[0003] Traditional view synthesis methods are usually based on geometric reconstruction or image interpolation, but they have certain limitations in practical applications. Geometric reconstruction relies on accurate depth estimation or point cloud data, and it is difficult to handle occluded areas and irregular surfaces. Image interpolation methods are prone to blurring or distortion, especially in cases of large parallax. These traditional methods are difficult to achieve high-quality and realistic view synthesis in real scenes.

[0004] In recent years, view synthesis methods based on deep learning have made significant progress, and can be roughly divided into two categories according to the scene modeling method and image generation mechanism: one is the explicit modeling method based on three-dimensional scene geometric reconstruction; the other is the implicit generation method based on image latent space representation. The former usually combines depth estimation and volume rendering technology to reconstruct three-dimensional structures, while the latter utilizes the generation ability of latent space features to achieve efficient novel view image synthesis. Both have their own advantages in terms of precision, efficiency and generalization ability, and constitute an important branch of current novel view synthesis research.

[0005] Among them, the generation method based on the diffusion model is a hot research direction of the latter. Some works use the strong prior ability of the diffusion model to handle the problems of sparse view input and scene occlusion; some works enhance the consistency of model generation by adding image feature embedding guidance; some works enhance the view change ability of model generation by using the time layer of video diffusion model. These works show that diffusion model, especially video diffusion model, has advantages in new view synthesis tasks such as high image quality, strong generalization ability and adaptation to sparse views. However, the accuracy of the view and the generation of three-dimensional consistency are required in the new view synthesis task, and in many application scenarios, relying on explicit three-dimensional structure will bring conditional restrictions or introduce additional errors. In order to solve these problems, how to introduce camera pose control diffusion model generation process without relying on explicit three-dimensional structure to realize highly consistent new view images and video sequences in real scenes is still a major challenge in current research. The current new view synthesis method based on diffusion model does not rely on explicit three-dimensional structure, and there is a problem of insufficient three-dimensional scene consistency constraint, which often causes three-dimensional consistency error in generated images; the current new view synthesis method based on diffusion model does not rely on explicit three-dimensional structure, and there is a problem of low camera pose control precision, which often causes camera pose deviation in generated images. SUMMARY

[0006] To achieve the above purpose, the present application is a new view synthesis method based on video diffusion model and feature embedding guidance, which proposes a double-branch camera pose embedding guidance scheme, one branch uses variable resolution camera pose light field coding to directly splice the camera pose to the U-Net input end, and the other branch uses a camera trajectory encoder to encode the one-dimensional camera trajectory into a camera pose condition embedding.

[0007] The present application proposes an image feature encoder which can extract cross-view image features from the images of the input view. The present application proposes a scene feature encoder which can fuse cross-view image features to generate a scene feature condition embedding. The present application proposes a camera pose and scene feature jointly guided video diffusion model framework which uses camera pose and scene feature to guide the generation process of the video diffusion model at the same time, and strengthens the three-dimensional consistency and view accuracy of the video diffusion model generation result.

[0008] Specifically, the technical scheme of the present application is as follows: comprising the following steps:

[0009] Step 1: Construct a multi-scene view dataset containing multiple views of multiple scenes and camera poses, and divide it into a training set and a test set;

[0010] Step 2: Preprocess the viewpoint data in the training dataset in step 1, resize and linearly normalize the viewpoint images, and one-dimensionize the camera poses to obtain camera trajectories, and simultaneously camera pose light field encoding is performed on the camera poses to obtain camera pose light field;

[0011] Step 3: Input the viewpoint images obtained in step 2 into the encoder part of the VAE autoencoder to generate image latent features of the input viewpoint;

[0012] Step 4: Randomly sample a real noise and randomly generate a time step;

[0013] Step 5: Add noise to the image latent features obtained in step 3 based on the real noise and time step obtained in step 4 to obtain noise latent features;

[0014] Step 6: Concatenate the camera pose light field obtained in step 2 and the noise latent features obtained in step 5 in the channel dimension to obtain U-Net input features;

[0015] Step 7: Pass the camera trajectory obtained in step 2 through a camera trajectory encoder to obtain camera pose conditional embedding;

[0016] Step 8: Extract the first and last frames of the viewpoint images obtained in step 2, and sequentially pass them through an image feature encoder and a scene feature encoder to obtain scene feature conditional embedding;

[0017] Step 9: Input the U-Net input features obtained in step 6 into the U-Net network of the video diffusion model, and use the camera pose conditional embedding obtained in step 7 and the scene feature conditional embedding obtained in step 8 as condition guidance, respectively embedded in the spatial attention layer and the temporal attention layer of the video diffusion model U-Net, and finally output the predicted noise;

[0018] Step 10: Freeze the parameters of the VAE encoder, calculate the loss value by comparing the predicted noise obtained in step 9 with the real noise obtained in step 4, and then optimize the loss to the global minimum value, and train to obtain a new viewpoint synthesis video diffusion model. The parameters of this model include a camera trajectory encoder, an image feature encoder, a scene feature encoder, and a video diffusion model U-Net network;

[0019] Step 11: Freeze the parameters of the new viewpoint synthesis video diffusion model obtained in step 10, and obtain a set of test data from step 1, and obtain viewpoint images, camera trajectories and camera pose light fields according to the processing in step 2. The first and last frames of the viewpoint images are selected as input viewpoint images, and the remaining images are selected as target viewpoint images;

[0020] Step 12: Randomly sample a set of Gaussian noise and set a fixed time step sequence;

[0021] Step 13: The input view image obtained in step 11 is processed to generate image latent features according to step 3, and the first and last noises in the set of Gaussian noises obtained in step 12 are replaced with the maximum time step in the fixed time step sequence to obtain a set of noise latent features;

[0022] Step 14: The noise latent features obtained in step 13 and the input view image, camera trajectory and camera pose light field obtained in step 11 are processed according to the method of steps 6-9 to obtain predicted noise, and the noise latent features obtained in step 13 are denoised based on the predicted noise;

[0023] Step 15: The denoising process of step 14 is repeatedly performed according to the time step sequence obtained in step 12 to finally obtain denoised image latent features;

[0024] Step 16: The denoised image latent features obtained in step 15 are processed through the decoder part of the VAE autoencoder to obtain generated target view images.

[0025] Preferably, a double-branch camera pose embedding guidance scheme is used in steps 6 and 7;

[0026] The first branch uses camera pose light field encoding to directly splice the camera pose into the input end of the U-Net, and uses a variable resolution pose light field encoding, which is expressed as follows:

[0027] (K, R, t)→(o, d)∈R H×W×6 →LF∈R H×W×C (1),

[0028] Where K is the camera pose intrinsic matrix, R is the camera pose extrinsic rotation matrix, and t is the camera pose extrinsic translation vector; LF is the target feature light field; (o, d) is the channel structure of the target feature light field, where o is the light field origin and d is the light field direction; H is the length of the target feature light field; W is the width of the target feature light field; and C is the number of channels of the target feature light field.

[0029] More specifically,

[0030] o i ∈R H×W×3 =T (2),

[0031] d i ∈R H×W×3 =R·K -1· (u,v,1) (3),

[0032] LF s =(o,d) (4),

[0033] LF = [sin(2 0 πLF s ), cos(2 0 πLF s ),..., sin(2 L-1 πLF s ), cos(2 L-1 πLF s )] (5),

[0034] where u, v are the pixel coordinates of the image, LFs is obtained by concatenating o, d in the channel dimension, and L is the length of the cosine position encoding, which is set to L = 15 by default.

[0035] The second branch uses a camera trajectory encoder to encode the one-dimensional camera trajectory CT into the camera pose condition embedding Cpose. This process can be represented as:

[0036] C pose pose = E CT (CT) (6),

[0037] where the camera trajectory CT is defined as a series of time-ordered camera poses CT = {P1, P2, …, PN}, each camera pose Pi is composed of a rotation matrix Ri ∈ R3x3 expanded into a one-dimensional vector and a translation vector ti ∈ R3, with a length of 12, representing the direction and position of the camera in the i-th frame of the video sequence; the camera pose encoder ECT is composed of 6 self-attention blocks, each containing a self-attention layer and an MLP layer. In the self-attention layer, the camera trajectory CT is converted into Q, K, V through a linear layer, and finally into the output feature, the process is as follows:

[0038]

[0039] Preferably, an image feature encoder Eimage proposed in step 8 extracts the cross-view image feature FV from the input viewpoint image, and the process can be represented as:

[0040] F V V = E image (I 1,…,N ) (8),

[0041] where the cross-view image feature FV is a sequence of image features, which can be represented as:

[0042] F V = [F V1 , F V2 , …, F VN ] (9),

[0043] The image feature encoder Eimage contains three parts, first using an encoder to encode the image into the feature space, then using a cross decoder to extract cross features from the image features of other input views, and finally obtaining the cross-input image features of the input image through a scene decoder.

[0044] In the image encoder part, a ViT-Large-based encoder architecture is used, which is composed of 24 self-attention blocks, each containing a self-attention layer and an MLP layer, with a hidden layer dimension of 1024, and 16 attention heads for attention mechanism. The two image encoders share the same network parameters; in the cross decoder, a ViT-Base-based decoder architecture is used, which has a total of 12 cross-attention blocks, each containing a self-attention layer, a cross-attention layer and an MLP layer, with a hidden layer dimension of 768, and 12 attention heads for attention mechanism; in the scene decoder module, 4 attention blocks are used, of which the first is a self-attention block, and the remaining 3 layers are cross-attention blocks, each block is based on the same architecture of ViT-Base, with a hidden layer dimension of 768 and 12 attention heads. The inputs of the 4 attention blocks are shown in the figure, respectively from the outputs of the image encoders and the uniformly sampled outputs of the 3 cross decoders from back to front, which are connected to the scene decoder module in an inverted manner. The detailed model structure of the cross-attention layer is shown in the figure. Figure 4

[0045] Preferably, a scene feature encoder Escene proposed in step 8 fuses and encodes the independent cross-view image features into a scene feature condition embedding Cscene, which can be represented as:

[0046] C scene =E scene (F V )(10),

[0047] The scene feature encoder Escene is composed of 6 self-attention blocks, each containing a self-attention layer and an MLP layer. In the self-attention layer, the cross-view image feature sequence FV is converted into Q, K, V through a linear layer, and finally into the output feature, the process is as follows:

[0048]

[0049] ​Preferably, a camera pose and scene feature co-embedding guided video diffusion model framework is adopted in step 9, and in the U-Net part of the video diffusion model, the scene feature embedding Cscene and the camera pose embedding Cpose will be embedded into the spatial attention layer and the temporal attention layer respectively, and the specific process is as follows:

[0050]

[0051] wherein, is the output of the self-attention layer in the temporal attention layer, Z Spa is the output of the self-attention layer in the spatial attention layer, and are the camera trajectory condition embedding and the scene feature condition embedding respectively.

[0052] Preferably, the loss function adopted in step 10 can be represented as:

[0053]

[0054] In the formula, ∈ represents the actual added real noise, t is the time step, Zt is the noisy latent feature at time t, LF is the camera pose light field, Cpose and Cscene are the camera pose condition embedding and the scene feature condition embedding respectively, ∈ θ is the noise predicted by the diffusion model U-Net under the above conditions, and the prediction error is minimized by mean square error loss.

[0055] Preferably, the channel splicing method used in step 6 modifies the channel dimension of the U-Net input feature, and according to the actual splicing dimension, such as the splicing dimension 184 used in the present application, the dimension of the first linear layer of the U-Net network is modified to adapt to the actual dimension size.

[0056] Preferably, the camera pose light field spliced by the channel splicing method used in step 6 only participates in the noise prediction of the U-Net network, and does not participate in the denoising step, and after each denoising process is completed, the original input is spliced with the new noise latent feature to participate in the noise prediction step in the next denoising process.

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] The present application not only utilizes the powerful prior of the pre-trained video diffusion model, but also proposes a scene feature condition embedding guided scheme, which effectively improves the generated image quality of the model without using explicit three-dimensional structure conditions, especially improves the three-dimensional consistency of the generated image, and compared with the recent method, the image quality index is improved by an average of 30%;

[0059] The application uses a double-branch camera pose embedding scheme, effectively improves the control accuracy of the model to the viewpoint without using an explicit three-dimensional structure condition, and the ablation experiment shows that the camera pose error of the generated viewpoint is reduced by an average of 20%. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The algorithm training framework of the application is shown in the figure;

[0061] Figure 2 The algorithm inference framework of the application is shown in the figure;

[0062] Figure 3 The self-attention layer structure of the application is shown in the figure;

[0063] Figure 4 The cross-attention layer structure of the application is shown in the figure;

[0064] Figure 5 The image feature encoder architecture of the application is shown in the figure;

[0065] Figure 6 The subjective result comparison of the indoor real scene dataset in the experiment of the application is shown in the figure;

[0066] Figure 7 The subjective result comparison of the outdoor real scene dataset in the experiment of the application is shown in the figure;

[0067] Figure 8 The camera pose light field ablation experiment of the application is shown in the figure. DETAILED DESCRIPTION

[0068] The application will be further illustrated in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and not to limit the scope of the application.

[0069] Embodiment: A scene new viewpoint synthesis method based on video diffusion model and feature embedding guidance, comprising the following steps:

[0070] Step 1: Construct a multi-scene viewpoint dataset containing multiple viewpoints of multiple scenes and camera poses, and divide it into a training set and a test set;

[0071] Step 2: Preprocess the viewpoint data in the training dataset in step 1, resize and linearly normalize the viewpoint image, one-dimensionize the camera pose to obtain the camera trajectory, and camera pose light field encoding to obtain the camera pose light field;

[0072] Step 3: Input the viewpoint image obtained in step 2 into the encoder part of the VAE autoencoder to generate the image latent features of the input viewpoint;

[0073] Step 4: Randomly sample a real noise, and randomly generate a time step;

[0074] Step 5: Add noise to the image latent feature obtained in step 3 based on the real noise obtained in step 4 and the time step to obtain a noise latent feature;

[0075] Step 6: Concatenate the camera pose light field obtained in step 2 and the noise latent feature obtained in step 5 in the channel dimension to obtain a U-Net input feature;

[0076] Step 7: Pass the camera trajectory obtained in step 2 through a camera trajectory encoder to obtain a camera pose conditional embedding;

[0077] Step 8: Extract the first and last frames of the view image obtained in step 2, and sequentially pass them through an image feature encoder and a scene feature encoder to obtain a scene feature conditional embedding;

[0078] Step 9: Input the U-Net input feature obtained in step 6 into the U-Net network of the video diffusion model, and use the camera pose conditional embedding obtained in step 7 and the scene feature conditional embedding obtained in step 8 as a condition guide, respectively embedded in the spatial attention layer and the time attention layer of the video diffusion model U-Net, and finally output a predicted noise;

[0079] Step 10: Freeze the parameters of the VAE encoder, calculate the loss value by comparing the predicted noise obtained in step 9 with the real noise obtained in step 4, and then optimize the loss to be a global minimum, and train to obtain a new view synthesis video diffusion model. The parameters of this model include a camera trajectory encoder, an image feature encoder, a scene feature encoder, and a video diffusion model U-Net network;

[0080] Step 11: Freeze the parameters of the new view synthesis video diffusion model obtained in step 10, obtain a test data set from step 1, and obtain view images, camera trajectories, and camera pose light fields according to the processing of step 2. The first and last frames of the view images are selected as input view images, and the remaining images are selected as target view images;

[0081] Step 12: Randomly sample a set of Gaussian noise, and set a fixed time step sequence;

[0082] Step 13: Generate image latent features for the input view images obtained in step 11 according to step 3, add noise to the first and last noise in the set of Gaussian noise obtained in step 12 and the maximum time step in the fixed time step sequence, replace the first and last noise in the set of Gaussian noise in step 12 with the obtained noise latent feature, and then obtain a set of noise latent features;

[0083] Step 14: Process the noise latent feature obtained in step 13 and the input view image, camera trajectory and camera pose light field obtained in step 11 according to the method of steps 6-9 to obtain predicted noise, and denoise the noise latent feature obtained in step 13 based on the predicted noise;

[0084] Step 15: Repeat the denoising process of step 14 according to the time step sequence obtained in step 12 to finally obtain the denoised image latent feature;

[0085] Step 16: Pass the denoised image latent feature obtained in step 15 through the decoder part of the VAE autoencoder to obtain the generated target view image.

[0086] Further, a double-branch camera pose embedding guidance scheme is used in steps 6 and 7;

[0087] The first branch uses camera pose light field encoding to directly splice the camera pose into the input end of the U-Net, which uses a variable resolution pose light field encoding, the expression is as follows:

[0088] (K, R, t)→(o, d)∈R H×W×6 →LF∈R H×W×C (1),

[0089] Where K is the camera pose intrinsic matrix, R is the camera pose extrinsic rotation matrix, t is the camera pose extrinsic translation vector; LF is the target feature light field; (o, d) is the channel structure of the target feature light field, where o is the light field origin and d is the light field direction; H is the length of the target feature light field; W is the width of the target feature light field; C is the number of channels of the target feature light field.

[0090] More specifically,

[0091] o i ∈R H×W×3 =T(2),

[0092] d i ∈R H×W×3 =R·K -1· (u,v,1) (3),

[0093] LF s =(o,d) (4),

[0094] LF=[sin(2 0 πLF s ),cos(2 0 πLF s ),...sin(2 L-1 πLF s ),cos(2 L-1 πLFs )] (5),

[0095] where u, v are the pixel coordinates of the image, LFs is concatenated by o, d in the channel dimension, and L is the length of the used cosine position encoding, which is set as L = 15 by default.

[0096] The second branch uses a camera trajectory encoder to encode the one-dimensional camera trajectory CT into the camera pose condition embedding Cpose. This process can be represented as:

[0097] C pose =E CT (CT) (6),

[0098] where the camera trajectory CT is defined as a series of time-ordered camera poses CT = {P1, P2, …, PN}, each camera pose Pi is composed of a rotation matrix Ri∈R3×3 expanded into a one-dimensional vector and a translation vector ti∈R3, with a length of 12, representing the direction and position of the camera in the i-th frame of the video sequence; the camera pose encoder ECT is composed of 6 self-attention blocks, each of which contains a self-attention layer and an MLP layer. The detailed model structure of the self-attention layer is shown in Figure 3 . In the self-attention layer, the camera trajectory CT is converted into Q, K, V through a linear layer, and finally becomes the output feature, the process is as follows:

[0099]

[0100] Further, an image feature encoder Eimage proposed in step 8 extracts the cross-view image feature FV from the input viewpoint image, the process can be represented as:

[0101] F V =E image (I 1,…,N ) (8),

[0102] where the cross-view image feature FV is a sequence of image features, which can be represented as:

[0103] F V =[F V1 ,F V2 ,…,F VN ](9),

[0104] The image feature encoder Eimage contains three parts, first, the image is encoded into the feature space using the encoder, then a cross decoder is used to extract cross features from the image features of other input viewpoints, and finally a scene decoder is used to obtain the cross-input image features of the input image. The specific model structure is shown in Figure 5 .

[0105] In the image encoder part, a ViT-Large-based encoder architecture is used, which consists of 24 self-attention Blocks, each containing a self-attention layer and an MLP layer, with a hidden layer dimension of 1024, and 16 attention heads for the attention mechanism. The two image encoders share the same network parameters. In the cross decoder, a ViT-Base-based decoder architecture is used, which has a total of 12 cross-attention Blocks, each containing a self-attention layer, a cross-attention layer, and an MLP layer, with a hidden layer dimension of 768 and 12 attention heads for the attention mechanism. In the scene decoder module, 4 attention Blocks are used, of which the first is a self-attention Block, and the remaining 3 are cross-attention Blocks. Each Block is based on the same architecture of ViT-Base, with a hidden layer dimension of 768 and 12 attention heads. The inputs of the 4 attention Blocks are shown in the figure, which come from the outputs of the image encoders and the uniformly sampled outputs of the 3 cross decoders from back to front, which are connected to the scene decoder module in an inverted manner. The detailed model structure of the cross-attention layer is shown in Figure 4 .

[0106] Further, a scene feature encoder Escene proposed in step 8 encodes the independent cross-view image features into a scene feature condition embedding Cscene. The process can be represented as:

[0107] C scene scene = Escene(FV) (9), scene V (10),

[0108] The scene feature encoder Escene consists of 6 self-attention Blocks, each containing a self-attention layer and an MLP layer. In the self-attention layer, the cross-view image feature sequence FV is converted into Q, K, and V through a linear layer, and finally becomes the output feature, the process is shown as follows:

[0109]

[0110] Further, a camera pose and scene feature co-embedding guided video diffusion model framework is adopted in step 9. In the U-Net part of the video diffusion model, the scene feature embedding Cscene and the camera pose embedding Cpose will be embedded into the spatial attention layer and the temporal attention layer, respectively. The specific process is as follows:

[0111]

[0112] wherein,​ is the output of the self-attention layer in the temporal attention layer, Z Spa is the output of the self-attention layer in the spatial attention layer, and are the camera trajectory condition embedding and the scene feature condition embedding, respectively.

[0113] Further, the loss function used in step 10 can be represented as:

[0114]

[0115] In the formula, ∈ represents the actual added real noise, t is the time step, Zt is the noise-added latent feature at time t, LF is the camera pose light field, Cpose and Cscene are the camera pose condition embedding and the scene feature condition embedding, respectively, and ∈ θ is the noise predicted by the diffusion model U-Net under the above conditions, and the prediction error is minimized by mean square error loss.

[0116] Further, the channel concatenation method used in step 6 modifies the channel dimension of the U-Net input feature, according to the actual concatenation dimension, such as the concatenation dimension 184 used in the present application, the first layer linear layer dimension of the U-Net network is modified to adapt to the actual dimension size.

[0117] Further, the camera pose light field concatenated by the channel concatenation method used in step 6 only participates in the noise prediction of the U-Net network, and does not participate in the denoising step, and after each denoising process is completed, the original input is concatenated with the new noise latent feature to participate in the noise prediction step in the next denoising process.

[0118] In this embodiment, the experimental architecture is implemented using Pytorch, where the VAE autoencoder part uses the same version of the pre-trained model of the latent diffusion model (LDM), the image encoder part combines the pre-trained model of DUSt3R, and the video diffusion model U-Net part uses the pre-trained model of DynamiCrafter, where the VAE autoencoder and the image latent space encoding part of DUSt3R are parameter frozen and do not participate in training, and the rest are fine-tuned based on the pre-trained model.

[0119] In the experiment, a machine with an NVIDIA RTX8000 GPU was used to implement the training and inference of the model.

[0120] Subjective comparison and analysis of the quality of the synthesized view images:

[0121] On large real scene dataset, subjective result comparison experiments are carried out with MotionCtrl[3], ViewCrafter[1]. Among them, MotionCtrl is a video generation method based on video diffusion model that provides single frame extrapolation and double frame interpolation, and supports conditional control of camera trajectory and motion control of picture content. Since this method is not specially trained for new view synthesis, the picture motion control will affect the three-dimensional consistency of the result. ViewCrafter is a new view synthesis method based on point cloud reconstruction rendering and video diffusion model generation. Although it has achieved the best image synthesis quality in real scenes, it is very dependent on the effectiveness of point cloud reconstruction, and once there is an estimation error, it will cause serious result distortion.

[0122] Both of these two methods and the present application use two input viewpoints as known conditions in the experiment. The target viewpoint images are generated based on the unified target viewpoint trajectory in indoor and outdoor scenes, and subjective comparison is carried out according to the generated image results. The comparison results are shown in Figure 6 and Figure 7 .

[0123] Figure 6 The test comparison results of indoor real scene dataset are shown. Figure 6 (d), (e), (f), (g) and (k), (l), (m), (n) in FIG. 13 show that the generation quality of the present application is higher than that of the previous methods in relatively open indoor scenes. MotionCtrl and ViewCrafter both produce distortion in some detail parts, such as shadows, water cups, etc., while the present application performs well, indicating that the encoded unified scene features can provide better three-dimensional consistency and texture details. Figure 6 (r), (s), (t), (u) in FIG. 14 show that the generation quality of the present application is also higher than that of the previous methods in relatively complex indoor scenes. In this type of scene, MotionCtrl has obvious distortion, which is caused by the conflict between the object motion module and the cross-frame movement caused by the change of viewpoint in complex static scenes. ViewCrafter does not have serious distortion phenomenon, but has structural errors in the reconstruction of pendant lamps, which is due to the fact that the three-dimensional reconstruction space composed of point clouds is sparse and discrete compared with the feature latent space, and is difficult to optimize together with the diffusion step. In contrast, the present application extracts scene features from the image latent space and embeds them, which can better learn the three-dimensional structure and texture details of the scene, without distortion and structural errors. Overall, the image quality generated by the present application is better than that of the previous methods in indoor real scenes.

[0124] Figure 7 The test comparison results of outdoor real scene dataset are shown. Figure 7(d), (e), (f), (g) and (k), (l), (m), (n) in FIG. 11 show that the application generates higher quality than previous methods in relatively open outdoor scenes. In such scenes, the point cloud reconstruction used by ViewCrafter is more difficult due to the greater span of viewpoints, more complex background, and deeper scene scale of outdoor scenes, and errors and distortions are more obvious, such as the appearance of serious artifacts in the vertical rod part, and some motion distortion in MotionCtrl. This is because MotionCtrl also adds picture content movement guidance, which allows it to generate dynamic scenes, but in static scenes, it is more prone to motion distortion caused by the cross-frame movement of static elements in the scene, such as the unnatural extension of the umbrella and the railing. In contrast, the application does not have obvious distortion and can generate relatively clear and realistic target viewpoint images. Figure 7 (r), (s), (t), (u) in FIG. 11 show that the application also generates higher quality than previous methods in outdoor scenes with more interference. In such scenes, MotionCtrl and ViewCrafter both have more serious distortion, which is due to the lack of unified scene feature information and the acquisition of semantic information from independent viewpoint images, which is easily disturbed. In contrast, although the application also has distortion, the degree is lighter and does not affect the overall structure of the scene, and performs better. Overall, the image quality generated by the application in outdoor real scenes is better than previous methods.

[0125] Comparison and analysis of objective results of viewpoint image synthesis quality:

[0126] On a large real scene dataset, the application is compared with IBRNet [2] , ZeroNVS [4] , CameraCtrl [5] , MotionCtrl [3] , ViewCrafter [1] to expand the objective result comparison experiment. In indoor and outdoor scenes, the target viewpoint image is generated based on the unified target viewpoint condition, and the PSNR, SSIM, and LPIPS indicators are used to evaluate the quality of the generated results. The objective results of the viewpoint image synthesis quality are shown in Table 1.

[0127] Table 1-Image quality evaluation results of different methods on real scene dataset

[0128]

[0129] As can be seen from Table 1, the synthetic image PSNR value, SSIM value and LPIPS value of the present application on the real scene data set all achieve the optimal result, reaching the most advanced level at present; compared with ZeroNVS, the PSNR value is improved by an average of 54.28% in the result, the SSIM value is improved by an average of 50.65% in the result, and the LPIPS value (the lower the better) is reduced by an average of 64.27% in the result; compared with IBRNet, the PSNR value is improved by an average of 98.82% in the result, the SSIM value is improved by an average of 206.46% in the result, and the LPIPS value (the lower the better) is reduced by an average of 73.31% in the result; compared with CameraCtrl, the PSNR value is improved by an average of 42.38% in the result, the SSIM value is improved by an average of 55.60% in the result, and the LPIPS value (the lower the better) is reduced by an average of 65.24% in the result; compared with MotionCtrl, the PSNR value is improved by an average of 34.87% in the result, the SSIM value is improved by an average of 37.31% in the result, and the LPIPS value (the lower the better) is reduced by an average of 60.10% in the result; compared with ViewCrafter, the PSNR value is improved by an average of 0.73% in the result, the SSIM value is improved by an average of 1.26% in the result, and the LPIPS value (the lower the better) is reduced by an average of 13.33% in the result. The above results show that the present application is superior to other previous methods in terms of image quality and color, lighting and structure of the generated image, and can generate results closer to real images.

[0130] Camera pose embedding ablation experiment:

[0131] In the present application, in order to improve the control accuracy of the camera pose on the result, the camera pose light field embedding method is used to add the camera pose directly to the U-Net input noise part of the video diffusion model, so as to distinguish from the cross attention module inside the U-Net, so as to enhance the control ability and constraint force of the camera pose. In the ablation experiment, this setting is cancelled, and the copy noise in the classic video diffusion model is used instead, and the error between the actual results generated by the two methods and the target camera trajectory is compared to test whether the setting has the ability to improve the control accuracy of the camera pose.

[0132] Subjectively, the experiment observes the camera pose error of the generated result by drawing a three-dimensional visual diagram of the camera trajectory and comparing it with the real view image. Part of the subjective comparison result diagram is as shown in Figure 8

[0133] Figure 8 ​In the middle, (a), (b) and (f), (g) show the error of the generated results on the camera trajectory when there is a camera pose light field and no camera pose light field, the red camera pose represents the real camera pose, and the blue camera pose represents the camera pose of the generated results. It can be seen that the trajectory error of the generated results when there is no camera pose light field is larger; (c), (d), (e) and (h), (i), (j) show the generated image results when there is a camera pose light field and no camera pose light field. By comparison, it can be seen that the generated image when there is a camera pose light field is closer to the real image in the viewpoint position, and when there is no camera pose light field, the generated image is obviously farther away in the viewpoint position.

[0134] Objectively, the experiment calculates the cumulative error between the camera pose parameters of the generated results and the actual real camera pose parameters in the rotation matrix R and the translation vector t to judge the accuracy of the camera pose of the generated results. The numerical results of objective comparison are shown in Table 2:

[0135] Table 2-Comparison of numerical results of camera pose light field ablation experiment

[0136]

[0137] As can be seen from Table 2, under the condition of camera pose light field, the average error value of the rotation matrix R of the camera trajectory of the generated results and the real camera trajectory error value decreases by 25.25%, and the average error value of the translation vector t decreases by 14.57%, and the overall error is obviously improved.

[0138] The above results show that by directly adding the camera pose to the U-Net input noise part of the video diffusion model, thereby distinguishing from the cross-attention module inside the U-Net, the control ability and constraint of the camera pose on the camera pose of the generated results of the video diffusion model can be enhanced.

[0139] [1] Yu W, Xing J, Yuan L, et al. View crafter: Taming video diffusion models for high-fidelity novel view synthesis [J]. arXiv preprint arXiv:2409.02048, 2024.

[0140] [2] Wang Q, Wang Z, Genova K, et al. Ibrnet: Learning multi-view image-based rendering [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 4690-4699.

[0141] [3] Wang Z, Yuan Z, Wang X, et al. Motionctrl: A unified and flexible motion controller for video generation [C] / / ACM SIGGRAPH 2024 Conference Papers. 2024: 1-11.

[0142] [4] Sargent K, Li Z, Shah T, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single image [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 9420-9429.

[0143] [5] He H, Xu Y, Guo Y, et al. Cameractrl: Enabling camera control for text-to-video generation [J]. arXiv preprint arXiv: 2404.02101, 2024.

[0144] Need to explain that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. For ordinary skilled in the art, under the premise of not departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the scope of protection of the claims of the present application.

Claims

1. A method for novel view synthesis based on video diffusion model and feature embedding guidance, characterized in that, The method comprises the following steps: Step one, constructing a multi-scene viewpoint dataset containing multiple viewpoints and camera poses under multiple scenes, and dividing it into a training set and a test set; Step two, preprocessing the viewpoint data in the training dataset in step one, resizing and linearly normalizing the viewpoint images, and one-dimensionizing the camera poses to obtain camera trajectories, and simultaneously camera pose light field encoding the camera poses to obtain camera pose light fields; Step three, inputting the viewpoint images obtained in step two into the encoder part of the VAE autoencoder to generate image latent features of the input viewpoint; Step four, randomly sampling a real noise and randomly generating a time step; Step five, adding noise to the image latent features obtained in step three based on the real noise and time step obtained in step four to obtain noise latent features; Step six, concatenating the camera pose light field obtained in step two and the noise latent features obtained in step five in the channel dimension to obtain U-Net input features; Step seven, passing the camera trajectory obtained in step two through a camera trajectory encoder to obtain a camera pose conditional embedding; Step eight, extracting the first and last frames of the viewpoint images obtained in step two, and sequentially passing them through an image feature encoder and a scene feature encoder to obtain a scene feature conditional embedding; Step nine, inputting the U-Net input features obtained in step six into the U-Net network of the video diffusion model, and using the camera pose conditional embedding obtained in step seven and the scene feature conditional embedding obtained in step eight as a condition guide, respectively embedded in the spatial attention layer and the time attention layer of the U-Net of the video diffusion model, and finally outputting a predicted noise; Step ten, freezing the parameters of the VAE encoder, comparing the predicted noise obtained in step nine with the real noise obtained in step four to calculate the loss value, and then optimizing the loss to the global minimum to train a new viewpoint synthesis video diffusion model, the parameter adjustment part of which includes a camera trajectory encoder, an image feature encoder, a scene feature encoder, and a video diffusion model U-Net network; Step eleven, freezing the parameters of the new viewpoint synthesis video diffusion model obtained in step ten, obtaining a test dataset from step one, and obtaining viewpoint images, camera trajectories, and camera pose light fields according to the processing of step two, wherein the first and last frames of the viewpoint images are selected as input viewpoint images, and the remaining images are selected as target viewpoint images; Step twelve, randomly sampling a set of Gaussian noise and setting a fixed time step sequence; Step thirteen, generating image latent features from the input viewpoint images obtained in step eleven according to step three, adding noise to the first and last noises in the set of Gaussian noise obtained in step twelve and the maximum time step in the fixed time step sequence, replacing the first and last noises in the set of Gaussian noise in step twelve with the obtained noise latent features, and then obtaining a set of noise latent features; Step fourteen, processing the noise latent features obtained in step thirteen and the input viewpoint images, camera trajectories, and camera pose light fields obtained in step eleven according to the methods of steps six to nine to obtain predicted noise, and denoising the noise latent features obtained in step thirteen based on the predicted noise; Step fifteen, the denoising process of step fourteen is repeated according to the time step sequence obtained in step twelve, and finally the denoised image latent feature is obtained; Step sixteen, the denoised image latent feature obtained in step fifteen is input into the decoder part of the VAE autoencoder to obtain the generated target view image.

2. The video diffusion model and feature embedding guided novel view synthesis method of claim 1, wherein, A double-branch camera pose embedding guide scheme is used in steps six and seven; The first branch uses camera pose light field encoding to directly splice the camera pose into the input end of the U-Net, which uses a variable resolution pose light field encoding, expressed as follows: (K, R, t) → (o, d) e R H×W×6 → LF e R H×W×C (1), Where K is the camera pose intrinsic matrix, R is the camera pose extrinsic rotation matrix, t is the camera pose extrinsic translation vector, LF is the target feature light field, (o, d) is the channel structure of the target feature light field, where o is the light field origin and d is the light field direction, H is the length of the target feature light field, W is the width of the target feature light field, and C is the number of channels of the target feature light field, More specifically, o i ∈R H×W×3 = T(2), d i ∈R H×W×3 =R·K -1 ·(u,v,1) (3), LF s = (o,d) (4), LF = [sin(2 0 πLF s ), cos(2 0 πLF s ),... sin(2 L-1 πLF s ), cos(2 L-1 πLF s )] (5), Where u, v are pixel coordinates of the image, LFs is obtained by splicing o, d in the channel dimension, and L is the length of the cosine position encoding, which is set to L=15 by default; The second branch uses a camera trajectory encoder to encode the one-dimensional camera trajectory CT into a camera pose conditional embedding Cpose, which is represented as: C pose = E CT (CT) (6), Where the camera trajectory CT is defined as a series of time-ordered camera poses CT={P1, P2, …, PN}, each camera pose Pi is composed of a rotation matrix Ri∈R3×3 expanded into a one-dimensional vector and a translation vector ti∈R3, with a length of 12, representing the direction and position of the camera at the i-th frame of the video sequence, and the camera pose encoder ECT is composed of 6 self-attention blocks, each containing a self-attention layer and an MLP layer. In the self-attention layer, the camera trajectory CT is converted into Q, K and V through a linear layer, and finally becomes the output feature, the process is as follows:

3. The method of claim 1, wherein: An image feature encoder Eimage is proposed in step eight, which extracts cross-view image features FV from the input view image, which is represented as: F V = E image (I 1,…,N ) (8), Where the cross-view image feature FV is a sequence of image features, which can be represented as: F V = [F V1 , F V2 ,..., F VN ] (9), The image feature encoder Eimage contains three parts: first, the image is encoded into the feature space using an encoder; second, a cross decoder is used to extract cross features from image features of other input views; and third, a scene decoder is used to obtain the cross-input image features of the input image. In the image encoder part, the encoder architecture based on ViT-Large is used, which is composed of 24 self-attention blocks, each containing a self-attention layer and an MLP layer, with a hidden layer dimension of 1024, and 16 attention heads for attention mechanism. The two image encoders share the same network parameters. In the cross decoder, the decoder architecture based on ViT-Base is used, which has a total of 12 cross-attention blocks, each containing a self-attention layer, a cross-attention layer, and an MLP layer, with a hidden layer dimension of 768, and 12 attention heads for attention mechanism. In the scene decoder module, 4 attention blocks are used, the first of which is a self-attention block, and the remaining 3 layers are cross-attention blocks. Each block is based on the same architecture of ViT-Base, with a hidden layer dimension of 768 and 12 attention heads. The input of the 4 attention blocks comes from the output of the image encoder and the uniformly sampled output of the 3 cross decoders from back to front, which are connected to the scene decoder module in an inverted manner.

4. The method of claim 1, wherein: A scene feature encoder Escene is proposed in step eight, which encodes the independent cross-view image features into scene feature conditional embedding Cscene. The process is represented as: C scene = E scene (F V )10, The scene feature encoder Escene is composed of 6 self-attention blocks, each containing a self-attention layer and an MLP layer. In the self-attention layer, the cross-view image feature sequence FV is converted into Q, K, and V through a linear layer, and finally becomes the output feature. The process is represented as:

5. The method of claim 1, wherein: In step nine, a camera pose and scene feature co-embedding guided video diffusion model framework is adopted. In the U-Net part of the video diffusion model, the scene feature embedding Cscene and the camera pose embedding Cpose are embedded into the spatial attention layer and the temporal attention layer, respectively. The specific process is as follows: wherein, is the output of the self-attention layer in the temporal attention layer, Z Spa is the output of the self-attention layer in the spatial attention layer, and are the camera trajectory condition embedding and the scene feature condition embedding, respectively.

6. The method of claim 1, wherein: The loss function used in step ten is represented as: In the formula, ∈ represents the actual added real noise, t is the time step, Zt is the latent feature with noise added at time t, LF is the camera pose light field, Cpose and Cscene are the camera pose condition embedding and scene feature condition embedding, respectively, ∈ θ is the noise predicted by the diffusion model U-Net under the above conditions, and the prediction error is minimized by mean square error loss.

7. The method of claim 1, wherein: The channel splicing method used in step six modifies the channel dimension of the U-Net input feature. According to the actual splicing dimension, the first layer linear layer dimension of the U-Net network is modified to adapt to the actual dimension size.

8. The method of claim 1, wherein: The camera pose light field spliced by the channel splicing method used in step six only participates in the noise prediction of the U-Net network, but does not participate in the denoising step. After each denoising process is completed, the original input and the new noise latent feature are spliced to participate in the noise prediction step in the next denoising process.

Images