Camera condition guide viewpoint synthesis method based on video diffusion model

Abstract

The invention relates to a video diffusion model-based camera condition-guided viewpoint synthesis method, which belongs to the technical field of computer vision, and comprises the following steps of: taking a stable video diffusion model SVD as a basic generator, and jointly inputting Gaussian noise and encoded image potential representation to obtain a video diffusion model SVD-based viewpoint synthesis model; meanwhile, a pose encoder based on a time attention mechanism is used for encoding camera parameters represented by Plcarbon coordinates, and then the camera parameters are used as pose conditions to be embedded into a time attention layer in a video diffusion model de-noising U-Net; pixel-level features of an input single image are extracted through an image coding embedding network and serve as image feature conditions to be embedded into a space attention layer in a de-noised U-Net of a video diffusion model, and a pre-trained video diffusion model is finely adjusted; the space consistency of the synthetic image and the input image and the track consistency of the synthetic image and the camera parameters are improved, and the generation diversity is increased while the generation quality is improved.

Description

TECHNICAL FIELD

[0001] The application relates to a synthesis method, in particular to a camera condition guided view synthesis method based on a video diffusion model, and belongs to the technical field of computer vision. BACKGROUND

[0002] Image-based rendering such as view interpolation and layered depth images is synthesized by image-to-image rendering to synthesize new view images, the rendering result is good and the efficiency is not slow, but it is only suitable for small baseline motion. New view synthesis methods based on deep learning, such as neural radiance field NeRF and Gaussian sputtering, have shown impressive new view image synthesis results in various scenes, however, these methods rely on dense multi-view images and known camera poses, and need to be optimized independently for each instance, which limits the generality of the model, and due to the complexity of calculation, these methods have high calculation cost. SUMMARY

[0003] The application is exactly aimed at the technical problems in the prior art, and provides a camera condition guided view synthesis method based on a video diffusion model, which generates a stable video diffusion model SVD-based generator, jointly inputs Gaussian noise and encoded image latent representation, and encodes camera parameters represented by Plucker coordinates using a pose encoder based on a time attention mechanism, and then embeds the pose conditions into a time attention layer in a video diffusion model denoising U-Net. The pixel-level features of the input single image are extracted by an image encoding embedding network, and are embedded into a spatial attention layer in the video diffusion model denoising U-Net as image feature conditions. The pre-trained video diffusion model is fine-tuned to improve the spatial consistency of the synthesized image and the input image and the trajectory consistency of the synthesized image and the camera parameters, thereby improving the generation quality and increasing the generation diversity.

[0004] In order to achieve the above purpose, the technical scheme of the application is as follows, a camera condition guided view synthesis method based on a video diffusion model,

[0005] Step 1: preprocessing a multi-view data set used for training, including video, camera pose and text description, and performing preprocessing operations such as cropping, normalization and data enhancement on the video to obtain video clip segments;

[0006] Step 2: obtaining camera poses corresponding to the video image frame sequence, i.e. L camera parameters P1~P L , represented by Plucker coordinates, to obtain Plucker coordinate representation P;

[0007] Step 3: Input the preprocessed video from Step 1 into the viewpoint image synthesis model. The viewpoint image synthesis model includes a pre-trained video diffusion model, a pose encoder Φ, and an image encoding embedding network ψ. The pre-trained video diffusion model includes a VAE autoencoder and a diffusion denoising U-Net network. The VAE autoencoder includes an encoder and a decoder. The pre-trained VAE encoder is used to process the image frame sequence I1~I2 of the input video. L (L represents the number of images, i.e., the number of video frames) The compressed encoding is used to obtain a latent representation, which is then subjected to forward diffusion to add noise, resulting in the noisy latent code z. t Simultaneously, the first frame image I1 is copied after being processed by the VAE encoder, and compared with the noise latent code z. t The sum is used as the diffusion input z′ t ;

[0008] Step 4: Input the camera pose Plücker coordinate representation P obtained in Step 2 into the attitude encoder Φ. Φ only accepts the camera pose Plücker coordinate representation. As input, 1 represents the batch size, L represents the number of video frames, and 512×512 represents the processed image size. The final output is multi-scale camera pose features p. t ;

[0009] Step 5: Input the first frame image I1 of the input video into the image coding embedding network ψ, which includes an encoder and an intermediate block, and finally obtain multi-scale image semantic features F;

[0010] Step 6: Diffusion input z′ obtained in Step 3 t The input is a diffusion denoising U-Net network, which includes an encoder, an intermediate block, and a decoder. Each of the encoder, intermediate block, and decoder contains multiple camera pose feature fusion modules and image semantic feature fusion modules. The latent features at different scales obtained by the encoder, intermediate block, and decoder are combined with the camera pose features at different scales obtained in step 4. t Input the camera pose feature fusion module, and input the latent features of different scales obtained by the encoder, intermediate block and decoder of the diffusion denoising U-Net network and the image semantic features F of different scales obtained in step 5 into the image semantic feature fusion module, and control the diffusion denoising U-Net network to achieve iterative denoising;

[0011] Step 7: Iterate the denoising operation in Step 6 for T steps, and finally the diffusion denoising U-Net network predicts the clean, noise-free image sequence latent code Z0;

[0012] Step 8: input the clean, noise-free image sequence latent code Z0 obtained in step 7 into the VAE decoder to finally generate a new view video consistent with the preset camera trajectory;

[0013] Step 9: calculate the loss value of the noise added in step 3 and the noise predicted by the denoising U-Net network in step 7 each iteration diffusion, and perform network hyperparameter iterative optimization to obtain a fine-tuned denoising video diffusion model.

[0014] In step 1, the RealEstate10K dataset is used for training, the videos in the dataset are split into multiple frames of images, the size is cropped and standardized to the same resolution size of 512x512x3, and linear normalization is used to make the pixel value range between [0, 1]; the camera pose includes the camera intrinsic matrix corresponding to the video frame image and the extrinsic matrix [R, T], where represents the rotation matrix, and the translation vector.

[0015] In step 2, the Plücker coordinate representation P of the camera pose is calculated as follows:

[0016]

[0017] d = RK -1 [u, v, 1] T

[0018] m = -R·d + T

[0019] where is the direction vector, and (u, v) represents the two-dimensional pixel point coordinates. This step first represents the camera pose as Plücker coordinates, and then performs multi-scale coding embedding. Compared with other algorithms that directly concatenate camera parameter matrices into diffusion denoising U-Net, this method can better perceive the camera pose in the latent space and more accurately control the view point.

[0020] In step 3, the network structure diagram of the view image synthesis model is as shown in Figure 2 The denoising U-Net network of the pre-trained video diffusion model and the VAE autoencoder are frozen during training and testing, the pose encoder, image coding embedding network and feature embedding layer are trained, and the diffusion input z' t obeys a Gaussian distribution.

[0021] In step 4, the pose encoder Φ includes 4 Resnet modules and a temporal attention layer, and the network structure diagram is as shown in Figure 3As shown, the pose encoder is trained when fine-tuning the diffusion model, and its initial weight parameters are initialized from zero.

[0022] In step 5, the image encoding embedding network ψ contains the same encoder and intermediate blocks as the diffusion denoising U-Net network, and when fine-tuning the diffusion model, the image encoding embedding network is trained, and its initial weight parameters are initialized from the pre-trained diffusion denoising U-Net network checkpoint, and the network structure diagram is as shown in Figure 4 The method can provide more accurate pose control for new view image synthesis in the diffusion model latent space. And the method extracts multi-scale features of the reference image through the image encoding embedding network, and corresponds to each layer of the diffusion denoising U-Net network, which can ensure more accurate semantic control in image generation, and realize the spatial structure consistent between the synthesized image and the input reference image.

[0023] In steps 6 and 7, when fine-tuning the diffusion denoising U-Net network, its encoder, intermediate block and decoder are all imported from the U-Net model weight of Stable Video Diffusion, and are frozen and do not participate in training, and only the camera pose feature fusion module and the image semantic feature fusion module are trained, and the initial weight is zero. Because the alignment of the conditional control signal and the noise input is not enough to accurately control the camera trajectory and retain the details of the original image input. Therefore, in the camera pose feature fusion module and the image semantic feature fusion module, the attention mechanism is applied to adjust. Specifically, as shown in Figure 5 For the camera pose feature fusion module, the denoising output of the U-Net network is spliced with the camera pose feature, and then passes through a feedforward network layer and a time attention layer; for the image semantic feature fusion module, the denoising output of the U-Net network first passes through a cross-attention layer, in which the query (Q) comes from the previous self-attention layer, and the image semantic feature F serves as the key (K) and the value (V), and then passes through a feedforward layer and a linear layer, and uses a normalization operation and a residual connection to obtain the fused feature,

[0024] The diffusion input z' of the video diffusion model t and the camera pose feature p t are directly spliced along the channel dimension, and then pass through a feedforward network layer, and the output is directly fed to the trainable diffusion time attention layer. The whole embedding fusion process can be described by the following formula:

[0025]

[0026] Q=W Q ·x t , K=W K ·x t , V=W V·x t

[0027] x=TA(x t )

[0028] where denote the concatenation (along the channel dimension), W Q , W K , W V are learnable projection matrices, TA denotes the temporal attention layer, and FFN denotes the feed-forward layer.

[0029] For the image semantic feature F output by the image encoding embedding network, cross-attention is directly used for feature fusion with the diffusion input z′ t , and then a feed-forward network layer and a linear layer are used, and finally a residual connection is used to promote feature reuse and enhance the model generalization capability. The entire embedding fusion process can be described by the formula,

[0030]

[0031] Q = W Q ·z′ t , K = W K ·F, and V = W V ·F

[0032] z″ t = CSA(z′ t ) + z′ t

[0033] y = LN(FFN(z″ t ) + z″ t ) + z′ t

[0034] where W Q , W K , W V are learnable projection matrices, CSA denotes the cross-attention layer, LN denotes the linear layer, and FFN denotes the feed-forward layer.

[0035] The loss function used for the noise added to the input video in step 9 and the predicted noise is defined as follows:

[0036]

[0037] where z0 is the latent feature of the input video frame sequence, t is the time step, c t is the image condition embedding extracted from the reference image frame, P represents the Plücker coordinate embedding of the camera parameter matrix, Φ represents the pose encoder, is Gaussian noise subject to a standard normal distribution, z′ tis a noisy latent code of a video frame sequence, also obeys Gaussian distribution, represents a neural network for predicting the addition of noise at each step.

[0038] The above steps are the model training method, and the test steps are slightly different, as follows: all model weights are frozen, and the input is only a single image. The input of the diffusion model uses a pre-trained VAE encoder to compress and encode the reference image and then adds random Gaussian noise. The test network is as shown in Figure 6

[0039] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the camera condition guided view synthesis method based on the video diffusion model when executing the program.

[0040] A computer readable storage medium having computer instructions stored thereon, the computer instructions being executed by a processor to implement the camera condition guided view synthesis method based on the video diffusion model.

[0041] Compared with the prior art, the present application has the following advantages: 1. The present application uses a stable video diffusion model SVD as a basic generator, and jointly controls the embedding network to optimize the diffusion model. It not only utilizes the rich prior of the diffusion model, but also greatly improves the generalization ability of the model. First, the control module can extract camera pose features and image semantic features, and provide accurate condition embedding, so that the model can better learn the features of the preset camera trajectory and input image. Second, the diffusion model performs well in image generation tasks. It has strong ability in approximate diversification and complex distribution, and the pre-trained Stable Video Diffusion has rich prior, which can sample in the feature space to generate very realistic images, and the near objects and surface details are more fine and clear.

[0042] 2. The present application does not use a pre-trained CLIP image encoder, but designs an image coding embedding network to extract input image features and fine-tune the pre-trained Stable Video Diffusion to control the diffusion model to generate, ensuring that the results are consistent with the input image space. At the same time, the cross-attention module is used to align the image pixel feature and noise input denoising output, which can well preserve the image structure information, so as to generate more detailed and clear images.

[0043] ​3、The application adopts two control modules, one of which extracts input multi-scale features through a learnable image coding embedding network, and the other of which realizes continuous representation of camera parameters through a pose encoder based on differential manifold. The two conditional branches are dynamically fused through a spatiotemporal feature coupling module, which not only retains the spatial consistency of generating new view content, but also ensures the accurate controllability of camera motion, achieving a better balance between geometric accuracy and visual quality.

[0044] 4、The application fine-tunes the potential diffusion model (Stable Video Diffusion), which can not only utilize its rich priori, but also improve performance on specific tasks. Through a double-path control network, the generation ability of the diffusion model is realized to accurately control the pose and semantics, and the feature fusion module also realizes the control of the generation results of the diffusion model at the detail and pixel level. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the new view synthesis method of the application,

[0046] Figure 2 The structure diagram of the new view synthesis method of the application,

[0047] Figure 3 The structure diagram of the pose encoder of the application,

[0048] Figure 4 The structure diagram of the image coding embedding network of the application,

[0049] Figure 5 The structure diagram of the feature fusion module of the application,

[0050] Figure 6 The structure diagram of the test network of the application. DETAILED DESCRIPTION

[0051] In order to deepen the understanding of the application, the embodiments will be described in detail below with reference to the drawings.

[0052] Embodiment 1: A camera condition guided view synthesis method based on a video diffusion model,

[0053] Step 1: Preprocess the multi-view data set used for training, including video, camera pose and text description, and perform preprocessing operations such as cropping, normalization and data enhancement on the video to obtain video clip segments;

[0054] Step 2: Obtain the camera pose obtained in step 1, i.e. L camera parameters P1~P L , expressed as Plucker coordinates, to obtain Plucker coordinate representation P;

[0055] Step 3: Input the preprocessed video from Step 1 into the viewpoint image synthesis model. The viewpoint image synthesis model includes a pre-trained video diffusion model, a pose encoder Φ, and an image encoding embedding network ψ. The pre-trained video diffusion model includes a VAE autoencoder and a diffusion denoising U-Net network. The VAE autoencoder includes an encoder and a decoder. The pre-trained VAE encoder is used to process the image frame sequence I1~I2 of the input video. L (L represents the number of images, i.e., the number of video frames) The compressed encoding is used to obtain a latent representation, which is then subjected to forward diffusion to add noise, resulting in the noisy latent code z. t Simultaneously, the first frame image I1 is copied after being processed by the VAE encoder, and compared with the noise latent code z. t The sum is used as the diffusion input z′ t ;

[0056] Step 4: Input the camera pose Plücker coordinate representation P obtained in Step 2 into the attitude encoder Φ. Φ only accepts the camera pose Plücker coordinate representation. As input, 1 represents the batch size, L represents the number of video frames, and 512×512 represents the processed image size. The final output is multi-scale camera pose features p. t ;

[0057] Step 5: Input the first frame image I1 of the input video into the image coding embedding network ψ, which includes an encoder and an intermediate block, and finally obtain multi-scale image semantic features F;

[0058] Step 6: Diffusion input z′ obtained in Step 3 t The input is a diffusion denoising U-Net network, which includes an encoder, an intermediate block, and a decoder. Each of the encoder, intermediate block, and decoder contains multiple camera pose feature fusion modules and image semantic feature fusion modules. The latent features at different scales obtained by the encoder, intermediate block, and decoder are combined with the camera pose features at different scales obtained in step 4. t Input the camera pose feature fusion module, and input the latent features of different scales obtained by the encoder, intermediate block and decoder of the diffusion denoising U-Net network and the image semantic features F of different scales obtained in step 5 into the image semantic feature fusion module, and control the diffusion denoising U-Net network to achieve iterative denoising;

[0059] Step 7: Iterate the denoising operation in Step 6 for T steps, and finally the diffusion denoising U-Net network predicts the clean, noise-free image sequence latent code Z0;

[0060] Step 8: input the clean, noise-free image sequence latent code Z0 obtained in step 7 into the VAE decoder to finally generate a new view video consistent with the preset camera trajectory;

[0061] Step 9: calculate the loss value of the noise added in step 3 and the noise predicted by the denoising U-Net network in step 7 each iteration diffusion, and perform network hyperparameter iterative optimization to obtain a fine-tuned denoising video diffusion model.

[0062] In step 1, the RealEstate10K dataset is used for training, the video in the dataset is split into multiple frames of images, the size is cropped and standardized to the same resolution size of 512x512x3, and linear normalization is used to make the pixel value range between [0, 1]; the camera pose includes the camera intrinsic matrix corresponding to the video frame image and the extrinsic matrix [R, T], where represents the rotation matrix, and the translation vector.

[0063] In step 2, the Plücker coordinate representation P of the camera pose is calculated as follows:

[0064]

[0065] d=RK -1 [u,v,1] T

[0066] m=-R·d+T

[0067] where is the direction vector, is the position vector, and (u, v) represents the two-dimensional pixel point coordinates.

[0068] In step 3, the network structure diagram of the view image synthesis model is as shown in Figure 2 , the denoising U-Net network of the pre-trained video diffusion model and the VAE autoencoder are frozen during training and testing, the pose encoder, image encoding embedding network and feature embedding layer are trained, and the diffusion input z' t obeys a Gaussian distribution.

[0069] In step 4, the pose encoder Φ includes 4 Resnet modules and a temporal attention layer, and the network structure diagram is as shown in Figure 3 , the pose encoder is trained during fine-tuning of the diffusion model, and its initial weight parameters are initialized from zero.

[0070] In step 5, the image encoding embedding network ψ contains the same encoder and intermediate blocks as the diffusion denoising U-Net network, and when fine-tuning the diffusion model, the image encoding embedding network is trained, whose initial weight parameters are initialized from the pre-trained diffusion denoising U-Net network checkpoint, and the network structure diagram is as shown in Figure 4 ;

[0071] In steps 6 and 7, when fine-tuning the diffusion denoising U-Net network, its encoder, intermediate blocks and decoder are imported into the U-Net model weights of Stable Video Diffusion, and are frozen and do not participate in training, and only the camera pose feature fusion module and the image semantic feature fusion module are trained, and the initial weights are all zero. Since the conditional control signal and the noise input are not aligned, it is not enough to accurately control the camera trajectory and retain the details of the original image input. Therefore, in the camera pose feature fusion module and the image semantic feature fusion module, the attention mechanism is applied to adjust. Specifically, as shown in Figure 5 , for the camera pose feature fusion module, the denoising output of the U-Net network is spliced with the camera pose feature, and then passes through a feedforward network layer and then through a temporal attention layer; for the image semantic feature fusion module, the denoising output of the U-Net network first passes through a cross-attention layer, where the query (Q) comes from the previous self-attention layer, and the image semantic feature F serves as the key (K) and the value (V), and then passes through a feedforward layer and a linear layer, and uses a normalization operation and a residual connection, to obtain the fused feature,

[0072] The diffusion input z' of the video diffusion model t and the camera pose feature p t are directly spliced along the channel dimension, and then pass through a feedforward network layer, and the output is directly fed to the trainable diffusion temporal attention layer, and the entire embedding fusion process can be described by the following formula:

[0073]

[0074] Q = W Q · x t , K = W K · x t , V = W V · x t

[0075] x = TA(x t )

[0076] where represents splicing (along the channel dimension), W Q , W K , W Vare learnable projection matrices, TA denotes a temporal attention layer, and FFN denotes a feed-forward layer.

[0077] For the image semantic feature F output by the image encoding embedding network, cross-attention is directly used with the diffusion input z' t Feature fusion is performed, then a feed-forward network layer and a linear layer are passed, and finally a residual connection is used to facilitate feature reuse and enhance the model generalization capability. The entire embedding fusion process can be described by the formula

[0078]

[0079] Q = W Q · z' t , K = W K · F, and V = W V · F

[0080] z" t = CSA(z' t ) + z' t

[0081] y = LN(FFN(z" t ) + z" t ) + z' t

[0082] where W Q , W K , and W V are learnable projection matrices, CSA denotes a cross-attention layer, LN denotes a linear layer, and FFN denotes a feed-forward layer.

[0083] The loss function used for the noise added to the input video in step 9 and the predicted noise is defined as follows:

[0084]

[0085] where z0 is the latent feature of the input video frame sequence, t is the time step, c t is the image condition embedding extracted from the reference image frame, P denotes the Plücker coordinate embedding of the camera parameter matrix, and Φ denotes the pose encoder. is Gaussian noise subject to a standard normal distribution, z' t is the noisy latent code of the video frame sequence, which is also subject to a Gaussian distribution, denotes a neural network that predicts the noise added at each step.

[0086] The above steps are the model training method, and the test steps are slightly different, as follows: all model weights are frozen, and only a single image is input. The input of the diffusion model is compressed and encoded by the pre-trained VAE encoder, and then random Gaussian noise is added. Figure 6 as shown in the test network.

[0087] It should be noted that the above embodiments are not intended to limit the scope of protection of the present application, and any equivalent transformations or substitutions made on the basis of the above technical solutions fall within the scope of protection of the claims of the present application.

Claims

1. A camera-conditionally guided viewpoint synthesis method based on a video diffusion model, characterized in that, The method includes the following steps: Step 1: Preprocess the multi-view dataset used for training, including videos, camera poses, and text descriptions, and perform preprocessing operations such as cropping, normalization, and data augmentation on the videos to obtain video clip segments; Step 2: The camera pose obtained in Step 1, i.e., the L camera parameters P1 to P2 corresponding to the video image frame sequence, is... L , represented as Plücker coordinates, yields the Plücker coordinate representation P; Step 3: Input the preprocessed video from Step 1 into the viewpoint image synthesis model. The viewpoint image synthesis model includes a pre-trained video diffusion model, a pose encoder Φ, and an image encoding embedding network ψ. The pre-trained video diffusion model includes a VAE autoencoder and a diffusion denoising U-Net network. The VAE autoencoder includes an encoder and a decoder. The pre-trained VAE encoder is used to process the image frame sequence I1~I2 of the input video. L Where L represents the number of video frames, is compressed into a latent representation, and then forward-spreading is performed to add noise, resulting in the noise latent code z. t Simultaneously, the first frame image I1 is copied after being processed by the VAE encoder, and compared with the noise latent code z. t The sum is used as the diffusion input z′ t ; Step 4: Input the camera pose Plücker coordinate representation P obtained in Step 2 into the attitude encoder Φ. Φ only contains the camera pose Plücker coordinate representation. As input, 1 represents the batch size, L represents the number of video frames, and 512×512 represents the processed image size. The final output is multi-scale camera pose features p. t ; Step 5: Input the first frame image I1 of the input video into the image encoding embedding network ψ, which includes an encoder and an intermediate block, and finally obtain the multi-scale image semantic features F; Step 6: Diffusion input z′ obtained in Step 3 t The input is a diffusion denoising U-Net network, which includes an encoder, an intermediate block, and a decoder. Each encoder, intermediate block, and decoder contains multiple camera pose feature fusion modules and image semantic feature fusion modules. The latent features at different scales obtained by the encoder, intermediate block, and decoder are combined with the camera pose features at different scales obtained in step 4. t The latent features of different scales obtained by the encoder, intermediate block and decoder of the diffusion denoising U-Net network are input into the image semantic feature fusion module, and the image semantic feature F of different scales obtained in step 5 is input into the image semantic feature fusion module to control the diffusion denoising U-Net network to achieve iterative denoising. Step 7: Iterate the denoising operation in Step 6 for T steps, and finally the diffusion denoising U-Net network predicts the clean, noise-free image sequence latent code Z0; Step 8: Input the clean, noise-free image sequence latent code Z0 obtained in Step 7 into the VAE decoder to finally generate a new viewpoint video that is consistent with the preset camera trajectory; Step 9: Calculate the loss value of the noise added in each step of Step 3 and the noise predicted by the U-Net network in each step of Step 7 for each iteration of diffusion denoising, and perform iterative optimization of the network hyperparameters to obtain the finely tuned and trained denoised video diffusion model.

2. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In step 1, the RealEstate10K dataset is used for training. The videos in the dataset are split into multiple frames, their sizes are cropped and normalized to the same resolution of 512×512×3, and linearly normalized to ensure that the pixel values ​​are in the range of [0,1]. The camera pose includes the camera intrinsic parameter matrix corresponding to the video frame image. and the extrinsic parameter matrix [R,T], where Represents the rotation matrix. This represents the translation vector.

3. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In step 2, the Plücker coordinate representation P of the camera pose is calculated as follows: d=RK -1 [u,v,1] T m=-R·d+T in It is a direction vector. Let (u,v) be the position vector, representing the coordinates of a two-dimensional pixel.

4. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In step 3, in the viewpoint image synthesis model, the denoising U-Net network and VAE autoencoder of the pre-trained video diffusion model are frozen during training and testing, while the pose encoder, image encoding embedding network, and feature embedding layer are trained, and the diffusion input z′ is used. t It follows a Gaussian distribution.

5. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In step 4, the pose encoder Φ comprises four ResNet modules and a temporal attention layer. During fine-tuning of the diffusion model, the pose encoder is trained, and its initial weight parameters are initialized from zero. This pose encoder consists of four ResNet modules and a temporal attention module. The temporal attention module captures the temporal relationships of camera poses throughout the video segment. The pose encoder only incorporates the Plücker embedding sequence. As input, 1 represents the batch size, L represents the number of video frames, and 512×512 represents the processed image size. The output is multi-scale embedding features, which are then embedded into the denoising U-Net network through a cross-attention mechanism according to the corresponding scale. The encoding process of the pose encoder Φ can be expressed by the formula: p t =Φ(P) Where p t Representing camera pose features, all intermediate features, i.e. multi-scale features, when the pose encoder Φ encodes the Plücker embedding sequence P are used as embeddings in the U-Net network for video diffusion model denoising.

6. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In step 5, the image encoding embedding network ψ contains the same encoder and intermediate blocks as the diffusion denoising U-Net network, and when fine-tuning the diffusion model, the image encoding embedding network is trained, and its initial weight parameters are initialized from the checkpoints of the pre-trained diffusion denoising U-Net network, as follows: The first frame of the input video is used as the reference image I1∈R H×W×3 After preprocessing, the feature vector is first passed through a convolutional layer to obtain a feature vector with a shape of 1×4×64×64. Then, it is expanded according to the number of video frames L to obtain a feature vector of 1×4×L×64×64, which is used as the input to the image coding embedding network ψ. The structure of the image coding embedding network ψ is consistent with the downsampling module (Downblock) of the video diffusion model U-Net network. During training, its model weight parameters are updated iteratively. The weights are initialized from the pre-trained U-Net network parameters, while the weight parameters of the convolutional layers are initialized to zero. ψ outputs multi-scale embedding features, which are then embedded into the denoising U-Net network through a cross-attention mechanism according to the corresponding scale. Finally, the sampling time step t is determined and modulation coefficients are generated through sinusoidal position encoding, which are also embedded into the U-Net network for diffusion input z′. t Perform T-step iterative denoising to obtain a clean image latent vector z0.

7. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, In steps 6 and 7, when fine-tuning the diffusion denoising U-Net network, its encoder, intermediate blocks, and decoder all use U-Net model weights imported from Stable Video Diffusion, which are frozen and not used in training. Only the camera pose feature fusion module and the image semantic feature fusion module are trained, and their weights are initialized to zero. Specifically, for the camera pose feature fusion module, the denoised output of the U-Net network is concatenated with the camera pose features, then passed through a feedforward network layer and a temporal attention layer; for the image semantic feature fusion module, the denoised output of the U-Net network first passes through a cross-attention layer, where the query (Q) comes from the previous self-attention layer, and the image semantic features F serve as the key (K) and value (V), then through a feedforward layer and a linear layer, using normalization and residual connections to obtain the fused features. The diffusion input z′ of the video diffusion model t and camera pose features p t The data is directly concatenated along the channel dimension, and then passed through a feedforward network layer. Its output is directly fed into a trainable diffusion-time attention layer. The entire embedding and fusion process can be described by the following formula: Q=W Q ·x t ,K=W K ·x t ,V=W V ·x t x=TA(x t ) in Indicates splicing (along the channel dimension), W Q W K W V These are the learnable projection matrices, where TA represents the temporal attention layer and FFN represents the feedforward layer. For the image semantic features F output by the image encoding embedding network, cross-attention and diffuse input z′ are directly used. t Feature fusion is performed, followed by feedforward and linear layers, and finally residual connections are used to promote feature reuse and enhance the model's generalization ability. The entire embedding and fusion process can be described by a formula. Q=W Q ·z′ t ,K=W K ·F,V=W V ·F With" t =CSA(z′ t )+z′ t y=LN(FFN(z″ t )+z″ t )+z′ t Among them W Q W K W V These are the learnable projection matrices, where CSA represents the cross-attention layer, LN represents the linear layer, and FFN represents the feedforward layer.

8. The camera-conditionally guided viewpoint synthesis method based on a video diffusion model according to claim 1, characterized in that, The loss function used for the noise added to the input video and the predicted noise in step 9 is defined as follows: Where z0 is the latent feature of the input video frame sequence, i.e., the latent code of the image sequence, t is the time step, and c is the time step. t It is an image conditional embedding extracted from a reference image frame, where P represents the Plücker coordinate embedding of the camera parameter matrix, and Φ represents the pose encoder. It is Gaussian noise that follows a standard normal distribution, z′ t It is the noisy latent code of the video frame sequence, i.e., the diffused input, and also follows a Gaussian distribution. This represents a neural network that predicts the addition of noise at each step.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the camera-conditionally guided viewpoint synthesis method based on a video diffusion model as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the camera-conditionally guided viewpoint synthesis method based on a video diffusion model as described in any one of claims 1-6.

Images