Absolute scale camera track image-to-video generation method suitable for real scene

Abstract

The invention discloses an absolute scale camera track image-to-video generation method suitable for a real scene. The method mainly comprises the following steps: reconstructing a relative scale 3D point cloud by using a multi-view reconstruction method; reconstructing an absolute-scale 3D point cloud by using a monocular depth prediction method; calculating a scene scale zoom factor from the relative scale to the absolute scale, and aligning the relative scale camera pose to the absolute scale camera pose; the absolute scale camera track is injected into the diffusion model, model training is carried out based on denoising loss, and a scene scale consistency diffusion model is obtained; in a high-noise stage, a scene constraint method without training is used for shaping a picture layout of a scene scale consistency diffusion model, the dynamic consistency is improved, and a real scene video is generated. According to the method, high-precision camera mirror movement and scene scale consistency training are supported, and the problems of inaccurate camera control, inconsistent scene scale and the like in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for generating absolute scale camera trajectory images to videos applicable to real scenes. Background Art

[0002] Image-to-video generation technology has received extensive attention in recent years in the fields of virtual reality, movie special effects, and digital content creation. Its core goal is to generate dynamic videos from static images. However, the camera motion control accuracy of existing technologies is low, and they are mostly based on simple text descriptions or rough trajectory inputs, which are difficult to meet the needs of complex scenes. Secondly, relying on multi-view data or predefined depth information, single image input often lacks these conditions. Existing technologies are mostly trained under relative scale assumptions, and the generated scenes are easily distorted and lack global consistency of absolute scale. In addition, it is difficult for existing technologies to keep the depth, layout and other constraints of scene objects consistent during motion, and the generated results lack a sense of naturalness. Therefore, the field of image-to-video generation urgently needs a technical solution to solve the problems of lack of depth information, insufficient camera motion control, and scale consistency. Summary of the invention

[0003] The purpose of the present invention is to solve the problems existing in the prior art and provide an absolute scale camera trajectory image to video generation method suitable for real scenes. The present invention uses monocular depth estimation to achieve 3D scene reconstruction, supports high-precision camera motion control and scene scale consistency training, solves the problems of inconsistent scale and poor scene control in the prior art, and realizes realistic dynamic video generation.

[0004] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0005] A method for generating absolute scale camera trajectory images to videos suitable for real scenes, comprising the following steps:

[0006] S1: Decompose the original video in the training data frame by frame to obtain the original image, obtain the camera intrinsic parameter matrix in the training data and the camera extrinsic parameter matrix corresponding to each original image, and use the multi-view reconstructor to reconstruct the relative scale 3D point cloud according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix;

[0007] S2: Take any original image in the original video as a conditional frame, reconstruct the conditional frame using a monocular depth prediction method, and obtain an absolute scale 3D point cloud;

[0008] S3: Compute the scene scale scaling factor from relative scale to absolute scale by computing the relative scale 3D point cloud and the absolute scale 3D point cloud, and align the relative scale camera pose in the training data to the absolute scale camera pose by the scene scale scaling factor;

[0009] S4: injecting the absolute scale camera pose into the pre-trained diffusion model to calculate the denoising loss of the diffusion model, updating a pose encoder based on minimizing the denoising loss of the diffusion model, and combining the diffusion model and the trained pose encoder to form a scene scale consistency diffusion model;

[0010] S5: using the image used for video generation as a new conditional frame, obtaining the absolute scale 3D point cloud corresponding to the new conditional frame according to step S2 and using it as the reconstructed 3D point cloud, obtaining the absolute scale camera pose used for video generation and using it as the reconstructed pose, inputting the reconstructed 3D point cloud and the reconstructed pose into a renderer, and obtaining a reference video;

[0011] S6: Input the reference video into the scene scale consistency diffusion model to generate a real scene video that conforms to the absolute scale camera pose.

[0012] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0013] Preferably, in step S1, the camera intrinsic parameter matrix K is composed of the lateral focal length f x , vertical focal length f y , horizontal main point c x and the longitudinal principal point c y According to the following formula:

[0014]

[0015] The i-th original image V i The camera extrinsic matrix The rotation matrix R corresponding to the i-th original image i and the translation vector T i According to the following formula:

[0016]

[0017] Preferably, in step S2, the specific process of reconstructing the absolute scale 3D point cloud is as follows:

[0018] S21: Processing the conditional frame through an absolute scale predictor to obtain an absolute scale value of each pixel in the conditional frame;

[0019] S22: Obtaining homogeneous coordinate points corresponding to each pixel in the conditional frame and using them as first homogeneous coordinate points, and projecting the first homogeneous coordinate points of each pixel in the conditional frame to normalized image coordinate points in a normalized image coordinate system through a camera intrinsic parameter matrix;

[0020] Among them, the normalized image coordinate point P corresponding to the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame is n The calculation process is as follows:

[0021] P n =K -1 (u,v,1) T

[0022] Where: -1 represents matrix inversion operation; (u,v,1) T represents the homogeneous coordinate point of the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame; the superscript T represents the matrix transpose;

[0023] S23: The absolute scale value of each pixel in the conditional frame is multiplied by the corresponding normalized image coordinate point to obtain the camera coordinate point of each pixel in the conditional frame in the camera coordinate system, and the camera coordinate points of all pixels in the conditional frame constitute an absolute scale 3D point cloud.

[0024] Preferably, the specific process of step S3 is as follows:

[0025] S31: The homogeneous coordinate point P in the relative scale 3D point cloud obtained in step S1 is rel As the second homogeneous coordinate point, project the second homogeneous coordinate point to the relative scale camera pose RT rel In the corresponding camera coordinate system, the first homogeneous coordinate points of each pixel in the conditional frame are used to calculate the corresponding relative scale values;

[0026] S32: taking the ratio between the absolute scale value and the relative scale value of each pixel in the conditional frame as the scene scale alignment scaling factor of the corresponding pixel;

[0027] S33: Take the median of the scene scale alignment scaling factors of all pixels in the conditional frame, and use the obtained median as the scene scale scaling factor s, and align the relative scale camera pose to the absolute scale camera pose RT according to the scene scale scaling factor as follows: abs :

[0028]

[0029] Where R and T are the rotation and translation components of the relative scale camera pose, respectively.

[0030] Preferably, in step S31, the relative scale value D corresponding to the pixel with the horizontal coordinate u and the vertical coordinate v in the conditional frame is rel The calculation process of (u,v) is as follows:

[0031] D rel (u,v)·(u,v,1) T =RT rel ·P rel .

[0032] Preferably, in step S4, the specific process of obtaining the denoising loss is as follows:

[0033] S41: The original video is converted to the latent space through the variational encoder of the variational autoencoder in the diffusion model to obtain the first latent space feature z 0 ;

[0034] S42: Add the Gaussian noise ∈ of the first latent space feature to the t-time step to obtain the first noise image z of the t-time step in the latent space t :

[0035] z t =α t z 0 +σ t ∈

[0036]

[0037] Where: α t represents the signal coefficient; σ t represents the noise factor;

[0038] S43: Obtain the text description corresponding to the original video and process it sequentially through the pre-trained CLIP model to obtain text conditions, and sequentially process the conditional frame through the pre-trained CLIP model and the pre-trained Q-Former model to obtain image conditions, and pass the absolute scale camera pose through the pose encoder to obtain camera conditions;

[0039] S44: calculating the Plücker coordinate point of each pixel in the conditional frame from the absolute scale camera pose and the first homogeneous coordinate point of each pixel in the conditional frame, and forming a Plücker line image from the Plücker coordinate points of all pixels in the conditional frame;

[0040] S45: inputting the Plücker line bundle image into the pose encoder to obtain a Plücker position coding feature;

[0041] S46: inputting the absolute scale camera pose, the first noise image, the text condition and its corresponding unconditional guided strength, the image condition and its corresponding unconditional guided strength, the camera condition and its corresponding unconditional guided strength, and the time step into the diffusion model, and injecting the Plücker position encoding feature into the original input data of each network layer of the diffusion model in the ControlNet paradigm to obtain new input data of each network layer of the diffusion model, which is processed by each network layer of the diffusion model, and finally the diffusion model outputs the predicted noise;

[0042] S47: Predicted noise based on the diffusion model output of step S46 And the sampled Gaussian noise ∈ calculates the denoising loss corresponding to the first noisy image

[0043]

[0044] Where: Indicates expectation; c txt 、c img and c cam denote the unconditional guidance strength of text condition, image condition and camera condition respectively; ∈ denotes the sampled Gaussian noise; Represents the square of the L2 norm; ∈ θ (·) represents the diffusion model; t represents the time step index.

[0045] Preferably, in step S43, the pose encoder is implemented using a fully connected layer.

[0046] Preferably, in step S44, the Plücker coordinate point r corresponding to the pixel with the horizontal coordinate u and the vertical coordinate v in the conditional frame is<m,d> The calculation process is as follows:

[0047] m=(-R -1 ·sT)×d

[0048] d=R -1 ·K -1 (u,v,1) T

[0049] Where: Indicates the direction of light; Indicates torque.

[0050] Preferably, in step S46, the original input data of a network layer of the diffusion model is added element by element to the Plücker position coding feature to obtain a first fused coding feature, and the first fused coding feature is pre-normalized, and the original input data is added to the pre-normalized first fused coding feature in the form of a residual connection to form new input data for the network layer.

[0051] As a preferred embodiment, the specific process of generating a real scene video is as follows:

[0052] S61: Convert the reference video into a latent space by using the variational encoder to obtain a second latent space feature;

[0053] S62: adding Gaussian noise of time step t to the second latent space feature to obtain a second noise image of time step t in the latent space;

[0054] S63: obtaining a new Plücker line bundle image formed by the Plücker coordinate points of all pixels in the new conditional frame, and inputting the new Plücker line bundle image into the trained pose encoder to obtain a new Plücker position encoding feature;

[0055] S64: adding the second noise image and the new Plück position coding feature element by element to obtain a second fused coding feature, pre-normalizing the second fused coding feature, and adding the second noise image to the pre-normalized second fused coding feature in the form of residual connection to form a third noise image;

[0056] S65: When the current time step index is greater than the preset noise shaping time step threshold, the first noise image z t With the third noise image Perform the calculation as follows to obtain the latent space features after noise shaping.

[0057]

[0058] Where: mask represents the position mask of the replaced pixel;

[0059] S66: The latent space features after noise shaping are converted to the image space through the variational decoder of the variational autoencoder in the diffusion model to obtain the real scene video.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention generates a 3D scene with absolute proportional consistency through monocular depth estimation and depth alignment technology, thereby improving the realism of video generation and solving the problem of proportional distortion in the prior art. The present invention introduces scene constraints in the high noise stage, and effectively improves the dynamic consistency and visual quality of the generated video by optimizing the diffusion model generation process. The present invention does not require multi-view data or predefined scene depth information, can adapt to diverse scenarios of single image input, and greatly expands the scope of application of the technology. Experimental results show that the present invention is significantly superior to the prior art in terms of generation quality, dynamic consistency, user experience, etc., with higher camera controllability and visual quality, and has good application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the steps of the method of the present invention;

[0063] Figure 2 A schematic diagram of the method of the present invention in training a pose encoder;

[0064] Figure 3 It is a schematic diagram of aligning a relative scale D point cloud to an absolute scale D point cloud according to the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0066] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0067] like Figure 1 As shown, in a preferred implementation of the present invention, the above absolute scale camera trajectory image to video generation method applicable to real scenes includes the following steps S1 to S6. The specific implementation process is described in detail below.

[0068] S1: For the original video in the training data The original video is decomposed frame by frame to obtain the original image, the camera intrinsic parameter matrix in the training data and the camera extrinsic parameter matrix corresponding to each original image are obtained, and the multi-view reconstruction is performed using the multi-view reconstruction tool f mvs Reconstruct the relative scale 3D point cloud PC based on the camera intrinsic parameter matrix and camera extrinsic parameter matrix rel Where: F is the number of frames of the original video, H is the height of an original image, W is the width of an original image, and C is the number of channels of an original image.

[0069] It should be noted that in step S1 of the embodiment of the present invention, the multi-view reconstructor uses the existing computer vision software COLMAP (Camera-Lens Model for Augmented Reality and Photogrammetry), and its implementation method belongs to the prior art. The training data provides the true value, and the multi-view reconstructor does not need to predict the intrinsic parameter matrix and the extrinsic parameter matrix by itself. It only needs to use the intrinsic parameter matrix and the extrinsic parameter matrix of the training data, and then project the original video back to the 3D space, and finally reconstruct the scene point cloud, that is, the relative scale 3D point cloud. The specific process of reconstructing the relative scale 3D point cloud by the multi-view reconstructor here belongs to the prior art, so it will not be repeated.

[0070] It should be noted that in step S1 of the embodiment of the present invention, the camera intrinsic parameter matrix By the horizontal focal length fx , vertical focal length f y , horizontal main point c x , longitudinal principal point c y According to the following formula:

[0071]

[0072] The i-th original image The camera extrinsic matrix The rotation matrix corresponding to the i-th original image and translation vectors According to the following formula:

[0073]

[0074] S2: Take any original image in the original video As the conditional frame, the monocular scale prediction method is used to reconstruct the conditional frame to obtain the absolute scale 3D point cloud PC abs .

[0075] It should be noted that, in step S2 of the present invention, the specific process of reconstructing the absolute scale 3D point cloud is as follows:

[0076] S21: The conditional frame is passed through the absolute scale predictor f depth Processing to obtain the absolute scale value of each pixel in the conditional frame, the process is as follows:

[0077] D abs (u,v)=f depth (V cond )

[0078] Where: D abs (u, v) represents the absolute scale value of the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame; u=1,2,…,W and v=1,2,…,H are the horizontal coordinate and vertical coordinate of the pixel in the conditional frame respectively.

[0079] In step S21 of the embodiment of the present invention, the absolute scale predictor uses an existing model (such as Depth AnythingV2). It is found in experiments that the model can already meet the monocular depth estimation accuracy requirements, so no additional training is required. Of course, other suitable models can also be selected by those skilled in the art, and whether to train or fine-tune the model or the like can be determined according to the implementation effect, so it is not limited in the present invention.

[0080] S22: Obtain the homogeneous coordinate point corresponding to each pixel in the conditional frame and use it as the first homogeneous coordinate point, and project the first homogeneous coordinate point of each pixel in the conditional frame to the normalized image coordinate point in the normalized image coordinate system through the camera intrinsic parameter matrix.

[0081] Among them, the normalized image coordinate point P corresponding to the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame is n The calculation process is as follows:

[0082] P n =K -1 (u,v,1) T

[0083] Where: -1 represents matrix inversion operation; (u,v,1) T Represents the homogeneous coordinate point of the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame; the superscript T represents matrix transposition.

[0084] S23: The absolute scale value of each pixel in the conditional frame is multiplied by the corresponding normalized image coordinate point to obtain the camera coordinate point of each pixel in the conditional frame in the camera coordinate system, and the camera coordinate points of all pixels in the conditional frame constitute an absolute scale 3D point cloud.

[0085] In step S23 of the embodiment of the present invention, the absolute scale value D of the pixel with the horizontal coordinate u and the vertical coordinate v in the conditional frame is abs (u, v) Project the normalized image coordinate point obtained in step S22 to the camera coordinate point P in the camera coordinate system c :

[0086] P c =D abs (u,v)·P n

[0087] Traversing u=1,2,…,W and v=1,2,…,H, that is, each pixel in the conditional frame obtains its corresponding camera coordinate point, and the set of camera coordinate points is used as the absolute scale 3D point cloud.

[0088] S3: Relative scale 3D point cloud PC obtained by step S1 rel And the absolute scale 3D point cloud PC obtained in step S2 abs , calculate the scene scale scaling factor s from the relative scale to the absolute scale, and use the scene scale scaling factor to convert the relative scale camera pose RT in the training data rel Align to the absolute scale camera pose RT abs .

[0089] It should be noted that the specific process of step S3 of the present invention is as follows:

[0090] S31: The homogeneous coordinate point P in the relative scale 3D point cloud obtained in step S1 is rel ∈PC relAs the second homogeneous coordinate point, project the second homogeneous coordinate point to the relative scale camera pose In the corresponding camera coordinate system, the first homogeneous coordinate points of each pixel in the conditional frame are used to calculate the corresponding relative scale values.

[0091] Among them, the relative scale value D corresponding to the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame is rel The calculation process of (u,v) is as follows:

[0092] D rel (u,v)·(u,v,1) T =RT rel ·P rel

[0093]

[0094] Where: and They are the relative scale camera pose RT rel The rotation and translation components of .

[0095] S32: The absolute scale value D of each pixel in the conditional frame abs (u,v) and relative scale value D rel The ratio between (u, v) is used as the scene scale alignment scaling factor s(u, v) of the corresponding pixel:

[0096]

[0097] S33: Take the median of the scene scale alignment scaling factors of all pixels in the conditional frame, and use the obtained median as the scene scale scaling factor s, and use the scene scale scaling factor to convert the relative scale camera pose RT into rel Align to absolute scale camera pose

[0098]

[0099] s=MEDIANs(u,v)

[0100] Where: MEDIAN represents the median operator.

[0101] like Figure 3 As shown, the yellow point cloud is the relative scale 3D point cloud, and the colored point cloud is the absolute scale 3D point cloud. After the above process, the relative scale 3D point cloud is finally aligned to the absolute scale D point cloud.

[0102] S4: The absolute scale camera pose RTF obtained in step S3 absInject into the pre-trained diffusion model DM to calculate the denoising loss of the diffusion model, based on minimizing the denoising loss of the diffusion model A pose encoder is updated, and the diffusion model and the trained pose encoder are combined to form a scene scale consistent diffusion model DM′.

[0103] It should be noted that in step S4 of the present invention, the diffusion model structure and the specific network used are not limited, as long as they contain a variational autoencoder and can cooperate with the posture encoder of the present invention.

[0104] It should be noted that, in step S4 of the present invention, Figure 2 As shown, the specific process of obtaining the denoising loss is as follows:

[0105] S41: The original video V in step S1 is converted to the latent space through the variational encoder ε(·) of the variational autoencoder in the diffusion model to obtain the first latent space feature z 0 =ε(V).

[0106] S42: The first latent space feature z obtained in step S41 is 0 Add Gaussian noise ∈ of time step t to obtain the first noise image z of time step t in the latent space t :

[0107] z t =α t z 0 +σ t ∈

[0108] Where: α t represents the signal coefficient; Represents the noise figure.

[0109] In step S42 of the embodiment of the present invention, the time step index t is an integer in the range [1, 1000].

[0110] S43: Obtain the text description corresponding to the original video and process it sequentially through the pre-trained CLIP model to obtain text conditions, and process the conditional frame sequentially through the pre-trained CLIP model and the pre-trained Q-Former model to obtain image conditions, and pass the absolute scale camera pose obtained in step S3 through the pose encoder to obtain camera conditions.

[0111] In step S43 of the embodiment of the present invention, the implementation of the CLIP model and the Q-Former model belongs to the prior art, so it is not repeated here. The pose encoder used in this step is a fully connected layer, and its weights and biases are trainable parameters of the present invention. The weights and biases here will be updated later by minimizing the denoising loss.

[0112] S44: Calculate the Plücker coordinate point of each pixel in the conditional frame from the absolute scale camera pose and the first homogeneous coordinate point of each pixel in the conditional frame, and construct a Plücker line image from the Plücker coordinate points of all pixels in the conditional frame.

[0113] The Plücker coordinate point corresponding to the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame is The calculation process is as follows:

[0114] m=(-R -1 ·sT)×d

[0115] d=R -1 ·K -1 (u,v,1) T

[0116] Where: Indicates the direction of light; Indicates torque.

[0117] In step S44 of the embodiment of the present invention, u=1, 2, ..., W and v=1, 2, ..., H are traversed, that is, each pixel in the conditional frame obtains its corresponding Plücker coordinate point, and the set of Plücker coordinate points is used as the Plücker line beam image.

[0118] S45: Input the Plücker line image obtained in step S44 into the pose encoder to obtain the Plücker position coding feature

[0119] In step S45 of the embodiment of the present invention, the pose encoder is implemented by a fully connected layer, so the process of obtaining the Plücker position encoding feature can be expressed as:

[0120] e=W l ·P l +b l

[0121] in, and represents the trainable parameters in the pose encoder; D represents the encoding dimension.

[0122] S46: The absolute scale camera pose obtained in step S3, the first noise image, the text condition generated in step S43 and its corresponding unconditional guide strength, the image condition generated in step S43 and its corresponding unconditional guide strength, the camera condition generated in step S43 and its corresponding unconditional guide strength, and the time step are input into the diffusion model ∈ θAt the same time, the Plücker position coding feature obtained in step S45 is injected into the original input data of each network layer of the diffusion model in the ControlNet paradigm to obtain new input data of each network layer of the diffusion model, which is processed by each network layer of the diffusion model and finally outputs the predicted noise by the diffusion model.

[0123] It should be noted that, in step S46 of the present invention, taking a network layer of the diffusion model as an example, the original input data x of the network layer of the diffusion model is added element by element to the Plücker position coding feature to obtain a fused coding feature, and the fused coding feature is pre-normalized, and the original input data is added to the pre-normalized fused coding feature in the form of residual connection to form a new input data of the network layer

[0124]

[0125] Where: PreNorm represents the pre-normalization operator.

[0126] In the embodiment of the present invention, step S46 is to adjust the original input data of each network layer in the diffusion model, and the specific structure of the diffusion model is not limited. Taking the first network layer of the diffusion model as an example, its original input data is composed of the absolute scale camera pose, the first noise image, the text condition generated in step S43 and its corresponding unconditional guide strength, the image condition generated in step S43 and its corresponding unconditional guide strength, the camera condition generated in step S43 and its corresponding unconditional guide strength and the time step. The Plücker position coding feature is introduced according to the above processing method to obtain the new input data of the first network layer. Therefore, the data processing process performed inside each network layer is not mentioned in the present invention. In other words, for those skilled in the art, it is only necessary to understand how the new input data of each network layer is obtained, and then replace the original input data of the diffusion model with this part of the new input data, and obtain the final output after being processed by the corresponding network layer of the diffusion model.

[0127] S47: Predicted noise based on the diffusion model output of step S46 And the sampled Gaussian noise ∈ calculates the denoising loss corresponding to the first noisy image

[0128]

[0129] in, Indicates expectation; c txt 、c img and c cam denote the unconditional guidance strength of text condition, image condition and camera condition respectively; ∈ denotes the sampled Gaussian noise; Represents the square of the L2 norm; ∈θ (·) represents the diffusion model; t represents the time step index.

[0130] S5: Take the image used for video generation as a new conditional frame, obtain the absolute scale 3D point cloud corresponding to the new conditional frame according to step S2 and use it as the reconstructed 3D point cloud, obtain the absolute scale camera pose used for video generation and use it as the reconstructed pose, and input the reconstructed 3D point cloud and the reconstructed pose into the renderer f render , get the reference video

[0131]

[0132] It should be noted that in step S5 of the present invention, the conditional frame in step S2 is replaced by the new conditional frame in step S5, and the new conditional frame is reconstructed according to the monocular depth prediction method, that is, the absolute scale 3D point cloud corresponding to the new conditional frame is obtained. The renderer uses existing tools (such as Open3D, OpenGL, etc.), and other suitable models or tools can certainly be selected by those skilled in the art, so it is not limited in the present invention.

[0133] S6: Input the reference video into the scene scale consistency diffusion model to generate a real scene video that conforms to the absolute scale camera pose.

[0134] It should be noted that in step S6 of the present invention, at the high noise time step in the sampling phase, a noise shaping method without training is used to constrain the scene layout of the scene scale consistency diffusion model, and finally a real scene video that conforms to the absolute scale camera trajectory is generated. The specific process is as follows:

[0135] S61: Convert the reference video to the latent space through the variational encoder to obtain the second latent space feature z′ 0 =ε(V′).

[0136] S62: The second latent space feature z′ obtained in step S61 0 Add Gaussian noise ∈ of time step t to obtain the second noise image z of time step t in the latent space t ′:

[0137] z t ′=α t z′ 0 +σ t ∈

[0138] S63: Obtain a new Plücker line bundle image formed by the Plücker coordinate points of all pixels in the new conditional frame, and input the new Plücker line bundle image into the trained pose encoder to obtain a new Plücker position encoding feature.

[0139] In step S63 of the embodiment of the present invention, the conditional frame in step S44 is replaced with a new conditional frame, and the absolute scale camera pose is replaced with a reconstructed pose, so that the Plücker coordinate points of each pixel in the new conditional frame can be obtained according to the process in step S44, thereby forming a new Plücker line beam image. The trained pose encoder then outputs the new Plücker position coding features corresponding to the new Plücker line beam image.

[0140] S64: Transform the second noise image z t ′ is added element by element to the new Plucker position coding feature e′ to obtain the second fused coding feature, and the second fused coding feature is pre-normalized, and the second noise image is added to the pre-normalized second fused coding feature in the form of residual connection to form a third noise image

[0141]

[0142] S65: When the current time step index is greater than the preset noise shaping time step threshold t thres When the first noise image z t With the third noise image Perform calculations to obtain the latent space features after noise shaping

[0143]

[0144] Where: mask represents the position mask of the replaced pixel.

[0145] In step S65 of the embodiment of the present invention, if and only if time step t>t thres In the high noise stage, the third noise image obtained in step S64 is compared with the first noise image z obtained in step S42. t Perform operations to obtain the latent space features after noise shaping Here, the noise shaping time step threshold t thres =900.

[0146] S66: Variational Decoder via Variational Autoencoder in Diffusion Model The latent space features after noise shaping Convert to image space to get real scene video

[0147] The present invention will use a specific example below to demonstrate the application effect of the absolute scale camera trajectory image to video generation method applicable to real scenes described in S1 to S6 in the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0148] Example

[0149] The specific implementation process of the absolute scale camera trajectory image-to-video generation method applicable to real scenes adopted in this embodiment is as described above and will not be repeated here.

[0150] This embodiment uses a RealEstate10K dataset with true value annotations, which contains 60,000 videos and corresponding relative scale camera trajectories. The RealEstate10K dataset is divided into the RealEstate10K training set and the RealEstate10K test set, and then the model is trained on the RealEstate10K training set, and the model is tested on the RealEstate10K test set. The verification indicators selected are FVD, TransErr, RotError and CamMC. Among them, FVD (Fréchet Video Distance) is a commonly used indicator for measuring generation quality in the field of video generation. It is used to calculate the distribution distance between two video sets. The present invention uses VideoGPT and StyleGAN to extract video set distributions for calculating the FVD indicator. TranErr, RotErr and CamMC are all camera pose indicators, which respectively measure the rotation component error, translation component error and overall pose error accumulated frame by frame of the generated video. The calculation process is as follows:

[0151]

[0152] Where F represents the number of video frames; tr represents the matrix trace operation; Represents the transpose of the rotation component of the i-th frame of the video; represents the true value of the rotation component of the i-th frame of the video; T i Represents the translation component of the i-th frame of the video; Represents the true value of the translation component of the i-th frame of the video; s i Represents the scene scale scaling factor of the i-th frame of the video; Indicates the true value of the scene scale scaling factor of the i-th frame of the video; RT i Represents the overall pose of the i-th frame of the video; Represents the true value of the overall pose of the i-th frame of the video.

[0153] In order to verify the necessity of scale alignment consistency training, this embodiment takes the scene scale scaling coefficients of the real video and the generated video respectively, and calculates the relative TransErr and relative CamMC; the scene scale scaling coefficients of the real video are taken for both the real video and the generated video, and the absolute TransErr and absolute CamMC are calculated. This embodiment randomly selects 1000 samples from the RealEstate10K test set, and each sample is independently measured 5 times to take the average. DynamiCrafter is selected as the video base model, and the most advanced existing technology is reproduced on the same video base model for fair comparison. The test results of the RotErr indicator are shown in Table 1, the test results of the TranErr indicator are shown in Table 2, the test results of the CamMC indicator are shown in Table 3, and the test results of the FVD indicator are shown in Table 4.

[0154] Table 1. RotErr index test results

[0155]

[0156] According to the test results in Table 1, it can be found that in terms of rotation error (RotErr), the error value of the present invention is 0.405, which is further reduced by 18.5% compared with the existing optimal method CamI2V (0.497), showing the accuracy of camera rotation control.

[0157] Table 2. TranErr index test results

[0158]

[0159] According to the experimental results in Table 2, it can be found that in terms of translation error (TransErr), the present invention has achieved the best results in both relative indicators (1.309) and absolute indicators (2.471), indicating that deep training at the absolute scale significantly improves the ability to accurately capture translation in real scenes.

[0160] According to the test results in Table 3, it can be found that in terms of camera motion consistency (CamMC), the absolute index of the present invention is 2.610, which is significantly improved by 27.1% compared with 3.579 of CamI2V. This result verifies the role of introducing scene constraints and optimizing noise shaping in promoting dynamic consistency.

[0161] Table 3. CamMC index test results

[0162]

[0163]

[0164] According to the test results in Table 4, it can be found that in terms of the key indicator of generated video quality FVD (Frechet Video Distance), the results of the present invention are 55.229 (VideoGPT) and 48.080

[0165] (StyleGAN), surpassing CamI2V’s 63.869 and 55.276 respectively, showing the advantages of generated videos in visual consistency and dynamic realism.

[0166] Table 4. FVD index test results

[0167]

[0168] From the test results in Tables 1 to 4, it can be seen that the present invention has achieved significant improvements in multiple indicators compared with the prior art. These improvements are due to the diffusion model optimization strategy of the present invention that combines absolute scale alignment consistency training and noise shaping scene constraints, which enables the camera control to show high stability and realism in large-scale movements and fast switching. The present invention adapts to diverse scene inputs and shows excellent robustness and compatibility in real-world applications.

[0169] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for generating absolute scale camera trajectory images to videos suitable for real scenes, characterized in that: The following steps are involved: S1: Decompose the original video in the training data frame by frame to obtain the original image, obtain the camera intrinsic parameter matrix in the training data and the camera extrinsic parameter matrix corresponding to each original image, and use the multi-view reconstructor to reconstruct the relative scale 3D point cloud according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix; S2: Take any original image in the original video as a conditional frame, reconstruct the conditional frame using a monocular depth prediction method, and obtain an absolute scale 3D point cloud; S3: Compute the scene scale scaling factor from relative scale to absolute scale by computing the relative scale 3D point cloud and the absolute scale 3D point cloud, and align the relative scale camera pose in the training data to the absolute scale camera pose by the scene scale scaling factor; S4: injecting the absolute scale camera pose into the pre-trained diffusion model to calculate the denoising loss of the diffusion model, updating a pose encoder based on minimizing the denoising loss of the diffusion model, and combining the diffusion model and the trained pose encoder to form a scene scale consistency diffusion model; S5: using the image used for video generation as a new conditional frame, obtaining the absolute scale 3D point cloud corresponding to the new conditional frame according to step S2 and using it as the reconstructed 3D point cloud, obtaining the absolute scale camera pose used for video generation and using it as the reconstructed pose, inputting the reconstructed 3D point cloud and the reconstructed pose into a renderer, and obtaining a reference video; S6: Input the reference video into the scene scale consistency diffusion model to generate a real scene video that conforms to the absolute scale camera pose.

2. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 1, characterized in that: In step S1, the camera intrinsic parameter matrix K is composed of the horizontal focal length f x , vertical focal length f y , horizontal main point c x and the longitudinal principal point c y According to the following formula: The i-th original image V i The camera extrinsic matrix The rotation matrix R corresponding to the i-th original image i and the translation vector T i According to the following formula:

3. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 2, characterized in that: In step S2, the specific process of reconstructing the absolute scale 3D point cloud is as follows: S21: Processing the conditional frame through an absolute scale predictor to obtain an absolute scale value of each pixel in the conditional frame; S22: Obtaining homogeneous coordinate points corresponding to each pixel in the conditional frame and using them as first homogeneous coordinate points, and projecting the first homogeneous coordinate points of each pixel in the conditional frame to normalized image coordinate points in a normalized image coordinate system through a camera intrinsic parameter matrix; Among them, the normalized image coordinate point P corresponding to the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame is n The calculation process is as follows: Where: -1 represents matrix inversion operation; represents the homogeneous coordinate point of the pixel with horizontal coordinate u and vertical coordinate v in the conditional frame; the superscript T represents the matrix transpose; S23: The absolute scale value of each pixel in the conditional frame is multiplied by the corresponding normalized image coordinate point to obtain the camera coordinate point of each pixel in the conditional frame in the camera coordinate system, and the camera coordinate points of all pixels in the conditional frame constitute an absolute scale 3D point cloud.

4. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 3, characterized in that: The specific process of step S3 is as follows: S31: The homogeneous coordinate point P in the relative scale 3D point cloud obtained in step S1 is rel As the second homogeneous coordinate point, project the second homogeneous coordinate point to the relative scale camera pose RT rel In the corresponding camera coordinate system, the first homogeneous coordinate points of each pixel in the conditional frame are used to calculate the corresponding relative scale values; S32: taking the ratio between the absolute scale value and the relative scale value of each pixel in the conditional frame as the scene scale alignment scaling factor of the corresponding pixel; S33: Take the median of the scene scale alignment scaling factors of all pixels in the conditional frame, and use the obtained median as the scene scale scaling factor s, and align the relative scale camera pose to the absolute scale camera pose RT according to the scene scale scaling factor as follows: abs : Where R and T are the rotation and translation components of the relative scale camera pose, respectively.

5. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 4, characterized in that: In step S31, the relative scale value D corresponding to the pixel with the horizontal coordinate u and the vertical coordinate v in the conditional frame is rel The calculation process of (u,v) is as follows:

6. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 4, characterized in that: In step S4, the specific process of obtaining the denoising loss is as follows: S41: converting the original video into a latent space through a variational encoder of a variational autoencoder in a diffusion model to obtain a first latent space feature z0; S42: Add the Gaussian noise ∈ of the first latent space feature to the t-time step to obtain the first noise image z of the t-time step in the latent space t : z t =a t z0+σ t ∈ Where: α t represents the signal coefficient; σ t represents the noise factor; S43: Obtain the text description corresponding to the original video and process it sequentially through the pre-trained CLIP model to obtain text conditions, and sequentially process the conditional frame through the pre-trained CLIP model and the pre-trained Q-Former model to obtain image conditions, and pass the absolute scale camera pose through the pose encoder to obtain camera conditions; S44: calculating the Plücker coordinate point of each pixel in the conditional frame from the absolute scale camera pose and the first homogeneous coordinate point of each pixel in the conditional frame, and forming a Plücker line image from the Plücker coordinate points of all pixels in the conditional frame; S45: inputting the Plücker line bundle image into the pose encoder to obtain a Plücker position coding feature; S46: inputting the absolute scale camera pose, the first noise image, the text condition and its corresponding unconditional guided strength, the image condition and its corresponding unconditional guided strength, the camera condition and its corresponding unconditional guided strength, and the time step into the diffusion model, and injecting the Plücker position encoding feature into the original input data of each network layer of the diffusion model in the ControlNet paradigm to obtain new input data of each network layer of the diffusion model, which is processed by each network layer of the diffusion model, and finally the diffusion model outputs the predicted noise; S47: Predicted noise based on the diffusion model output of step S46 And the sampled Gaussian noise ∈ calculates the denoising loss corresponding to the first noisy image Where: Indicates expectation; c txt 、c img and c cam denote the unconditional guidance strength of text condition, image condition and camera condition respectively; ∈ denotes the sampled Gaussian noise; Represents the square of the L2 norm; ∈ θ (·) represents the diffusion model; t represents the time step index.

7. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 6, characterized in that: In step S43, the pose encoder is implemented using a fully connected layer.

8. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 6, characterized in that: In step S44, the Plücker coordinate point r corresponding to the pixel with the horizontal coordinate u and the vertical coordinate v in the conditional frame is<m,d> The calculation process is as follows: m=(-R -1 ·sT)×d d=R -1 ·K -1 ·(u,v,1) T Where: Indicates the direction of light; Indicates torque.

9. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 6, characterized in that: In step S46, the original input data of a network layer of the diffusion model is added element by element to the Plücker position coding feature to obtain a first fused coding feature, and the first fused coding feature is pre-normalized, and the original input data is added to the pre-normalized first fused coding feature in the form of a residual connection to form new input data for the network layer.

10. The absolute scale camera trajectory image to video generation method applicable to real scenes as claimed in claim 6, characterized in that: The specific process of generating real scene videos is as follows: S61: Convert the reference video into a latent space by using the variational encoder to obtain a second latent space feature; S62: adding Gaussian noise of time step t to the second latent space feature to obtain a second noise image of time step t in the latent space; S63: obtaining a new Plücker line bundle image formed by the Plücker coordinate points of all pixels in the new conditional frame, and inputting the new Plücker line bundle image into the trained pose encoder to obtain a new Plücker position encoding feature; S64: adding the second noise image and the new Plück position coding feature element by element to obtain a second fused coding feature, pre-normalizing the second fused coding feature, and adding the second noise image to the pre-normalized second fused coding feature in the form of residual connection to form a third noise image; S65: When the current time step index is greater than the preset noise shaping time step threshold, the first noise image z t With the third noise image Perform the calculation as follows to obtain the latent space features after noise shaping. Where: mask represents the position mask of the replaced pixel; S66: The latent space features after noise shaping are converted to the image space through the variational decoder of the variational autoencoder in the diffusion model to obtain the real scene video.

Images