Video generation method, device, electronic device, storage medium and computer program product

Abstract

The present application provides a video generation method, device, electronic device, storage medium and computer program product; including: obtaining a target image and camera trajectory information; determining the image coding features of the target image and the trajectory coding features of the camera trajectory information; generating a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence includes a plurality of scene images obtained by predicting the camera movement of the target image based on the camera trajectory information; generating a target video according to the target image and the reference image sequence; the first frame of the target video is consistent with the target image, and the camera movement mode of the target video is consistent with the camera trajectory information. The present application generates a target video by combining the camera trajectory information and the target image, which can improve the authenticity of the target video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, electronic device, storage medium and computer program product. Background Art

[0002] Videos can be seen everywhere in daily life. People can not only shoot videos anytime and anywhere through various devices (such as mobile phones, cameras, etc.), but also make various videos according to their own creativity. Moreover, with the rise of artificial intelligence, people can generate the videos they want through intelligent algorithms.

[0003] In the related technology, users can input their ideas into the intelligent algorithm in the form of text, and the intelligent algorithm can generate a corresponding video based on the input text. However, the video generated only by text description cannot well display the scene details in the video, which will reduce the authenticity of the generated video. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, device, electronic device, storage medium and computer program product, which can improve the authenticity of the generated target video by combining camera trajectory information and a target map to generate a target video.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a video generation method, the method comprising: obtaining a target image and camera trajectory information; determining image coding features of the target image and trajectory coding features of the camera trajectory information; generating a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence comprises a plurality of scene images obtained by predicting the camera movement of the target image based on the camera trajectory information; generating a target video according to the target image and the reference image sequence; comparing the first frame of the target video with the first frame of the target video; Figure 1 The camera movement mode of the target video is consistent with the camera trajectory information.

[0007] The embodiment of the present application provides a video generation device, comprising: an acquisition module, used to acquire a target image and camera trajectory information; a determination module, used to determine the image coding features of the target image and the trajectory coding features of the camera trajectory information; a first generation module, used to generate a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence includes a plurality of scene images obtained by predicting the movement of the target image based on the camera trajectory information; a second generation module, used to generate a target video according to the target image and the reference image sequence; the first frame of the target video is the same as the first frame of the target video; Figure 1The camera movement mode of the target video is consistent with the camera trajectory information.

[0008] In the above scheme, the camera trajectory information includes at least one of a camera trajectory graph and a camera trajectory description text; the determination module is further used to: when the camera trajectory information includes the camera trajectory graph, perform image encoding on the camera trajectory graph to obtain the trajectory encoding feature; when the camera trajectory information includes the camera trajectory description text, generate a camera trajectory graph based on the camera trajectory description text, perform image encoding on the camera trajectory graph, and obtain the trajectory encoding feature; when the camera trajectory information includes the camera trajectory graph and the camera trajectory description text, perform trajectory update on the camera trajectory in the camera trajectory graph based on the camera trajectory description text to obtain an updated camera trajectory graph; and perform image encoding on the updated camera trajectory graph to obtain the trajectory encoding feature.

[0009] In the above scheme, the first generation module is also used to: input the image coding features and the trajectory coding features into an image generation network; the image generation network includes multiple self-attention mechanism layers and multiple camera motion attention mechanism layers, wherein the self-attention mechanism layers are interlaced with the camera motion attention mechanism layers; the image coding features are input into the first self-attention mechanism layer of the image generation network, and the trajectory coding features are input into the first camera motion attention mechanism layer of the image generation network; through the multiple self-attention mechanism layers and the multiple camera motion attention mechanism layers, the image coding features and the trajectory coding features are subjected to first feature processing to generate the reference image sequence.

[0010] In the above scheme, the first generation module is also used to: perform a first attention processing on the image coding feature through the first self-attention mechanism layer to obtain a first image feature; perform a second attention processing on the first image feature and the trajectory coding feature through the first camera motion attention mechanism layer to obtain a second image feature; and perform a first feature processing on the second image feature in sequence through a self-attention mechanism layer and a camera motion attention mechanism layer other than the first self-attention mechanism layer and the first camera motion attention mechanism layer to obtain the reference image sequence.

[0011] In the above scheme, the first generation module is also used to: perform feature fusion on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a first fused feature; and perform the second attention processing on the first fused feature to obtain the second image feature.

[0012] In the above scheme, the second generation module is also used to: identify the moving object in the target image and obtain the motion state characteristics of the moving object; obtain random Gaussian noise; splice the random Gaussian noise and the motion state characteristics in the channel dimension to obtain a first splicing feature; generate the target video according to the first splicing feature and the reference image sequence.

[0013] In the above scheme, the second generation module is also used to: input the first splicing feature and the reference image sequence into a video generation network, and perform multiple diffusion processes based on the first splicing feature and the reference image sequence through the video generation network; wherein the output of the last diffusion process is the first latent space expression of the target video; and decode the first latent space expression to obtain the target video.

[0014] An embodiment of the present application provides a training method for a video generation model, wherein the video generation model includes a coding network, an image generation network, and a video generation network. The training method includes: inputting the acquired sample label video, sample image, and sample camera trajectory information into the coding network to determine a second latent space expression of the sample label video, a sample image coding feature of the sample image, and a sample trajectory coding feature of the sample camera trajectory information; inputting the sample image coding feature and the sample trajectory coding feature into the image generation network to generate a sample reference image sequence; determining a loss result of the video generation network according to the sample image, the sample reference image sequence, and the second latent space expression; and adjusting the network parameters of the image generation network according to the loss result to obtain a trained video generation model.

[0015] In the above scheme, the video generation model also includes a motion state prediction network; the device also includes a model training module, which is used to: identify the sample motion object in the sample image through the motion state prediction network, and obtain the sample motion state feature of the sample motion object; splice the sample motion state feature and the second latent space expression to obtain a second splicing feature; input the second splicing feature and the sample reference image sequence into the video generation network to obtain the loss result.

[0016] In the above scheme, the training module is also used to: adjust the network parameters of the image generation network and the network parameters of the motion state prediction network according to the loss result to obtain a trained video generation model.

[0017] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the video generation method provided in the embodiment of the present application, or implementing the training method of the video generation model provided in the embodiment of the present application, when executing the computer-executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or executable instructions. When the computer program or computer executable instructions are executed by a processor, the video generation method provided in the embodiment of the present application is implemented, or the training method of the video generation model provided in the embodiment of the present application is implemented.

[0019] An embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the computer executable instructions from the computer-readable storage medium and executes the computer executable instructions, the video generation method provided in the embodiment of the present application is implemented, or the training method of the video generation model provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] The video generation method in the embodiment of the present application ensures that the content of the generated target video is highly consistent with the preset camera motion trajectory by extracting and processing the trajectory coding features of the camera trajectory information, thereby improving the control accuracy of the camera; and obtains a reference image sequence by performing image processing on the image coding features and the trajectory coding features, and generates the final target video by combining the reference image sequence and the target image, so that the generated target video can accurately represent the scene details, thereby improving the authenticity of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an optional flowchart of the video generation method provided in the embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the interlaced connection between the self-attention mechanism layer and the camera motion attention mechanism layer provided in an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of a process of obtaining a reference image sequence through multiple self-attention mechanism layers and multiple camera motion attention mechanism layers provided in an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of a process of generating a target video according to a target image and a reference image sequence provided by an embodiment of the present application;

[0026] Figure 5 It is a flowchart of a training method for a video generation model provided in an embodiment of the present application;

[0027] Figure 6 It is a structural framework diagram of the video generation method provided in the embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of three-dimensional modeling provided in an embodiment of the present application;

[0029] Figure 8 is a structural block diagram of a video generating device provided in an embodiment of the present application;

[0030] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] In order to better understand the video generation method provided in the embodiment of the present application, the video generation method in the related art is first described below.

[0037] In the related art, the video generation method is limited to text guidance plus camera trajectory control. The text guidance plus camera trajectory control method can only be limited to specific directions, such as moving forward, backward or left, etc. However, the input text description is relatively limited, and it is difficult to accurately describe the user's needs. At the same time, it is also unable to display the scene details well, resulting in the generated video having low authenticity and failing to meet the user's personalized needs.

[0038] Based on the problems existing in the related art, an embodiment of the present application provides a video generation method. When generating a video, first, a target image and camera trajectory information are obtained; then, image coding features of the target image and trajectory coding features of the camera trajectory information are determined; then, a reference image sequence is generated according to the image coding features and the trajectory coding features; wherein the reference image sequence includes multiple scene images obtained by predicting the movement of the target image based on the camera trajectory information; finally, a target video is generated according to the target image and the reference image sequence; wherein the first frame of the target video is the same as the first frame of the target video. Figure 1 The target video's camera movement is consistent with the camera trajectory information. In this way, by extracting and processing the trajectory coding features of the camera trajectory information, it is ensured that the content of the generated target video is highly consistent with the preset camera motion trajectory, thereby improving the camera's control accuracy; and by performing image processing on the image coding features and the trajectory coding features, a reference image sequence is obtained, and the final target video is generated by combining the reference image sequence and the target image, so that the generated target video can well describe the detailed scene, thereby improving the authenticity of the target video.

[0039] The video generation method provided in the embodiment of the present application can be applied to electronic devices such as laptop computers, tablet computers, and desktop computers. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0040] The video generation method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0041] Figure 1 is an optional flow chart of the video generation method provided in the embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0042] Step S101, obtaining target image and camera trajectory information.

[0043] The target image refers to image data containing a target object. The target object refers to the subject in the target image, such as a person, an object or an animal. The type of the target object is not limited in this application.

[0044] The camera trajectory refers to the motion path of the camera in three-dimensional space, including the changes in the camera's position and direction over time. The camera trajectory information is a description of the camera's motion path and is usually stored in the form of time series data.

[0045] The target map and camera trajectory information can be directly obtained from a pre-saved database or manually input by the user.

[0046] Step S102, determining the image coding features of the target image and the trajectory coding features of the camera trajectory information.

[0047] Image coding features refer to feature vectors or feature representations extracted from target images by image encoders, which are used to describe image content. Image coding features can capture key information such as color, texture, shape, spatial relationship, etc. of target images.

[0048] Trajectory coding features are coded representations of camera motion trajectory information, usually used to compress trajectory data or extract key camera motion patterns. Trajectory coding features can describe properties such as camera motion speed, direction change, and path curve.

[0049] In some embodiments, the camera trajectory information includes at least one of a camera trajectory graph and a camera trajectory description text; wherein, in step S102, determining the trajectory coding feature of the camera trajectory information may include at least the following three situations:

[0050] Case 1: When the camera trajectory information includes a camera trajectory graph, image encoding is performed on the camera trajectory graph to obtain a trajectory encoding feature.

[0051] The camera trajectory diagram refers to an image that represents the camera trajectory information in a visual form, where the camera trajectory information is used to characterize the changes in camera motion parameters such as camera position, camera orientation, and camera motion speed. The visual form has no input threshold and does not require familiarity with camera parameters, that is, the camera trajectory information can be reflected through a simple marking method. For example, the visual forms include: trajectory line, trajectory point sequence, trajectory color coding, etc., where the trajectory line refers to a method of representing the camera trajectory information through a line or a combination of a line and an arrow, the trajectory point sequence refers to a method of representing the camera trajectory information through a collection of some points or a collection of some icons, and the trajectory color coding refers to a method of representing the camera trajectory information through some color information.

[0052] In this embodiment, the camera trajectory diagram may include a two-dimensional trajectory diagram and / or a three-dimensional trajectory diagram; a two-dimensional trajectory diagram refers to a curve or a broken line representing the motion path of the camera in a plane, and only displays horizontal or vertical motion; a three-dimensional trajectory diagram refers to a motion path of the camera in a three-dimensional coordinate system, usually represented by a curve plus an arrow, and the specific representation of the camera trajectory diagram is not limited in this application. Image encoding refers to the process of converting an image into a feature representation.

[0053] In some embodiments, under the conditions of situation one, the camera trajectory map can be image encoded by a traditional feature extraction algorithm, such as extracting the shape features of the camera trajectory by the canny algorithm or the sobel algorithm, extracting the local features of the camera trajectory map by the scale-invariant feature transform algorithm (SIFT, Scale-invariant feature transform), etc.; the camera trajectory map can also be image encoded by a deep learning algorithm, such as a residual network (ResNet), an efficient neural network (Effi cient Net), etc. The specific algorithm is not limited in this application.

[0054] Case 2: When the camera trajectory information includes a camera trajectory description text, a camera trajectory graph is generated based on the camera trajectory description text, and image encoding is performed on the camera trajectory graph to obtain a trajectory encoding feature.

[0055] The camera trajectory description text refers to the natural language description of the camera trajectory, including the camera's position, direction, and moving speed. For example, the camera starts from point (0, 0, 0), moves along the X-axis at a speed of 2 cm per second for 5 seconds, and then moves along the Z-axis at a speed of 1 cm per second for 3 seconds.

[0056] In some embodiments, under the conditions of situation two, the camera trajectory description text can be parsed and converted through a pre-trained text graph model to generate a camera trajectory graph; then, the camera trajectory graph is image encoded through a traditional feature extraction algorithm or a deep learning algorithm to obtain trajectory encoding features. The traditional feature algorithm or deep learning algorithm used has been introduced in situation one and will not be repeated in this application.

[0057] Case 3: When the camera trajectory information includes a camera trajectory graph and a camera trajectory description text, the camera trajectory in the camera trajectory graph is updated based on the camera trajectory description text to obtain an updated camera trajectory graph; and the updated camera trajectory graph is image encoded to obtain a trajectory encoding feature.

[0058] In some embodiments, under the conditions of situation three, the camera trajectory in the camera trajectory map is updated based on the camera trajectory description text to obtain an updated camera trajectory map, which can be achieved by the following method: first, the camera trajectory description text is parsed, and key information (such as the newly added moving direction, moving speed, and moving time, etc.) is extracted from the camera trajectory description text; then, the camera trajectory points are calculated or modified according to the extracted key information, and the trajectory points are inserted or replaced into the existing trajectory map to achieve the update of the camera trajectory map.

[0059] As an example of case three, suppose the existing camera trajectory is that the camera moves from point (0, 0, 0) along the X-axis to (10, 0, 0); the camera trajectory description text is "starting to rise along the Z-axis at (10, 0, 0) at a speed of 1 cm per second for three seconds"; the camera trajectory text is parsed, and the newly added trajectory points (10, 0, 1), (10, 0, 2) and (10, 0, 3) are calculated, and these three points are added to the existing trajectory map to obtain an updated camera trajectory map; the updated camera trajectory map is image encoded by a traditional feature extraction algorithm or a deep learning algorithm to obtain a trajectory encoding feature.

[0060] Step S103, generating a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence includes a plurality of scene images obtained by performing camera movement prediction on the target image based on the camera trajectory information.

[0061] There is spatial continuity between each adjacent image frame in the reference image sequence; the reference image sequence contains both the visual information of the image and the dynamic information of the camera motion trajectory.

[0062] Camera motion prediction refers to predicting the shooting effects of the camera at different positions and angles based on the camera trajectory information, thereby generating images that conform to the camera motion trajectory. Specifically, camera motion prediction uses camera trajectory information (such as the camera's spatial position, direction of movement, speed, rotation angle, etc.) to calculate the change in the camera's shooting angle, and generates scene graphs of different angles based on the change in the camera's shooting angle.

[0063] In some embodiments, step S103 can be implemented by the following method: first, the image coding features and the trajectory coding features are input into the image generation network; the image generation network includes multiple self-attention mechanism layers and multiple camera motion attention mechanism layers, wherein the self-attention mechanism layers are interlaced with the camera motion attention mechanism layers; the image coding features are input into the first self-attention mechanism layer of the image generation network, and the trajectory coding features are input into the first camera motion attention mechanism layer of the image generation network; then, through multiple self-attention mechanism layers and multiple camera motion attention mechanism layers, the image coding features and the trajectory coding features are subjected to first feature processing to generate a reference image sequence.

[0064] The image generation network refers to a network model that has been pre-trained on a large-scale data set; the image generation network can be implemented as U-Net, and the specific implementation method is not limited in this application.

[0065] The self-attention mechanism layer is a network layer used to capture global dependencies within the image encoding features. The self-attention mechanism layer can dynamically adjust the weight of each pixel to enhance the image feature representation capability, enabling the pre-trained image generation network to capture the contextual relationship within the image.

[0066] The camera motion attention mechanism layer is used to learn the correlation between each spatiotemporal point in the camera motion trajectory. The camera motion attention mechanism layer can dynamically adjust the influence weight of the camera motion trajectory on image generation according to the trajectory encoding features.

[0067] The first feature processing refers to the process of optimizing and enhancing image encoding features and trajectory encoding features under the action of multiple self-attention mechanisms and camera motion attention mechanisms.

[0068] In some embodiments, interleaving the self-attention mechanism layer with the camera motion attention mechanism layer refers to a connection form in which the self-attention mechanism layer and the camera motion attention mechanism layer are arranged and connected in a specific order; see Figure 2 , Figure 2 This is a schematic diagram of the interlaced connection between the self-attention mechanism layer and the camera motion attention mechanism layer provided in an embodiment of the present application. It can be seen that the first layer of the encoder part (that is, the part represented by the trapezoid on the left) in the pre-trained image generation network is the self-attention mechanism layer, the second layer is the camera motion attention mechanism layer, the third layer is the self-attention mechanism layer, the fourth layer is the camera motion attention mechanism layer, and so on. This connection method is called an interlaced connection.

[0069] In some embodiments, see Figure 3 , Figure 3 It is a schematic diagram of a process of obtaining a reference image sequence through multiple self-attention mechanism layers and multiple camera motion attention mechanism layers provided in an embodiment of the present application; Figure 3 It is shown that the first feature processing is performed on the image coding features and the trajectory coding features through multiple self-attention mechanism layers and multiple camera motion attention mechanism layers to generate a reference image sequence, which can be achieved by the following steps S201 to S203:

[0070] Step S201, performing a first attention process on the image encoding feature through the first self-attention mechanism layer to obtain a first image feature.

[0071] The first attention processing refers to the feature optimization operation on the image coding features through the first layer self-attention mechanism; the first image feature refers to the optimized feature generated after the first attention processing, which contains the global context information of the image coding features.

[0072] In some embodiments, step S201 can be implemented by the following method: first, a linear transformation is performed on the input image coding features to obtain a query vector, a key vector and a value vector, wherein the query vector represents the target feature that needs to be paid attention to, the key vector represents the reference point between the features, and the value vector contains complete information of the image coding features; then, an attention score matrix is ​​generated by calculating the similarity between the query vector and the key vector, and the score matrix expresses the degree of attention of each query point to other feature points; then, the scores in the attention score matrix are normalized to generate attention weights, and the attention weights are used to measure the importance of each feature; finally, the value vectors are weighted and summed according to the attention weights, and related features are aggregated to obtain the first image feature.

[0073] Step S202: Perform a second attention process on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a second image feature.

[0074] The second image feature refers to the feature generated after the second attention processing, and the second image feature contains the visual information of the first image feature and the motion information of the trajectory encoding feature.

[0075] In some embodiments, step S202 can be implemented by the following method: first, through the first camera motion attention mechanism layer, the first image feature and the trajectory encoding feature are feature fused to obtain a first fused feature; then, the first fused feature is subjected to a second attention process to obtain a second image feature.

[0076] Feature fusion refers to splicing the first image feature and the trajectory coding feature in a specific dimension, such as splicing in the channel dimension, the shape of the first image feature is (H, W, C), where H is the height of the first image feature, W is the width of the first image feature, and C is the number of channels of the first image feature; the shape of the trajectory coding feature is (H, W, D), where H is the height of the trajectory coding feature, W is the width of the trajectory coding feature, and D is the number of channels of the trajectory coding feature; the first image feature and the trajectory coding feature are spliced ​​in the channel dimension, and the shape of the first fused feature obtained is (H, W, C+D). The processing process of the second attention processing is the same as the processing process of the first attention processing, and this application will not repeat it here.

[0077] Step S203, performing the first feature processing on the second image feature in sequence through the self-attention mechanism layer and the camera motion attention mechanism layer except the first self-attention mechanism layer and the first camera motion attention mechanism layer to obtain a reference image sequence.

[0078] In some embodiments, the number of camera motion attention mechanism layers and self-attention mechanism layers is K; step S203 can be implemented by the following method: first, the second image feature is processed by the first feature through the interlaced K-1 self-attention mechanism layers and K-1 camera motion attention mechanism layers to obtain the third image feature output by each camera motion attention mechanism layer; then, the second image feature output by the first camera motion attention mechanism layer and the third image feature output by each camera motion attention mechanism layer in the K-1 camera motion attention mechanism layers are feature spliced ​​to obtain a high-dimensional spliced ​​feature; then, the high-dimensional spliced ​​feature is compressed through a fully connected layer or an autoencoder to obtain a compressed feature, and the compressed feature is converted into an embedded feature through a pre-trained time series model (such as a long short-term memory artificial neural network (LSTM, Long Short-Term Memory)); finally, the embedded feature can be restored to an image format through deconvolution to obtain a reference image sequence.

[0079] As an example of step S203, assume that the number of camera motion attention mechanism layers and self-attention mechanism layers are both 5; the second image feature is input into the second self-attention mechanism layer for feature extraction to obtain output 1, and output 1 is used as the input of the second camera motion attention mechanism layer. The second camera motion attention mechanism layer performs feature extraction on output 1 to obtain a third image feature 1; next, the third image feature 1 is input into the third self-attention mechanism layer for feature extraction to obtain output 2, and output 2 is used as the input of the third camera motion attention mechanism layer. The attention mechanism layer extracts features from output 2 to obtain the third image feature 2, and so on to obtain the third image feature 3 and the third image feature 4 respectively; finally, the second image feature and the third image feature 1 to the third image feature 4 are feature spliced ​​to obtain a reference image sequence; for example: the second image feature is [1, 2, 3, 4], the third image feature 1 is [2, 1, 3, 5], the third image feature 2 is [1, 0, 2, 3], the third image feature 3 is [1, 4, 2, 3], and the third image feature 4 is [2, 2, 3, 0]. The high-dimensional splicing features obtained after image splicing are:

[0080]

[0081] Next, the high-dimensional splicing features are compressed through the fully connected layer to obtain compressed features, and the compressed features are converted into embedded features through the pre-trained LSTM model; finally, the embedded features are restored to image format through deconvolution to obtain the reference image sequence.

[0082] Step S104, generating a target video according to the target image and the reference image sequence; the first frame of the target video is compared with the target image. Figure 1 The target video’s camera movement is consistent with the camera’s trajectory information.

[0083] The target video refers to the video captured by the shooting device according to the camera movement method corresponding to the camera trajectory information, and contains the movement process of the target object. The first frame of the target video is the same as the target object. Figure 1 Assuming that the target object is an elephant, in the generated target video, the camera shoots the elephant according to the camera trajectory, and the elephant moves while the camera moves.

[0084] In some embodiments, see Figure 4 , Figure 4 It is a schematic diagram of a process of generating a target video according to a target image and a reference image sequence provided by an embodiment of the present application; Figure 4 It is shown that step S104 can be implemented by the following steps S1041 to S1044:

[0085] Step S1041, identifying the moving object in the target image, and obtaining the motion state characteristics of the moving object.

[0086] A moving object refers to an object that can change position in the target graph, such as people, animals, and movable objects (such as cars) in the target graph; a moving object has dynamics relative to time.

[0087] The motion state characteristics are used to describe the changing characteristics of the moving object in time and space, where the motion state may include: speed, motion direction, acceleration, motion trajectory, etc.

[0088] It should be noted that the target image may include not only moving objects but also non-moving objects, that is, stationary objects such as tables, water cups, etc.

[0089] In some embodiments, step S1041 may be implemented by the following method: first, the target image is identified by the target detection model to obtain the bounding box and category of the moving object in the target image; then, the gradient direction consistency is calculated in the area where the moving object is located (which may be within the range of the bounding box), and the motion state characteristics such as the moving direction and speed of the moving object are inferred. Among them, the gradient direction consistency is an indicator to measure the overall consistency of the pixel gradient direction in a certain area in the image, and the gradient direction refers to the direction of the change in the brightness of the image pixels.

[0090] Step S1042, obtaining random Gaussian noise.

[0091] Random Gaussian noise is a kind of random noise that conforms to the Gaussian distribution. Adding random Gaussian noise to the image can simulate the interference under natural shooting conditions and is often used for data enhancement. Random Gaussian noise is usually determined by two parameters: mean (indicating the central value of the noise intensity) and variance (controlling the degree of noise diffusion).

[0092] Step S1043, concatenate the random Gaussian noise and the motion state feature in the channel dimension to obtain a first concatenated feature.

[0093] The channel dimension is often used to store color information. For example, an RGB image contains three channels (red, blue, and green). An image tensor is a multidimensional array that represents image data. The dimensions of the tensor represent the structural information of the image, such as width, height, and color channels.

[0094] In some embodiments, step S1042 can be implemented by the following method: first, since the acquired random Gaussian noise does not have a channel dimension, it is necessary to expand the random Gaussian noise into a single channel (for example, a shape of H×X×1) so that the random Gaussian noise can be aligned with the motion state feature in the channel dimension; then, the expanded random Gaussian noise is spliced ​​with the motion state feature in the channel dimension, such as: the motion state feature contains two channels and has a shape of H×X×2. After splicing the expanded random Gaussian noise with the motion state feature in the channel dimension, a first spliced ​​feature is obtained, and the shape of the first spliced ​​feature is H×X×3.

[0095] Step S1044: Generate a target video according to the first splicing feature and the reference image sequence.

[0096] In some embodiments, step S1044 can be implemented by the following method: first, the first splicing feature and the reference image sequence are input into a video generation network, and multiple diffusion processes are performed based on the first splicing feature and the reference image sequence through the video generation network; wherein the output of the last diffusion process is the first latent space expression of the target video; then, the first latent space expression is decoded to obtain the target video.

[0097] The video generation network is a network model trained on a large dataset and has the ability to perform feature fusion and video generation. The video generation network can be implemented as a diffusion model.

[0098] Decoding refers to the process of restoring the first latent space expression to a visible form of image frame through a decoder. The decoder can be implemented as a convolutional neural network, an adversarial network, etc.

[0099] Diffusion processing refers to the operation of generating or processing data by simulating the gradual addition or removal of noise. It should be noted that the diffusion processing in step S1044 refers to the noise removal process.

[0100] The first latent space expression refers to the feature vector obtained after multiple diffusion processes are performed on the first spliced ​​feature and the reference image sequence.

[0101] As an example of step S1044, the video generation network is implemented as a diffusion model, and the first splicing feature and the reference image sequence are diffused for the first time through the diffusion model to obtain the first diffusion feature; then, the first diffusion feature, the first splicing feature and the reference image sequence are input into the diffusion model again to obtain the second diffusion feature; and so on, the N-1th diffusion feature, the first splicing feature and the reference image sequence are input into the diffusion model for the Nth time to obtain the first latent space expression, where the specific value of N can be set in advance; then, the first latent space expression is decoded by the decoder to obtain the target video.

[0102] In some embodiments, the diffusion model includes multiple data processing modules, wherein each data processing module includes a self-attention mechanism layer, a reference attention mechanism layer, a target object attention mechanism layer and a temporal attention mechanism layer connected in sequence; at this time, the first splicing feature and the reference image sequence are subjected to a first diffusion processing through the diffusion model to obtain a first diffusion feature, which can be achieved by the following method: first, the first splicing feature is subjected to a third attention processing through the self-attention mechanism layer in the first data processing module to obtain a fourth image feature; then, the fourth image feature and the reference image sequence are subjected to a fourth attention processing through the reference attention mechanism layer in the first data processing module to obtain a fifth image feature; then, the fifth image feature is subjected to a fifth attention processing through the target object attention mechanism layer in the first data processing module to obtain a sixth image feature; finally, the sixth image feature is subjected to a sixth attention processing through the temporal attention mechanism layer in the first data processing module to obtain the first module processed data, and the first module processed data is sequentially input into the remaining data processing modules except the first data processing module for attention processing to obtain the first diffusion feature.

[0103] After obtaining the first diffusion feature, the first diffusion feature, the first splicing feature and the reference image sequence are input into the diffusion model, and the first diffusion feature, the first splicing feature and the reference image sequence are subjected to a second diffusion process through multiple data processing modules in the diffusion model. Specifically: the first diffusion feature and the first splicing feature are input into the self-attention mechanism layer in the first data processing module, and the seventh attention process is performed to obtain the seventh image feature; then, the seventh image feature and the reference image sequence are input into the reference attention mechanism layer in the first data processing module, and the eighth attention process is performed to obtain the eighth image feature; then, the eighth image feature is subjected to the ninth attention process through the target object attention mechanism layer in the first data processing module to obtain the ninth image feature; finally, the ninth image feature is subjected to the tenth attention process through the temporal attention mechanism layer in the first data processing module to obtain the second module processing data, and the second module processing data is sequentially input into the remaining data processing modules except the first data processing module for attention processing to obtain the second diffusion feature. By analogy, the N-1th diffusion feature, the first splicing feature and the reference image sequence are input into the diffusion model for the Nth time to obtain the first latent space expression.

[0104] It should be noted that the input of the first diffusion process includes the first splicing feature and the reference image sequence, and the input of each subsequent diffusion process includes the first splicing feature, the reference image sequence and the diffusion feature obtained by the previous diffusion process.

[0105] Here, the reference attention mechanism layer is a cross-modal or cross-feature attention mechanism, which is used to combine features from two different sources, calculate the correlation between the features and generate fused features.

[0106] The target object attention mechanism layer is an attention mechanism for feature targets, which is used to highlight and strengthen the features of specific regions or objects.

[0107] The temporal attention mechanism layer is used to analyze the correlation of features in the temporal dimension, ensure the dynamic consistency between the generated image frames, and avoid frame-to-frame jumps. The specific steps of attention processing through the attention mechanism have been described in detail in step S201, and this application will not repeat them here.

[0108] It should be noted that the embodiment of the present application also provides a method for training a video generation model, wherein the video generation model includes a coding network, an image generation network and a video generation network. Figure 5 , Figure 5 3 is a flow chart of a method for training a video generation model provided in an embodiment of the present application. The training of the video generation model can be implemented by following steps S301 to S304:

[0109] Step S301: input the obtained sample label video, sample graph and sample camera trajectory information into the encoding network to determine the second latent space expression of the sample label video, the sample image encoding features of the sample graph and the sample trajectory encoding features of the sample camera trajectory information.

[0110] The sample image is the image data containing the target object in the training data set. The target object may be an object or a person in a specific scene. The sample camera trajectory information is the data containing the camera motion trajectory in the training data set. The sample camera trajectory information includes at least one of the sample camera trajectory image and the sample camera trajectory description text. The sample label video refers to the video shot according to the camera movement method described in the sample trajectory information for the sample in the sample image. The first frame of the sample label video is the same as the sample image. Figure 1 To.

[0111] The coding network refers to a neural network structure that extracts features from input data. The coding network can be implemented as a convolutional neural network, a recursive neural network, etc., which is not limited in this application. The coding network in the embodiment of the present application includes three sub-coding networks, namely a sample image encoder, a sample trajectory information encoder, and a sample label video encoder, wherein the sample image encoder is used to encode the sample image, the sample trajectory information encoder is used to encode the sample trajectory information, and the sample label video encoder is used to encode the sample label video.

[0112] Here, encoding the sample label video by the sample label video encoder can be achieved by the following method: first, preprocessing the sample label video, including frame extraction, video frame size adjustment, normalization, etc., and the specific preprocessing method is not limited in this application; then, the preprocessed sample label video is subjected to feature extraction by the sample label video encoder to obtain a second latent space expression, wherein the sample video label encoder can be implemented as a convolutional neural network, a long short-term memory network, etc. The method for determining the sample image coding features and the sample camera trajectory coding features in step S301 refers to step S102, and this application will not repeat them here.

[0113] Step S302: input the sample image coding features and the sample trajectory coding features into the image generation network to generate a sample reference image sequence.

[0114] Here, the method and process of generating the sample image reference sequence refer to step S103, and this application will not repeat them here.

[0115] Step S303, determining the loss result of the video generation network according to the sample image, the sample reference image sequence and the second latent space expression.

[0116] In some embodiments, the video generation model also includes a motion state prediction network; in step S303, the loss result of the video generation network is determined according to the sample image, the sample reference image sequence and the second latent space expression, which can be achieved by the following method: first, the sample motion object in the sample image is identified by the motion state prediction network, and the sample motion state feature of the sample motion object is obtained; then, the sample motion state feature and the second latent space expression are spliced ​​to obtain a second splicing feature; finally, the second splicing feature and the sample reference image sequence are input into the video generation network to obtain the loss result.

[0117] Here, the motion state prediction network refers to a network model that identifies moving objects in an image and extracts the motion features of the moving objects. The motion state prediction network can be implemented as a target detection network, a timing model (such as a long short-term memory network, a converter network, etc.), and this application does not limit this.

[0118] The sample motion object refers to an object that can change position in the sample graph. The sample motion state feature is used to describe the temporal and spatial variation characteristics of the sample motion object. The implementation process of identifying the sample motion object in the sample graph through the motion state prediction network and obtaining the sample motion state feature of the sample motion object is shown in step S1041, and this application will not be repeated here.

[0119] In some embodiments, the video generation network can be implemented as a diffusion model. The structure of the diffusion model is shown in step S1044, and this application will not repeat it here. The second splicing feature and the sample reference image sequence are input into the video generation network to obtain the loss result, which can be achieved by the following method: first, the second splicing feature and the sample reference image sequence are input into the diffusion model; then, the second splicing feature and the sample reference image sequence are diffused through the multi-layer attention mechanism layer in the diffusion model to obtain the diffusion processing feature. It should be noted that the diffusion processing here is a noise addition process; finally, the error between the diffusion processing feature and the Gaussian noise is calculated according to the diffusion loss function, and the calculated error is determined as the loss result.

[0120] Here, the diffusion loss function refers to the function used to measure the difference between the model-generated data and the real data during the diffusion model training process, that is, the difference between the noise-added result of the model output and the target noise; the diffusion loss function can be reconstruction error loss, log-likelihood loss, perceptual loss, etc. The specific diffusion loss function is not limited in this application.

[0121] Step S304: According to the loss result, the network parameters of the image generation network are adjusted to obtain a trained video generation model.

[0122] Here, parameter adjustment refers to updating the weights of the image generation network through an optimization algorithm (such as gradient descent method); through multiple iterative training, the parameters of the image generation network are continuously optimized until the loss result stabilizes within a preset range, and the training is terminated to obtain a trained video generation model.

[0123] In some embodiments, when the video generation model also includes a motion state prediction network, step S304 can be implemented by the following method: according to the loss result, the network parameters of the image generation network and the network parameters of the motion state prediction network are adjusted to obtain a trained video generation model.

[0124] When the video generation model includes a motion state prediction network, after the loss result is calculated, the network parameters of the image generation network and the network parameters of the motion state prediction network can be optimized simultaneously through an optimization algorithm based on the loss result until the loss result stabilizes within a preset range. The training is terminated to obtain the trained video generation network.

[0125] It should be noted that the training method of the video generation model provided in the embodiment of the present application, by training the image generation network and the motion state prediction network without adjusting the network parameters of the video generation network, can avoid affecting the parameters of the video generation network when training the image generation network and the motion state prediction network, thereby affecting the video generation effect of the video generation network; moreover, only adjusting the network parameters of the image generation network and the motion state prediction network can reduce the computing overhead of the server, shorten the training time, and improve the model training efficiency.

[0126] The video generation method provided in the embodiment of the present application can be widely used in a variety of scenarios. For example, in virtual reality and augmented reality scenarios, a virtual scene can be generated by the video generation method provided in the embodiment of the present application to simulate the movement of objects in the real world and the changes in the shooting angle, so that users can see scenes that highly restore the real world in a virtual environment, thereby improving the user's immersive experience. In the film production, animation and game development scenarios, the video generation method provided in the embodiment of the present application can automatically generate a video containing the movement process of the target object, and through the precise simulation of the camera motion trajectory and the motion state of the target object, it supports more flexible lens shooting and motion capture. The movement of the target object in the target video can be accurately mapped to the real world, providing a high-quality virtual shooting experience.

[0127] The video generation method provided in the embodiment of the present application ensures that the content of the generated target video is highly consistent with the preset camera motion trajectory by extracting and processing the trajectory coding features of the camera trajectory information, thereby improving the control accuracy of the camera; and obtains a reference image sequence by performing image processing on the image coding features and the trajectory coding features, and generates a final target video by combining the reference image sequence and the target image, so that the generated target video can well describe the detailed scene, thereby improving the authenticity of the target video; and, by extracting the camera trajectory information through the image generation network, there is no need to process the camera trajectory information through the video generation network, which will not affect the video generation capability of the video generation network, thereby improving the video generation efficiency.

[0128] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0129] The video generation method provided in the embodiment of the present application mainly solves the following three technical problems:

[0130] Technical problem 1: Generating videos through text descriptions cannot describe scene details well.

[0131] Technical problem 2: Camera control is not precise enough.

[0132] Technical problem 3: The camera moves in a large range, and the generated video content is incoherent.

[0133] The reason for the technical problem 3 is that there are problems in the modeling of the video generation method in the related technology. The modeling in the related technology is: user input encoding + video generation model, that is, after encoding the text or camera motion parameters input by the user, input it into the video generation model to generate the corresponding video. This modeling method requires fine-tuning the video generation model (that is, the video generation model is mostly based on the diffusion model as the base model. After the video generation model is pre-trained with a large number of pre-training data sets, the video generation model has a certain video generation ability, but the video generation model at this time generally does not have the ability to generate videos with specified motion trajectories; therefore, it is necessary to prepare a fine-tuning data set and perform fine-tuning training on the pre-trained video generation model to generate specified motion trajectories). The fine-tuning training not only changes the base model, but also fine-tunes the entire video generation model, resulting in low quality of the generated video. In particular, when the camera movement range is relatively large, the generated video has the problem of incoherent content.

[0134] See also Figure 6 , Figure 6301 (the above-mentioned camera trajectory information) is input into the posture encoder 302 (the above-mentioned traditional feature extraction algorithm or deep learning algorithm) to obtain trajectory coding features; the object graph 303 (the above-mentioned target graph) is input into the image encoder 304 (the above-mentioned image encoder) to obtain image coding features; then, the image coding features are input into the first self-attention mechanism layer in the image generation network 305 for feature extraction to obtain the first image features, the first image features and the trajectory coding features are input into the first camera motion attention mechanism layer in the image generation network 305 to obtain the second image features, and the second image features are sequentially input into the subsequent multiple self-attention mechanisms The first image processing method comprises the following steps: first, inputting the object graph 303 into the motion state prediction network 307 to obtain the motion state feature of the moving object, and splicing the motion state feature with the Gaussian noise 308 to obtain the splicing feature (the above-mentioned first splicing feature); then, inputting the splicing feature and the reference image sequence 306 into the video generation network 309, and performing multiple diffusion processes on the splicing feature and the reference image sequence 306 through the video generation network 309 to obtain the latent space expression (the above-mentioned first latent space expression); finally, the latent space expression is decoded by the decoder 310 to obtain the target video.

[0135] Regarding technical issue 1: The video generation method provided in the embodiment of the present application adopts an image-guided video generation method, and text is only used as a supplement; the user input includes two pictures, one is a camera trajectory picture, and the other is an object picture.

[0136] Regarding technical problem 2: jointly model the camera motion and the target object motion to avoid the ambiguity caused by modeling the camera motion and the target object motion separately (such as the target object moves forward and the camera moves backward).

[0137] Regarding technical issue 3: Use the image generation network to guide the video generation network, that is, use the output of the image generation network as the input of the video generation network. The image generation network and the video generation network are separate, which will not affect the general capabilities of the video generation network and preserve the video generation capabilities of the video generation network to the greatest extent.

[0138] The video generation method provided in the embodiment of the present application can greatly improve the video shooting in the film and television industry, without the need for the video shooter to provide a detailed video shooting process; and can improve sparse 3D reconstruction. The user only needs to shoot a scene image, and then the entire scene can be generated according to the video generation method provided in the embodiment of the present application, and then reconstructed by 3D reconstruction; if further 3D modeling of the object is required, only a reference image and a camera trajectory image representing a 360-degree surround can be input to obtain a video shot by the camera around the target object. Figure 7 , Figure 7 It is a schematic diagram of three-dimensional modeling provided in an embodiment of the present application.

[0139] Based on the video generation method described in the above embodiment, Figure 8 A structural block diagram of a video generating device 100 provided in an embodiment of the present application is shown. The video generating device may be a device in an electronic device (e.g., a server). The video generating device may be implemented in software, which may be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 101, a determination module 102, a first generation module 103, and a second generation module 104. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented.

[0140] Among them, the acquisition module 101 is used to acquire the target image and the camera trajectory information; the determination module 102 is used to determine the image coding features of the target image and the trajectory coding features of the camera trajectory information; the first generation module 103 is used to generate a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence includes a plurality of scene graphs obtained by predicting the movement of the target image based on the camera trajectory information; the second generation module 104 is used to generate a target video according to the target image and the reference image sequence; the first frame of the target video is the same as the first frame of the target video; Figure 1 The camera movement mode of the target video is consistent with the camera trajectory information.

[0141] In some embodiments, the camera trajectory information includes at least one of a camera trajectory map and a camera trajectory description text; the determination module is further used to: when the camera trajectory information includes the camera trajectory map, perform image encoding on the camera trajectory map to obtain the trajectory encoding feature; when the camera trajectory information includes the camera trajectory description text, generate a camera trajectory map based on the camera trajectory description text, perform image encoding on the camera trajectory map, and obtain the trajectory encoding feature; when the camera trajectory information includes the camera trajectory map and the camera trajectory description text, perform trajectory update on the camera trajectory in the camera trajectory map based on the camera trajectory description text to obtain an updated camera trajectory map; and perform image encoding on the updated camera trajectory map to obtain the trajectory encoding feature.

[0142] In some embodiments, the first generation module is further used to: input the image coding features and the trajectory coding features into an image generation network; the image generation network includes multiple self-attention mechanism layers and multiple camera motion attention mechanism layers, wherein the self-attention mechanism layers are interlaced with the camera motion attention mechanism layers; the image coding features are input into the first self-attention mechanism layer of the image generation network, and the trajectory coding features are input into the first camera motion attention mechanism layer of the image generation network; through the multiple self-attention mechanism layers and the multiple camera motion attention mechanism layers, the image coding features and the trajectory coding features are subjected to first feature processing to generate the reference image sequence.

[0143] In some embodiments, the first generation module is further used to: perform a first attention processing on the image coding feature through the first self-attention mechanism layer to obtain a first image feature; perform a second attention processing on the first image feature and the trajectory coding feature through the first camera motion attention mechanism layer to obtain a second image feature; and perform a first feature processing on the second image feature in sequence through a self-attention mechanism layer and a camera motion attention mechanism layer other than the first self-attention mechanism layer and the first camera motion attention mechanism layer to obtain the reference image sequence.

[0144] In some embodiments, the first generation module is further used to: perform feature fusion on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a first fused feature; and perform the second attention processing on the first fused feature to obtain the second image feature.

[0145] In some embodiments, the second generation module is also used to: identify the moving object in the target image and obtain the motion state characteristics of the moving object; obtain random Gaussian noise; splice the random Gaussian noise and the motion state characteristics in the channel dimension to obtain a first splicing feature; generate the target video according to the first splicing feature and the reference image sequence.

[0146] In some embodiments, the second generation module is further used to: input the first splicing feature and the reference image sequence into a video generation network, and perform multiple diffusion processes based on the first splicing feature and the reference image sequence through the video generation network; wherein the output of the last diffusion process is the first latent space expression of the target video; and decode the first latent space expression to obtain the target video.

[0147] An embodiment of the present application provides a training method for a video generation model, wherein the video generation model includes a coding network, an image generation network, and a video generation network. The training method includes: inputting the acquired sample label video, sample image, and sample camera trajectory information into the coding network to determine a second latent space expression of the sample label video, a sample image coding feature of the sample image, and a sample trajectory coding feature of the sample camera trajectory information; inputting the sample image coding feature and the sample trajectory coding feature into the image generation network to generate a sample reference image sequence; determining a loss result of the video generation network according to the sample image, the sample reference image sequence, and the second latent space expression; and adjusting the network parameters of the image generation network according to the loss result to obtain a trained video generation model.

[0148] In the above scheme, the video generation model also includes a motion state prediction network; the device also includes a model training module, which is used to: identify the sample motion object in the sample image through the motion state prediction network, and obtain the sample motion state feature of the sample motion object; splice the sample motion state feature and the second latent space expression to obtain a second splicing feature; input the second splicing feature and the sample reference image sequence into the video generation network to obtain the loss result.

[0149] In the above scheme, the training module is also used to: adjust the network parameters of the image generation network and the network parameters of the motion state prediction network according to the loss result to obtain a trained video generation model.

[0150] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0151] An embodiment of the present application provides an electronic device, Fig. 9 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig. 9 As shown, the electronic device 130 includes: at least one processor 131 ( Fig. 9 Only one is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the executable instructions 133, the steps of any of the above-mentioned video generation method embodiments are implemented.

[0152] The electronic device may include but is not limited to a processor 131 and a memory 132. Those skilled in the art may understand that: Fig. 9 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0153] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0154] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard disk or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the electronic device 130. Further, the memory 132 may also include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 132 may also be used to temporarily store data that has been output or is to be output.

[0155] The present application embodiment provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the video generation method provided by the present application embodiment, for example, Figure 1 The video generation method is shown.

[0156] The embodiment of the present application provides a computer program product, which includes computer executable instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the video generation method described in the embodiment of the present application.

[0157] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0158] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0159] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0160] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0161] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Get target image and camera trajectory information; Determining image coding features of the target image and trajectory coding features of the camera trajectory information; Generate a reference image sequence according to the image coding features and the trajectory coding features; wherein the reference image sequence includes a plurality of scene images obtained by performing camera movement prediction on the target image based on the camera trajectory information; A target video is generated according to the target image and the reference image sequence; the first frame of the target video is consistent with the target image, and the camera movement mode of the target video is consistent with the camera trajectory information.

2. The method according to claim 1, characterized in that The camera trajectory information includes at least one of a camera trajectory graph and a camera trajectory description text; Determining the trajectory coding features of the camera trajectory information includes: In a case where the camera trajectory information includes the camera trajectory graph, performing image encoding on the camera trajectory graph to obtain the trajectory encoding feature; In a case where the camera trajectory information includes the camera trajectory description text, generating a camera trajectory graph based on the camera trajectory description text, performing image encoding on the camera trajectory graph, and obtaining the trajectory encoding feature; In a case where the camera trajectory information includes the camera trajectory graph and the camera trajectory description text, the camera trajectory in the camera trajectory graph is updated based on the camera trajectory description text to obtain an updated camera trajectory graph; and image encoding is performed on the updated camera trajectory graph to obtain the trajectory encoding feature.

3. The method according to claim 1, characterized in that The step of generating a reference image sequence according to the image coding feature and the trajectory coding feature comprises: The image encoding feature and the trajectory encoding feature are input into an image generation network; the image generation network comprises a plurality of self-attention mechanism layers and a plurality of camera motion attention mechanism layers, wherein the self-attention mechanism layers are interlaced with the camera motion attention mechanism layers; the image encoding feature is input into a first self-attention mechanism layer of the image generation network, and the trajectory encoding feature is input into a first camera motion attention mechanism layer of the image generation network; The first feature processing is performed on the image encoding features and the trajectory encoding features through the multiple self-attention mechanism layers and the multiple camera motion attention mechanism layers to generate the reference image sequence.

4. The method according to claim 3, characterized in that The step of performing first feature processing on the image coding features and the trajectory coding features through the multiple self-attention mechanism layers and the multiple camera motion attention mechanism layers to generate the reference image sequence includes: Performing a first attention process on the image encoding feature through the first self-attention mechanism layer to obtain a first image feature; Performing a second attention process on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a second image feature; The reference image sequence is obtained by sequentially performing first feature processing on the second image features through self-attention mechanism layers and camera motion attention mechanism layers other than the first self-attention mechanism layer and the first camera motion attention mechanism layer.

5. The method according to claim 4, characterized in that The step of performing a second attention process on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a second image feature includes: Performing feature fusion on the first image feature and the trajectory encoding feature through the first camera motion attention mechanism layer to obtain a first fused feature; The second attention processing is performed on the first fusion feature to obtain the second image feature.

6. The method according to claim 1, characterized in that The step of generating a target video according to the target image and the reference image sequence comprises: Identify the moving object in the target image and obtain the motion state characteristics of the moving object; Get random Gaussian noise; Splicing the random Gaussian noise and the motion state feature in a channel dimension to obtain a first splicing feature; The target video is generated according to the first splicing feature and the reference image sequence.

7. The method according to claim 6, characterized in that The step of generating the target video according to the first splicing feature and the reference image sequence includes: Inputting the first splicing feature and the reference image sequence into a video generation network, and performing multiple diffusion processes based on the first splicing feature and the reference image sequence through the video generation network; wherein the output of the last diffusion process is a first latent space expression of the target video; The first latent space expression is decoded to obtain the target video.

8. A method for training a video generation model, wherein the video generation model is used to implement the video generation method according to any one of claims 1 to 7, wherein the video generation model comprises a coding network, an image generation network and a video generation network, wherein: The training method comprises: Inputting the acquired sample label video, sample graph and sample camera trajectory information into the encoding network, determining the second latent space expression of the sample label video, the sample image encoding feature of the sample graph and the sample trajectory encoding feature of the sample camera trajectory information; Inputting the sample image encoding features and the sample trajectory encoding features into the image generation network to generate a sample reference image sequence; Determining a loss result of the video generation network according to the sample graph, the sample reference image sequence and the second latent space expression; According to the loss result, the network parameters of the image generation network are adjusted to obtain a trained video generation model.

9. The method according to claim 8, characterized in that The video generation model also includes a motion state prediction network; The step of determining the loss result of the video generation network according to the sample graph, the sample reference image sequence and the second latent space expression includes: Identifying a sample motion object in the sample image through the motion state prediction network, and obtaining a sample motion state feature of the sample motion object; Splicing the sample motion state feature and the second latent space expression to obtain a second splicing feature; The second splicing feature and the sample reference image sequence are input into the video generation network to obtain the loss result.

10. The method according to claim 9, characterized in that The adjusting the network parameters of the image generation network according to the loss result to obtain the trained video generation model comprises: According to the loss result, the network parameters of the image generation network and the network parameters of the motion state prediction network are adjusted to obtain a trained video generation model.

11. A video generating device, characterized in that: include: The acquisition module is used to obtain the target image and camera trajectory information; A determination module, used to determine the image coding features of the target image and the trajectory coding features of the camera trajectory information; A first generating module generates a reference image sequence according to the image coding feature and the trajectory coding feature; wherein the reference image sequence includes a plurality of scene images obtained by performing camera movement prediction on the target image based on the camera trajectory information; The second generating module is used to generate a target video according to the target image and the reference image sequence; the first frame of the target video is consistent with the target image, and the camera movement mode of the target video is consistent with the camera trajectory information.

12. An electronic device, characterized in that: include: A memory for storing computer executable instructions; The processor is used to implement the video generation method described in any one of claims 1 to 7 when executing the computer executable instructions stored in the memory, or to implement the video generation model training method described in any one of claims 8 to 10.

13. A computer-readable storage medium, characterized in that: Computer executable instructions or computer programs are stored, which are used to cause a processor to execute the computer executable instructions or the computer program to implement the video generation method described in any one of claims 1 to 7, or to implement the video generation model training method described in any one of claims 8 to 10.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the video generation method described in any one of claims 1 to 7 is implemented, or the training method of the video generation model described in any one of claims 8 to 10 is implemented.

Images