Video generation method and apparatus, and electronic device, computer-readable storage medium and computer program product

Abstract

A video generation method and apparatus, and an electronic device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a target trajectory map, and mapping the target trajectory map on the basis of a target rank decomposition matrix configured for a motion control network, so as to obtain a target motion feature, wherein the target rank decomposition matrix is used for controlling the target motion feature to constrain a motion trajectory of a video lens and / or a motion trajectory of a video display object during a video generation process; acquiring target reference content required during the video generation process, and extracting a first content feature of the target reference content; and acquiring a target noise image stack, and by means of a diffusion model, using the target motion feature and the first content feature as constraint conditions to perform denoising processing on the target noise image stack, so as to obtain a target video in which the motion trajectory of the video lens and / or the motion trajectory of the video display object is constrained.

Description

Video generation method and device, electronic device, computer-readable storage medium, and computer program product

[0001] Cross-reference to Related Applications

[0002] Embodiments of the present application are based on and claim priority from Chinese Patent Application No. 202410783125.X filed on June 17, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computer technology, and in particular, to a video generation method and device, electronic device, computer-readable storage medium, and computer program product. BACKGROUND

[0004] Film production or animation production is an important form of visual art. In the creation process of video media, professional directors often need advanced photography techniques to carefully plan the movement of video shots and video display objects to ensure the coherence of the story and achieve exquisite visual effects.

[0005] With the development of generative artificial intelligence technology, videos can be generated based on artificial intelligence technology. However, in related technologies, the movement of video shots and video display objects is random when generating videos based on artificial intelligence technology, which reduces the quality of the generated videos. SUMMARY

[0006] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of protection of the claims.

[0007] Embodiments of the present disclosure provide a video generation method, device, electronic device, computer-readable storage medium, and computer program product, which can finely control the movement of video shots and / or video display objects, thereby improving the quality of the generated videos.

[0008] Embodiments of the present disclosure provide a video generation method applied to an electronic device, comprising:

[0009] Obtaining a target trajectory graph, mapping the target trajectory graph based on a target rank decomposition matrix configured by a motion control network to obtain a target motion feature, wherein the target rank decomposition matrix is used to control the target motion feature to constrain the movement trajectory of the video shots and / or the movement trajectory of the video display objects in a video generation process;

[0010] Obtaining a target reference content required in the video generation process, and extracting a first content feature of the target reference content;

[0011] obtain a target video by denoising a target noise image stack through a diffusion model with the target motion feature and the first content feature as constraint conditions.

[0012] The embodiments of the present disclosure further provide a video generation apparatus, comprising:

[0013] a motion control module configured to obtain a target trajectory map, map the target trajectory map based on a target rank decomposition matrix of the motion control network to obtain a target motion feature, wherein the target rank decomposition matrix is used to control the target motion feature to constrain a motion trajectory of a video shot and / or a motion trajectory of a video display object in a video generation process;

[0014] a content control module configured to obtain a target reference content required in the video generation process, and extract a first content feature of the target reference content;

[0015] a generation module configured to obtain a target noise image stack, and denoise the target noise image stack through a diffusion model with the target motion feature and the first content feature as constraint conditions to obtain a target video constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object.

[0016] The embodiments of the present disclosure further provide an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the video generation method described above when executing the computer program.

[0017] The embodiments of the present disclosure further provide a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the video generation method described above.

[0018] The embodiments of the present disclosure further provide a computer program product, which comprises a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to enable the computer device to implement the video generation method described above.

[0019] The embodiments of the present disclosure at least have the following beneficial effects: the target trajectory graph is mapped based on the target rank decomposition matrix configured in the motion control network to obtain target motion features, the first content features of the target reference content are extracted, and the target noise image stack is denoised by taking the target motion features and the first content features as constraint conditions through the diffusion model to obtain the target video which is constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object. Since the target rank decomposition matrix is used to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process, by configuring the target rank decomposition matrix in the motion control network, the motion decoupling of the video shot and the video display object can be realized in the processing process of the diffusion model, the fine control of the motion of the video shot and / or the video display object is achieved, and thus the quality of the generated video is improved.

[0020] Other features and advantages of the present disclosure will be described in the following description, and some will become apparent from the description, or will be learned through implementation of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are included to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used together with the embodiments of the present disclosure to explain the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0022] FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure;

[0023] FIG. 2 is a flowchart of a video generation method provided by an embodiment of the present disclosure;

[0024] FIG. 3 is a structural schematic diagram of a target rank decomposition matrix provided by an embodiment of the present disclosure;

[0025] FIG. 4 is an architectural schematic diagram of a video generation method provided by an embodiment of the present disclosure;

[0026] FIG. 5 is a flowchart of generating a target video provided by an embodiment of the present disclosure;

[0027] FIG. 6 is a generated result schematic diagram of a target video under two motion types provided by an embodiment of the present disclosure;

[0028] FIG. 7 is another generated result schematic diagram of a target video under two motion types provided by an embodiment of the present disclosure;

[0029] FIG. 8 is a generated result schematic diagram of a target video under different second adjustment weights provided by an embodiment of the present disclosure;

[0030] FIG. 9 is a flowchart of adjusting a first rank decomposition matrix provided by an embodiment of the present disclosure;

[0031] FIG. 10 is a flow diagram illustrating a method for adjusting a second rank decomposition matrix according to an embodiment of the present disclosure;

[0032] FIG. 11 is a diagram illustrating a generated result of controlling video shot motion according to an embodiment of the present disclosure;

[0033] FIG. 12 is a diagram illustrating a generated result of controlling video display object motion according to an embodiment of the present disclosure;

[0034] FIG. 13 is a diagram illustrating a structure of a video generation apparatus according to an embodiment of the present disclosure;

[0035] FIG. 14 is a diagram illustrating a partial structure of a terminal according to an embodiment of the present disclosure;

[0036] FIG. 15 is a diagram illustrating a partial structure of a server according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, further detailed descriptions will be given to the present disclosure in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and should not be used to limit the present disclosure.

[0038] It should be noted that in each of the specific embodiments of the present disclosure, when it is necessary to perform relevant processing according to data related to the characteristics of a target object, such as target object attribute information or attribute information set, permission or consent of the target object will be obtained first, and the collection, use and processing of the data will comply with relevant laws, regulations and standards. The target object can be a user. In addition, when the specific embodiments of the present disclosure need to obtain target object attribute information, a separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the target object, the necessary target object related data for enabling the specific embodiments of the present disclosure to operate normally will be obtained.

[0039] In the specific embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0040] In order to facilitate understanding of the technical solutions provided by the specific embodiments of the present disclosure, some key terms used in the specific embodiments of the present disclosure will be explained first.

[0041] Filmmaking and animation are important forms of visual arts. In the creation of video media, professional directors typically require advanced cinematography techniques to meticulously plan the movement of video shots and objects to ensure a coherent storyline and achieve sophisticated visual effects. With the development of generative artificial intelligence (AI) technology, videos can be generated based on AI. However, in AI-based video generation, the movement of video shots and objects is often random, reducing the quality of the generated video.

[0042] Based on this, the present disclosure provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can perform fine control over the movement of video lenses and / or video display objects, thereby improving the quality of the generated video.

[0043] Referring to Figure 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0044] For example, server 102 can obtain the target trajectory map sent by terminal 101, input the target trajectory map into a motion control network configured with an original parameter matrix and a target rank decomposition matrix, and map the target trajectory map based on the target rank decomposition matrix configured in the motion control network to obtain target motion features. The target rank decomposition matrix is ​​used to control the target motion features to constrain the motion trajectory of the video lens and / or the motion trajectory of the video display object during the video generation process. Server 102 can obtain the target reference content sent by terminal 101, extract the first content feature of the target reference content, obtain the target noise image stack, input the target noise image stack, target motion features, and the first content feature into a diffusion model, and perform denoising processing on the target noise image stack using the target motion features and the first content feature as constraints through the diffusion model to obtain a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object. Server 102 sends the target video to terminal 101.

[0045] The server 102 obtains a target trajectory graph, inputs the target trajectory graph into a motion control network configured with an original parameter matrix and a target rank decomposition matrix, maps the target trajectory graph based on the target rank decomposition matrix configured in the motion control network, obtains a target motion feature, obtains a target reference content required in a video generation process, extracts a first content feature of the target reference content, obtains a target noise image stack, inputs the target noise image stack, the target motion feature and the first content feature into a diffusion model, and performs denoising processing on the target noise image stack by the diffusion model with the target motion feature and the first content feature as constraint conditions to obtain a target video constrained by a motion trajectory of a video shot and / or a motion trajectory of a video display object. Since the target rank decomposition matrix is used to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process, by configuring the target rank decomposition matrix in the motion control network, the motion decoupling of the video shot and the video display object can be realized in the processing process of the diffusion model, the fine control of the motion of the video shot and / or the video display object is achieved, and thus the quality of the generated video is improved.

[0046] The server 102 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Networks), and basic cloud computing services such as big data and artificial intelligence platforms. In addition, the server 102 can also be a node server in a blockchain network.

[0047] The terminal 101 can be a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, and the like, but is not limited thereto. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, and the present disclosure is not limited thereto.

[0048] The method provided by the present disclosure can be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving scenes.

[0049] Referring to FIG. 2, FIG. 2 is a flowchart of a video generation method provided by an embodiment of the present disclosure. The video generation method can be executed by a server, or can be executed by a terminal, or can be executed by a server in cooperation with a terminal. The video generation method includes, but is not limited to, the following steps 201 to 203.

[0050] Step 201: Obtain a target trajectory graph, and map the target trajectory graph based on a target rank decomposition matrix configured in the motion control network to obtain target motion features.

[0051] The target rank decomposition matrix is used to control the target motion features to constrain the motion trajectory of the video lens and / or the motion trajectory of the video display object in the video generation process. The target rank decomposition matrix can be a learnable parameter matrix optimized by low-rank decomposition in the motion control network. The target rank decomposition matrix realizes compact representation and efficient calculation of motion information by projecting high-dimensional trajectory data (i.e., the target trajectory graph) into a low-rank feature space (i.e., the target motion feature). The target rank decomposition matrix can be constructed using matrix decomposition techniques such as SVD or NMF, has the characteristics of dimensionality reduction and feature decoupling, and can convert the spatial-temporal information in the original trajectory graph (such as the target trajectory graph) into motion basis vectors (such as the target motion feature) with clear physical meaning. These basis vectors can be linearly combined to generate various complex motion patterns, providing an interpretable motion constraint mechanism for video generation.

[0052] Here, the target trajectory graph is a graphical representation for indicating the motion trajectory of the video lens and / or the motion trajectory of the video display object, and is used to indicate the motion trajectory of the video lens and / or the motion trajectory of the video display object. The target trajectory graph contains two types of trajectories that can be used independently or in combination: the first type of trajectory is used to define the motion parameters of the video lens (including translation direction, movement distance, rotation angle, and scaling ratio, etc. camera motion attributes); the second type of trajectory is used to specify the motion path of the video display object (such as a person, a vehicle, or other visual elements). The starting point of this trajectory is geometrically associated with the spatial position (usually the center point or feature area) of the video display object, ensuring that the motion trajectory of the video display object strictly corresponds to the physical position of the video display object. For example, the target trajectory graph can be provided with a first motion trajectory (i.e., the first type of trajectory) of the video lens, which indicates the motion direction and motion distance of the video lens through the first motion trajectory. For another example, the target trajectory graph can be provided with a second motion trajectory (i.e., the second type of trajectory) of the video display object, which indicates the motion direction and motion distance of the video display object through the second motion trajectory. The starting end of the second motion trajectory is located in the area where the video display object is located, such as the center of the video display object, which means that the starting end of the second motion trajectory is associated with the area where the video display object is located. The motion trajectory of the video lens refers to the motion path of the camera in the video generation process, which is defined by the first motion trajectory in the target trajectory graph, such as controlling the translation, rotation, or scaling of the lens in the direction and distance. The motion trajectory of the video display object refers to the motion path of a specific object (such as a person or an object) in the video, which is defined by the second motion trajectory in the target trajectory graph, and its starting end is associated with the object area, used to constrain the moving direction and distance of the object.

[0053] Here, the target motion feature is a dense vector representation obtained by mapping the target trajectory graph through a target rank decomposition matrix, encoding the motion rules of the video shot and / or the video display object in the spatio-temporal dimension. The target motion feature contains dynamic information such as motion direction, speed curve, and acceleration pattern, which is subsequently embedded in the form of a latent space vector in the denoising process of the diffusion model, ensuring that the generated video sequence strictly follows the constraint conditions of the target trajectory graph in terms of spatial transformation (shot motion) and object displacement (video display object motion) in each frame.

[0054] wherein the motion control network is a modular architecture based on neural networks, which maps the input target trajectory graph into structured motion features through a target rank decomposition matrix (composed of a low-rank matrix A and a low-rank matrix B). The network achieves motion constraints through the following mechanisms: trajectory encoding: encoding the shot motion trajectory (such as camera pose parameters) and / or object motion trajectory (such as object displacement path) in the target trajectory graph into low-dimensional motion basis vectors; rank constraint mapping: using the low-rank property of the target rank decomposition matrix ABT to compress the trajectory information into compact motion features, ensuring that the inter-frame motion of the generated video conforms to physical laws; diffusion model collaboration: the output target motion features and content features jointly guide the diffusion model denoising process, achieving spatio-temporal alignment of motion trajectories and generated content. It should be noted that the embodiments of the present disclosure are not limited to the structure of the motion control network, which can be based on an attention mechanism structure, or based on a graph neural network structure, etc.

[0055] For example, since the target trajectory graph carries key information indicating the motion trajectory of the video shot and / or the motion trajectory of the video display object, and the motion control network loaded with the target rank decomposition matrix has the ability to capture the key information from the target trajectory graph, which can guide the diffusion model to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object during video generation, the target trajectory graph can be mapped using the motion control network loaded with the target rank decomposition matrix to obtain the target motion feature containing the key information. Since the key information can guide the diffusion model to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object during video generation, the target motion feature containing the key information can guide the diffusion model to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object during video generation, thereby obtaining the target video with the motion trajectory of the video shot and / or the motion trajectory of the video display object constrained.

[0056] For a video lens, i.e. a lens of a camera, constraining a motion trajectory of the video lens changes the whole video picture, and the constraint on the video lens specifically refers to a constraint on a target view angle of the video lens, the target view angle refers to a specific observation point and direction of the video lens to a video display object in a three-dimensional space, and the target view angle includes a position of the video lens, an orientation of the video lens, and a focal length of the video lens. For example, when a motion trajectory of the position of the video lens is a first motion trajectory, the generated video can present a picture that translates based on the first motion trajectory. For another example, when a motion trajectory of the orientation of the video lens is a second motion trajectory, the generated video can present a picture that rotates based on the second motion trajectory. For another example, when a motion trajectory of the focal length of the video lens is a third motion trajectory, the generated video can present a picture that advances or zooms out based on the third motion trajectory.

[0057] For a video display object, i.e. a person or thing appearing in a video, constraining a motion trajectory of the video display object changes a position or state of the video display object in a video picture. For example, the video display object can move linearly, move in a curve, or move in a combination of linear and curve motions, or can move in other manners, which are not limited in the embodiments of the present disclosure.

[0058] Taking a car as an example of a video display object, when a motion trajectory of the car is a straight-line trajectory, the generated video can present a picture that the car moves linearly along the straight-line trajectory. When multiple video display objects appear in the video, the motion trajectories of the video display objects can be constrained simultaneously, and the generated video can present a picture that the multiple video display objects move simultaneously.

[0059] Based on this, the target trajectory graph is mapped based on the target rank decomposition matrix, and a target motion feature carrying key information is extracted from the target trajectory graph, which is helpful for fine control of the motion of the video lens and / or the video display object in the video generation process, thereby improving the quality of the generated video.

[0060] In some embodiments, the motion control network can include one or more (i.e. at least two) processing layers. When the motion control network includes multiple cascaded processing layers, each processing layer of the motion control network is used to learn a feature representation of different levels and degrees of abstraction. A lower-level processing layer can learn a relatively simple feature representation, and as the level increases, a higher-level processing layer can combine the feature representation learned by the lower-level processing layer to form a more abstract and complex feature representation.

[0061] In this system, some or all of the processing layers are configured with an original parameter matrix. In each processing layer configured with an original parameter matrix, some or all of the processing layers are also configured with a target rank decomposition matrix. The target rank decomposition matrix is ​​used to sum with the original parameter matrix and then map the input of the processing layer to obtain the output of the processing layer. During the training of the motion control network, only the target rank decomposition matrix can be adjusted and the original parameter matrix can be frozen, which can greatly reduce the number of parameters adjusted when training the motion control network and improve the training efficiency of the motion control network.

[0062] For example, referring to Figure 3, Figure 3 is a schematic diagram of a target rank decomposition matrix provided in an embodiment of this disclosure.

[0063] The original parameter matrix W has a dimension of d×d. In the low-rank decomposition structure, the first target rank matrix A has a dimension of d×r, and the second target rank matrix B can also have a dimension of d×r. Multiplying the target rank matrix A by the transpose of the target rank matrix B yields the target rank decomposition matrix AB. T The dimension of the target rank decomposition matrix is ​​d×d. Typically, the target rank decomposition matrix is ​​set to a low rank, meaning r is much smaller than d, where r is the rank of the target rank decomposition matrix. The dimension of the target rank decomposition matrix is ​​the same as the dimension of the original parameter matrix W, so it does not change the dimension of the output data. Therefore, when performing full parameter adjustment on the parameter update matrix of the processing layer, the number of parameters adjusted is d*d, while when adjusting the target rank decomposition matrix inserted into the processing layer, the number of parameters adjusted is 2*d*r. Thus, the number of parameters adjusted by the processing layer is reduced from d*d to 2*d*r. This effectively reduces the number of parameters adjusted by each processing layer, effectively reducing the model training time and space, thereby reducing the additional GPU memory usage during inference. Furthermore, before training begins, the first target rank matrix A is initialized as a random normal matrix, and the second target rank matrix B is initialized as a zero matrix. Therefore, the weight parameter matrix of the processing layer can be defined as: W′=W+AB T

[0064] Where W′ is the weight parameter matrix of the processing layer, W is the original parameter matrix, A is the first target rank matrix in the target rank decomposition matrix, and B is the second target rank matrix in the target rank decomposition matrix. T This represents the transpose of B.

[0065] Step 202: Obtain the target reference content required in the video generation process, and extract the first content feature of the target reference content.

[0066] The target reference content represents single-modal data or multi-modal data (such as text, image, or audio) used to guide the video generation in the video generation process, that is, the data required for reference in the video generation process, and the semantic information carried by the target reference content will constrain the subject content and scene performance of the generated video. The target reference content is the guiding data in the video generation process, and the target reference content carries information for guiding the generation of specific content of the video, for example, the target reference content can be "a car driving", and the target video generated based on the target reference content will show a car driving. The target reference content can be single-modal data or multi-modal data, for example, the target reference content can include one or more of text data, image data, audio data, etc. When the target reference content is multi-modal data, the target reference contents of different modalities are associated, and there is usually complementarity between the target reference contents of different modalities, which can improve the quality of the generated video through the target reference contents of different modalities.

[0067] Here, the first content feature is a structured representation obtained by encoding the target reference content, and the first content feature carries semantic, visual, or auditory key information of the target reference content, which is used to guide the diffusion model to generate a content-consistent video. Based on this, the feature extraction network can be used to extract features from the target reference content, and more abstract and informative first content features can be extracted from the target reference content. The first content feature is a feature of great value to the video generation process, and the quality of the generated video can be improved through the first content feature. The feature extraction network is a neural network architecture used to encode the input target reference content (such as text, image, audio, etc.) into a high-dimensional structured representation (i.e., the first content feature). The feature extraction network is used to extract and compress the semantic, visual, or auditory key information of the input data, so that it becomes a compact feature representation that the diffusion model can understand, thereby guiding the content consistency of video generation. It should be noted that the disclosed embodiments are not limited to the structure of the feature extraction network, for example, it can be Bidirectional Encoder Representations from Transformers (BERT) to extract content features in text type target reference content; it can also be a Convolutional Neural Networks (CNN) to extract content features in image type target reference content.

[0068] Taking the target reference content including text data and image data as an example, the text data is used to describe the content in the image data, the first feature extraction network can be used to extract features of the text data to obtain the first content features corresponding to the text data, and the second feature extraction network can be used to extract features of the image data to obtain the first content features corresponding to the image data. For example, the first feature extraction network includes a contrastive language-image pre-training (CLIP) model, a word embedding network, a BERT model, etc., and the second feature extraction network can include a convolutional neural network, a residual network, etc.

[0069] Step 203: obtaining a target noise image stack, and using the target motion feature and the first content feature as constraint conditions to denoise the target noise image stack through the diffusion model to obtain a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object.

[0070] The target noise image stack is formed by sequentially stacking a plurality of randomly generated target noise images, and each target noise image is a two-dimensional image with the same dimensions. Therefore, the target noise image stack can form a three-dimensional matrix with dimensions of HxWxT. For the target noise image stack, H represents the height of the target noise image, W represents the width of the target noise image, and T represents the total number of target noise images.

[0071] The diffusion model has two main processes: a forward process and a backward process. The forward process is also known as a diffusion process. The diffusion model gradually adds noise to the image in the forward process until the image becomes random noise. For example, the diffusion process as a whole can be a parameterized Markov chain. The backward process is also known as an inverse diffusion process. The diffusion model uses a series of Markov chains to gradually remove predicted noise at each time step in the backward process, thereby recovering data from noise. Therefore, the diffusion model can denoise the target noise image stack based on the target motion feature and the first content feature in the backward process, thereby generating the target video.

[0072] The specific generation process of the target video is described in detail below.

[0073] Firstly, the noise in the target noise image stack is removed step by step by using the diffusion model through multiple time steps, and in the process of removing the noise, the target motion feature and the first content feature are used as constraint conditions at the same time, so that a high-quality and detailed target video frame stack can be recovered. Since the constraint conditions include the target motion feature and the first content feature, the target motion feature and the first content feature are injected into the diffusion model when predicting the noise at each time step, so that the diffusion model considers the target motion feature and the first content feature in each step of the inverse diffusion process, to ensure that the target video frame stack matches the target motion feature and the first content feature respectively, and to achieve more natural video lens motion and / or video display object motion.

[0074] It can be known that since the target video frame stack is generated based on the target noise image stack with the dimension of HxWxT, the dimension of the target video frame stack is also HxWxT, and the target video frame stack is stacked by T target video frames with the dimension of HxW, therefore, the stacking order of each target video frame in the target video frame stack can be used as the playing order of each target video frame in the video, for example, the first target video frame in the target video frame stack is the first playing picture, and the second target video frame in the target video frame stack is the second playing picture.

[0075] Then, based on the video frame rate and the stacking order of each target video frame in the target video frame stack, each target video frame in the target video frame stack is synthesized into a target video, to realize high-quality generation of the target video, wherein the video frame rate can be preset or input by relevant personnel, and the video frame rate is the number of target video frames displayed per second in the target video. The higher the video frame rate, the higher the smoothness of the target video.

[0076] It should be noted that the target video frame can be various types of images, for example, the target video frame can be a red-green-blue three-channel image (RGB image), a grayscale image, a binary image, etc., and the embodiments of the present disclosure are not limited herein. Therefore, the target video synthesized by the target video frame can also be various types of videos, for example, the target video can be a dynamic picture sequence composed of RGB images, grayscale images or binary images, etc. The target video can be applied in various fields and scenes such as entertainment, education, advertising, intelligent transportation, etc.

[0077] Based on this, the target rank decomposition matrix configured based on the motion control network is used to map the target trajectory graph to obtain the target motion feature, the first content feature of the target reference content is extracted, and the target noise image stack is denoised by taking the target motion feature and the first content feature as constraint conditions through the diffusion model to obtain the target video constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object. Since the target rank decomposition matrix is used to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process, by configuring the target rank decomposition matrix in the motion control network, the motion decoupling of the video shot and the video display object can be realized in the processing process of the diffusion model, the fine control of the motion of the video shot and / or the video display object is achieved, and thus the quality of the generated video is improved.

[0078] The process of inputting the target motion feature and the first content feature into the diffusion model is described in detail below.

[0079] For example, referring to FIG. 4, FIG. 4 is an architecture schematic diagram of the video generation method provided in the embodiment of the present disclosure.

[0080] The target reference content includes a target reference image and a target reference text, the target reference text is used to describe the content of the target reference image, the target text feature of the target reference text can be extracted through the first feature extraction network 401, the target image feature of the target reference image can be extracted through the second feature extraction network 402, and the target text feature and the target image feature are both target content features.

[0081] Then, the architecture of the diffusion model, the motion control network and the second feature extraction network is described. The diffusion model 403 includes a downsampling network and an upsampling network, the downsampling network is used to obtain the target noise image stack, the upsampling network can include multiple levels of upsampling blocks, and an attention layer is added between adjacent two levels of upsampling blocks. The motion control network 404 can include multiple cascaded first linear layers, and the second feature extraction network 402 can include multiple cascaded second linear layers.

[0082] Then, the output of each first linear layer in the motion control network 404 can be used as the target motion feature, and the output of each first linear layer in the motion control network can be input into the corresponding level of the upsampling block in the upsampling network in addition to being input into the next level of the first linear layer, for example, the output of the first level of the first linear layer in the motion control network is input into the first level of the upsampling block in the upsampling network.

[0083] Similar to the target motion features, the output of each second linear layer in the second feature extraction network 402 can be regarded as a target image feature, and the output of each second linear layer at each level in the second feature extraction network 402 can be input into the upsampling block at the corresponding level in the upsampling network in addition to being input into the second linear layer at the next level.

[0084] Then, the motion control network 404 inputs the time embedding determined by the time step into each upsampling block, each of which includes a first convolutional layer and a second convolutional layer. The input of the first convolutional layer is the output of the previous attention layer, the output of the first convolutional layer, the time embedding, the corresponding target motion feature, and the corresponding target motion feature are added and input into the second convolutional layer, and the output of the second convolutional layer is input into the next attention layer.

[0085] Then, the attention layer includes a self-attention sublayer and a cross-attention sublayer. The self-attention sublayer can perform self-attention processing on the output of the second convolutional layer. The cross-attention sublayer can construct a query matrix based on the output of the self-attention sublayer, then construct a key matrix and a value matrix based on the target text feature, and obtain the input of the first convolutional layer in the next upsampling block by performing cross-attention processing on the query matrix, the key matrix and the value matrix through the cross-attention sublayer.

[0086] In some embodiments, the motion control network 404 is configured with a first rank decomposition matrix and / or a second rank decomposition matrix. The first rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video shot in the video generation process, and the second rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video display object in the video generation process. For this purpose, before step 201, an operation instruction is obtained; the first rank decomposition matrix and / or the second rank decomposition matrix are loaded according to the operation instruction, and the loaded first rank decomposition matrix and / or the second rank decomposition matrix are used as the target rank decomposition matrix.

[0087] Wherein, the operation instruction refers to a specific control signal input by the user or the system in the video generation process, which is used to dynamically select and load different rank decomposition matrices (including the first rank decomposition matrix and / or the second rank decomposition matrix) in the motion control network, so as to accurately configure the constraint mode of the video shot motion trajectory and / or the object motion trajectory. The operation instruction can be generated through an interactive interface (such as clicking options), a programmed parameter (such as script calling), or automatic analysis (such as scene demand recognition). Finally, the combination form of the target rank decomposition matrix is determined to realize flexible control of the motion mode of the generated video.

[0088] The type of the operation instruction can be various, for example, the operation instruction can be a button instruction, a checkbox instruction, and the like graphical interface instruction, or the operation instruction can be a voice instruction, a gesture instruction, a command line instruction, and the like other type of instruction, which is not limited herein. Therefore, the first rank decomposition matrix and / or the second rank decomposition matrix can be loaded as the target rank decomposition matrix through different operation instructions.

[0089] Taking the graphical interface instruction as an example, the display interface can display a first button control and a second button control, the first button control is associated with the first rank decomposition matrix, and the second button control is associated with the second rank decomposition matrix. The relevant personnel can generate a first button instruction by triggering the first button control, and the first button instruction is used to load the first rank decomposition matrix as the target rank decomposition matrix. The relevant personnel can also generate a second button instruction by triggering the second button control, and the second button instruction is used to load the second rank decomposition matrix as the target rank decomposition matrix.

[0090] Based on this, the motion control network can be simultaneously configured with the first rank decomposition matrix and the second rank decomposition matrix. In the inference stage, the operation instruction is first acquired, and when the target trajectory graph needs to be mapped, the specific target rank decomposition matrix is loaded according to the operation instruction. When the first rank decomposition matrix is loaded, the target motion feature determined based on the first rank decomposition matrix can constrain the motion trajectory of the video lens in the video generation process. When the second rank decomposition matrix is loaded, the target motion feature determined based on the second rank decomposition matrix can constrain the motion trajectory of the video display object in the video generation process. It can be seen that the motion of the video lens and the video display object can be decoupled in the processing process of the diffusion model, and the fine control of the motion of the video lens and / or the video display object is achieved, thereby improving the quality of the generated video.

[0091] In other words, the embodiments of the present disclosure can selectively load the first rank decomposition matrix and / or the second rank decomposition matrix through the operation instruction, and dynamically combine the two matrices as the target rank decomposition matrix based on the operation instruction, thereby achieving modular and flexible control of the motion constraint mechanism in the video generation process. This design allows users or systems to independently regulate the video lens motion trajectory and the object motion trajectory according to actual needs, which not only retains the efficiency of the low-rank decomposition technology (reduces computational complexity through matrix decomposition), but also provides fine-grained motion control capability, thereby significantly improving the flexibility and controllability of video generation.

[0092] Referring to FIG. 5, FIG. 5 is a flowchart of generating a target video according to an embodiment of the present disclosure. When the first rank decomposition matrix 501 is loaded by the operation instruction, the diffusion model can generate a target video that conforms to the target reference content and in which the video lens moves according to the motion trajectory indicated by the target trajectory graph. In addition, when the second rank decomposition matrix 502 is loaded by the operation instruction, the diffusion model can generate a target video that conforms to the target reference content and in which the video display object moves according to the motion trajectory indicated by the target trajectory graph.

[0093] For example, referring to FIG. 6, FIG. 6 is a diagram of a generation result of a target video under two motion types according to an embodiment of the present disclosure. The content 601 in the first column of FIG. 6 is used to indicate the target reference content and the target trajectory graph, the content 602 in the second column to the fifth column of FIG. 6 is used to indicate the frames of the target video, the content 603 in the first row of FIG. 6 shows the generation result of the target video in which the video lens moves, and the content 604 in the second row of FIG. 6 shows the generation result of the target video in which the video display object moves.

[0094] As can be seen, when the target trajectory graph is provided with a single motion trajectory, in the target video in which the video lens moves, the video lens can move based on the motion trajectory; and in the target video in which the video display object moves, the video display object at the starting end of the motion trajectory can move along the motion trajectory.

[0095] For another example, referring to FIG. 7, FIG. 7 is another diagram of a generation result of a target video under two motion types according to an embodiment of the present disclosure. The content 701 in the first column of FIG. 7 is used to indicate the target reference content and the target trajectory graph, the content 702 in the second column to the fifth column of FIG. 7 is used to indicate the frames of the target video, the content 703 in the first row of FIG. 7 shows the generation result of the target video in which the video lens moves, and the content 704 in the second row of FIG. 7 shows the generation result of the target video in which the video display object moves.

[0096] As can be seen, when the target trajectory graph is provided with two opposite motion trajectories, in the target video in which the video lens moves, the video lens can zoom out based on the two motion trajectories; and in the target video in which the video display object moves, the video display object at the starting end of one of the motion trajectories can move along the motion trajectory, and the video display object at the starting end of the other motion trajectory can move along the motion trajectory.

[0097] In a possible implementation, the motion control network can also be configured only with the first rank decomposition matrix, in which case the motion control network defaults to loading the first rank decomposition matrix as the target rank decomposition matrix, and the target motion feature determined by the motion control network can constrain the motion trajectory of the video lens in the video generation process.

[0098] In addition, the motion control network can also be configured only with the second rank decomposition matrix, at this time, the motion control network defaults to load the second rank decomposition matrix as the target rank decomposition matrix, and the target motion feature determined based on the motion control network can constrain the motion trajectory of the video display object in the video generation process.

[0099] In some embodiments, when "de-noising the target noise image stack by the diffusion model with the target motion feature and the first content feature as constraint conditions to obtain a target video constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object" in step 203, it can be achieved by the following way: predicting the third predicted noise by the diffusion model with the target motion feature and the first content feature as constraint conditions; predicting the fourth predicted noise by the diffusion model on the target noise image stack; adjusting the third predicted noise based on the fourth predicted noise, and de-noising the target noise image stack based on the adjusted third predicted noise to obtain the target video.

[0100] Among them, the dimension of the third predicted noise is the same as the dimension of the target noise image stack, assuming that the target noise image stack is stacked by 100 noise images, then the third predicted noise is also stacked by 100 noise images, and the size of each noise image is the same. Similar to the third predicted noise, the dimension of the fourth predicted noise is also the same as the dimension of the target noise image stack.

[0101] It should be noted that since the target motion feature is obtained by mapping the target trajectory graph based on the target rank decomposition matrix, when the target motion feature and the first content feature are used as constraint conditions, the third predicted noise predicted by the diffusion model can make the diffusion model generate a video with fine control motion. Therefore, the third predicted noise is the noise predicted by the diffusion model under the double constraint conditions (target motion feature and first content feature), which encodes the generation direction of video content semantics and motion trajectory at the same time. The third predicted noise accurately controls the shot / object motion (such as camera translation or object displacement) through the target motion feature mapped by the target rank decomposition matrix, and constrains the consistency of the scene subject through the first content feature, thereby realizing the collaborative generation of content and motion.

[0102] In addition, the target noise image stack is input into the diffusion model for noise prediction, which is equivalent to the diffusion model performing noise prediction under unconstrained conditions. At this time, the fourth predicted noise predicted by the diffusion model can enable the diffusion model to generate an unconstrained video. Therefore, the fourth predicted noise is noise directly predicted by the diffusion model under unconstrained conditions on the target noise image stack, and the fourth predicted noise reflects the natural distribution characteristics of the data itself. The fourth predicted noise is not limited by motion or content conditions and can contain a mixed mode of camera motion and object motion.

[0103] Based on this, the third predicted noise is predicted by the diffusion model under the condition that both content and motion conditions are present, and the third predicted noise can control the generation of both content and motion types in the target video. In some special scenarios, when one of the motion types, i.e., camera motion or object motion, needs to be controlled by the third predicted noise, the other motion type will inevitably be introduced. At this time, the fourth predicted noise can be predicted by the diffusion model under unconstrained conditions. Since the fourth predicted noise can control both motion types, adjusting the third predicted noise based on the fourth predicted noise can eliminate the other motion type introduced by the third predicted noise.

[0104] For example, when the target rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video display object in the video generation process, the third predicted noise controls the motion of the video display object. In some special scenarios, the third predicted noise will inevitably introduce camera motion. Adjusting the third predicted noise based on the fourth predicted noise can eliminate the camera motion introduced by the third predicted noise.

[0105] For another example, when the target rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the camera in the video generation process, the third predicted noise controls the motion of the camera. In some special scenarios, the third predicted noise will inevitably introduce object motion. Adjusting the third predicted noise based on the fourth predicted noise can eliminate the object motion introduced by the third predicted noise.

[0106] On this basis, when the target noise image stack is denoised based on the adjusted third predicted noise to obtain a target video, the motion intensity of the target video can be flexibly and seamlessly enhanced, achieving fine control of the motion of the camera and / or the video display object, and thus improving the quality of the generated video.

[0107] In some embodiments, the adjusting the third predicted noise based on the fourth predicted noise can be achieved by: mapping the target trajectory graph based on an original parameter matrix of the motion control network to obtain initial motion features; performing noise prediction on the initial motion features and the first content features as constraint conditions based on a diffusion model to obtain a fifth predicted noise; and adjusting the third predicted noise based on the fourth predicted noise and the fifth predicted noise.

[0108] wherein the original parameter matrix is an initial full parameter weight matrix W (dimension dxd) of the motion control network which is not adjusted by the low-rank decomposition (AB T ) and directly acts on the spatial transformation of the target trajectory graph to generate a baseline motion feature which is not subject to specific motion constraints. The original parameter matrix is learned in the model pre-training stage and remains in a frozen state in the fine-tuning process as a baseline parameter for calculating unconditioned motion features.

[0109] wherein the initial motion feature is a motion representation obtained by directly mapping the target trajectory graph based on the original parameter matrix (non-target rank decomposition matrix), and the initial motion feature reflects the basic geometric transformation mode (such as affine parameters) of the input trajectory, but does not incorporate specific motion constraints (such as the motion trajectory of the video lens and / or the motion trajectory of the video display object) introduced by the low-rank decomposition. The initial motion feature serves as a reference benchmark for motion condition generation, and is used to quantify the motion control increment effect brought by the target rank decomposition.

[0110] wherein, similar to the third predicted noise, the dimension of the fifth predicted noise is also the same as that of the target noise image stack. The fifth predicted noise is the noise predicted by the diffusion model with the initial motion feature and the first content feature as joint constraint conditions, and the fifth predicted noise encodes the coupling relationship between the motion mode not modulated by the low-rank decomposition and the content semantics.

[0111] Based on this, the initial motion feature is obtained by mapping the target trajectory graph based on the original parameter matrix, i.e., the initial motion feature is mapped by the motion control network based on the non-loaded target rank decomposition matrix. Based on the adjustment of the third predicted noise based on the fourth predicted noise, the adjustment of the third predicted noise based on the fifth predicted noise can adjust the similarity between the content in the target video and the target reference content, and achieve the diversity of the content in the target video.

[0112] Specifically, taking an example of using a target rank decomposition matrix for controlling a target motion feature to constrain a motion trajectory of a video display object in a video generation process, the target rank decomposition matrix is a second rank decomposition matrix, when adjusting the third prediction noise based on the fourth prediction noise and the fifth prediction noise, a first adjustment weight and a second adjustment weight can be introduced, and a determination formula of the adjusted third prediction noise is as shown in formula (1):

[0113] wherein, is the adjusted third prediction noise, is the third prediction noise before adjustment, is the fourth prediction noise, is the fifth prediction noise, θ0includes parameters of a pre-trained diffusion model and a motion control network, θ trajs includes parameters of the diffusion model and the motion control network loaded only with the second rank decomposition matrix, c all is a first content feature and an initial motion feature, c trajs_all is the first content feature and a target motion feature, is used to indicate an unconstrained condition, x t is a target noise image stack after the t-th denoising processing.

[0114] wherein, λ cfg is a preset first adjustment weight, λ trajs is a preset second adjustment weight, for example, when λ cfg and λ trajs are both preset as 4, the determination formula of the adjusted third prediction noise can be simplified as formula (2):

[0115] At this time, the formula (2) only contains the third prediction noise and the fourth prediction noise, which is equivalent to the case of adjusting the third prediction noise based on the fourth prediction noise described above.

[0116] It should be noted that λ trajs may be less than or equal to λ cfg , λ cfg and λ trajs can also be preset as other values, and the embodiments of the present disclosure are not limited here, and the values of λ cfg and λ trajs can be set according to actual conditions, when λ cfg is larger, the content in the target video controlled by the adjusted third prediction noise is more similar to the target reference content, that is, the diversity of the content in the target video is lower; on the contrary, when λ cfg is smaller, the content in the target video controlled by the adjusted third prediction noise is less similar to the target reference content, that is, the diversity of the content in the target video is higher.

[0117] In addition, the role of the second adjustment weight is described in detail below.

[0118] Referring to FIG. 8, FIG. 8 is a schematic diagram of a generation result of a target video under different second adjustment weights according to an embodiment of the present disclosure. In FIG. 8, the content 801 in the first column is used to indicate a target reference content and a target trajectory graph, the content 802 in the second column to the fifth column is used to indicate a picture of a video frame of the target video, the content 803 in the first row is used to show a generation result of the target video when λ trajs = 1, the content 804 in the second row is used to show a generation result of the target video when λ trajs = 1.1, and the content 805 in the third row is used to show a generation result of the target video when λ trajs = 2. In any row, the playing order of the video frame of the target video is the second column, the third column, the fourth column, and the fifth column.

[0119] It can be seen that, in the case where the target rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video display object in the video generation process, when λ trajs = 1, the video shot of the target video appears a leftward panning motion, which is equivalent to introducing another motion type, when λ trajs = 1.1, the motion of the video display object can be preliminarily enhanced, and when λ trajs = 2, the motion of the video display object becomes more obvious and clear.

[0120] The training process of the motion control network is described in detail below.

[0121] In an embodiment, before step 201, a first video sample is obtained, a first trajectory graph sample of the first video sample is extracted, wherein the first video sample is a video in which a video shot is moving and a video display object is static; the motion control network is controlled to load a first rank decomposition matrix, and the first rank decomposition matrix loaded by the motion control network is adjusted based on the first trajectory graph sample and the first video sample; a second video sample is obtained, a second trajectory graph sample of the second video sample is extracted, wherein the second video sample is a video in which both a video shot and a video display object are moving; the motion control network is controlled to load a second rank decomposition matrix, and the second rank decomposition matrix loaded by the motion control network is adjusted based on the second trajectory graph sample and the second video sample.

[0122] The first video sample can be of various types, such as a sequence of dynamic images composed of RGB images, grayscale images, or binary images. The first video sample can be obtained from sources such as movies, animations, and photographic works. This embodiment of the present disclosure does not limit the types of videos. Similarly, the second video sample can also be of various types and can be obtained from various sources.

[0123] The first video sample is a video in which the video lens moves and the video display object is stationary. Therefore, by extracting the trajectory map of the first video sample, a first trajectory map can be obtained to indicate the movement trajectory of the video lens. For example, the first video sample can be obtained by moving the camera to shoot the still subject. The camera is equivalent to the video lens and the subject is equivalent to the video display object.

[0124] The second video sample is a video in which both the video lens and the video object are in motion. Therefore, by extracting the trajectory map of the second video sample, a second trajectory map can be obtained to indicate the motion trajectory of the video lens and the motion trajectory of the video object. For example, the second video sample can be obtained by moving the camera to shoot the subject in motion.

[0125] Based on this, during the training process, firstly, the motion control network loads the first rank decomposition matrix. The first video sample and the corresponding first trajectory map sample are used to adjust the first rank decomposition matrix loaded by the motion control network. The first rank decomposition matrix needs to learn how to capture the first key information in the trajectory map. The first key information can guide the diffusion model to constrain the motion trajectory of the video lens during the video generation process. It can be seen that the adjusted first rank decomposition matrix can enable the motion features to constrain the motion trajectory of the video lens during the video generation process. Based on the constraint of the motion features, the diffusion model can accurately control the motion of the video lens when generating the video, achieving fine control of the motion of the video lens.

[0126] Then, the motion control network is controlled to reload the second rank decomposition matrix, and the second rank decomposition matrix loaded by the motion control network is adjusted by using the second video sample and the corresponding second trajectory pattern sample, without adjusting the first rank decomposition matrix. Since the adjusted first rank decomposition matrix can capture the second key information from the second trajectory pattern sample, the second rank decomposition matrix does not need to learn how to capture the first key information from the trajectory pattern, and only needs to learn how to capture the second key information from the trajectory pattern. The second key information can guide the diffusion model to constrain the motion trajectory of the video display object in the video generation process. It can be seen that the adjusted second rank decomposition matrix can make the motion feature constrain the motion trajectory of the video display object in the video generation process, and based on the constraint of the motion feature, the diffusion model accurately controls the motion of the video display object when generating the video, achieving fine control of the motion of the video display object.

[0127] In this way, the embodiments of the present disclosure realize decoupled optimization of the video camera motion and object motion constraint capability by independently training the first rank decomposition matrix and the second rank decomposition matrix using the first video sample (i.e., the pure camera motion sample) and the second video sample (i.e., the mixed motion sample) respectively, thereby improving the accurate modeling capability of the motion control network for different types of motion.

[0128] It should be noted that the second rank decomposition matrix can be adjusted in the following manner in addition to the above-mentioned manner.

[0129] In a possible implementation, before step 201, a third video sample is obtained, and a third trajectory pattern sample of the third video sample is extracted; the motion control network is controlled to load the second rank decomposition matrix, and the second rank decomposition matrix loaded by the motion control network is adjusted based on the third trajectory pattern sample and the third video sample.

[0130] The third video sample is a video in which the video camera is static and the video display object moves, and therefore, by extracting the trajectory pattern of the third video sample, the third trajectory pattern indicating the motion trajectory of the video display object can be obtained.

[0131] Based on this, in the training process, the motion control network is controlled to load the second rank decomposition matrix, and the second rank decomposition matrix is adjusted by using the third video sample and the corresponding third trajectory pattern sample. The second rank decomposition matrix needs to learn how to capture the second key information in the trajectory pattern. The second key information can guide the diffusion model to constrain the motion trajectory of the video display object in the video generation process. It can be seen that the adjusted second rank decomposition matrix can make the motion feature constrain the motion trajectory of the video display object in the video generation process, and based on the constraint of the motion feature, the diffusion model accurately controls the motion of the video display object when generating the video, achieving fine control of the motion of the video display object.

[0132] In some embodiments, referring to FIG. 9, FIG. 9 is a flowchart for adjusting the first rank decomposition matrix according to an embodiment of the present disclosure; the “adjusting the first rank decomposition matrix loaded by the motion control network based on the first trajectory pattern sample and the first video sample” can be implemented in the following manner: mapping the first trajectory pattern sample through the motion control network loaded with the first rank decomposition matrix to obtain the first motion feature sample; obtaining the first reference content sample associated with the first video sample, and extracting the second content feature of the first reference content sample; performing noise prediction on the first video sample added with the first noise sample, the first motion feature sample, and the second content feature through the diffusion model to obtain the first predicted noise; determining the first loss value based on the difference between the first noise sample and the first predicted noise, and adjusting the first rank decomposition matrix loaded by the motion control network based on the first loss value.

[0133] In the process of training the first rank decomposition matrix, the first trajectory pattern sample is input into the motion control network loaded with only the first rank decomposition matrix, and the first trajectory pattern sample is mapped based on the first rank decomposition matrix. Since the first trajectory pattern is used to indicate the motion trajectory of the video shot, the first motion feature sample extracted from the first trajectory pattern sample can guide the diffusion model to constrain the motion trajectory of the video shot in the video generation process.

[0134] The first reference content sample carries information guiding the generation of specific content of the video, and the first reference content sample can be single-modal data or multi-modal data. For example, the first reference content sample can include one or more of text data, image data, audio data, etc. Assuming that the first reference content sample includes a first sample reference image and a first sample reference text, the first video sample can be used as the first sample reference image, and the text describing the content of the first sample reference image can be used as the first sample reference text, which can ensure the correlation between the first reference content sample and the first video sample. Based on the first reference content sample, the diffusion model can predict appropriate first predicted noise, which is used to denoise the first video sample added with the first noise sample to restore the original first video sample. The first reference content sample is similar to the target reference content described above, and will not be described here.

[0135] Based on this, in the process of training the first rank decomposition matrix, the first predicted noise is obtained by predicting the diffusion model, and then the first noise sample is taken as the label data, the first loss value is determined by the difference between the first noise sample and the first predicted noise, and the first rank decomposition matrix is adjusted based on the first loss value, that is, only the first rank decomposition matrix is optimized, so that the motion control network loaded with the adjusted first rank decomposition matrix can learn how to extract the motion features carrying the first key information from the trajectory graph. The first key information can guide the diffusion model to constrain the motion trajectory of the video shot in the video generation process, and then help the diffusion model better understand the motion trajectory of the video shot, so as to predict more accurate first predicted noise. The accurate first predicted noise can accurately control the motion of the video shot when the diffusion model generates the video, and achieve fine control of the motion of the video shot.

[0136] For example, since the diffusion model and the original parameter matrix are frozen when the first rank decomposition matrix is adjusted based on the first loss value, the update gradient of the first loss value to the parameters of the first rank decomposition matrix can be calculated based on the back propagation algorithm, and the first gradient is obtained. Then, the parameters of the first rank decomposition matrix are updated based on the first gradient; when the first rank decomposition matrix is adjusted based on the first loss value, the adjustment formula of the parameters of the diffusion model and the motion control network is as shown in formula (3): cam θ cam (3)

[0137] wherein, θ cam includes the parameters of the diffusion model and the motion control network loaded with only the first rank decomposition matrix, θ0 includes the pre-trained parameters of the diffusion model and the motion control network, and Δθ cam is the first rank decomposition matrix.

[0138] In addition, the first video sample can include a plurality of video frames, each video frame being a two-dimensional image. The first video sample can be regarded as a first original image stack stacked by a plurality of video frames in the order of playing. The operation of adding the first noise sample to the first video sample can be regarded as a noise adding process on the first original image stack.

[0139] It should be noted that when the diffusion model adopts an implicit diffusion manner or a non-implicit diffusion manner, the first original image stack needs to be processed by different ways for noise adding and determination of the first loss. The implicit diffusion manner means that the diffusion model performs noise adding and denoising processes in a low-dimensional latent space (such as an implicit space compressed by a VAE encoding network), and the implicit diffusion manner balances the generation quality and the calculation efficiency by operating on the implicit features rather than the original pixels. The non-implicit diffusion manner directly models the diffusion in the original high-dimensional pixel space (such as the RGB image space), and preserves the complete spatial details but requires higher computational resources. The core differences between the two are: 1) operation domain: the implicit diffusion manner processes compressed implicit variables (such as 64x64x4 dimensions), and the non-implicit diffusion manner processes original resolution images (such as 512x512x3); 2) noise adding strategy: noise scheduling in the implicit space is completed for the implicit diffusion manner, and noise is directly applied in the pixel space for the non-implicit diffusion manner; 3) loss calculation: the implicit diffusion manner uses implicit space feature distance (such as LPIPS or latent variable MSE), and the non-implicit diffusion manner uses pixel-level loss (such as L2 loss in the RGB space).

[0140] The implicit diffusion manner is described in detail below, and an encoding network and a decoding network need to be introduced. The encoding network is used to map the input of the diffusion model in the original data space to the target implicit space, and the decoding network is used to restore the output of the diffusion model in the target implicit space to the original data space.

[0141] Therefore, after determining the first original image stack, first, the first original image stack is encoded by the encoding network to obtain a first encoded image stack.

[0142] Then, the first encoded image stack is processed by noise based on the first sample time step and the first noise sample randomly obtained, to obtain a second encoded image stack, wherein the second encoded image stack is equivalent to a noise latent variable of a video only constrained by video shot motion, the first noise sample is randomly generated, and the first noise sample has the same dimension as the first original image stack.

[0143] Then, the second encoded image stack, the first motion feature sample, and the second content feature are input into the diffusion model, and the first sample time step also needs to be input into the diffusion model, and the first predicted noise is obtained by the diffusion model with the first motion feature sample and the second content feature as the constraint condition for noise prediction.

[0144] The determination formula of the second encoded image stack is shown in formula (4):

[0145] wherein z t,cam is the second encoded image stack, t is the first sample time step, and z0,cam is the first noise sample; the determination formula (4) is equivalent to adding noise to the first encoded image stack based on the first sample time step and the first noise sample to obtain a second encoded image stack, and the second encoded image stack is equivalent to the first video sample to which the first noise sample is added.

[0146] wherein, α t is determined in the following manner: β1 to β t are hyperparameters, β1 to β t are used to define the proportion of noise introduced at each step in the diffusion process, β1 to β t are monotonically decreasing, β t The value of α M can be 500, and the disclosure embodiment does not make specific limitations on the value of β t Therefore, as t increases, the value of α t decreases, that is, the value of decreases and the value of increases, is used to represent the degree of preservation of the first original image stack, is used to represent the degree of addition of the first noise sample, and it can be seen that, and are determined by the first sample time step.

[0147] On this basis, the calculation formula of the first loss value is shown in formula (5):

[0148] wherein, is the first loss value, z t,cam is the second encoded image stack, t is the first sample time step, z 0,cam is the first encoded image stack, ∈1 is the first noise sample, θ cam includes the parameters of the diffusion model and the motion control network loaded only with the first rank decomposition matrix, c txt is the first sample text feature, c img is the first sample image feature, and the first sample text feature and the first sample image feature are both second content features, c trajs is the first motion feature sample, is the first predicted noise, is used to calculate the square of the two-norm.

[0149] The non-implicit diffusion mode is described in detail below. After determining the first original image stack, first, based on the first sample time step and the first noise sample obtained randomly, the first original image stack is added with noise to obtain a first added noise image stack, wherein the first noise sample is specifically an image stack stacked by a plurality of randomly generated noise sample images, the first noise sample has the same dimension as the first original image stack, assuming that the first original image stack is stacked by 100 video sample frames, the first noise sample is stacked by 100 noise sample images, and the size of the noise sample image is the same as the size of the video sample frame; then, the first added noise image stack, the first motion feature sample and the second content feature are input into the diffusion model, and the first sample time step also needs to be input into the diffusion model, and the diffusion model is used to predict noise with the first motion feature sample and the second content feature as constraint conditions to obtain a first predicted noise.

[0150] The determination formula of the first added noise image stack is shown in formula (6):

[0151] wherein x t,cam is the first added noise image stack, t is the first sample time step, x 0,cam is the first original image stack, ∈1 is the first noise sample, and α t The determination method can be consistent with the above; the determination formula (6) is equivalent to adding noise to the first original image stack based on the first sample time step and the first noise sample to obtain the first added noise image stack.

[0152] On this basis, the calculation formula of the first loss value is shown in formula (7):

[0153] wherein, is the first loss, x t,cam is the first added noise image stack, t is the first sample time step, x 0,cam is the first original image stack, ∈1 is the first noise sample, θ cam includes the diffusion model and the parameters of the motion control network loaded with only the first rank decomposition matrix, c txt is the first sample text feature, c img is the first sample image feature, the first sample text feature and the first sample image feature are both the second content feature, c trajs is the first motion feature sample, is the first predicted noise, is used to calculate the square of the two norms.

[0154] In a possible implementation, referring to FIG. 10, FIG. 10 is a flowchart for adjusting the second rank decomposition matrix according to an embodiment of the present disclosure; the “adjusting the second rank decomposition matrix loaded by the motion control network based on the second trajectory pattern sample and the second video sample” can be implemented in the following manner: mapping the second trajectory pattern sample through the motion control network loaded with the second rank decomposition matrix and the adjusted first rank decomposition matrix to obtain a second motion feature sample; obtaining a second reference content sample associated with the second video sample, and extracting a third content feature of the second reference content sample; performing noise prediction on the second video sample added with the second noise sample, the second motion feature sample, and the third content feature through the diffusion model to obtain a second predicted noise; determining a second loss value based on the difference between the second noise sample and the second predicted noise, freezing the adjusted first rank decomposition matrix, and adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value.

[0155] In the training of the second rank decomposition matrix, the second trajectory pattern sample is input into the motion control network loaded with the second rank decomposition matrix and the adjusted first rank decomposition matrix, and the second trajectory pattern sample is mapped based on the original parameter matrix, the first rank decomposition matrix, and the second rank decomposition matrix. Since the second trajectory pattern is used to indicate the motion trajectory of the video shot and the motion trajectory of the video display object, the second motion feature sample extracted from the second trajectory pattern sample can guide the diffusion model to simultaneously constrain the motion trajectory of the video shot and constrain the motion trajectory of the video display object in the video generation process.

[0156] The second reference content sample carries information for guiding the generation of specific content of the video, and the second reference content sample can be single-modal data or multi-modal data. For example, the second reference content sample can include one or more of text data, image data, audio data, etc. Assuming that the second reference content sample includes a second sample reference image and a second sample reference text, the first video frame in the second video sample can be taken as the second sample reference image, and the text for describing the content of the second sample reference image can be taken as the second sample reference text, so as to ensure the correlation between the second reference content sample and the second video sample. Based on the second reference content sample, the diffusion model can predict a suitable second predicted noise, which is used to denoise the second video sample added with the second noise sample to restore the original second video sample. The second reference content sample is similar to the target reference content described above, and will not be described here again.

[0157] Based on this, in the process of training the second rank decomposition matrix, the second predicted noise is obtained through the diffusion model prediction, and then the second noise sample is taken as the label data, the second loss value is determined through the difference between the second noise sample and the second predicted noise, and the second loss value is used to guide the optimization of the first rank decomposition matrix and the second rank decomposition matrix, so that the diffusion model can predict more suitable predicted noise, and can generate a video in which the video shot and the video display object are both moving based on the predicted noise. Since the first rank decomposition matrix has been adjusted, the first rank decomposition matrix can be frozen, and only the second rank decomposition matrix is adjusted based on the second loss, so that the motion control network loaded with only the second rank decomposition matrix can learn how to extract the motion features carrying the second key information from the trajectory graph. The second key information can guide the diffusion model to constrain the motion trajectory of the video display object during video generation, thereby helping the diffusion model to better understand the motion trajectory of the video display object, so as to predict more accurate second predicted noise. The accurate second predicted noise can enable the diffusion model to accurately control the motion of the video display object during video generation, achieving fine control of the motion of the video display object.

[0158] For example, since the diffusion model, the original parameter matrix and the first rank decomposition matrix are all frozen when the second rank decomposition matrix is adjusted based on the second loss value, the update gradient of the second loss value on the parameters of the second rank decomposition matrix can be calculated based on the back propagation algorithm, and the second gradient is obtained. Then, the parameters of the second rank decomposition matrix are updated based on the second gradient; when the second rank decomposition matrix is adjusted based on the second loss value, the parameters of the diffusion model and the motion control network are adjusted according to the following formula: mixed = θ0+ sg[Δθ cam ]+Δθ obj

[0159] Wherein, θ mixed includes the parameters of the diffusion model and the motion control network loaded with the first rank decomposition matrix and the second rank decomposition matrix, θ0 includes the parameters of the pre-trained diffusion model and the motion control network, Δθ cam is the first rank decomposition matrix, and Δθ obj is the second rank decomposition matrix, and the gradient flow of Δθ cam is prevented by the sg[] function to freeze the first rank decomposition matrix.

[0160] In addition, since the second video sample includes a plurality of video frames, each video frame is a two-dimensional image, and the second video sample can be regarded as a second original image stack stacked by a plurality of video frames in the playing order. The operation of adding the second noise sample to the second video sample can be regarded as a noise adding process on the second original image stack.

[0161] It should be noted that when the diffusion model adopts the implicit diffusion mode and the non-implicit diffusion mode, the second original image stack needs to be processed by different ways, and the second loss needs to be determined.

[0162] The implicit diffusion mode is described in detail below, and an encoding network and a decoding network need to be introduced. The encoding network is used to map the input of the diffusion model located in the original data space to the target latent space, and the decoding network is used to restore the output of the diffusion model located in the target latent space to the original data space.

[0163] Therefore, after determining the second original image stack, first, the second original image stack is processed by the encoding network to obtain a third encoded image stack;

[0164] Then, the third encoded image stack is processed by adding noise based on the randomly obtained second sample time step and the second noise sample to obtain a fourth encoded image stack, wherein the fourth encoded image stack is equivalent to simultaneously constraining the noise latent variable of the video when the video shot motion and the video display object motion are constrained. The second noise sample is randomly generated, and the second noise sample has the same dimension as the second original image stack;

[0165] Then, the fourth encoded image stack, the second motion feature sample, and the third content feature are input into the diffusion model, and the second sample time step also needs to be input into the diffusion model. The second motion feature sample and the third content feature are used as constraint conditions, and the diffusion model is used for noise prediction to obtain the second predicted noise.

[0166] Specifically, the determination formula of the fourth encoded image stack is as formula (8):

[0167] Wherein, z t,mixed is the fourth encoded image stack, t is the second sample time step, z 0,mixed is the third encoded image stack, ε2 is the second noise sample, and the current α t is determined in the same way as above, and the difference is that the current α t is determined based on the second sample time step; the determination formula (8) is equivalent to adding noise to the third encoded image stack based on the second sample time step and the second noise sample to obtain the fourth encoded image stack, and the fourth encoded image stack is equivalent to the second video sample added with the second noise sample.

[0168] On this basis, the calculation formula of the second loss value is as formula (9):

[0169] Wherein, is the second loss value, z t,mixedis a fourth encoded image stack, t is a second sample time step, z 0,mixed is a third encoded image stack, ∈2 is a second noise sample, θ mixed including a diffusion model and parameters of a motion control network loaded with the first rank decomposition matrix and the second rank decomposition matrix, c txt is a second sample text feature, c img is a second sample image feature, the second sample text feature and the second sample image feature are third content features, c trajs is a first motion feature sample, is a second predicted noise, is used to calculate the square of the two-norm.

[0170] The non-implicit diffusion method is described in detail below. After determining the second original image stack, first, based on the randomly obtained second sample time step and the second noise sample, the second original image stack is added with noise to obtain a second added noise image stack, wherein the first noise sample is an image stack stacked by a plurality of randomly generated noise sample images, the second noise sample has the same dimension as the second original image stack. Assuming that the second original image stack is stacked by 100 video sample frames, the second noise sample is stacked by 100 noise sample images, and the size of the noise sample image is the same as that of the video sample frame. Then, the second added noise image stack, the second motion feature sample, and the third content feature are input into the diffusion model, and the second sample time step also needs to be input into the diffusion model. The diffusion model performs noise prediction with the second motion feature sample and the third content feature as the constraint condition to obtain the second predicted noise.

[0171] wherein the determination formula of the second added noise image stack is shown in formula (10):

[0172] wherein x t,mixed is the second added noise image stack, t is the second sample time step, x 0,mixed is the second original image stack, ∈2 is the second noise sample, and the current a t is determined in a similar manner as above, except that the current a t is determined based on the second sample time step; the determination formula is equivalent to adding noise to the second original image stack based on the second sample time step and the second noise sample to obtain the second added noise image stack.

[0173] On this basis, the calculation formula of the second loss value is formula (11):

[0174] wherein, is the second loss, x t,mixedis a second noisy image stack, t is a second sample time step, x 0,mixed is a second original image stack, ∈2 is a second noise sample, θ mixed including a diffusion model and parameters of a motion control network loaded with the first rank decomposition matrix and the second rank decomposition matrix, c txt is a second sample text feature, c img is a second sample image feature, the second sample text feature and the second sample image feature are third content features, c trajs is a first motion feature sample, is a second predicted noise, is used to calculate the square of the two-norm.

[0175] In some embodiments, the motion control network includes a plurality of cascaded processing layers, and adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value can be implemented by performing the following processing for any layer of the processing layers: applying an orthogonality constraint on the adjusted first rank decomposition matrix and the loaded second rank decomposition matrix located at the processing layer to obtain a third loss value; and adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value and the third loss value.

[0176] Wherein, the processing layer in the motion control network can be a linear layer, or an up-sampling block, a pooling layer, etc., which are not limited in the embodiments of the present disclosure; each processing layer can be configured with an original parameter matrix, the processing layer configured with the first rank decomposition matrix and the second rank decomposition matrix is defined as a target layer, and it is assumed that the number of target layers is N, which is less than or equal to the total number of processing layers.

[0177] Here, the orthogonality constraint is a mathematical constraint condition applied between the first rank decomposition matrix (A) and the second rank decomposition matrix (B) in the same processing layer of the motion control network, which forces the row vector space of the first rank decomposition matrix (A) and the second rank decomposition matrix (B) to maintain an orthogonal relationship (i.e., A·B T ≈0). The first rank decomposition matrix (A) and the second rank decomposition matrix (B) play a role through the following mechanisms: 1) promoting effective propagation of parameter update gradient, avoiding mutual interference of matrix parameter update; 2) accelerating model convergence speed by constructing orthogonal subspace structure; 3) significantly improving training efficiency (through the third loss) while ensuring the optimization effect of the second rank decomposition matrix (controlled by the second loss). The mathematical implementation of the orthogonality constraint can adopt the Frobenius norm form as a regularization term together with the original loss to guide network optimization.

[0178] Based on this, in the process of adjusting the second rank decomposition matrix, by imposing orthogonality constraints on the first rank decomposition matrix and the second rank decomposition matrix located in the same processing layer, the flow of the update gradient of the parameters of the second rank decomposition matrix can be promoted, and the convergence speed can be accelerated. Therefore, adjusting the second rank decomposition matrix based on the second loss and the third loss can improve the training efficiency of the second rank decomposition matrix based on improving the optimization effect of the second rank decomposition matrix by the second loss.

[0179] The calculation formula of the third loss is shown in formula (12):

[0180] Wherein, is the third loss value, W cam is the first rank decomposition matrix of all target layers, W traj is the second rank decomposition matrix of all target layers, W i,cam is the first rank decomposition matrix located in the i-th target layer, W i,traj is the second rank decomposition matrix located in the i-th target layer, is the transpose of W i,traj , I is the unit matrix, is used to calculate the square of the two norms.

[0181] In this way, the embodiments of the present disclosure can force the motion feature spaces of the lens and the object to be decoupled by imposing orthogonality constraints on the adjusted first rank decomposition matrix (controlling the lens motion) and the second rank decomposition matrix to be trained (controlling the object motion) in each processing layer, and jointly optimizing the second loss value (noise prediction error) and the third loss value, thereby avoiding the mutual interference of the two types of motion control. The gradient propagation direction is guided by the orthogonal constraint, which significantly accelerates the training convergence speed of the second rank decomposition matrix. Under the premise of maintaining the performance of the first rank decomposition matrix (frozen), the independence and accuracy of object motion trajectory learning are ensured, and finally the stability and physical reasonableness of the multi-motion type cooperative control in the generated video are improved.

[0182] In some embodiments, "adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value and the third loss value" can be realized by the following way: weighting and summing the second loss value and the third loss value to obtain a fourth loss value; adjusting the second rank decomposition matrix based on the fourth loss value. Based on this, by weighting processing, the contribution degree of the second loss value and the third loss value to the fourth loss value can be adjusted, and the training effect of the second rank decomposition matrix can be improved, thereby improving the performance of the diffusion model.

[0183] The calculation formula of the fourth loss value is shown in formula (13):

[0184] Wherein, is a fourth loss value, is a second loss value, is a third loss value, λ ortho is a weight of the third loss value.

[0185] In some embodiments, the "extracting a first trajectory map sample of the first video sample" can be implemented by: identifying a plurality of candidate motion trajectories in the first video sample; determining a motion intensity of each candidate motion trajectory, determining a sampling probability of each candidate motion trajectory based on the motion intensity, and sampling a preset number of candidate motion trajectories as target motion trajectories based on the sampling probability; determining a difference between trajectory points in the target motion trajectories, and generating the first trajectory map sample of the first video sample based on the difference result.

[0186] The candidate motion trajectories in the first video sample are motion trajectories of video shots of the first video sample, and the motion intensity is used to indicate the distance between the trajectory points in the candidate motion trajectories. The sampling probability of each candidate motion trajectory can be determined according to the motion intensity, which can be the sampling probability of each candidate motion trajectory obtained by normalizing the motion intensity. When sampling randomly based on the sampling probability, the preset number can be between 1 and 8, so as to match the order of magnitude of the motion trajectory of the user input image in the actual generation process. It can be understood that the present disclosure does not limit the preset number. The target motion trajectories can be randomly sampled according to the motion intensity, which can reduce the number of motion trajectories while retaining the target motion trajectories with better quality, thereby improving the generation efficiency of the first trajectory map sample and the sample quality of the finally generated first trajectory map sample.

[0187] In addition, the format of the target motion trajectories randomly sampled can be represented as (n, t, 2), where n is the number of trajectory points, t is the number of frames involved in the target motion trajectory, and 2 represents the coordinates (x, y) of each trajectory point in each frame. In order to improve the orderliness of the target motion trajectories and conform to the input format of the motion control network, the first trajectory map sample of the first video sample is generated according to the calculated difference result, so that the format of the target motion trajectories is converted to (h, w, t-1, 2), where h is the height of the first trajectory map sample, w is the width of the first trajectory map sample, t-1 is the number of time steps between the first frame and the current frame, and 2 represents the coordinates (x, y) of each trajectory point in each frame.

[0188] Thus, the embodiments of the present disclosure achieve the following beneficial effects by dynamically calculating the sampling probability in the plurality of candidate motion trajectories based on the motion intensity and selecting the significantly different trajectory points to generate the trajectory graph sample: 1) adaptively selecting the most representative motion trajectory to exclude noise or small jitter and other low-value motion data, thereby improving the quality of the trajectory graph sample; 2) ensuring the diversity of the distribution of the sampled trajectory points in the time-space dimension through difference analysis, thereby enhancing the generalization ability of the motion control network; and 3) combining the probabilistic sampling mechanism based on the motion intensity, so that the generated trajectory graph sample focuses more on the dominant motion pattern in the video, thereby significantly improving the accuracy of subsequent motion feature extraction and the control accuracy of the motion in the video generation.

[0189] It can be understood that when the second trajectory graph sample of the second video sample is extracted, a similar manner to that of extracting the first trajectory graph sample of the first video sample can also be used, and details are not repeated here.

[0190] In some embodiments, the "obtaining the first video sample" can be implemented by: obtaining an original video sample, identifying a transition frame in the original video sample, wherein the transition frame is used to indicate that a scene in the original video sample changes; based on the transition frame, extracting a target segment showing the same scene from the original video sample, and generating the first video sample based on the target segment.

[0191] The transition frame refers to a video frame in the original video sample that marks a significant change in the scene, and the transition frame indicates the discontinuity of the scenes of the front and rear video paragraphs through visual content mutation (such as lens switching, scene conversion, or subject replacement), and serves as a boundary basis for segmenting the video sample. The transition frame can be automatically identified by detecting picture features (such as HSV histogram mutation, light flow dramatic change, or deep learning feature difference), ensuring that the extracted target segment has strict space-time consistency, thereby eliminating the interference of cross-scene data on the training of the motion control network.

[0192] The original video sample can be a long video, and the transition frame is used to indicate that the scene in the original video sample changes. Generally, since the generative model is sensitive to changes between different video shots, different video shots represent changes in the scene of the video, which easily causes the model to overfit. Therefore, by identifying the transition frame in the original video sample, the video frames located between two adjacent transition frames show consistent video scenes. At this time, the video frames between the two adjacent transition frames can be extracted to obtain a target segment showing the same scene, thereby improving the scene consistency of the first video sample obtained based on the target segment, and effectively improving the training effect when training based on the first video sample.

[0193] For example, assuming that transition frame A, transition frame B, transition frame C and transition frame D are identified in the original video sample in sequence, at this time, the video frames before the transition frame A show the same scene as a target segment a, the video frames between the transition frame A and the transition frame B show the same scene as a target segment b, the video frames between the transition frame B and the transition frame C show the same scene as a target segment c, and the video frames between the transition frame C and the transition frame D show the same scene as a target segment d.

[0194] Thus, the embodiments of the present disclosure generate the first video sample by identifying the transition frames in the original video sample and extracting the target segments with consistent scenes, and achieve the following beneficial effects: 1) effectively eliminating the data distribution difference caused by different scene switching, avoiding the overfitting problem of the model caused by scene mutation; 2) ensuring that the training sample (the first video sample) has strict scene continuity, significantly improving the learning effect of the motion control network on the motion law of the lens in a single scene; 3) by constructing a scene-consistent dataset, enhancing the smoothness and physical rationality of the lens motion in the generated video, and finally improving the stability and controllability of the video generation quality.

[0195] In some embodiments, “generating the first video sample based on the target segments” can be directly taking the target segments as the first video sample, or can be implemented in the following way: reducing the resolution of the target segments, and determining the optical flow between adjacent video frames in the target segments after the resolution is reduced; determining the frame motion score between adjacent video frames based on the norm of the optical flow, and determining the segment motion score of the target segment based on the mean value of multiple motion scores; sorting the multiple target segments in descending order based on multiple segment motion scores, eliminating the target segments located after the preset ranking, and determining the target segments that are not eliminated as the first video sample.

[0196] Among them, the optical flow between adjacent video frames is used to indicate the moving direction and speed of the object in the two continuous video frames, and in the above target segment, the optical flow is used to indicate the moving direction and speed of the video lens. In order to effectively improve the determination efficiency of the optical flow, the resolution of the target segment can be reduced first, for example, the number of pixels contained in the short side of the target segment can be reduced.

[0197] Then, the optical flow between the adjacent video frames can be determined based on a neural network-based optical flow estimator. For example, features of the video frames can be extracted, which can capture information such as texture, color, and shape in the video frames, and then convolution processing can be performed on the extracted features to estimate the motion of the pixels, thereby obtaining the optical flow between the adjacent video frames. After obtaining the optical flow between the adjacent video frames, the frame motion score between the adjacent video frames can be determined according to the norm of the optical flow. At this time, the frame motion score can be used to measure whether the adjacent video frames have undergone significant motion. Then, the average of a plurality of frame motion scores can be calculated to obtain the segment motion score of the entire target segment. At this time, the segment motion score can be used to measure whether the entire target segment has undergone significant motion. Finally, by sorting the segment motion scores, the target segments with relatively insignificant motion can be removed, thereby improving the sample quality of the first video sample. For example, the preset ranking can be 25% of the total number of target segments. It can be understood that the preset ranking is not limited in the embodiments of the present disclosure.

[0198] For example, assuming that the target segment includes video frame Z1, video frame Z2, video frame Z3, and video frame Z4, the frame motion score z1 between video frame Z1 and video frame Z2, the frame motion score z2 between video frame Z2 and video frame Z3, and the frame motion score z3 between video frame Z3 and video frame Z4 are determined respectively, and then the average of the frame motion score z1, the frame motion score z2, and the frame motion score z3 is calculated, thereby obtaining the segment motion score of the target segment.

[0199] It can be understood that when the second video sample is obtained, a similar manner as obtaining the first video sample can be used. At this time, the optical flow between the adjacent video frames in the target segment is used to indicate the moving direction and speed of the video lens and the video display object. The optical flow can be calculated respectively for the video lens and the video display object, and then the average can be calculated as the optical flow between the adjacent video frames in the target segment.

[0200] The complete process of the video generation method will be described in detail below. The video generation method includes a training phase and an inference phase.

[0201] The training phase will be described in detail below.

[0202] First, an original video sample is obtained, and a transition frame in the original video sample is identified. The transition frame is used to indicate that the scene in the original video sample changes.

[0203] Then, based on the transition frame, a target clip showing the same scene is extracted from the original video sample, optical flow between adjacent video frames in the target clip is determined after reducing the resolution of the target clip, frame motion scores between the adjacent video frames are determined according to the norm of the optical flow, clip motion scores of the target clip are determined according to the mean of the plurality of motion scores, the plurality of clip motion scores are sorted in descending order, target clips corresponding to the clip motion scores after the preset ranking are removed, and the target clips that are not removed are determined as a first video sample, wherein the first video sample is a video in which a video shot motion and a video display object are static.

[0204] Then, a plurality of candidate motion trajectories are identified in the first video sample.

[0205] Then, motion intensities of the candidate motion trajectories are determined, sampling probabilities of the candidate motion trajectories are determined according to the motion intensities, and a preset number of candidate motion trajectories are randomly sampled as target motion trajectories based on the sampling probabilities.

[0206] Then, differences between trajectory points in the target motion trajectories are calculated frame by frame, and a first trajectory map sample of the first video sample is generated according to the calculated difference results.

[0207] Then, the motion control network is controlled to load a first rank decomposition matrix.

[0208] Then, the first trajectory map sample is input into the motion control network loaded with the first rank decomposition matrix to obtain a first motion feature sample.

[0209] Then, a first reference content sample associated with the first video sample is obtained, and a second content feature of the first reference content sample is extracted.

[0210] Then, the first video sample to which the first noise sample is added, the first motion feature sample, and the second content feature are input into a diffusion model to perform noise prediction to obtain a first predicted noise.

[0211] Then, a first loss value is determined based on the difference between the first noise sample and the first predicted noise, and the first rank decomposition matrix is adjusted based on the first loss value.

[0212] Then, a second video sample is obtained, and a second trajectory map sample of the second video sample is extracted, wherein the second video sample is a video in which a video shot and a video display object are both moving.

[0213] Then, the motion control network is controlled to simultaneously load a second rank decomposition matrix.

[0214] Then, the second trajectory map sample is input into the motion control network loaded with the adjusted first rank decomposition matrix and the second rank decomposition matrix to obtain a second motion feature sample.

[0215] Then, a second reference content sample associated with the second video sample is obtained, and a third content feature of the second reference content sample is extracted.

[0216] Next, the second video sample to which the second noise sample is added, the second motion feature sample, and the third content feature are input into the diffusion model to perform noise prediction, and a second predicted noise is obtained.

[0217] Then, a second loss value is determined based on a difference between the second noise sample and the second predicted noise, and the first rank decomposition matrix is frozen.

[0218] Next, an orthogonality constraint is applied to the first rank decomposition matrix and the second rank decomposition matrix located in the same processing layer to obtain a third loss value.

[0219] Then, the second rank decomposition matrix is adjusted based on the second loss value and the third loss value.

[0220] The inference stage is described in detail below.

[0221] First, a target trajectory graph is obtained, and the target trajectory graph is input into a motion control network configured with a target rank decomposition matrix, wherein the motion control network is configured with a first rank decomposition matrix for controlling a target motion feature to constrain a motion trajectory of a video shot in a video generation process, and a second rank decomposition matrix for controlling the target motion feature to constrain a motion trajectory of a video display object in the video generation process.

[0222] Then, an operation instruction is obtained.

[0223] Next, the first rank decomposition matrix and / or the second rank decomposition matrix are loaded as the target rank decomposition matrix according to the operation instruction.

[0224] Then, the target trajectory graph is mapped based on the target rank decomposition matrix configured by the motion control network to obtain a target motion feature, wherein the target rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process.

[0225] Next, a target reference content required in the video generation process is obtained, and a first content feature of the target reference content is extracted.

[0226] Then, a target noise image stack is obtained.

[0227] Next, the target noise image stack, the target motion feature, and the first content feature are input into the diffusion model, and a third predicted noise is obtained by performing noise prediction on the diffusion model with the target motion feature and the first content feature as constraint conditions.

[0228] Then, the target noise image stack is input into the diffusion model for noise prediction to obtain fourth predicted noise.

[0229] Then, the third predicted noise is adjusted based on the fourth predicted noise, and the target noise image stack is denoised based on the adjusted third predicted noise to obtain the target video.

[0230] Then, the target noise image stack and the first content feature are input into the diffusion model, and the diffusion model is used to predict noise with the initial motion feature and the first content feature as constraint conditions to obtain fifth predicted noise.

[0231] Then, the third predicted noise is adjusted based on the fourth predicted noise and the fifth predicted noise.

[0232] Finally, the target noise image stack is denoised based on the adjusted third predicted noise to obtain the target video.

[0233] Therefore, the target rank decomposition matrix configured in the motion control network is used to map the target trajectory graph to obtain the target motion feature, the first content feature of the target reference content is extracted, and the diffusion model is used to denoise the target noise image stack with the target motion feature and the first content feature as constraint conditions to obtain the target video constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object. Since the target rank decomposition matrix is used to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process, by configuring the target rank decomposition matrix in the motion control network, the motion decoupling of the video shot and the video display object can be realized in the processing process of the diffusion model, the fine control of the motion of the video shot and / or the video display object is achieved, and thus the quality of the generated video is improved.

[0234] The evaluation process between the video generation method provided by the present disclosure and other generation methods is described in detail below.

[0235] In the first evaluation process, the evaluation indicators of the benchmark model, the motion control model, and the model provided by the embodiments of the present disclosure are determined respectively, as shown in Table 1 below:

[0236] Table 1

[0237] CamMC is used to indicate the matching degree of the motion trajectory of the video shot and the motion trajectory of the target trajectory graph, ObjMC is used to indicate the matching degree of the motion trajectory of the video display object and the motion trajectory of the target trajectory graph, FID is a kind of measurement of the distance between the feature vector of the image of the real video and the feature vector of the image of the generated target video, when FID is lower, the image quality of the target video is higher, FVD is the full name of Frechet Video Distance, FID is a kind of measurement of the distance between the feature vector of the real video and the feature vector of the generated target video, when FVD is lower, the video quality of the target video is higher.

[0238] It can be seen that the CamMC and the ObjMC of the model provided by the embodiment of the present disclosure are better than the benchmark model, the benchmark model refers to the diffusion model and the motion control network without loading the target rank decomposition matrix, the FID and the FVD of the model provided by the embodiment of the present disclosure are the lowest, so the quality of the target video generated by the model provided by the embodiment of the present disclosure is the highest, and the performance of the model provided by the present disclosure is the best.

[0239] In the second evaluation process, the DragUWA model, the DragAnything model, the motion control model and the model provided by the present disclosure are respectively determined to test.

[0240] Referring to FIG. 11 and FIG. 12, FIG. 11 is a generated result schematic diagram of controlling video shot motion provided by the embodiment of the present disclosure, and FIG. 12 is a generated result schematic diagram of controlling video display object motion provided by the embodiment of the present disclosure. It can be seen that in the case of controlling video shot motion, the target video generated by the model provided by the embodiment of the present disclosure can realize more natural video shot motion; and in the case of controlling video display object motion, the target video generated by the model provided by the embodiment of the present disclosure can realize more natural and accurate video display object motion.

[0241] It can be understood that although each step in each of the above flowcharts is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified in the embodiment, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0242] Referring to FIG. 13, FIG. 13 is a structural schematic diagram of a video generation apparatus provided by an embodiment of the present disclosure, the video generation apparatus 1300 comprises:

[0243] a motion control module 1301 configured to obtain a target trajectory graph, and map the target trajectory graph based on a target rank decomposition matrix configured in a motion control network to obtain a target motion feature, wherein the target rank decomposition matrix is used to control the target motion feature to constrain a motion trajectory of a video lens and / or a motion trajectory of a video display object in a video generation process;

[0244] a content control module 1302 configured to obtain a target reference content required in the video generation process, and extract a first content feature of the target reference content;

[0245] a generation module 1303 configured to obtain a target noise image stack, and perform denoising processing on the target noise image stack by taking the target motion feature and the first content feature as constraint conditions through a diffusion model to obtain a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object.

[0246] In some embodiments, the motion control network is configured with a first rank decomposition matrix and / or a second rank decomposition matrix, the first rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video lens in the video generation process, and the second rank decomposition matrix is used to control the target motion feature to constrain the motion trajectory of the video display object in the video generation process. The motion control module 1301 is further configured to: obtain an operation instruction; load the first rank decomposition matrix and / or the second rank decomposition matrix according to the operation instruction, and take the loaded first rank decomposition matrix and / or the second rank decomposition matrix as the target rank decomposition matrix.

[0247] In some embodiments, the video generation apparatus further comprises a training module (not shown in the figure), and the training module is specifically configured to: obtain a first video sample, and extract a first trajectory graph sample of the first video sample, wherein the first video sample is a video in which a video lens moves and a video display object is static; control the motion control network to load the first rank decomposition matrix, and adjust the first rank decomposition matrix loaded by the motion control network based on the first trajectory graph sample and the first video sample; obtain a second video sample, and extract a second trajectory graph sample of the second video sample, wherein the second video sample is a video in which both a video lens and a video display object move; control the motion control network to load the second rank decomposition matrix, and adjust the second rank decomposition matrix loaded by the motion control network based on the second trajectory graph sample and the second video sample.

[0248] In some embodiments, the training module is specifically configured to: map the first trajectory graph sample through the motion control network loaded with the first rank decomposition matrix to obtain a first motion feature sample; obtain a first reference content sample associated with the first video sample, and extract a second content feature of the first reference content sample; perform noise prediction on the first video sample added with a first noise sample, the first motion feature sample, and the second content feature through the diffusion model to obtain a first predicted noise; determine a first loss value based on a difference between the first noise sample and the first predicted noise, and adjust the first rank decomposition matrix loaded by the motion control network based on the first loss value.

[0249] In some embodiments, the training module is specifically configured to: map the second trajectory graph sample through the motion control network loaded with the second rank decomposition matrix and the adjusted first rank decomposition matrix to obtain a second motion feature sample; obtain a second reference content sample associated with the second video sample, and extract a third content feature of the second reference content sample; perform noise prediction on the second video sample added with a second noise sample, the second motion feature sample, and the third content feature through the diffusion model to obtain a second predicted noise; determine a second loss value based on a difference between the second noise sample and the second predicted noise, freeze the adjusted first rank decomposition matrix, and adjust the second rank decomposition matrix loaded by the motion control network based on the second loss value.

[0250] In some embodiments, the training module is specifically configured to: for any layer of the processing layer, perform the following processing: impose an orthogonality constraint on the adjusted first rank decomposition matrix and the loaded second rank decomposition matrix at the processing layer to obtain a third loss value; and adjust the second rank decomposition matrix loaded by the motion control network based on the second loss value and the third loss value.

[0251] In some embodiments, the training module is specifically configured to: identify a plurality of candidate motion trajectories in the first video sample; determine a motion intensity of each of the candidate motion trajectories, determine a sampling probability of each of the candidate motion trajectories based on the motion intensity, and sample a preset number of the candidate motion trajectories as target motion trajectories based on the sampling probability; determine a difference between trajectory points in the target motion trajectories, and generate a first trajectory graph sample of the first video sample based on a difference result.

[0252] In some embodiments, the training module is specifically configured to: obtain an original video sample, and identify a transition frame in the original video sample, wherein the transition frame is used to indicate a change in a scene in the original video sample;

[0253] Based on the transition frame, a target clip showing the same scene is extracted from the original video sample, and a first video sample is generated based on the target clip.

[0254] In some embodiments, the training module is specifically configured to: reduce the resolution of the target clip, and determine the optical flow between adjacent video frames in the target clip after the resolution is reduced; determine the frame motion score between the adjacent video frames based on the norm of the optical flow, and determine the clip motion score of the target clip based on the mean of a plurality of the motion scores; sort a plurality of the target clips in descending order based on a plurality of the clip motion scores, eliminate the target clips after a preset ranking, and determine the target clips that are not eliminated as the first video sample.

[0255] In some embodiments, the generation module 1303 is specifically configured to: perform noise prediction on the target motion feature and the first content feature as constraint conditions through the diffusion model to obtain third predicted noise; perform noise prediction on the target noise image stack through the diffusion model to obtain fourth predicted noise; adjust the third predicted noise based on the fourth predicted noise, and perform denoising processing on the target noise image stack based on the third predicted noise after the adjustment to obtain the target video.

[0256] In some embodiments, the generation module 1303 is specifically configured to: map the target trajectory graph through an original parameter matrix configured by the motion control network to obtain an initial motion feature; perform noise prediction on the initial motion feature and the first content feature as constraint conditions through the diffusion model to obtain fifth predicted noise; and adjust the third predicted noise based on the fourth predicted noise and the fifth predicted noise.

[0257] The video generation apparatus 1300 and the video generation method are based on the same inventive concept, and the target rank decomposition matrix configured in the motion control network is used to map the target trajectory graph to obtain a target motion feature, and then the first content feature of the target reference content is extracted, and then the target noise image stack is denoised by taking the target motion feature and the first content feature as constraint conditions through the diffusion model to obtain a target video constrained by the motion trajectory of the video shot and / or the motion trajectory of the video display object. Since the target rank decomposition matrix is used to constrain the motion trajectory of the video shot and / or the motion trajectory of the video display object in the video generation process, by configuring the target rank decomposition matrix in the motion control network, the motion decoupling of the video shot and the video display object can be realized in the processing process of the diffusion model, the fine control of the motion of the video shot and / or the video display object is achieved, and thus the quality of the generated video is improved.

[0258] The electronic device for executing the video generation method provided in the embodiments of the present disclosure can be a terminal. Referring to FIG. 14, FIG. 14 is a partial structural block diagram of a terminal provided in the embodiments of the present disclosure, which includes a camera assembly 1410, a first storage 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a first processor 1480, and a first power supply 1490, and the like. Those skilled in the art can understand that the terminal structure shown in FIG. 14 does not constitute a limitation on the terminal, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0259] The camera assembly 1410 can be configured to capture images or videos. Optionally, the camera assembly 1410 includes a front camera and a rear camera. Generally, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blur function of the fusion of the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function of the fusion of the main camera and the wide-angle camera, or other fusion shooting functions.

[0260] The first storage 1420 can be configured to store software programs and modules, and the first processor 1480 executes various function applications and data processing of the terminal by running the software programs and modules stored in the first storage 1420.

[0261] The input unit 1430 can be configured to receive input digital or character information and to generate a key signal corresponding to a user's setting or function control command of the terminal. Specifically, the input unit 1430 can include a touch panel 1431 and other input devices 1432.

[0262] The display unit 1440 can be configured to display input or output information and various menus of the terminal. The display unit 1440 can include a display panel 1441.

[0263] The audio circuit 1460, the speaker 1461, and the microphone 1462 can provide an audio interface.

[0264] The first power supply 1490 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.

[0265] The sensor 1450 can be one or more, and can include, but is not limited to, an acceleration sensor, a gyro sensor, a pressure sensor, an optical sensor, etc. Among them:

[0266] The acceleration sensor can detect the magnitude of acceleration in three coordinate axes of a coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravitational acceleration in three coordinate axes. The first processor 1480 can control the display unit 1440 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for game or user motion data collection.

[0267] The gyro sensor can detect the body direction and rotation angle of the terminal, and the gyro sensor can collect 3D user actions on the terminal in cooperation with the acceleration sensor. The first processor 1480 can implement the following functions according to the data collected by the gyro sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0268] The pressure sensor can be disposed on the side frame of the terminal and / or the lower layer of the display unit 1440. When the pressure sensor is disposed on the side frame of the terminal, the user's grip signal on the terminal can be detected, and the left and right hand recognition or shortcut operation can be performed by the first processor 1480 according to the grip signal collected by the pressure sensor. When the pressure sensor is disposed on the lower layer of the display unit 1440, the first processor 1480 can control the operable control on the UI interface according to the user's pressure operation on the display unit 1440. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0269] The optical sensor is configured to collect ambient light intensity. In one embodiment, the first processor 1480 can control the display brightness of the display unit 1440 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1440 is increased; when the ambient light intensity is low, the display brightness of the display unit 1440 is decreased. In another embodiment, the first processor 1480 can also dynamically adjust the shooting parameters of the camera assembly 1410 according to the ambient light intensity collected by the optical sensor.

[0270] In the embodiment, the first processor 1480 included in the terminal can execute the video generation method of the foregoing embodiment.

[0271] The electronic device for executing the video generation method provided in the embodiments of the present disclosure can also be a server. Referring to FIG. 15, FIG. 15 is a partial structural block diagram of a server provided in the embodiments of the present disclosure. The server can have great differences due to different configurations or performances. The server can include one or more second processors 1510 and a second memory 1530, and one or more storage media 1540 (for example, one or more mass storage devices) storing application programs 1543 or data 1542. The second memory 1530 and the storage media 1540 can be temporary storage or persistent storage. The programs stored in the storage media 1540 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the second processor 1510 can be configured to communicate with the storage media 1540 and execute the series of instruction operations in the storage media 1540 on the server.

[0272] The server can also include one or more second power supplies 1520, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1560, and / or one or more operating systems 1541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0273] The second processor 1510 in the server can be configured to execute the video generation method.

[0274] The embodiments of the present disclosure also provide a computer readable storage medium for storing a computer program, and the computer program is configured to execute the video generation method of the foregoing embodiments.

[0275] The embodiment of the present disclosure further provides a computer program product, which comprises a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program, so that the computer device performs the video generation method.

[0276] The terms "first", "second", "third", "fourth" and the like in the description of the present disclosure and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0277] It should be understood that in the present disclosure, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b and c can be single or multiple.

[0278] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. are not included in the number, above, below, within, etc. are understood to include the number.

[0279] In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0280] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0281] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software function unit.

[0282] If the integrated unit is implemented in the form of software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present disclosure essentially or the part that makes a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0283] It should also be understood that the various embodiments provided by the present disclosure can be combined in any manner to achieve different technical effects.

[0284] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit and scope of the present disclosure, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present disclosure.

Claims

1. A video generation method, applied to an electronic device, comprising: Obtain the target trajectory map, and map the target trajectory map based on the target rank decomposition matrix configured in the motion control network to obtain the target motion features. The target rank decomposition matrix is ​​used to control the target motion features to constrain the motion trajectory of the video lens and / or the motion trajectory of the video display object during the video generation process. Obtain the target reference content required in the video generation process, and extract the first content feature of the target reference content; A target noise image stack is obtained. The target motion features and the first content features are used as constraints by a diffusion model to denoise the target noise image stack, thereby obtaining a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object.

2. The video generation method according to claim 1, wherein, The motion control network is configured with a first rank decomposition matrix and / or a second rank decomposition matrix. The first rank decomposition matrix is ​​used to control the target motion features to constrain the motion trajectory of the video shot during video generation, and the second rank decomposition matrix is ​​used to control the target motion features to constrain the motion trajectory of the video display object during video generation. Before mapping the target trajectory map based on the target rank decomposition matrix configured in the motion control network to obtain the target motion features, the video generation method further includes: Obtain operation instructions; The first rank decomposition matrix and / or the second rank decomposition matrix are loaded according to the operation instructions, and the loaded first rank decomposition matrix and / or the second rank decomposition matrix are used as the target rank decomposition matrix.

3. The video generation method according to claim 2, wherein, Before mapping the target trajectory map to obtain the target motion features using the target rank decomposition matrix configured based on the motion control network, the video generation method further includes: Obtain a first video sample and extract a first trajectory map sample from the first video sample, wherein the first video sample is a video in which the video camera moves and the video display object is still; The motion control network is controlled to load the first rank decomposition matrix, and the first rank decomposition matrix loaded by the motion control network is adjusted based on the first trajectory map sample and the first video sample. Obtain a second video sample and extract a second trajectory map sample from the second video sample, wherein the second video sample is a video in which both the video camera and the video display object are in motion; The motion control network is controlled to load the second rank decomposition matrix, and the second rank decomposition matrix loaded by the motion control network is adjusted based on the second trajectory map samples and the second video samples.

4. The video generation method according to claim 3, wherein, The step of adjusting the first rank decomposition matrix loaded by the motion control network based on the first trajectory map samples and the first video samples includes: The first trajectory map sample is mapped by the motion control network loaded with the first rank decomposition matrix to obtain the first motion feature sample; Obtain a first reference content sample associated with the first video sample, and extract the second content features of the first reference content sample; Using the diffusion model, noise prediction is performed on the first video sample with added first noise sample, the first motion feature sample, and the second content feature to obtain the first predicted noise. Based on the difference between the first noise sample and the first predicted noise, a first loss value is determined, and based on the first loss value, the first rank decomposition matrix loaded by the motion control network is adjusted.

5. The video generation method according to any one of claims 3-4, wherein, The step of adjusting the second rank decomposition matrix loaded by the motion control network based on the second trajectory map samples and the second video samples includes: The motion control network loaded with the second rank decomposition matrix and the adjusted first rank decomposition matrix maps the second trajectory map sample to obtain the second motion feature sample. Obtain a second reference content sample associated with the second video sample, and extract a third content feature from the second reference content sample; Using the diffusion model, noise prediction is performed on the second video sample with added second noise sample, the second motion feature sample, and the third content feature to obtain the second predicted noise. A second loss value is determined based on the difference between the second noise sample and the second predicted noise. The adjusted first rank decomposition matrix is ​​frozen. Based on the second loss value, the second rank decomposition matrix loaded by the motion control network is adjusted.

6. The video generation method according to claim 5, wherein, The motion control network includes multiple cascaded processing layers. Adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value includes: For any layer of the processing layer, the following processing is performed: An orthogonality constraint is applied to the adjusted first rank decomposition matrix and the loaded second rank decomposition matrix located in the processing layer to obtain a third loss value; Based on the second loss value and the third loss value, the second rank decomposition matrix loaded by the motion control network is adjusted.

7. The video generation method according to claim 6, wherein, The step of adjusting the second rank decomposition matrix loaded by the motion control network based on the second loss value and the third loss value includes: The second loss value and the third loss value are weighted and summed to obtain the fourth loss value; The second rank decomposition matrix is ​​adjusted based on the fourth loss value.

8. The video generation method according to any one of claims 3-7, wherein, The extraction of the first trajectory map sample from the first video sample includes: Multiple candidate motion trajectories were identified in the first video sample; Determine the motion intensity of each candidate motion trajectory, determine the sampling probability of each candidate motion trajectory based on the motion intensity, and sample a preset number of candidate motion trajectories as the target motion trajectory based on the sampling probability. The differences between trajectory points in the target motion trajectory are determined, and based on the difference results, a first trajectory map sample of the first video sample is generated.

9. The video generation method according to any one of claims 3-8, wherein, The acquisition of the first video sample includes: Acquire original video samples and identify transition frames in the original video samples, wherein the transition frames are used to indicate changes in the scene in the original video samples; Based on the transition frame, a target segment displaying the same scene is extracted from the original video sample, and a first video sample is generated based on the target segment.

10. The video generation method according to claim 9, wherein, The step of generating a first video sample based on the target segment includes: Reduce the resolution of the target segment and determine the optical flow between adjacent video frames in the target segment after the resolution reduction; Based on the norm of the optical flow, the frame motion score between adjacent video frames is determined, and based on the mean of multiple motion scores, the segment motion score of the target segment is determined. Based on the motion scores of multiple segments, the multiple target segments are sorted in descending order, and the target segments ranked after a preset order are removed. The remaining target segments are then determined as the first video sample.

11. The video generation method according to any one of claims 1-10, wherein, The step of using a diffusion model to apply the target motion features and the first content features as constraints to denoise the target noise image stack, thereby obtaining a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object, includes: The third predicted noise is obtained by using the diffusion model to predict noise by taking the target motion features and the first content features as constraints. The target noise image stack is predicted using the diffusion model to obtain the fourth predicted noise; Based on the fourth predicted noise, the third predicted noise is adjusted, and the target noise image stack is denoised based on the adjusted third predicted noise to obtain the target video.

12. The video generation method according to claim 11, wherein, The adjustment of the third prediction noise based on the fourth prediction noise includes: The target trajectory map is mapped using the original parameter matrix configured in the motion control network to obtain initial motion features; The fifth predicted noise is obtained by using the diffusion model to predict noise by taking the initial motion features and the first content features as constraints. The third prediction noise is adjusted based on the fourth and fifth prediction noises.

13. A video generation apparatus, comprising: The motion control module is configured to acquire a target trajectory map, and map the target trajectory map based on the target rank decomposition matrix configured in the motion control network to obtain target motion features. The target rank decomposition matrix is ​​used to control the target motion features to constrain the motion trajectory of the video lens and / or the motion trajectory of the video display object during the video generation process. The content control module is configured to acquire the target reference content required during the video generation process and extract the first content feature of the target reference content; The generation module is configured to acquire a target noise image stack, and use a diffusion model to perform denoising processing on the target noise image stack, using the target motion features and the first content features as constraints, to obtain a target video constrained by the motion trajectory of the video lens and / or the motion trajectory of the video display object.

14. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the video generation method according to any one of claims 1-12.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-12.

Images