Video generation method, motion video generation method of virtual object, video editing method, video generation model training method and information processing method based on video generation model

By introducing timing attention units and parameter cross-attention units in the video generation model, learning camera parameters and timing information, the controllability and stability of video mirrors are achieved, and the difficulty in controlling video mirrors in the existing technology is solved. Problem, the generated video is more realistic and natural, and is suitable for high-standard film and television production.

CN119031207APending Publication Date: 2024-11-26ALIBABA (CHINA) CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410833166.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing video generation technology is difficult to achieve the controllability of video mirror operation, affecting the presentation of video effects and the expression of content. The existing Wensheng video model has shortcomings in controllability, which is difficult to meet the needs of high-standard film and television production.

Method used

Using a video generation model, combining the timing attention unit and the parameter cross-attention unit, through the correlation learning camera parameter control of virtual camera parameters and video frames, a video with a large amplitude and stable timing is generated.

Benefits of technology

The real and natural mirror control of the target video is achieved, and the controllability and stability of video generation is improved, making the generated video more realistic and natural, and suitable for high-standard film and television production scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031207A_ABST
    Figure CN119031207A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a video generation method, a motion video generation method of a virtual object, a video editing method, a video generation model training method and an information processing method based on a video generation model. The video generation method comprises the following steps: acquiring video generation data; and inputting the video generation data into a video generation model to obtain a target video of the target object, the video generation model is obtained by performing parameter adjustment on a parameter coding unit, a parameter cross-attention unit and a time sequence attention unit in the pre-training generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video; the pre-training generation model is obtained by performing parameter adjustment on an object coding unit, an action coding unit and a generation unit in the initial generation model based on a sample video. Virtual camera parameters are input to a model comprising a timing attention unit and a parameter trans-attention unit, thereby generating a timing-stable video conforming to the camera motion trajectory.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to video generation, motion video generation of virtual objects, video editing, video generation model training, and information processing methods based on video generation models. Background Art

[0002] With the development of computer technology, video generation has gradually become an important research topic in multimodality and vision in generative artificial intelligence (AIGC, AI Generated Content), and has broad prospects and application value in scenarios such as digital humans, e-commerce content, and film and television production.

[0003] In the video generation scenario, in addition to keeping the characters' roles consistent and their movements controllable, we also hope to further achieve the controllability of the video camera movement. Good camera movement is crucial to the presentation of video effects and the expression of content. However, since the controllability of video camera movement is relatively complex and a difficult problem to solve, a video generation solution with controllable video camera movement is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides a video generation method. One or more embodiments of the present specification also relate to a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of the present specification, a video generation method is provided, comprising: obtaining video generation data, wherein the video generation data comprises virtual camera parameters, a reference action sequence of a target object, and an object image of the target object; inputting the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video.

[0006] According to a second aspect of an embodiment of the present specification, a method for generating motion video of a virtual object is provided, comprising: obtaining motion video generation data of the virtual object, wherein the motion video generation data comprises virtual camera parameters, a reference action sequence of the virtual object and an image of the virtual object; inputting the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

[0007] According to a third aspect of the embodiments of this specification, a video editing method is provided, including: obtaining an original video and target editing data, wherein the target editing data includes at least one of a target object image, a target camera parameter, and a target action sequence; parsing the original video based on the target editing data to determine the original video data in the original video; inputting the target editing data and the original video data into a video generation model to obtain a target edited video, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video.

[0008] According to a fourth aspect of the embodiments of this specification, a video generation model training method is provided, including: obtaining a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video; inputting the sample object image, the sample action sequence and the sample camera parameter into a pre-trained generation model to obtain a first prediction feature output by a generation unit in the pre-trained generation model, wherein the pre-trained generation model is obtained by adjusting parameters of an object coding unit, an action coding unit and a generation unit in an initial generation model based on the sample video; inputting the sample video into an image coding unit in the pre-trained generation model to obtain a first sample feature; adjusting unit parameters of a parameter coding unit, a temporal attention unit and a parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0009] According to a fifth aspect of the embodiments of this specification, there is provided an information processing method based on a video generation model, comprising: receiving a task generation request, wherein the task generation request comprises request information; based on the request information, obtaining a video generation model, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video; generating task information based on the video generation model, wherein the task information is used to execute a target video task.

[0010] According to a sixth aspect of an embodiment of this specification, a task platform is provided, including a request interface and a response unit; the request interface is used to receive a task generation request, wherein the task generation request includes request information; the response unit is used to obtain a video generation model based on the request information, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video; based on the video generation model, task information is generated, wherein the task information is used to execute a target video task.

[0011] According to a seventh aspect of the embodiments of this specification, a video generation device is provided, comprising: a first acquisition module, configured to acquire video generation data, wherein the video generation data comprises virtual camera parameters, a reference action sequence of a target object and an object image of the target object; a first input module, configured to input the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

[0012] According to an eighth aspect of the embodiments of this specification, a motion video generation device for a virtual object is provided, comprising: a second acquisition module, configured to acquire motion video generation data of the virtual object, wherein the motion video generation data comprises virtual camera parameters, a reference action sequence of the virtual object and an image of the virtual object; a second input module, configured to input the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

[0013] According to a ninth aspect of an embodiment of the present specification, a video editing device is provided, comprising: a third acquisition module, configured to acquire an original video and target editing data, wherein the target editing data comprises at least one of a target object image, a target camera parameter and a target action sequence; a parsing module, configured to parse the original video based on the target editing data to determine the original video data in the original video; a third input module, configured to input the target editing data and the original video data into a video generation model to obtain a target edited video, wherein the video generation model adjusts parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model adjusts parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

[0014] According to the tenth aspect of the embodiments of the present specification, a video generation model training device is provided, including: a fourth acquisition module, configured to acquire a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video; a fourth input module, configured to input the sample object image, the sample action sequence and the sample camera parameter into a pre-trained generation model, and obtain a first prediction feature output by a generation unit in the pre-trained generation model, wherein the pre-trained generation model is obtained by adjusting parameters of an object coding unit, an action coding unit and a generation unit in an initial generation model based on the sample video; a fifth input module, configured to input the sample video into an image coding unit in the pre-trained generation model, and obtain a first sample feature; an adjustment module, configured to adjust unit parameters of a parameter coding unit, a temporal attention unit and a parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature, and obtain a trained video generation model.

[0015] According to the eleventh aspect of the embodiments of this specification, there is provided an information processing device based on a video generation model, comprising: a receiving module, configured to receive a task generation request, wherein the task generation request includes request information; a fifth acquisition module, configured to acquire the video generation model based on the request information, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video; and a generation module, configured to generate task information based on the video generation model, wherein the task information is used to execute a target video task.

[0016] According to the twelfth aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions, and the computer program / instructions, when executed by the processor, implement the steps of the method provided in the above-mentioned first aspect or second aspect or third aspect or fourth aspect or fifth aspect.

[0017] According to the thirteenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method provided in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect are implemented.

[0018] According to the fourteenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect.

[0019] The video generation method provided by an embodiment of the present specification includes: obtaining video generation data, wherein the video generation data includes virtual camera parameters, a reference action sequence of a target object, and an object image of the target object; inputting the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model is obtained by adjusting the parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting the parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video. By inputting virtual camera parameters into the video generation model, the camera movement control of the target video is realized, and since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target video that supports a large camera movement effect and has stable timing, making the target video more realistic and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is an architecture diagram of a video generation system provided by an embodiment of this specification;

[0021] Figure 2 is a flow chart of a video generation method provided by an embodiment of this specification;

[0022] Figure 3 is a processing flow chart of a video generation model training method provided by an embodiment of this specification;

[0023] Figure 4 is a processing flow chart of a pre-training generative model training method provided by an embodiment of this specification;

[0024] Figure 5 is a flow chart of a method for generating a motion video of a virtual object provided by one embodiment of the present specification;

[0025] Figure 6 is a flow chart of a video editing method provided by one embodiment of this specification;

[0026] Figure 7 is a flow chart of a video generation model training method provided by an embodiment of this specification;

[0027] Figure 8 is a flowchart of an information processing method based on a video generation model provided by an embodiment of this specification;

[0028] Fig. 9 It is a structural diagram of a task platform provided by an embodiment of this specification;

[0029] Fig.10 is a structural schematic diagram of a video generating device provided by an embodiment of this specification;

[0030] Fig.11 It is a structural schematic diagram of a motion video generation device of a virtual object provided by an embodiment of the present specification;

[0031] Fig.12 is a structural schematic diagram of a video editing device provided by an embodiment of this specification;

[0032] Fig.13 It is a structural diagram of a video generation model training device provided by an embodiment of this specification;

[0033] Fig.14 is a structural schematic diagram of an information processing device based on a video generation model provided by an embodiment of this specification;

[0034] Fig.15 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0035] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0036] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model (Foundation Model 1). The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model 1), a multi-modal pre-training model (multi-modal pre-training model 1), etc.

[0040] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0041] First, the terms involved in one or more embodiments of this specification are explained.

[0042] Cross Attention: An attention mechanism that handles dependencies between two different data sources or sequences, such as injecting data of different modalities into an image / video generation network. The cross-attention mechanism allows the model to selectively focus on relevant parts of the input data when generating output, thereby more effectively utilizing contextual information.

[0043] Deep Self-Attention (Transformer) Model: It is a network structure based on a multi-head self-attention mechanism module, mainly used to process sequence data. The Transformer model consists of a number of encoders and decoders that can be repeatedly stacked. This design allows the Transformer to efficiently learn long-term dependencies and is suitable for a variety of natural language processing tasks including machine translation, text summarization, and question-answering systems.

[0044] Stable Diffusion Model: It is a generative model based on generative adversarial networks. By gradually increasing the noise disturbance, the generated image becomes clearer and more realistic.

[0045] Low-Rank Adaptation (Lora): It is a low-rank adaptive expression in large models, which can use a smaller number of parameters to allow large models to add additional parameters and functions.

[0046] Text to video (T2V): Automatically convert written text content into video content containing visual and auditory elements.

[0047] Convolutional Neural Network (CNN): A deep learning model suitable for processing data with grid structure, such as images, videos, and speech signals.

[0048] Camera trajectory: refers to the path of the camera's position and direction changes relative to time in three-dimensional space. The camera trajectory contains a series of camera poses that change over time. Each pose is usually composed of camera parameters such as position (transition) and rotation (rotation), which can be used to describe the camera's movement in the scene and the change of the viewing direction.

[0049] In recent years, thanks to the rapid development and accumulation of model frameworks such as Transformer and Stab le Diffus ion, graphics processing unit (GPU) hardware, and big data content, AIGC has made great progress in generation effects. Taking the Vincent video model as an example, the Vincent video model can generate corresponding videos based on text input, and it can be relatively long-term, stable, and the effect can be indistinguishable from the real thing. However, the current Vincent video model has poor controllability results, and many Vincent video models often need to run multiple times to get a relatively satisfactory result. Ensuring the effect and strong controllability are the problems that the generation model needs to solve in actual implementation. Therefore, controllable video generation has gradually become a research focus, especially for downstream content production, film and television production and other high-standard scenes, the controllability of video content is a very important hard requirement. Among the various controllable input conditions, the camera motion trajectory is a very important parameter. In the process of video production, camera motion control is a very important component. Good camera movement is crucial to the presentation of video effects and the expression of content.

[0050] Based on this, the embodiment of this specification proposes a video generation scheme using a video generation model, which includes a temporal attention unit and a parameter cross-attention unit. Through the parameter cross-attention unit, the model can associate virtual camera parameters and video frames in the feature dimension, learn the control and generation of camera parameters, and thus support a large-scale camera movement effect. Through the temporal attention unit, the video generation model can learn the timing information, so that the generated video timing transitions smoothly, making it more realistic and natural.

[0051] In this specification, a video generation method is provided. This specification also involves a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0052] See also Figure 1 , Figure 1 The architecture diagram of a video generation system provided by an embodiment of the present specification is shown. The video generation system may include a client 100 and a server 200;

[0053] The client 100 is used to send video generation data to the server 200, wherein the video generation data includes virtual camera parameters, a reference action sequence of a target object, and an object image of the target object;

[0054] The server 200 is used to input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model is obtained by adjusting the parameters of the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video; sending the target video to the client 100;

[0055] The client 100 is also used to receive the target video sent by the server 200.

[0056] By applying the solution of the embodiments of this specification, the camera movement control of the target video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target video that supports a large-amplitude camera movement effect and has stable timing, making the target video more realistic and natural.

[0057] In practical applications, the video generation system may include multiple clients 100. Multiple clients 100 may establish a communication connection through a server 200. In the video generation scenario, the server 200 is used to provide video generation services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and realize communication through the server 200. Users may interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the video generation scenario, the user may publish a data stream to the server 200 through the client 100, and the server 200 may generate a target video according to the data stream and push the target video to other clients that have established communication. Among them, the client 100 and the server 200 are connected through a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be processed by encoding, transcoding, compression, etc. before being issued to the server 200. It is worth noting that the video generation method provided in the embodiment of this specification is generally performed by the server, but, in other embodiments of this specification, the client can also have similar functions with the server, thereby performing the video generation method provided by the embodiment of this specification. In other embodiments, the video generation method provided by the embodiment of this specification can also be performed by the client and the server together. Next, the video generation method proposed by the server performing the embodiment of this specification is described in detail.

[0058] See also Figure 2 , Figure 2 A flowchart of a video generation method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0059] Step 202: Acquire video generation data, wherein the video generation data includes virtual camera parameters, a reference action sequence of the target object, and an object image of the target object.

[0060] In one or more embodiments of the present specification, when generating a video, video generation data may be obtained, and a target video meeting the requirements may be generated based on the video generation data.

[0061] Specifically, video generation data refers to a data set used to generate a new target video. Video generation data can be data of different video tasks, including but not limited to video generation tasks, video camera adjustment tasks, video object replacement tasks, and video action replacement tasks. Video tasks can be tasks in motion video scenes, tasks in film and television video editing scenes, tasks in multimedia content production scenes, and the like. Target objects refer to objects that appear in target videos, and target objects can be people, animals, virtual characters, and the like. Object images refer to images of target objects, and can also be understood as video foreground images. For example, object images can be images of people standing. The number of object images can be one or more, for example, the object image includes a front image and a side image of the target object. The reference action sequence includes multiple reference action images, and each reference action image contains multiple action key points. By referring to the action sequence, the action or posture (such as standing, lying flat) to be displayed by the target object in the target video can be defined. Virtual camera parameters can also be called virtual camera trajectory parameters. Virtual camera parameters are a series of data used to describe the motion state of the virtual camera itself when the virtual camera moves or shoots dynamic scenes. The motion state of the virtual camera itself can be controlled by the virtual camera parameters, thereby controlling the camera trajectory of the generated video. Specifically, the virtual camera parameters include the camera external parameters at each moment t in the entire T time period, such as the parameters of the position and posture of the virtual camera in the world coordinate system, namely the rotation matrix (Rotat ion Matrix) and the translation vector (Trans l at ion Vector). Among them, the rotation matrix is ​​used to describe the rotation direction and angle of the camera coordinate system relative to the world coordinate system. It maintains the distance and direction relationship in space to ensure that no scale deformation occurs during the rotation process. The translation vector represents the position of the origin of the camera coordinate system in the world coordinate system. Through the translation vector and the rotation matrix, any point in the world coordinate system can be converted to the camera coordinate system by translation and rotation.

[0062] In practical applications, there are many ways to obtain video generation data, which can be selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. In the first possible implementation of this specification, the video generation data sent by the user through the client can be received. In the second possible implementation of this specification, since it is difficult for the user to directly construct a reference action sequence and virtual camera parameters, multiple candidate action sequences and candidate camera parameters can be provided to the user to determine the reference action sequence selected by the user from the multiple candidate action sequences and the virtual camera parameters selected from the multiple candidate camera parameters. In the third possible implementation of this specification, the user can send a reference video and an object image through the client, and the server can parse the reference action sequence and / or virtual camera parameters from the reference video.

[0063] In an optional embodiment of the present specification, the obtaining of video generation data may include the following steps:

[0064] receiving a reference video and an object image of a target object sent by a client;

[0065] Input the reference video into the action extraction model to obtain a reference action sequence;

[0066] The reference video input parameter extraction model is used to obtain the virtual camera parameters.

[0067] Specifically, the action extraction model is used to extract action sequences in the input video. The action extraction model can be a deep learning model trained with the CNN model as the base. The parameter extraction model is used to extract camera parameters in the input video. The action extraction model and the parameter extraction model can be deep learning models trained with the CNN model as the base. For example, the action extraction model can be trained based on sample videos and sample action sequences carried by the sample videos. The parameter extraction model can be trained based on sample videos and sample camera parameters carried by the sample videos.

[0068] It should be noted that the reference action sequence and / or virtual camera parameters can be parsed from the reference video, so the reference video may include the target object or may not include the target object. For example, if the target object is object A, the reference video may be a dancing video of object A, the reference video may also be a dancing video of object B, and similarly, the reference video may also be a landscape video that does not include any object.

[0069] By applying the solution of the embodiments of this specification, the user only needs to provide a reference video and an object image of the target object, and the server determines a reference action sequence and virtual camera parameters based on the reference video, thereby reducing the difficulty of user data provision and lowering the threshold for video generation.

[0070] Step 204: Input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model adjusts the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video.

[0071] In one or more embodiments of the present specification, after obtaining the video generation data, further, the reference action sequence, virtual camera parameters and object image of the target object may be input into the video generation model to obtain the target video of the target object.

[0072] Specifically, the target video refers to a video in which the action of the target object conforms to the reference action sequence and the video camera effect conforms to the virtual camera parameters. The training methods of the video generation model and the pre-trained generation model are both supervised training. The video generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a parameter encoding unit, a generation unit and a decoding unit. The video generation model is obtained by adjusting the parameters of the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video.

[0073] In an optional embodiment of the present specification, the structure of the video generation model is described. That is, the video generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a parameter encoding unit, a generation unit, and a decoding unit; the above-mentioned inputting the video generation data into the video generation model to obtain the target video of the target object may include the following steps:

[0074] Inputting the object image into the image coding unit to obtain image coding features;

[0075] Inputting the image coding feature into the object coding unit to obtain the object coding feature;

[0076] Inputting the reference action sequence into the action encoding unit to obtain an action encoding sequence;

[0077] Inputting the virtual camera parameters into the parameter encoding unit to obtain parameter encoding features;

[0078] Inputting the object coding feature, the action coding sequence and the parameter coding feature into the generation unit to obtain the video coding feature;

[0079] The video encoding features are input into the decoding unit to obtain the target video of the target object.

[0080] It should be noted that the image coding unit is connected to the object coding unit, the object coding unit is connected to the generation unit, the action coding unit is connected to the generation unit, the parameter coding unit is connected to the generation unit, and the generation unit is connected to the decoding unit. The image coding unit, the action coding unit and the parameter coding unit are composed of CNN network layers. The object coding unit and the generation unit are composed of Transformer network modules. The generation unit can be constructed based on the UNet network of Stab le Diffus ion, or can be constructed based on Video Stab le Diffus ion. The image coding unit is used to extract features from the object image to obtain image coding features. The object coding unit is used to perform feature processing on the image coding features to obtain object coding features. The action coding unit is used to extract features from the reference action sequence to obtain an action coding sequence. The parameter coding unit is used to extract features from the virtual camera parameters to obtain parameter coding features.

[0081] By applying the solution of the embodiment of this specification, an object coding unit is connected after the image coding unit to perform object control on the video coding features, thereby ensuring that the object in each frame of the generated target video is consistent with the target object.

[0082] In an optional embodiment of the present specification, the structure of the object coding unit is described. That is, the object coding unit includes multiple coding blocks; the above-mentioned inputting the image coding feature into the object coding unit to obtain the object coding feature may include the following steps:

[0083] Inputting the image coding feature into a first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is a first coding block among the multiple coding blocks;

[0084] Inputting the coding feature output by the previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks;

[0085] The object coding feature is determined according to the coding features respectively output by the multiple coding blocks.

[0086] Specifically, the coding block includes a self-attention unit and a cross-attention unit. When the image coding feature is input into the first coding block to obtain the first coding feature output by the first coding block, the image coding feature can be input into the self-attention unit in the coding block to obtain the image self-attention feature, and the image self-attention feature can be input into the cross-attention unit in the coding block to obtain the first coding feature.

[0087] Exemplarily, it is assumed that the object coding unit includes three sequentially connected coding blocks, which are coding block 1, coding block 2, and coding block 3. The process of inputting the image coding feature into the object coding unit may include: first, inputting the image coding feature into coding block 1 to obtain coding feature 1; then, inputting coding feature 1 into coding block 2 to obtain coding feature 2; finally, inputting coding feature 2 into coding block 3 to obtain coding feature 3. Coding feature 1, coding feature 2, and coding feature 3 together constitute the object coding feature.

[0088] By applying the solution of the embodiment of this specification, an object coding unit is connected after the image coding unit to perform object control on the video coding features, thereby ensuring that the object in each frame of the generated target video is consistent with the target object.

[0089] In an optional embodiment of the present specification, the structure of the generation unit is described. That is, the generation unit includes multiple generation blocks, the object coding unit includes multiple coding blocks, and the generation blocks and the coding blocks correspond to each other one by one; the above-mentioned inputting the object coding feature, the action coding sequence and the parameter coding feature into the generation unit to obtain the video coding feature may include the following steps:

[0090] According to the corresponding relationship between the generated block and the encoded block, the encoding feature corresponding to the generated block is determined;

[0091] Inputting the action coding sequence, the parameter coding feature, and the coding feature corresponding to the first generation block into the first generation block to obtain a first generation feature output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks;

[0092] Inputting the generated feature output by the previous generation block of the second generation block, the coding feature corresponding to the second generation block, and the parameter coding feature into the second generation block to obtain the second generated feature output by the second generation block, wherein the second generation block is any generation block other than the first generation block among the multiple generation blocks;

[0093] In a case where the second generation block is the last generation block among the multiple generation blocks, a second generation feature output by the second generation block is determined as the video encoding feature.

[0094] It should be noted that, by extracting the feature expressions of object images at different layers through the object encoding unit, and using the cross-attention mechanism in the same layer of the generation unit to fuse the object features into the generation unit, the objects in each frame of the generated target video can be consistent with the target object.

[0095] Exemplarily, it is assumed that the object encoding unit includes three sequentially connected encoding blocks, which are encoding block 1, encoding block 2, and encoding block 3. The generation unit includes three sequentially connected generation blocks, which are generation block 1, generation block 2, and generation block 3. The image encoding feature is input into the object encoding unit, and the encoding feature 1 output by encoding block 1, the encoding feature 2 output by encoding block 2, and the encoding feature 3 output by encoding block 3 can be obtained. After encoding block 1 outputs encoding feature 1, the cross-attention mechanism can be used to fuse encoding feature 1 into the input of generation block 1. For generation block 1, the action encoding sequence, parameter encoding feature, and encoding feature 1 are input into generation block 1, and generation block 1 outputs generated feature 1. After encoding block 2 outputs encoding feature 2, the cross-attention mechanism can be used to fuse encoding feature 2 into the input of generation block 2. For generation block 2, the generated feature 1, parameter encoding feature, and encoding feature 2 are input into generation block 2, and generation block 2 outputs generated feature 2. After encoding block 3 outputs encoding feature 3, the cross-attention mechanism can be used to fuse encoding feature 3 into the input of generation block 3. For generation block 3, generation feature 2, parameter coding feature and coding feature 3 are input into generation block 3, and generation block 3 outputs generation feature 3. Since generation block 3 is the last generation block, generation feature 3 is determined as the video coding feature.

[0096] By applying the solution of the embodiment of this specification, by utilizing the cross-attention mechanism, the generation process of each generation block is constrained by the encoding features output by the corresponding encoding block, so that the generated video has object consistency.

[0097] In an optional embodiment of the present specification, the structure of the generation block in the generation unit is described. Each generation block includes a self-attention unit, a cross-attention unit, a temporal attention unit, and a parameter cross-attention unit. Taking the second generation block as an example, the second generation block includes a self-attention unit, a cross-attention unit, a temporal attention unit, and a parameter cross-attention unit; the above-mentioned input of the generation feature output by the previous generation block of the second generation block, the encoding feature corresponding to the second generation block, and the parameter encoding feature into the second generation block to obtain the second generation feature output by the second generation block can include the following steps:

[0098] Input the generated features output by the previous generation block of the second generation block into the self-attention unit to obtain the self-attention features;

[0099] Input the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain the cross-attention feature;

[0100] Input the cross-attention feature into the temporal attention unit to obtain the temporal attention feature;

[0101] The temporal attention features and parameter encoding features are input into the parameter cross-attention unit to obtain the second generated features.

[0102] Specifically, the self-attention unit is connected to the cross-attention unit, the cross-attention unit is connected to the temporal attention unit, and the temporal attention unit is connected to the parameter cross-attention unit. For example, in the second generation block, the temporal attention unit is after the cross-attention unit, the self-attention unit is before the cross-attention unit, and the parameter cross-attention unit is after the temporal attention unit. The self-attention unit is used to process data using the self-attention mechanism. The self-attention mechanism allows the self-attention unit to consider how each position in the input data (such as different regions of the image or different parts of the feature map) depends on all other positions. The self-attention mechanism can help the model identify and utilize the correlation between distant pixels. The cross-attention unit is used to process data using the cross-attention mechanism. The cross-attention mechanism can help the model guide the video generation process based on conditional information (such as parameter encoding features, action feature sequences). The cross-attention mechanism allows the model to align the conditional information with the image features to ensure that the generated image meets the given conditions. The temporal attention unit is used to utilize the attention mechanism in the time dimension so that the temporal order of the features is correlated with each other, making the generated video result more stable. The purpose of the parameter cross-attention unit is to enable the model to learn the camera motion trajectory. Given the virtual camera parameters, the virtual camera parameters are first processed by the parameter encoding unit to obtain parameter encoding features, and then the parameter encoding features are injected into the generation unit through the cross-attention mechanism, so that the video encoding features can learn the correspondence with the camera parameters, and the final generated video conforms to the input camera motion trajectory.

[0103] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target video that supports a large-amplitude camera movement effect and has stable timing, making the target video more realistic and natural.

[0104] In an optional embodiment of the present specification, after inputting the video generation data into the video generation model to obtain the target video of the target object, the following steps may also be included:

[0105] Send the target video to the client;

[0106] Receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client regarding the target video;

[0107] Based on the feedback information from the results, build a model to optimize the data;

[0108] Use model optimization data to adjust parameters of the video generation model.

[0109] Specifically, the result feedback information can be information that provides feedback on the content, quality, and completion of the target video, reflecting the client's true feelings and expectations of the target video. The result feedback information includes but is not limited to quality evaluation information, corrected accurate target video, and optimization areas of the model. Model optimization data refers to accurate optimized sample videos used to optimize the video generation model.

[0110] In practical applications, there are many ways to construct model optimization data based on the result feedback information, which are selected according to the actual situation. The embodiments of this specification do not limit this. In one possible implementation of this specification, the model optimization data can be automatically constructed directly based on the result feedback information. In another possible implementation of this specification, optimization prompt information can be generated based on the result feedback information, and the model optimization data sent by the client based on the optimization prompt information can be received.

[0111] Exemplarily, when generating optimization prompt information based on result feedback information, in one possible implementation of this specification, the preset prompt information can be directly obtained, and the result feedback information can be added to the preset prompt information to obtain the optimization prompt information, such as the preset prompt information is "I am very sorry for bringing you an inaccurate video. Please point out what is not accurate or provide an accurate video. I will correct and optimize my answer as soon as possible to better serve you." If the result feedback information is "poor video camera movement", the optimization prompt information is "Regarding the problem of poor video camera movement you reported, I am very sorry for bringing you a poor video. Please point out what is not accurate or provide an accurate video. I will correct and optimize my answer as soon as possible to better serve you." In another possible implementation of this specification, the result feedback information can be type identified to determine the information type of the result feedback information, and the information type can be further matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type as the information type is determined as the optimization prompt information.

[0112] Furthermore, when constructing model optimization data directly based on the result feedback information, if the result feedback information is the corrected accurate target video, the corrected accurate target video can be determined as the model optimization data. If the result feedback information is the optimization field of the model, such as the XXX field, multiple sample videos in the XXX field can be obtained, and the multiple sample videos in the XXX field can be determined as the model optimization data.

[0113] By applying the solution of the embodiments of this specification, the performance of the video generation model is continuously optimized by collecting and utilizing result feedback information to more accurately meet the actual needs of the client and improve the quality and accuracy of the final target video.

[0114] In an optional embodiment of the present specification, a training method of a video generation model is described. That is, before inputting the video generation data into the video generation model to obtain the target video of the target object, the following steps may also be included:

[0115] Obtaining sample videos and sample object images, sample action sequences, and sample camera parameters corresponding to the sample videos;

[0116] Inputting a sample object image, a sample action sequence, and a sample camera parameter into a pre-trained generative model to obtain a first prediction feature output by a generative unit in the pre-trained generative model;

[0117] Inputting the sample video into the image encoding unit in the pre-trained generative model to obtain a first sample feature;

[0118] According to the first prediction feature and the first sample feature, the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model are adjusted to obtain a trained video generation model.

[0119] Specifically, the first sample feature is the generation target of the generation unit in the pre-trained generation model, which is used to guide the training process of the pre-trained generation model. When training the pre-trained generation model, an additional temporal attention unit and a parameter cross-attention unit are added to each generation block of the generation unit in the pre-trained generation model, that is, the pre-trained generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a parameter encoding unit, a generation unit, and a decoding unit; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit, a cross-attention unit, a temporal attention unit, and a parameter cross-attention unit.

[0120] It should be noted that there are multiple ways to obtain sample videos and sample object images, sample action sequences, and sample camera parameters corresponding to the sample videos. In one possible implementation of this specification, sample videos and sample object images, sample action sequences, and sample camera parameters corresponding to the sample videos can be read from other data acquisition devices or databases. In another possible implementation of this specification, sample videos can be acquired, the sample videos can be parsed, and sample object images, sample action sequences, and sample camera parameters can be determined. Among them, there are multiple ways to acquire sample videos. In one possible implementation of this specification, multiple sample videos sent by a user through a client can be received. In another possible implementation of this specification, multiple sample videos can be read from other data acquisition devices or databases.

[0121] Furthermore, after obtaining the sample video, for the sample object image: any video frame including the sample object can be randomly selected from the sample video as the sample object image, or a video frame including the sample object with higher quality can be selected as the sample object image. For the sample action sequence: the sample video can be input into the action extraction model to obtain a sample action sequence; or the sample video frame can be extracted from the sample video, and for each sample video frame, the sample video frame can be input into the key point recognition model to obtain the sample action key point, and the sample action image including the object skeleton can be drawn. The sample action image corresponding to each sample video frame can constitute the sample action sequence. For the sample camera parameters: the sample video can be input into the parameter extraction model to obtain the sample camera parameters.

[0122] In practical applications, when adjusting the unit parameters of the parameter encoding unit, the temporal attention unit, and the parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature, the loss value can be calculated according to the first prediction feature and the first sample feature, and the unit parameters of the parameter encoding unit, the temporal attention unit, and the parameter cross-attention unit can be adjusted according to the loss value in a low-rank adaptive manner until the training process meets the preset stop condition, and a trained video generation model is obtained. Among them, there are many functions for calculating the loss value, such as the cross entropy loss function, the L1 norm loss function, the maximum loss function, the mean square error loss function, the logarithmic loss function, etc., which are selected according to the actual situation, and the embodiments of this specification do not make any restrictions on this. The preset stop conditions include but are not limited to the loss value being less than or equal to the preset threshold and the number of iterations reaching the preset number of iterations, wherein the preset threshold and the preset number of iterations are selected according to the actual situation, and the embodiments of this specification do not make any restrictions on this.

[0123] In a possible implementation of the present specification, after calculating the loss value, the loss value is compared with a preset threshold. Specifically, if the loss value is greater than the preset threshold, it means that the difference between the first prediction feature and the first sample feature is large, and the pre-trained generation model has poor prediction ability for video features. At this time, the unit parameters of the parameter encoding unit, the temporal attention unit, and the parameter cross-attention unit can be adjusted, and the sample object image, the sample action sequence, and the sample camera parameter are input into the pre-trained generation model to obtain the step of the first prediction feature output by the generation unit in the pre-trained generation model, and the pre-trained generation model is continuously trained until the loss value is less than or equal to the preset threshold, indicating that the difference between the target first prediction feature and the first sample feature is small, and the preset stop condition is met, and the trained video generation model is obtained.

[0124] In another possible implementation of the present specification, in addition to comparing the size relationship between the loss value and the preset threshold, the number of iterations can also be combined to determine whether the current pre-trained generation model has been trained. Specifically, if the loss value is greater than the preset threshold, the unit parameters of the parameter encoding unit, the temporal attention unit, and the parameter cross-attention unit are adjusted, and the sample object image, the sample action sequence, and the sample camera parameter are input into the pre-trained generation model to obtain the first prediction feature output by the generation unit in the pre-trained generation model. The pre-trained generation model is continued to be trained until the preset number of iterations is reached, and the iteration is stopped to obtain a video generation model that has been trained.

[0125] By applying the solution of the embodiments of the present specification, additional temporal attention units and parameter cross-attention units are added to each generation block of the generation unit in the pre-trained generation model, so that the trained video generation model can learn timing information and camera parameters, thereby generating a target video that supports a large-amplitude camera movement effect and has stable timing, making the target video more realistic and natural.

[0126] See also Figure 3 , Figure 3 A processing flow chart of a video generation model training method provided by an embodiment of the present specification is shown, and the video generation model is obtained by training the pre-trained generation model. This training stage can be called the video generation training stage. In this stage, the model training goal is to generate a video clip with stable time sequence, and at the same time be able to control the camera movement. Therefore, a temporal attention unit and a parameter cross-attention unit are introduced in the pre-trained generation model. By adding the temporal attention unit, the relevant associations between different frames in the temporal dimension can be learned, and the video generation model can further learn the temporal information, such as the fluttering hem and the continuous and stable temporal change characteristics. By adding the parameter cross-attention unit, the model learns the information of the camera trajectory and generates a video that meets the input camera parameters. That is, the pre-trained generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a parameter encoding unit, a generation unit and a decoding unit; the object encoding unit includes a plurality of encoding blocks, and the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and the generation block includes a self-attention unit, a cross-attention unit, a temporal attention unit and a parameter cross-attention unit. During the training process, only the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit are adjusted, while the unit parameters of other units except the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit remain fixed.

[0127] Next, the processing of the pre-trained generative model is described. Figure 3As shown, a sample video is obtained; the sample video is parsed to obtain a sample object image, a sample action sequence and a sample camera parameter; the sample object image is input into the image coding unit in the pre-trained generation model to obtain a sample image coding feature, and the image coding feature is input into the object coding unit in the pre-trained generation model to obtain a sample object coding feature; the sample action sequence is input into the action coding unit in the pre-trained generation model to obtain a sample action coding sequence; the sample camera parameter is input into the parameter coding unit in the pre-trained generation model to obtain a sample parameter coding feature; the sample object coding feature, the sample action coding sequence and the sample parameter coding feature are input into the generation unit in the pre-trained generation model to obtain a first prediction feature; the first prediction feature is input into the decoding unit in the pre-trained generation model to obtain a predicted video. The sample video is input into the image coding unit in the pre-trained generation model to obtain a first sample feature; according to the first prediction feature and the first sample feature, the unit parameters of the parameter coding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model are adjusted to obtain a trained video generation model.

[0128] By applying the scheme of the embodiments of this specification, when training the pre-trained generation model, sample object images and sample action sequences are used to ensure that the model can also ensure the generation effect when the input conditions are met; the model is trained using sample object images including sample objects, so that the effect of the model on the object (such as a person) video generation task is greatly improved. It is realized that one or more person reference images, the action sequence of the person, and the specified virtual camera parameters are input, and the model outputs a video that is consistent with the person role, conforms to the input action sequence, and has a specified camera trajectory, so that the entire video effect is real and stable, the timing is consistent, and it can support a relatively large camera movement effect.

[0129] In an optional embodiment of the present specification, a training method of a pre-trained generation model is described, that is, the sample video includes a plurality of sample video frames; before the sample object image, the sample action sequence and the sample camera parameter are input into the pre-trained generation model and the first prediction feature output by the generation unit in the pre-trained generation model is obtained, the following steps may also be included:

[0130] For a first sample video frame, determining a first sample object image and a first sample action image corresponding to the first sample video frame, wherein the first sample video frame is sampled from a plurality of sample video frames;

[0131] Inputting the first sample object image and the first sample action image into an initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model;

[0132] Inputting the first sample video frame into the image encoding unit in the initial generation model to obtain a second sample feature;

[0133] According to the second prediction feature and the second sample feature, the unit parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model are adjusted to obtain a pre-trained generation model that has been trained.

[0134] Specifically, the second sample feature is the generation target of the generation unit in the initial generation model, which is used to guide the training process of the initial generation model. Since the initial generation model is trained based on a single video frame and does not include timing information and camera motion trajectory, the initial generation model does not include a parameter encoding unit, and the generation unit of the initial generation model does not include a temporal attention unit and a parameter cross-attention unit, that is, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a generation unit, and a decoding unit; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit and a cross-attention unit.

[0135] It should be noted that when sampling from multiple sample video frames to obtain the first sample video frame, in one possible implementation, all sample video frames can be determined as the first sample video frame. In another possible implementation, since the similarity between adjacent frames is high, the first sample video frame can be extracted every preset time length.

[0136] In actual applications, the implementation method of "adjusting the unit parameters of the object coding unit, action coding unit and generation unit in the initial generation model according to the second prediction feature and the second sample feature to obtain a pre-trained generation model that has been trained" is the same as the above-mentioned implementation method of "adjusting the unit parameters of the parameter coding unit, temporal attention unit and parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature to obtain a trained video generation model", and the embodiments of this specification will not be repeated here.

[0137] Furthermore, according to the second prediction feature and the second sample feature, the unit parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model are adjusted. After the trained pre-trained generation model is obtained, the pre-trained generation model can be used to process the image generation task, that is, the object image and the action image of the target object are input into the pre-trained generation model to obtain the target image, the target image includes the target object, and the action of the target object is consistent with the action in the action image.

[0138] By applying the scheme of the embodiments of this specification, the unit parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model are adjusted according to the second prediction feature and the second sample feature, so that the pre-trained generation model can generate images with consistent objects and actions, thereby ensuring the generation capability of the pre-trained generation model.

[0139] See also Figure 4 , Figure 4 The flowchart of the processing process of a pre-trained generative model training method provided by an embodiment of the present specification is shown. The pre-trained generative model is obtained by training the initial generative model. This training stage can be called the image generation training stage. In this stage, the model training goal is to generate an image that is consistent with the object and conforms to the target action. The initial generative model can reuse the weights of Stab le Diffus ion, and add an action coding unit and an object coding unit on the basis of Stab le Diffus ion. That is, Figure 4 As shown in FIG, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a generation unit, and a decoding unit; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit and a cross-attention unit. During the training process, only the unit parameters of the object encoding unit, the action encoding unit, and the generation unit are adjusted, while the unit parameters of the image encoding unit and the decoding unit are fixed.

[0140] Next, the processing of the initial generation model is described. Figure 4 As shown, a sample video is obtained, wherein the sample video includes multiple sample video frames; for a first sample video frame, a first sample object image and a first sample action image corresponding to the first sample video frame are determined, wherein the first sample video frame is sampled from multiple sample video frames; the first sample object image and the first sample action image are input into an initial generation model to obtain a first predicted video frame output by the initial generation model. The first sample video frame is input into an image encoding unit in the initial generation model to obtain a second sample feature; according to the second sample feature and the second predicted feature output by the generation unit in the initial generation model, the unit parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model are adjusted to obtain a pre-trained generation model that has been trained.

[0141] The following combination Figure 5 Taking the application of the video generation method provided in this specification in the virtual object motion video generation scene as an example, the video generation method is further described. Figure 5 A flowchart of a method for generating a motion video of a virtual object provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0142] Step 502: acquiring motion video generation data of a virtual object, wherein the motion video generation data includes virtual camera parameters, a reference action sequence of the virtual object, and an image of the virtual object.

[0143] Step 504: Input the motion video generation data into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model adjusts the parameter encoding unit, parameter cross-attention unit and temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, sample action sequence and sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, action encoding unit and generation unit in the initial generation model based on the sample video.

[0144] It should be noted that the implementation of step 502 to step 504 is the same as the implementation of step 202 to step 204, and will not be described in detail in the embodiments of this specification.

[0145] In a virtual object motion video generation scenario, a virtual object image of a virtual object, an action sequence of a certain sport, and virtual camera parameters may be input into a video generation model to obtain a target motion video of the virtual object performing the sport.

[0146] By applying the solution of the embodiments of the present specification, the camera movement control of the target motion video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target motion video that supports a large-amplitude camera movement effect and has stable timing, making the target motion video more realistic and natural.

[0147] The following combination Figure 6 , taking the application of the video generation method provided in this specification in the video editing (such as replacement, adjustment) scenario as an example, the video generation method is further described. Figure 6 A flowchart of a video editing method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0148] Step 602: Acquire the original video and target editing data, wherein the target editing data includes at least one of a target object image, a target camera parameter, and a target action sequence.

[0149] Step 604: parse the original video based on the target editing data to determine the original video data in the original video.

[0150] Step 606: Input the target editing data and the original video data into the video generation model to obtain the target edited video, wherein the video generation model adjusts the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video.

[0151] It should be noted that the implementation of step 602 to step 606 is the same as the implementation of step 202 to step 204, and will not be described in detail in this embodiment of the specification.

[0152] In practical applications, if the video editing task is a video camera adjustment task, the target editing data is the target camera parameters, and the original video is parsed based on the target editing data to determine that the original video data in the original video includes the original action sequence and the original object image. If the video editing task is a video object editing task, the target editing data is the target object image, and the original video is parsed based on the target editing data to determine that the original video data in the original video includes the original camera parameters and the original action sequence. If the video editing task is a video action editing task, the target editing data is the target action sequence, and the original video is parsed based on the target editing data to determine that the original video data in the original video includes the original camera parameters and the original object image.

[0153] By applying the solution of the embodiments of the present specification, the camera movement control of the target edited video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target edited video that supports a large-amplitude camera movement effect and has stable timing, making the target edited video more realistic and natural.

[0154] See also Figure 7 , Figure 7 A flowchart of a video generation model training method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0155] Step 702: Obtain a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video.

[0156] Step 704: Input the sample object image, the sample action sequence and the sample camera parameters into the pre-trained generation model to obtain the first prediction feature output by the generation unit in the pre-trained generation model, wherein the pre-trained generation model is obtained by adjusting the parameters of the object coding unit, the action coding unit and the generation unit in the initial generation model based on the sample video.

[0157] Step 706: Input the sample video into the image encoding unit in the pre-trained generation model to obtain the first sample feature.

[0158] Step 708: According to the first prediction feature and the first sample feature, adjust the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model to obtain a trained video generation model.

[0159] It should be noted that the implementation method of steps 702 to 708 is the same as that of the above Figure 2 The training method of the video generation model in the provided video generation method is the same, and will not be repeated in the embodiments of this specification.

[0160] By applying the solution of the embodiments of the present specification, when training the pre-trained generation model, additional temporal attention units and parameter cross-attention units are added to each generation block included in the generation unit of the pre-trained generation model, so that the trained video generation model can learn timing information and camera parameters, thereby generating a target video that supports a larger range of camera movement effects and has stable timing, making the target video more realistic and natural.

[0161] See also Figure 8 , Figure 8 A flowchart of an information processing method based on a video generation model provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0162] Step 802: Receive a task generation request, wherein the task generation request includes request information.

[0163] Specifically, the information processing method based on the video generation model can be applied to the task platform, and the task platform can be deployed on the terminal device or the cloud device. The task generation request is used to request the task information for generating the target video task. The task generation request usually includes the task type, the expected output format, and the request information. The request information refers to the parameters or description information related to the target video task carried in the task generation request. The request information includes but is not limited to the task scene identifier of the target video task, the task model identifier, and multiple sample videos corresponding to the target video task.

[0164] Step 804: Based on the request information, obtain a video generation model, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on the sample video and the sample object image, the sample action sequence, and the sample camera parameters corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video.

[0165] In actual applications, there are multiple ways to obtain a video generation model based on the request information, and the specific method is selected according to the actual situation. The embodiments of this specification do not impose any limitation on this.

[0166] In an optional embodiment of the present specification, the obtaining of the video generation model based on the request information may include the following steps:

[0167] Based on the task scene identifier, determine a target scene template from a plurality of preset scene templates, and based on the target scene template, search for a video generation model from a model library, wherein the model library stores a plurality of video generation models, and the request information includes a task scene identifier of the target video task; or,

[0168] Based on the task model identifier, a video generation model is searched from a model library, wherein the request information includes the task model identifier of the target video task.

[0169] Specifically, the task scene identifier refers to a unique or specific label used to distinguish different task application scenarios. In the embodiment of this specification, the task scene identifier is a part of the request information. Through the task scene identifier, a target scene template that matches the request information can be selected from a series of preset scene templates to generate task information. For example, the task scene identifier is "video camera control", which means that the user wants to adjust the camera effect of the uploaded original video, then the video camera control scene template can be selected from multiple preset scene templates according to the task scene identifier.

[0170] The preset scene templates are predefined standard configuration scene templates for application scenarios of different target video tasks. Each template contains model information that matches the application scenario of the task and information such as the task processing flow. Through a series of preset scene templates, task generation requests for different scenarios can be quickly responded to. Different preset scene templates correspond to different task types, model information, and processing flows. For example, there may be a video motion control scene template specifically for video motion control tasks in the preset scene templates, which contains the model information and processing flow of the trained video motion control model.

[0171] The target scene template refers to the scene template that matches the task scene identifier. When parsing the task generation request, the corresponding target scene template can be located based on the task scene identifier, and the corresponding video generation model and other related configuration information can be selected from the model library based on the model information included in the target scene template. For example, when the task scene identifier is "video object control", the target scene template is a template that contains the model information and related configuration parameters of the video object control model.

[0172] The model library is a resource library that centrally stores video generation models. These models are trained and optimized to solve different target video tasks. In addition, the video generation models in the model library can be divided into different versions according to different applicable tasks. For example, the model library may store video camera control models for video camera control tasks, video object control models for video object control tasks, and video motion control models for video motion control tasks.

[0173] The task model identifier refers to a unique or specific label used to distinguish models applicable to different tasks. For example, the task model identifier may be "video camera control". Based on the task model identifier, a video camera control model applicable to the video camera control task can be found from the model library.

[0174] By applying the solution of the embodiments of this specification, the process of acquiring the video generation model is made more flexible, efficient and standardized through pre-defined task scenario templates, task model identifiers and model library resources.

[0175] In another optional embodiment of the present specification, in addition to selecting a video generation model from a model library, a pre-trained generation model in the model library may be trained according to a sample video in the request information and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video to obtain a video generation model, that is, the request information includes a sample video of a target video task and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video; the above-mentioned acquisition of a video generation model based on the request information may include the following steps:

[0176] Based on the sample videos and the sample object images, sample action sequences and sample camera parameters corresponding to the sample videos, a pre-trained generation model corresponding to the target video task is trained to obtain a trained video generation model.

[0177] It should be noted that the implementation method of "training the pre-trained generation model corresponding to the target video task based on the sample video and the sample object images, sample action sequences and sample camera parameters corresponding to the sample video to obtain the trained video generation model" is the same as the training method of the above-mentioned "video generation model", and will not be repeated in the embodiments of this specification.

[0178] By applying the solution of the embodiments of this specification, the pre-trained generation model corresponding to the target video task is trained based on the sample video to obtain a trained video generation model. While ensuring the accuracy of the video generation model, the video generation model is made more in line with the task scenario of the target video task.

[0179] Step 806: Generate task information based on the video generation model, where the task information is used to perform the target video task.

[0180] Specifically, the task information includes the model configuration and processing flow required to perform the target video task. The terminal device or other server-side components can correctly use the video generation model to process the target video task based on the task information. The target video tasks include but are not limited to video camera control of characters, video action control tasks, video object control tasks, and video generation tasks.

[0181] It should be noted that when generating task information based on the video generation model, the model parameters of the video generation model can be directly packaged to obtain the task information. Other model information of the video generation model can also be obtained to construct the task information based on the other model information, where the other model information includes the processing method of the model input data, the specifications of the expected output results, and the intermediate steps and other auxiliary information that may be involved.

[0182] By applying the solution of the embodiments of this specification and generating task information, it is possible to reduce system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provide users with convenient and efficient target video task processing services.

[0183] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a task platform provided by an embodiment of the present specification is shown, the task platform includes a request interface 902 and a response unit 904;

[0184] A request interface 902, used to receive a task generation request, wherein the task generation request includes request information;

[0185] Response unit 904 is used to obtain a video generation model based on the request information, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video; task information is generated based on the video generation model, wherein the task information is used to execute a target video task.

[0186] In an optional embodiment of the present specification, the task platform further includes a model library, wherein the model library stores a plurality of video generation models;

[0187] The response unit is specifically used to determine a target scene template from multiple preset scene templates based on a task scene identifier, and to search a video generation model from a model library based on the target scene template, wherein the request information includes the task scene identifier of the target video task; or, to search a video generation model from a model library based on a task model identifier, wherein the request information includes the task model identifier of the target video task.

[0188] By applying the solution of the embodiments of this specification, by generating task information of the target video task, it is possible to reduce system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provide users with convenient and efficient target video task processing services.

[0189] The above is a schematic scheme of a task platform of this embodiment. The technical scheme of the task platform and the technical scheme of the information processing method based on the video generation model are of the same concept. For details not described in detail in the technical scheme of the task platform, please refer to the description of the technical scheme of the information processing method based on the video generation model.

[0190] Corresponding to the above-mentioned video generation method embodiment, this specification also provides a video generation device embodiment, Fig.10 FIG. 2 shows a schematic diagram of the structure of a video generation device provided by an embodiment of the present specification. Fig.10 As shown, the device comprises:

[0191] A first acquisition module 1002 is configured to acquire video generation data, wherein the video generation data includes virtual camera parameters, a reference action sequence of a target object, and an object image of the target object;

[0192] The first input module 1004 is configured to input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model adjusts the parameters of the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video.

[0193] Optionally, the video generation model includes an image coding unit, an object coding unit, an action coding unit, a parameter coding unit, a generation unit and a decoding unit; the first input module 1004 is further configured to input the object image into the image coding unit to obtain image coding features; input the image coding features into the object coding unit to obtain object coding features; input the reference action sequence into the action coding unit to obtain an action coding sequence; input the virtual camera parameters into the parameter coding unit to obtain parameter coding features; input the object coding features, the action coding sequence and the parameter coding features into the generation unit to obtain video coding features; input the video coding features into the decoding unit to obtain a target video of the target object.

[0194] Optionally, the object coding unit includes multiple coding blocks; the first input module 1004 is further configured to input the image coding feature into the first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is the first coding block among the multiple coding blocks; input the coding feature output by the previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks; determine the object coding feature based on the coding features respectively output by the multiple coding blocks.

[0195] Optionally, the generation unit includes multiple generation blocks, the object coding unit includes multiple coding blocks, and the generation blocks and the coding blocks correspond one to one; the first input module 1004 is further configured to determine the coding features corresponding to the generation blocks according to the correspondence between the generation blocks and the coding blocks; the action coding sequence, the parameter coding features and the coding features corresponding to the first generation block are input into the first generation block to obtain the first generation features output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks; the generation features output by the previous generation block of the second generation block, the coding features corresponding to the second generation block and the parameter coding features are input into the second generation block to obtain the second generation features output by the second generation block, wherein the second generation block is any generation block other than the first generation block among the multiple generation blocks; when the second generation block is the last generation block among the multiple generation blocks, the second generation features output by the second generation block are determined as the video coding features.

[0196] Optionally, the second generation block includes a self-attention unit, a cross-attention unit, a temporal attention unit and a parameter cross-attention unit; the first input module 1004 is further configured to input the generation feature output by the previous generation block of the second generation block into the self-attention unit to obtain a self-attention feature; input the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain a cross-attention feature; input the cross-attention feature into the temporal attention unit to obtain a temporal attention feature; input the temporal attention feature and the parameter encoding feature into the parameter cross-attention unit to obtain a second generation feature.

[0197] Optionally, the device also includes: a first training module, configured to obtain a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video; input the sample object image, the sample action sequence and the sample camera parameter into a pre-trained generation model to obtain a first prediction feature output by a generation unit in the pre-trained generation model; input the sample video into an image encoding unit in the pre-trained generation model to obtain a first sample feature; and adjust the unit parameters of a parameter encoding unit, a temporal attention unit and a parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0198] Optionally, the sample video includes multiple sample video frames; the device also includes: a second training module, configured to determine, for a first sample video frame, a first sample object image and a first sample action image corresponding to the first sample video frame, wherein the first sample video frame is sampled from multiple sample video frames; input the first sample object image and the first sample action image into an initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model; input the first sample video frame into an image coding unit in the initial generation model to obtain a second sample feature; and adjust unit parameters of an object coding unit, an action coding unit and a generation unit in the initial generation model according to the second prediction feature and the second sample feature to obtain a pre-trained generation model that has been trained.

[0199] Optionally, the first acquisition module 1002 is further configured to receive a reference video and an object image of a target object sent by a client; input the reference video into an action extraction model to obtain a reference action sequence; and input the reference video into a parameter extraction model to obtain virtual camera parameters.

[0200] Optionally, the device also includes: a sending module configured to send the target video to the client; receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client on the target video; constructing model optimization data based on the result feedback information; and adjusting parameters of the video generation model using the model optimization data.

[0201] By applying the solution of the embodiments of this specification, the camera movement control of the target video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target video that supports a large-amplitude camera movement effect and has stable timing, making the target video more realistic and natural.

[0202] The above is a schematic scheme of a video generation device of this embodiment. It should be noted that the technical scheme of the video generation device and the technical scheme of the above video generation method belong to the same concept, and the details not described in detail in the technical scheme of the video generation device can be referred to the description of the technical scheme of the above video generation method.

[0203] Corresponding to the above-mentioned method embodiment for generating a motion video of a virtual object, this specification also provides an embodiment of a device for generating a motion video of a virtual object. Fig.11 FIG. 1 is a schematic diagram showing a structure of a motion video generation device for a virtual object provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0204] The second acquisition module 1102 is configured to acquire motion video generation data of the virtual object, wherein the motion video generation data includes virtual camera parameters, a reference action sequence of the virtual object, and an image of the virtual object;

[0205] The second input module 1104 is configured to input the motion video generation data into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model adjusts the parameters of the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video.

[0206] By applying the solution of the embodiments of the present specification, the camera movement control of the target motion video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target motion video that supports a large-amplitude camera movement effect and has stable timing, making the target motion video more realistic and natural.

[0207] The above is a schematic scheme of a motion video generation device for a virtual object of this embodiment. It should be noted that the technical scheme of the motion video generation device for a virtual object and the technical scheme of the motion video generation method for a virtual object are of the same concept, and the details not described in detail in the technical scheme of the motion video generation device for a virtual object can be referred to the description of the technical scheme of the motion video generation method for a virtual object.

[0208] Corresponding to the above-mentioned video editing method embodiment, this specification also provides a video editing device embodiment, Fig.12 FIG. 2 shows a schematic diagram of the structure of a video editing device provided by an embodiment of the present specification. Fig.12 As shown, the device comprises:

[0209] A third acquisition module 1202 is configured to acquire the original video and target editing data, wherein the target editing data includes at least one of a target object image, a target camera parameter, and a target action sequence;

[0210] The parsing module 1204 is configured to parse the original video based on the target editing data to determine the original video data in the original video;

[0211] The third input module 1206 is configured to input the target editing data and the original video data into the video generation model to obtain the target edited video, wherein the video generation model adjusts the parameter encoding unit, the parameter cross-attention unit and the temporal attention unit in the pre-trained generation model based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, and the pre-trained generation model adjusts the parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model based on the sample video.

[0212] By applying the solution of the embodiments of the present specification, the camera movement control of the target edited video is achieved by inputting virtual camera parameters into the video generation model. In addition, since the video generation model includes a temporal attention unit and a parameter cross-attention unit, the video generation model can learn the timing information and camera parameters, thereby generating a target edited video that supports a large-amplitude camera movement effect and has stable timing, making the target edited video more realistic and natural.

[0213] The above is a schematic scheme of a video editing device of this embodiment. It should be noted that the technical scheme of the video editing device and the technical scheme of the above video editing method belong to the same concept, and the details not described in detail in the technical scheme of the video editing device can be referred to the description of the technical scheme of the above video editing method.

[0214] Corresponding to the above-mentioned video generation model training method embodiment, this specification also provides a video generation model training device embodiment, Fig.13 FIG. 1 is a schematic diagram showing the structure of a video generation model training device provided by an embodiment of the present specification. Fig.13 As shown, the device comprises:

[0215] A fourth acquisition module 1302 is configured to acquire a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video;

[0216] The fourth input module 1304 is configured to input the sample object image, the sample action sequence and the sample camera parameter into the pre-trained generation model to obtain a first prediction feature output by a generation unit in the pre-trained generation model, wherein the pre-trained generation model is obtained by adjusting parameters of an object coding unit, an action coding unit and a generation unit in the initial generation model based on the sample video;

[0217] A fifth input module 1306 is configured to input the sample video into the image encoding unit in the pre-trained generative model to obtain a first sample feature;

[0218] The adjustment module 1308 is configured to adjust the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0219] By applying the solution of the embodiments of the present specification, when training the pre-trained generation model, additional temporal attention units and parameter cross-attention units are added to each generation block included in the generation unit of the pre-trained generation model, so that the trained video generation model can learn timing information and camera parameters, thereby generating a target video that supports a larger range of camera movement effects and has stable timing, making the target video more realistic and natural.

[0220] The above is a schematic scheme of a video generation model training device of this embodiment. It should be noted that the technical scheme of the video generation model training device and the technical scheme of the above-mentioned video generation model training method belong to the same concept, and the details not described in detail in the technical scheme of the video generation model training device can be referred to the description of the technical scheme of the above-mentioned video generation model training method.

[0221] Corresponding to the above-mentioned information processing method embodiment based on the video generation model, this specification also provides an information processing device embodiment based on the video generation model, Fig.14 FIG. 1 shows a schematic diagram of the structure of an information processing device based on a video generation model provided by an embodiment of the present specification. Fig.14 As shown, the device comprises:

[0222] The receiving module 1402 is configured to receive a task generation request, wherein the task generation request includes request information;

[0223] The fifth acquisition module 1404 is configured to acquire a video generation model based on the request information, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on the sample video and the sample object image, the sample action sequence, and the sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video;

[0224] The generation module 1406 is configured to generate task information based on the video generation model, wherein the task information is used to perform the target video task.

[0225] Optionally, the fifth acquisition module 1404 is further configured to determine a target scene template from multiple preset scene templates based on the task scene identifier, and search for a video generation model from a model library based on the target scene template, wherein the model library stores multiple video generation models and the request information includes the task scene identifier of the target video task; or, search for a video generation model from the model library based on the task model identifier, wherein the request information includes the task model identifier of the target video task.

[0226] Optionally, the request information includes a sample video of the target video task and a sample object image, a sample action sequence and a sample camera parameters corresponding to the sample video; the fifth acquisition module 1404 is further configured to train a pre-trained generation model corresponding to the target video task based on the sample video and the sample object image, the sample action sequence and the sample camera parameters corresponding to the sample video, so as to obtain a trained video generation model.

[0227] By applying the solution of the embodiments of this specification, by generating task information of the target video task, the system deployment and operation and maintenance costs are reduced while ensuring the processing quality and efficiency of the target video task, providing users with convenient and efficient task processing services.

[0228] The above is a schematic scheme of an information processing device based on a video generation model of this embodiment. It should be noted that the technical scheme of the information processing device based on the video generation model and the technical scheme of the information processing method based on the video generation model belong to the same concept, and the details not described in detail in the technical scheme of the information processing device based on the video generation model can be referred to the description of the technical scheme of the information processing method based on the video generation model.

[0229] Fig.15The block diagram of a computing device provided by one embodiment of the present specification is shown. The components of the computing device 1500 include but are not limited to a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and the database 1550 is used to store data.

[0230] The computing device 1500 also includes an access device 1540 that enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1540 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0231] In one embodiment of the present specification, the above components of the computing device 1500 and Fig.15 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.15 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0232] The computing device 1500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1500 may also be a mobile or stationary server.

[0233] Among them, processor 1520 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned video generation method or virtual object motion video generation method or video editing method or video generation model training method or information processing method based on video generation model.

[0234] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model belong to the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0235] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0236] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model belong to the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0237] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0238] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model are of the same concept. For details not described in detail in the technical scheme of the computer program product, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0239] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0240] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0241] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0242] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0243] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A video generation method, comprising: Acquire video generation data, wherein the video generation data includes virtual camera parameters, a reference action sequence of a target object, and an object image of the target object; The video generation data is input into a video generation model to obtain a target video of the target object, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

2. The method according to claim 1, wherein the video generation model comprises an image encoding unit, an object encoding unit, an action encoding unit, a parameter encoding unit, a generation unit and a decoding unit; The step of inputting the video generation data into a video generation model to obtain a target video of the target object comprises: Inputting the object image into the image encoding unit to obtain image encoding features; Inputting the image coding feature into the object coding unit to obtain the object coding feature; Inputting the reference action sequence into the action encoding unit to obtain an action encoding sequence; Inputting the virtual camera parameters into the parameter encoding unit to obtain parameter encoding features; Inputting the object coding feature, the action coding sequence and the parameter coding feature into the generating unit to obtain a video coding feature; The video coding feature is input into the decoding unit to obtain the target video of the target object.

3. The method according to claim 2, wherein the object coding unit comprises a plurality of coding blocks; The step of inputting the image coding feature into the object coding unit to obtain the object coding feature comprises: Inputting the image coding feature into a first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is a first coding block among the multiple coding blocks; Inputting a coding feature output by a previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks; The object coding feature is determined according to the coding features respectively output by the multiple coding blocks.

4. The method according to claim 2, wherein the generation unit comprises a plurality of generation blocks, the object coding unit comprises a plurality of coding blocks, and the generation blocks correspond to the coding blocks one by one; The step of inputting the object coding feature, the action coding sequence and the parameter coding feature into the generating unit to obtain the video coding feature comprises: Determining, according to a correspondence between the generation block and the encoding block, an encoding feature corresponding to the generation block; Inputting the action coding sequence, the parameter coding feature and the coding feature corresponding to the first generation block into the first generation block, obtaining a first generation feature output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks; Inputting a generation feature output by a previous generation block of a second generation block, a coding feature corresponding to the second generation block, and the parameter coding feature into the second generation block to obtain a second generation feature output by the second generation block, wherein the second generation block is any generation block other than the first generation block among the multiple generation blocks; In a case where the second generation block is the last generation block among the multiple generation blocks, a second generation feature output by the second generation block is determined as the video encoding feature.

5. The method according to claim 4, wherein the second generation block comprises a self-attention unit, a cross-attention unit, a temporal attention unit, and a parameter cross-attention unit; The step of inputting the generated feature output by a previous generation block of the second generation block, the encoding feature corresponding to the second generation block, and the parameter encoding feature into the second generation block to obtain the second generated feature output by the second generation block includes: Inputting the generated features output by the previous generation block of the second generation block into the self-attention unit to obtain the self-attention features; Inputting the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain a cross-attention feature; Inputting the cross-attention feature into the temporal attention unit to obtain a temporal attention feature; The temporal attention feature and the parameter encoding feature are input into the parameter cross-attention unit to obtain the second generated feature.

6. The method according to claim 1, before inputting the video generation data into a video generation model to obtain the target video of the target object, further comprising: Obtaining a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video; Inputting the sample object image, the sample action sequence and the sample camera parameter into the pre-trained generation model to obtain a first prediction feature output by a generation unit in the pre-trained generation model; Inputting the sample video into the image encoding unit in the pre-trained generation model to obtain a first sample feature; According to the first prediction feature and the first sample feature, the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model are adjusted to obtain the trained video generation model.

7. The method according to claim 6, wherein the sample video comprises a plurality of sample video frames; Before inputting the sample object image, the sample action sequence and the sample camera parameter into the pre-trained generation model to obtain the first prediction feature output by the generation unit in the pre-trained generation model, the method further includes: For a first sample video frame, determining a first sample object image and a first sample action image corresponding to the first sample video frame, wherein the first sample video frame is sampled from the plurality of sample video frames; Inputting the first sample object image and the first sample action image into the initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model; Inputting the first sample video frame into the image encoding unit in the initial generation model to obtain a second sample feature; According to the second prediction feature and the second sample feature, the unit parameters of the object encoding unit, the action encoding unit and the generation unit in the initial generation model are adjusted to obtain the trained pre-trained generation model.

8. The method according to claim 1, wherein obtaining video generation data comprises: Receiving a reference video and an object image of the target object sent by a client; Inputting the reference video into an action extraction model to obtain the reference action sequence; The reference video input parameter extraction model is used to obtain the virtual camera parameters.

9. The method according to claim 1, after inputting the video generation data into a video generation model to obtain the target video of the target object, further comprising: Sending the target video to the client; Receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client regarding the target video; According to the result feedback information, a model is constructed to optimize the data; The model optimization data is used to adjust parameters of the video generation model.

10. A method for generating a motion video of a virtual object, comprising: Acquire motion video generation data of a virtual object, wherein the motion video generation data includes virtual camera parameters, a reference action sequence of the virtual object, and a virtual object image; The motion video generation data is input into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

11. A video editing method, comprising: Acquire an original video and target editing data, wherein the target editing data includes at least one of a target object image, a target camera parameter, and a target action sequence; Parsing the original video based on the target editing data to determine original video data in the original video; The target editing data and the original video data are input into a video generation model to obtain a target edited video, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video.

12. A video generation model training method, comprising: Obtaining a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video; Inputting the sample object image, the sample action sequence and the sample camera parameter into a pre-trained generation model to obtain a first prediction feature output by a generation unit in the pre-trained generation model, wherein the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video; Inputting the sample video into the image encoding unit in the pre-trained generation model to obtain a first sample feature; According to the first prediction feature and the first sample feature, the unit parameters of the parameter encoding unit, the temporal attention unit and the parameter cross-attention unit in the pre-trained generation model are adjusted to obtain the trained video generation model.

13. An information processing method based on a video generation model, comprising: receiving a task generation request, wherein the task generation request includes request information; Based on the request information, a video generation model is obtained, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit, and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence, and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit, and a generation unit in an initial generation model based on the sample video; Based on the video generation model, task information is generated, wherein the task information is used to perform a target video task.

14. The method according to claim 13, wherein obtaining a video generation model based on the request information comprises: Based on the task scene identifier, determine a target scene template from a plurality of preset scene templates, and based on the target scene template, search for a video generation model from a model library, wherein the model library stores a plurality of video generation models, and the request information includes the task scene identifier of the target video task; or Based on the task model identifier, a video generation model is searched from the model library, wherein the request information includes the task model identifier of the target video task.

15. The method according to claim 13, wherein the request information includes a sample video of the target video task and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video; The acquiring of the video generation model based on the request information includes: Based on the sample video and the sample object images, sample action sequences and sample camera parameters corresponding to the sample video, the pre-trained generation model corresponding to the target video task is trained to obtain a trained video generation model.

16. A task platform, comprising a request interface and a response unit; The request interface is used to receive a task generation request, wherein: The task generation request includes request information; The response unit is used to obtain a video generation model based on the request information, wherein the video generation model is obtained by adjusting parameters of a parameter encoding unit, a parameter cross-attention unit and a temporal attention unit in a pre-trained generation model based on a sample video and a sample object image, a sample action sequence and a sample camera parameter corresponding to the sample video, and the pre-trained generation model is obtained by adjusting parameters of an object encoding unit, an action encoding unit and a generation unit in an initial generation model based on the sample video; task information is generated based on the video generation model, wherein the task information is used to execute a target video task.

17. The task platform according to claim 16 further comprises a model library, wherein: The model library stores a plurality of video generation models; The response unit is specifically used to determine a target scene template from multiple preset scene templates based on a task scene identifier, and to search for a video generation model from the model library based on the target scene template, wherein the request information includes the task scene identifier of the target video task; or, to search for a video generation model from the model library based on a task model identifier, wherein the request information includes the task model identifier of the target video task.

18. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15 are implemented.

19. A computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15.

20. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15.

Citation Information

Cited By

  • Information processing method and device, computer readable storage medium and electronic equipment

    CN119722269A

  • Data generation method and device, electronic equipment and storage medium

    CN120451347A

  • Artificial intelligence model-based room type video generation method, computing device and computer program product

    CN120602738A

  • Video generation method, motion video generation method for virtual object, video editing method, video generation model training method, and video generation model-based information processing method

    WO2026001197A1

  • Video generation method and related device

    WO2026113493A1