Video generation method, motion video generation method of virtual object, video editing method, video generation model training method and information processing method based on video generation model

By introducing timing attention units into the video generation model, using the attention mechanism of the time dimension, the video quality problem caused by timing disorder in the prior art is solved, and the timing stability and real and natural video generation are achieved.

CN119031208APending Publication Date: 2024-11-26ALIBABA (CHINA) CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410833799.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the existing video generation technology, due to data diversity, the timing disorders are caused by poor quality of generated videos, and the video generation is urgently needed to achieve stable video generation solutions.

Method used

A video generation model is adopted, including timing attention units, and data encoding features are correlated through the attention mechanism of the time dimension to generate stable timing, real and natural videos.

Benefits of technology

Improve the quality of the target video, ensure the consistency and nature of video timing, and enhance the controllability and authenticity of video generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031208A_ABST
    Figure CN119031208A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a video generation method, a motion video generation method of a virtual object, a video editing method, a video generation model training method and an information processing method based on a video generation model. The video generation method comprises the following steps: acquiring video generation data of a target object; the video generation data are input into a video generation model, a target video of the target object is obtained, the video generation model comprises a generation unit and different types of coding units, the different types of coding units are used for coding the video generation data to obtain data coding features, and the generation unit comprises a time sequence attention unit; and the time sequence attention unit is used for associating the data coding features by using an attention mechanism of a time dimension. The video generation model comprises the time sequence attention unit, so that the video generation model can learn time sequence information, the target video which is stable in time sequence and more real and natural is generated, and the quality of the target video is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to video generation, motion video generation of virtual objects, video editing, video generation model training, and information processing methods based on video generation models. Background Art

[0002] With the development of computer technology, video generation has gradually become an important research topic in multimodality and vision in generative artificial intelligence (AIGC, AI Generated Content), and has broad prospects and application value in scenarios such as digital humans, e-commerce content, and film and television production.

[0003] In the video generation scenario, since a large amount of data is used to generate the video, the generated video may have timing disorder problems, which further leads to poor quality of the generated video. Therefore, a video generation solution with stable timing is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides a video generation method. One or more embodiments of the present specification also relate to a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a video generation method is provided, including:

[0006] Obtaining video generation data of a target object;

[0007] The video generation data is input into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the video generation data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0008] According to a second aspect of an embodiment of this specification, a method for generating a motion video of a virtual object is provided, comprising:

[0009] Obtaining motion video generation data of a virtual object;

[0010] The motion video generation data is input into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0011] According to a third aspect of the embodiments of this specification, a video editing method is provided, including:

[0012] Get the original video and target editing data;

[0013] Parsing the original video to determine the original video data in the original video;

[0014] The target editing data and the original video data are input into a video generation model to obtain a target editing video, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using an attention mechanism in the time dimension.

[0015] According to a fourth aspect of the embodiments of this specification, a video generation model training method is provided, including:

[0016] Obtaining a sample video and sample generation data corresponding to the sample video;

[0017] Inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the sample encoding features using an attention mechanism in a time dimension;

[0018] Inputting the sample video into the encoding unit to obtain a first sample feature;

[0019] According to the first prediction feature and the first sample feature, the unit parameters of the encoding unit and the temporal attention unit are adjusted to obtain a trained video generation model.

[0020] According to a fifth aspect of the embodiments of this specification, there is provided an information processing method based on a video generation model, comprising:

[0021] receiving a task generation request, wherein the task generation request includes request information;

[0022] Based on the request information, a video generation model is obtained, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in a time dimension;

[0023] Based on the video generation model, task information is generated, wherein the task information is used to perform a target video task.

[0024] According to a sixth aspect of the embodiments of this specification, there is provided a task platform, including a request interface and a response unit;

[0025] A request interface, used to receive a task generation request, wherein the task generation request includes request information;

[0026] A response unit is used to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in the time dimension; based on the video generation model, task information is generated, wherein the task information is used to execute a target video task.

[0027] According to a seventh aspect of the embodiments of this specification, there is provided a video generating device, including:

[0028] A first acquisition module is configured to acquire video generation data of a target object;

[0029] The first input module is configured to input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0030] According to an eighth aspect of the embodiments of this specification, there is provided a motion video generating device for a virtual object, comprising:

[0031] A second acquisition module is configured to acquire motion video generation data of a virtual object;

[0032] The second input module is configured to input the motion video generation data into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the motion video generation data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0033] According to a ninth aspect of the embodiments of this specification, there is provided a video editing device, including:

[0034] A third acquisition module is configured to acquire the original video and the target editing data;

[0035] A parsing module is configured to parse the original video and determine the original video data in the original video;

[0036] The third input module is configured to input the target editing data and the original video data into the video generation model to obtain the target editing video, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0037] According to a tenth aspect of the embodiments of this specification, a video generation model training device is provided, including:

[0038] A fourth acquisition module is configured to acquire a sample video and sample generation data corresponding to the sample video;

[0039] a fourth input module, configured to input the sample generation data into the video generation model, and obtain a first prediction feature output by a generation unit in the video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the sample encoding features using an attention mechanism in a time dimension;

[0040] A fifth input module is configured to input the sample video into the encoding unit to obtain a first sample feature;

[0041] The adjustment module is configured to adjust the unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0042] According to an eleventh aspect of the embodiments of this specification, there is provided an information processing device based on a video generation model, including:

[0043] A receiving module is configured to receive a task generation request, wherein the task generation request includes request information;

[0044] a fifth acquisition module, configured to acquire a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in a time dimension;

[0045] The generation module is configured to generate task information based on the video generation model, wherein the task information is used to perform the target video task.

[0046] According to the twelfth aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, and the computer program / instructions, when executed by the processor, implement the steps of the method provided in the above-mentioned first aspect or second aspect or third aspect or fourth aspect or fifth aspect.

[0047] According to the thirteenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method provided in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect are implemented.

[0048] According to the fourteenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect.

[0049] A video generation method provided by an embodiment of the present specification includes: obtaining video generation data of a target object; inputting the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using an attention mechanism of the time dimension. Since the video generation model includes the temporal attention unit, the video generation model can learn the temporal information, thereby generating a target video with stable temporal order and more real and natural, thereby improving the quality of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is an architecture diagram of a video generation system provided by an embodiment of this specification;

[0051] Figure 2 is a flow chart of a video generation method provided by an embodiment of this specification;

[0052] Figure 3 is a processing flow chart of a video generation model training method provided by an embodiment of this specification;

[0053] Figure 4 is a processing flow chart of a pre-training generative model training method provided by an embodiment of this specification;

[0054] Figure 5 is a flow chart of a method for generating a motion video of a virtual object provided by one embodiment of the present specification;

[0055] Figure 6 is a flow chart of a video editing method provided by one embodiment of the present specification;

[0056] Figure 7 is a flow chart of a video generation model training method provided by an embodiment of this specification;

[0057] Figure 8 is a flowchart of an information processing method based on a video generation model provided by an embodiment of this specification;

[0058] Fig. 9 It is a structural diagram of a task platform provided by an embodiment of this specification;

[0059] Fig.10 is a structural schematic diagram of a video generating device provided by an embodiment of this specification;

[0060] Fig.11 It is a structural schematic diagram of a motion video generation device of a virtual object provided by an embodiment of the present specification;

[0061] Fig.12 is a structural schematic diagram of a video editing device provided by an embodiment of this specification;

[0062] Fig.13 It is a structural diagram of a video generation model training device provided by an embodiment of this specification;

[0063] Fig.14 is a structural schematic diagram of an information processing device based on a video generation model provided by an embodiment of this specification;

[0064] Fig.15It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0065] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0066] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0067] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0068] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0069] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model (Foundation Model 1). The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model 1), a multi-modal pre-training model (multi-modal pre-training model 1), etc.

[0070] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0071] First, the terms involved in one or more embodiments of this specification are explained.

[0072] Cross Attention: It is an attention mechanism that is mainly used to handle the dependency between two different data sources or sequences, such as injecting data of different modalities into image / video generation networks. The cross attention mechanism allows the model to selectively focus on relevant parts of the input data when generating output, thereby more effectively utilizing contextual information.

[0073] Deep Self-Attention (Transformer) Model: It is a network structure based on a multi-head self-attention mechanism module, mainly used to process sequence data. The Transformer model consists of a number of encoders and decoders that can be repeatedly stacked. This design allows the Transformer to efficiently learn long-term dependencies and is suitable for a variety of natural language processing tasks including machine translation, text summarization, and question-answering systems.

[0074] Stable Diffusion Model: It is a generative model based on generative adversarial networks. By gradually increasing the noise disturbance, the generated image becomes clearer and more realistic.

[0075] Low-Rank Adaptation of Large Language Models (Lora): It is a low-rank adaptive expression in large models, which can use a smaller number of parameters to allow large models to add additional parameters and functions.

[0076] Text to video (T2V): Automatically convert written text content into video content containing visual and auditory elements.

[0077] Relighting: is a computer graphics and computer vision technology that aims to modify or recalculate the lighting effects in existing images or three-dimensional scenes to generate pictures or videos with consistent foreground and background lighting. This process allows users to adjust the position, color, intensity, etc. of the light source without changing the geometry and material properties of the original scene, thereby re-arranging the scene lighting and creating different light and shadow effects.

[0078] Convolutional Neural Network (CNN): A deep learning model suitable for processing data with grid structure, such as images, videos, and speech signals.

[0079] In recent years, text-generated video models can generate relatively stable videos with realistic effects based on text input for a long time. Open-source image and video generation models such as Stab le Diffus ion and Video Stab le Diffus ion have greatly promoted the progress in the field of visual generation. However, the current image / video generation models have poor controllability results, and many image models often need to be run multiple times to get a relatively satisfactory result. Ensuring the effect and strong controllability are the problems that the generation model needs to solve in actual implementation. Therefore, controllable image / video generation has gradually become a research focus. Taking the field of character video generation as an example, it is necessary to pay attention not only to the character consistency and action controllability of foreground character generation, but also to the controllability of background generation.

[0080] Based on this, an embodiment of this specification proposes a video generation solution using a video generation model, which includes a temporal attention unit. Through the temporal attention unit, the video generation model can learn the lighting consistency of the foreground and background, so that the generated video can not only meet the controllability requirements of the background, but also the overall foreground and background lighting of the video is consistent, and the foreground can be harmoniously and naturally integrated into the given background, making the generated video real and natural.

[0081] In this specification, a video generation method is provided. This specification also involves a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0082] See also Figure 1 , Figure 1 The architecture diagram of a video generation system provided by an embodiment of the present specification is shown. The video generation system may include a client 100 and a server 200;

[0083] The client 100 is used to send the video generation data of the target object to the server 200;

[0084] The server 200 is used to input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the data encoding features using the attention mechanism of the time dimension; send the target video to the client 100;

[0085] The client 100 is also used to receive the target video sent by the server 200.

[0086] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, the video generation model can learn temporal information, thereby generating a target video with stable timing and more realistic and natural, thereby improving the quality of the target video.

[0087] In practical applications, the video generation system may include multiple clients 100. Multiple clients 100 may establish communication connections through the server 200. In the video generation scenario, the server 200 is used to provide video generation services between multiple clients 100. Multiple clients 100 may serve as senders or receivers, respectively, and achieve communication through the server 200.

[0088] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the video generation scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a target video based on the data stream and pushes the target video to other clients that have established communication. Among them, the client 100 and the server 200 are connected through a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being published to the server 200.

[0089] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0090] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content De l ivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0091] It is worth noting that the video generation method provided in the embodiments of this specification is generally performed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby performing the video generation method provided by the embodiments of this specification. In other embodiments, the video generation method provided by the embodiments of this specification can also be performed by the client and the server together. Next, the video generation method proposed by the server performing the embodiments of this specification is described in detail.

[0092] See also Figure 2 , Figure 2 A flowchart of a video generation method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0093] Step 202: Obtain video generation data of the target object.

[0094] In one or more embodiments of the present specification, when generating a video, video generation data may be obtained, and a target video meeting the requirements may be generated based on the video generation data.

[0095] Specifically, video generation data refers to a data set used to generate a new target video. Video generation data can be data of different video tasks, including but not limited to video generation tasks, video background replacement tasks, video object replacement tasks, and video action replacement tasks. Target objects refer to objects that appear in the target video, and target objects can be people, animals, virtual characters, etc. Video generation data can include reference action sequences, reference background sequences, and object images. Object images refer to images of target objects, and can also be understood as video foreground images. For example, object images can be images of people standing. The number of object images can be one or more. The reference action sequence includes multiple reference action images, and each reference action image contains multiple action key points. Through the reference action sequence, the action or posture (such as standing, lying flat) that the target object is to show in the target video can be defined. The reference background sequence includes multiple reference background images, and the multiple reference background images can be static image sequences, and the background in the generated target video is constant (such as grass). The multiple reference background images can also be dynamic image sequences, and the background in the generated target video can change over time (such as crowd movement).

[0096] In practical applications, there are many ways to obtain video generation data, which can be selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. In the first possible implementation of this specification, the video generation data sent by the user through the client can be received. In the second possible implementation of this specification, since it is difficult for the user to directly construct a reference action sequence and a reference background sequence, a plurality of candidate action sequences and candidate background sequences can be provided to the user to determine the reference action sequence selected by the user from the plurality of candidate action sequences and the reference background sequence selected from the plurality of candidate background sequences. In the third possible implementation of this specification, the user can send a reference video and an object image through the client, and the server can parse the reference action sequence and / or the reference background sequence from the reference video.

[0097] In an optional embodiment of the present specification, the video generation data includes a reference action sequence, a reference background sequence and an object image, and the above-mentioned acquisition of the video generation data of the target object may include the following steps:

[0098] Receive a reference video, background requirement information, and an object image of a target object sent by a client;

[0099] Input the reference video into the action extraction model to obtain a reference action sequence;

[0100] According to the background requirement information, the reference background sequence is determined.

[0101] Specifically, the background requirement information is used to describe the requirements for the video background during the video generation process. For example, the background requirement information can be "indoor" or "outdoor". The action extraction model is used to extract the action sequence in the input video. The action extraction model can be a deep learning model trained based on the CNN model. For example, the action extraction model can be trained based on the sample video and the sample action sequence carried by the sample video.

[0102] It should be noted that the reference action sequence can be parsed from the reference video, and the reference action sequence can also be parsed. Therefore, the reference video may include the target object or may not include the target object. For example, if the target object is object A, the reference video can be a dancing video of object A, or a dancing video of object B, or a landscape video that does not include any object.

[0103] In practical applications, there are multiple ways to determine the reference background sequence based on the background requirement information, and the specific selection is based on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, multiple candidate background sequences can be obtained, and each candidate background sequence carries background description information. The background requirement information and the background description information are matched, and the candidate background sequence with the same background description information as the background requirement information is determined as the reference background sequence. In another possible implementation of this specification, the background requirement information can be input into the image generation model to obtain a reference background sequence.

[0104] By applying the solution of the embodiments of this specification, the user only needs to provide a reference video, background requirement information and an object image of the target object. The server determines a reference action sequence based on the reference video and a reference background sequence based on the background requirement information, thereby reducing the difficulty of data provision by the user and lowering the threshold for video generation.

[0105] Step 204: Input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0106] In one or more embodiments of the present specification, after obtaining the video generation data of the target object, the video generation data may be further input into a video generation model to obtain a target video of the target object.

[0107] Specifically, the target video refers to a video whose video data conforms to the video generation data. For example, the target video may be a video whose action of the target object conforms to the reference action sequence and whose video background conforms to the reference background sequence. The training methods of the video generation model and the pre-trained generation model are both supervised training.

[0108] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, the video generation model can learn temporal information, thereby generating a target video with stable timing and more realistic and natural, thereby improving the quality of the target video.

[0109] In an optional embodiment of the present specification, the video generation data includes a reference action sequence, a reference background sequence and an object image, different types of coding units include an image coding unit, an object coding unit, an action coding unit and a background coding unit, the data coding features include an object coding feature, an action coding sequence and a background coding sequence, the video generation model also includes a decoding unit, and the above-mentioned inputting the video generation data into the video generation model to obtain the target video of the target object may include the following steps:

[0110] Inputting the object image into the image coding unit to obtain image coding features, and inputting the image coding features into the object coding unit to obtain object coding features;

[0111] Inputting the reference action sequence into the action encoding unit to obtain an action encoding sequence;

[0112] Inputting the reference background sequence into the background coding unit to obtain a background coding sequence;

[0113] Inputting the object coding feature, the action coding sequence and the background coding sequence into the generation unit to obtain the video coding feature;

[0114] The video encoding features are input into the decoding unit to obtain the target video of the target object.

[0115] It should be noted that the image coding unit is connected to the object coding unit, the object coding unit is connected to the generation unit, the action coding unit is connected to the generation unit, the background coding unit is connected to the generation unit, and the generation unit is connected to the decoding unit. The image coding unit, the action coding unit and the background coding unit are composed of CNN network layers. The object coding unit and the generation unit are composed of Transformer network modules. Since the background is aligned at the pixel level, the background coding sequence can be directly used as the input of the generation unit. The generation unit can be constructed based on the UNet network of Stab le Diffus ion, or it can be constructed based on Video Stab le Diffus ion.

[0116] By applying the solution of the embodiment of this specification, an object coding unit is connected after the image coding unit to perform object control on the video coding features, thereby ensuring that the object in each frame of the generated target video is consistent with the target object.

[0117] In an optional embodiment of the present specification, the structure of the object coding unit is described. That is, the object coding unit includes multiple coding blocks; the above-mentioned inputting the image coding feature into the object coding unit to obtain the object coding feature may include the following steps:

[0118] Inputting the image coding feature into a first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is a first coding block among the multiple coding blocks;

[0119] Inputting the coding feature output by the previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks;

[0120] The object coding feature is determined according to the coding features respectively output by the multiple coding blocks.

[0121] Specifically, the coding block includes a self-attention unit and a cross-attention unit. When the image coding feature is input into the first coding block to obtain the first coding feature output by the first coding block, the image coding feature can be input into the self-attention unit in the coding block to obtain the image self-attention feature, and the image self-attention feature can be input into the cross-attention unit in the coding block to obtain the first coding feature.

[0122] Exemplarily, it is assumed that the object coding unit includes three sequentially connected coding blocks, which are coding block 1, coding block 2, and coding block 3. The process of inputting the image coding feature into the object coding unit may include: first, inputting the image coding feature into coding block 1 to obtain coding feature 1; then, inputting coding feature 1 into coding block 2 to obtain coding feature 2; finally, inputting coding feature 2 into coding block 3 to obtain coding feature 3. Coding feature 1, coding feature 2, and coding feature 3 together constitute the object coding feature.

[0123] The solution of the embodiment of this specification is applied, by setting multiple encoding blocks in the encoding unit, so as to facilitate the correspondence with each generation block in the generation unit, and prepare for the subsequent cross-attention mechanism operation.

[0124] In an optional embodiment of the present specification, the structure of the generation unit is described. That is, the generation unit includes multiple generation blocks, the object coding unit includes multiple coding blocks, and the generation blocks and the coding blocks correspond to each other one by one; the above-mentioned inputting the object coding feature, the action coding sequence and the background coding sequence into the generation unit to obtain the video coding feature may include the following steps:

[0125] Determine, according to the correspondence between the plurality of generation blocks and the plurality of encoding blocks, encoding features corresponding to the plurality of generation blocks respectively;

[0126] Inputting the action coding sequence, the background coding sequence, and the coding features corresponding to the first generation block into the first generation block to obtain the first generation features output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks;

[0127] Inputting a generated feature output by a previous generated block of the second generated block and a coding feature corresponding to the second generated block into the second generated block to obtain a second generated feature output by the second generated block, wherein the second generated block is any generated block other than the first generated block among the multiple generated blocks;

[0128] In a case where the second generation block is the last generation block among the multiple generation blocks, a second generation feature output by the second generation block is determined as the video encoding feature.

[0129] It should be noted that, by extracting the feature expressions of object images at different layers through the object encoding unit, and using the cross-attention mechanism in the same layer of the generation unit to fuse the object features into the generation unit, the objects in each frame of the generated target video can be consistent with the target object.

[0130] Exemplarily, it is assumed that the object encoding unit includes three sequentially connected encoding blocks, which are encoding block 1, encoding block 2, and encoding block 3. The generation unit includes three sequentially connected generation blocks, which are generation block 1, generation block 2, and generation block 3. The image encoding feature is input into the object encoding unit, and the encoding feature 1 output by encoding block 1, the encoding feature 2 output by encoding block 2, and the encoding feature 3 output by encoding block 3 can be obtained. After encoding block 1 outputs encoding feature 1, the cross-attention mechanism can be used to fuse encoding feature 1 into the input of generation block 1. For generation block 1, the action encoding sequence, the background encoding sequence, and encoding feature 1 are input into generation block 1, and generation block 1 outputs generated feature 1. After encoding block 2 outputs encoding feature 2, the cross-attention mechanism can be used to fuse encoding feature 2 into the input of generation block 2. For generation block 2, the generated feature 1 and the encoded feature 2 are input into generation block 2, and generation block 2 outputs generated feature 2. After encoding block 3 outputs encoding feature 3, the cross-attention mechanism can be used to fuse encoding feature 3 into the input of generation block 3. For generation block 3, generation feature 2 and coding feature 3 are input into generation block 3, and generation block 3 outputs generation feature 3. Since generation block 3 is the last generation block, generation feature 3 is determined as the video coding feature.

[0131] By applying the solution of the embodiment of this specification, the coding features in the input generation block are determined according to the correspondence between multiple generation blocks and multiple encoding blocks, so that the generation process of each generation block has the coding features of the corresponding encoding block output as constraints, preparing for subsequent cross-attention mechanism operations.

[0132] In an optional embodiment of the present specification, the structure of the generation block in the generation unit is described. Each generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit. Taking the second generation block as an example, that is, the second generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit; the above-mentioned inputting the generation feature output by the previous generation block of the second generation block and the encoding feature corresponding to the second generation block into the second generation block to obtain the second generation feature output by the second generation block can include the following steps:

[0133] Input the generated features output by the previous generation block of the second generation block into the self-attention unit to obtain the self-attention features;

[0134] Input the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain the cross-attention feature;

[0135] The cross-attention features are input into the temporal attention unit to obtain the second generated features.

[0136] Specifically, the self-attention unit is connected to the cross-attention unit, and the cross-attention unit is connected to the temporal attention unit. For example, in the second generation block, the temporal attention unit is after the cross-attention unit, and the self-attention unit is before the cross-attention unit. The self-attention unit is used to process data using the self-attention mechanism. The self-attention mechanism allows the self-attention unit to consider how each position in the input data (such as different regions of the image or different parts of the feature map) depends on all other positions. The self-attention mechanism can help the model identify and exploit the correlation between distant pixels. The cross-attention unit is used to process data using the cross-attention mechanism. The cross-attention mechanism can help the model guide the video generation process based on conditional information (such as background feature sequences, action feature sequences). The cross-attention mechanism allows the model to align the conditional information with the image features to ensure that the generated image meets the given conditions. The temporal attention unit is used to utilize the attention mechanism of the time dimension so that the temporal sequence of the features is correlated with each other, making the generated video results more stable.

[0137] By applying the solution of the embodiments of this specification, a self-attention mechanism is used in the self-attention unit to process data, so as to realize the recognition and utilization of correlations between long-distance pixels. A cross-attention mechanism is used in the cross-attention unit to process data, so that the model aligns conditional information with image features to ensure that the generated image meets the given conditions. An attention mechanism in the time dimension is used in the time-series attention unit to process data, so that the time series of features are correlated with each other, making the generated video results more stable.

[0138] In an optional embodiment of the present specification, after inputting the video generation data into the video generation model to obtain the target video of the target object, the following steps may also be included:

[0139] Send the target video to the client;

[0140] Receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client regarding the target video;

[0141] Based on the feedback information from the results, build a model to optimize the data;

[0142] Use model optimization data to adjust parameters of the video generation model.

[0143] Specifically, the result feedback information can be information that provides feedback on the content, quality, and completion of the target video, reflecting the client's true feelings and expectations of the target video. The result feedback information includes but is not limited to quality evaluation information, corrected accurate target video, and optimization areas of the model. Model optimization data refers to accurate optimized sample videos used to optimize the video generation model.

[0144] In practical applications, there are many ways to construct model optimization data based on the result feedback information, which are selected according to the actual situation. The embodiments of this specification do not limit this. In one possible implementation of this specification, the model optimization data can be automatically constructed directly based on the result feedback information. In another possible implementation of this specification, optimization prompt information can be generated based on the result feedback information, and the model optimization data sent by the client based on the optimization prompt information can be received.

[0145] Exemplarily, when generating optimization prompt information based on result feedback information, in one possible implementation of this specification, the preset prompt information can be directly obtained, and the result feedback information can be added to the preset prompt information to obtain the optimization prompt information, such as the preset prompt information is "I am very sorry for bringing you an inaccurate video. Please point out what is not accurate or provide an accurate video. I will correct and optimize my answer as soon as possible to better serve you." If the result feedback information is "the video background is inaccurate", the optimization prompt information is "Regarding the issue of inaccurate video background you reported, I am very sorry for bringing you an inaccurate video. Please point out what is not accurate or provide an accurate video. I will correct and optimize my answer as soon as possible to better serve you." In another possible implementation of this specification, the result feedback information can be type identified to determine the information type of the result feedback information, and the information type can be further matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type as the information type is determined as the optimization prompt information.

[0146] Furthermore, when constructing model optimization data directly based on the result feedback information, if the result feedback information is a corrected and accurate target video, the corrected and accurate target video can be determined as the model optimization data. If the result feedback information is an optimization field of the model, such as a certain field, multiple sample videos of the certain field can be obtained and the multiple sample videos can be determined as the model optimization data.

[0147] By applying the solution of the embodiments of this specification, the performance of the video generation model is continuously optimized by collecting and utilizing result feedback information to more accurately meet the actual needs of the client and improve the quality and accuracy of the final target video.

[0148] In an optional embodiment of the present specification, a training method of a video generation model is described. That is, before inputting the video generation data into the video generation model to obtain the target video of the target object, the following steps may also be included:

[0149] Obtaining a sample video and sample generation data corresponding to the sample video;

[0150] Inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model;

[0151] Inputting the sample video into the encoding unit to obtain a first sample feature;

[0152] According to the first prediction feature and the first sample feature, the unit parameters of the encoding unit and the temporal attention unit are adjusted to obtain a trained video generation model.

[0153] Specifically, the sample generation data includes sample object images, sample action sequences, and sample background sequences. The sample generation data can be parsed from the sample video. The first sample feature is the generation target of the generation unit in the video generation model, which is used to guide the training process of the video generation model. The video generation model is obtained by adding an additional temporal attention unit to each generation block of the generation unit of the pre-trained generation model, that is, the video generation model includes different types of encoding units, generation units, and decoding units; different types of encoding units include image encoding units, object encoding units, action encoding units, and background encoding units; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit, a cross-attention unit, and a temporal attention unit.

[0154] It should be noted that there are multiple ways to obtain sample videos, and the specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, multiple sample videos sent by the user through the client can be received. In another possible implementation of this specification, multiple sample videos can be read from other data acquisition devices or databases. When the sample video is input into the encoding unit to obtain the first sample feature, the sample video can be input into the image encoding unit to obtain the first sample feature. When adjusting the unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature, only the unit parameters of the background encoding unit and the temporal attention unit can be adjusted.

[0155] Furthermore, after obtaining the sample video, for the sample object image: any video frame including the sample object can be randomly selected from the sample video as the sample object image, or a video frame including the sample object with higher quality can be selected as the sample object image. For the sample action sequence: the sample video can be input into the action extraction model to obtain the sample action sequence, or the sample video frame can be extracted (such as extracting frames at 30FPS), and for each sample video frame, the sample video frame is input into the key point recognition model to obtain the sample action key points, and the sample action image including the object skeleton is drawn. The sample action image corresponding to each sample video frame can constitute the sample background sequence. For the sample background sequence: the sample video can be extracted from the sample video frame, and for each sample video frame, the segmentation model can be used to obtain the masks of the foreground person and the background respectively, and then the filling model can be used to complete the background of the foreground part according to the known partial background to generate a sample background image. The sample background image corresponding to each sample video frame can constitute the sample background sequence.

[0156] In practical applications, according to the first prediction feature and the first sample feature, the unit parameters of the coding unit and the temporal attention unit are adjusted. When the trained video generation model is obtained, the loss value can be calculated according to the first prediction feature and the first sample feature, and a low-rank adaptive method is used to adjust the unit parameters of the coding unit and the temporal attention unit according to the loss value until the training process meets the preset stop condition, thereby obtaining the trained video generation model. Among them, there are many functions for calculating the loss value, such as the cross entropy loss function, the L1 norm loss function, the maximum loss function, the mean square error loss function, the logarithmic loss function, etc., which are selected according to the actual situation, and the embodiments of this specification do not make any restrictions on this. The preset stop conditions include but are not limited to the loss value being less than or equal to the preset threshold value and the number of iterations reaching the preset number of iterations, wherein the preset threshold value and the preset number of iterations are selected according to the actual situation, and the embodiments of this specification do not make any restrictions on this.

[0157] In a possible implementation of the present specification, after calculating the loss value, the loss value is compared with a preset threshold. Specifically, if the loss value is greater than the preset threshold, it means that the difference between the first prediction feature and the first sample feature is large, and the video generation model has poor prediction ability for the video feature. At this time, the unit parameters of the encoding unit and the temporal attention unit can be adjusted, and the step of inputting the sample generation data into the video generation model to obtain the first prediction feature output by the generation unit in the video generation model is returned to continue training the video generation model until the loss value is less than or equal to the preset threshold, indicating that the difference between the target first prediction feature and the first sample feature is small, and the preset stop condition is met, and the trained video generation model is obtained.

[0158] In another possible implementation of the present specification, in addition to comparing the size relationship between the loss value and the preset threshold, the number of iterations can also be combined to determine whether the current video generation model has been trained. Specifically, if the loss value is greater than the preset threshold, the unit parameters of the encoding unit and the temporal attention unit are adjusted, and the step of inputting the sample generation data into the video generation model to obtain the first prediction feature output by the generation unit in the video generation model is returned to continue training the video generation model until the preset number of iterations is reached, and the iteration is stopped to obtain a video generation model that has been trained.

[0159] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, a target video with stable temporal sequence and natural fusion of foreground and background can be generated, making the target video more realistic and natural.

[0160] See also Figure 3 , Figure 3A processing flow chart of a video generation model training method provided by an embodiment of the present specification is shown, and the training stage of the video generation model can be called the video generation training stage. In this stage, the model training goal is to generate a video clip with stable temporal sequence, and at the same time, the generation of the background can be controlled. Therefore, a temporal attention unit is introduced into the pre-trained generation model to obtain a video generation model. By adding the temporal attention unit, the relevant associations between different frames in the temporal dimension can be learned, and the video generation model can further learn temporal information, such as fluttering hems and continuous and stable temporal change features. That is, the video generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a background encoding unit, a generation unit and a decoding unit; the object encoding unit includes a plurality of encoding blocks, and the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and the generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit. During the training process, only the unit parameters of the background encoding unit and the temporal attention unit are adjusted, and the unit parameters of other units except the background encoding unit and the temporal attention unit are fixed.

[0161] Next, the training process of the video generation model is described. Figure 3 As shown, a sample video is obtained; the sample video is parsed to obtain a sample object image, a sample action sequence and a sample background sequence; the sample object image is input into the image coding unit in the video generation model to obtain a sample image coding feature, and the image coding feature is input into the object coding unit in the video generation model to obtain a sample object coding feature; the sample action sequence is input into the action coding unit in the video generation model to obtain a sample action coding sequence; the sample background sequence is input into the background coding unit in the video generation model to obtain a sample background coding sequence; the sample object coding feature, the sample action coding sequence and the sample background coding sequence are input into the generation unit in the video generation model to obtain a first prediction feature; the first prediction feature is input into the decoding unit in the video generation model to obtain a predicted video. The sample video is input into the image coding unit in the video generation model to obtain a first sample feature; according to the first prediction feature and the first sample feature, the unit parameters of the background coding unit and the temporal attention unit in the video generation model are adjusted to obtain a trained video generation model.

[0162] By applying the solution of the embodiment of this specification, when training the video generation model, the newly added temporal attention unit and background encoding unit are trained at the same time, so that the generated result can learn the consistency of foreground and background lighting while the temporal sequence is stable, making the result of the entire video more realistic and natural. Moreover, after the above model training, the final video generation model can generate video clips with consistent objects and consistent with the input action sequence and background sequence.

[0163] In an optional embodiment of the present specification, a training method of a pre-trained generation model is described, that is, the sample video includes a plurality of sample video frames; before the sample generation data is input into the video generation model and the first prediction feature output by the generation unit in the video generation model is obtained, the following steps may also be included:

[0164] For a first sample video frame, determining first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from a plurality of sample video frames;

[0165] Inputting the first sample generation data into the initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model;

[0166] Inputting the first sample video frame into the encoding unit to obtain a second sample feature;

[0167] According to the second prediction feature and the second sample feature, adjusting unit parameters of the encoding unit and the generating unit to obtain a pre-trained generating model;

[0168] In the generation unit of the pre-trained generation model, a temporal attention unit is constructed to obtain a video generation model.

[0169] Specifically, the first sample generation data includes a first sample object image, a first sample action image, and a first sample background image. The second sample feature is a generation target of a generation unit in the initial generation model, which is used to guide the training process of the initial generation model. Since the initial generation model is trained based on a single video frame and does not include timing information, the generation unit of the initial generation model does not include a temporal attention unit, that is, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a background encoding unit, a generation unit, and a decoding unit; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit and a cross-attention unit.

[0170] It should be noted that when sampling from multiple sample video frames to obtain a first sample video frame, in one possible implementation, all sample video frames can be determined as first sample video frames. In another possible implementation, since the similarity between adjacent frames is high, the first sample video frame can be extracted at preset time intervals. When the first sample video frame is input into the encoding unit to obtain the second sample feature, the first sample video frame can be input into the image encoding unit to obtain the second sample feature. When adjusting the unit parameters of the encoding unit and the generation unit according to the second prediction feature and the second sample feature, the unit parameters of the object encoding unit, the action encoding unit, the background encoding unit and the generation unit can be adjusted to obtain a pre-trained generation model.

[0171] In actual applications, the implementation method of "adjusting the unit parameters of the encoding unit and the generation unit according to the second prediction feature and the second sample feature to obtain a pre-trained generation model" is the same as the implementation method of "adjusting the unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature to obtain a trained video generation model" mentioned above, and will not be repeated in the embodiments of this specification.

[0172] Furthermore, according to the second prediction feature and the second sample feature, the unit parameters of the encoding unit and the generation unit in the initial generation model are adjusted. After the pre-trained generation model is obtained, the pre-trained generation model can be used to process the image generation task, that is, the object image, action image and background image of the target object are input into the pre-trained generation model to obtain the target image, the target image includes the target object, and the action of the target object is consistent with the action in the action image, and the background of the target image is consistent with the background image.

[0173] By applying the solution of the embodiments of the present specification, an additional temporal attention unit is added to each generation block of the generation unit in the pre-trained generation model to obtain a video generation model, so that the video generation model can learn temporal information, and according to the second prediction feature and the second sample feature, the unit parameters of the encoding unit and the generation unit in the initial generation model are adjusted, so that the pre-trained generation model can generate images with consistent objects, actions and backgrounds, thereby ensuring the generation capability of the pre-trained generation model.

[0174] See also Figure 4 , Figure 4 A flowchart of the processing process of a pre-trained generative model training method provided by an embodiment of the present specification is shown. The pre-trained generative model is obtained by adjusting the unit parameters of the object coding unit, action coding unit, background coding unit and generation unit in the initial generative model based on the sample video. This training stage can be called the image generation training stage. In this stage, the model training goal is to generate images that are consistent with the object, the target action and the target background. The initial generative model can reuse the weights of Stab le Diffus ion, and add action coding units, object coding units and background coding units on the basis of Stab le Diffus ion. In other words, Figure 4 As shown in the figure, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a background encoding unit, a generation unit and a decoding unit; the object encoding unit includes multiple encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes multiple generation blocks, and the generation block includes a self-attention unit and a cross-attention unit. During the training process, only the unit parameters of the object encoding unit, the action encoding unit, the background encoding unit and the generation unit are adjusted, while the unit parameters of the image encoding unit and the decoding unit are fixed.

[0175] Next, the processing of the initial generation model is described. Figure 4 As shown, a sample video is obtained, wherein the sample video includes multiple sample video frames; for a first sample video frame, a first sample object image, a first sample action image, and a first sample background image corresponding to the first sample video frame are determined, wherein the first sample video frame is sampled from multiple sample video frames; the first sample object image, the first sample action image, and the first sample background image are input into an initial generation model to obtain a first predicted video frame output by the initial generation model. The first sample video frame is input into an image encoding unit in the initial generation model to obtain a second sample feature; according to the second sample feature and the second predicted feature output by the generation unit in the initial generation model, the unit parameters of the object encoding unit, the action encoding unit, and the background encoding unit in the initial generation model are adjusted to obtain a pre-trained generation model.

[0176] The following combination Figure 5 Taking the application of the video generation method provided in this specification in the virtual object motion video generation scene as an example, the video generation method is further described. Figure 5 A flowchart of a method for generating a motion video of a virtual object provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0177] Step 502: Obtain motion video generation data of a virtual object.

[0178] Step 504: Input the motion video generation data into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, the generation unit includes a temporal attention unit, the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0179] It should be noted that the implementation of step 502 to step 504 is the same as the implementation of step 202 to step 204, and will not be described in detail in the embodiments of this specification.

[0180] In practical applications, in a virtual object motion video generation scenario, the virtual object image of the virtual object, the action sequence of a certain sport and the sports field background sequence can be input into a video generation model to obtain a target motion video of the virtual object performing a sport in the sports field.

[0181] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, the video generation model can learn temporal information, thereby generating a target motion video with stable timing and natural fusion of foreground and background, making the target motion video more realistic and natural.

[0182] The following combination Figure 6 , taking the application of the video generation method provided in this specification in the video editing (such as replacement, modification, relighting) scene as an example, the video generation method is further explained. Figure 6 A flowchart of a video editing method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0183] Step 602: Obtain original video and target editing data.

[0184] Step 604: parse the original video to determine the original video data in the original video.

[0185] Step 606: Input the target editing data and the original video data into the video generation model to obtain the target editing video, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0186] It should be noted that the implementation of step 602 to step 606 is the same as the implementation of step 202 to step 204, and will not be described in detail in this embodiment of the specification.

[0187] In practical applications, when parsing the original video and determining the original video data in the original video, the original video can be parsed based on the target editing data to determine the original video data in the original video. Specifically, if the video editing task is a video background editing task, the target editing data is the target background sequence, and the original video is parsed to determine that the original video data in the original video includes the original action sequence and the original object image. If the video editing task is a video object editing task, the target editing data is the target object image, and the original video is parsed to determine that the original video data in the original video includes the original background sequence and the original action sequence. If the video editing task is a video action editing task, the target editing data is the target action sequence, and the original video is parsed to determine that the original video data in the original video includes the original background sequence and the original object image.

[0188] By applying the solution of the embodiments of this specification, the editing of the actions, background, and objects of the original video is achieved by inputting the target editing data and the original video data into the video generation model. In addition, since the video generation model includes a temporal attention unit, the video generation model can learn the temporal information, thereby generating a target edited video with stable timing and natural fusion of the foreground and background, making the target edited video more realistic and natural.

[0189] See also Figure 7 , Figure 7 A flowchart of a video generation model training method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0190] Step 702: Obtain a sample video and sample generation data corresponding to the sample video.

[0191] Step 704: Input the sample generation data into the video generation model to obtain the first prediction feature output by the generation unit in the video generation model, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the sample generation data to obtain sample encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the sample encoding features using the attention mechanism of the time dimension.

[0192] Step 706: Input the sample video into the encoding unit to obtain the first sample feature.

[0193] Step 708: According to the first prediction feature and the first sample feature, adjust the unit parameters of the encoding unit and the temporal attention unit to obtain a trained video generation model.

[0194] It should be noted that the implementation method of steps 702 to 708 is the same as that of the above Figure 2 The training method of the video generation model in the provided video generation method is the same, and will not be repeated in the embodiments of this specification.

[0195] By applying the solution of the embodiments of this specification, an additional temporal attention unit is added to each generation block included in the generation unit of the pre-trained generation model to obtain a video generation model, so that the video generation model can learn temporal information, thereby generating a target video with stable timing and natural fusion of foreground and background, making the target video more realistic and natural.

[0196] See also Figure 8 , Figure 8 A flowchart of an information processing method based on a video generation model provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0197] Step 802: Receive a task generation request, wherein the task generation request includes request information.

[0198] Specifically, the information processing method based on the video generation model can be applied to the task platform, and the task platform can be deployed on the terminal device or the cloud device. The task generation request is used to request the task information for generating the target video task. The task generation request usually includes the task type, the expected output format, and the request information. The request information refers to the parameters or description information related to the target video task carried in the task generation request. The request information includes but is not limited to the task scene identifier of the target video task, the task model identifier, and multiple sample videos corresponding to the target video task.

[0199] Step 804: Based on the request information, obtain a video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in the time dimension.

[0200] In actual applications, there are multiple ways to obtain a video generation model based on the request information, and the specific method is selected according to the actual situation. The embodiments of this specification do not impose any limitation on this.

[0201] In an optional embodiment of the present specification, the obtaining of the video generation model based on the request information may include the following steps:

[0202] Based on the task scene identifier, determine a target scene template from a plurality of preset scene templates, and based on the target scene template, search for a video generation model from a model library, wherein the model library stores a plurality of video generation models, and the request information includes a task scene identifier of the target video task; or,

[0203] Based on the task model identifier, a video generation model is searched from a model library, wherein the request information includes the task model identifier of the target video task.

[0204] Specifically, the task scene identifier refers to a unique or specific label used to distinguish different task application scenarios. In the embodiment of this specification, the task scene identifier is part of the request information. Through the task scene identifier, a target scene template that matches the request information can be selected from a series of preset scene templates to generate task information. For example, the task scene identifier is "video background control", which means that the user wants to control and replace the background of the uploaded original video, then the video background control scene template can be selected from multiple preset scene templates according to the task scene identifier.

[0205] The preset scene templates are predefined standard configuration scene templates for application scenarios of different target video tasks. Each template contains model information that matches the application scenario of the task and information such as the task processing flow. Through a series of preset scene templates, task generation requests for different scenarios can be quickly responded to. Different preset scene templates correspond to different task types, model information, and processing flows. For example, there may be a video motion control scene template specifically for video motion control tasks in the preset scene templates, which contains the model information and processing flow of the trained video motion control model.

[0206] The target scene template refers to the scene template that matches the task scene identifier. When parsing the task generation request, the corresponding target scene template can be located based on the task scene identifier, and the corresponding video generation model and other related configuration information can be selected from the model library based on the model information included in the target scene template. For example, when the task scene identifier is "video object control", the target scene template is a template that contains the model information and related configuration parameters of the video object control model.

[0207] The model library is a resource library that centrally stores video generation models. These models are trained and optimized to solve different target video tasks. In addition, the video generation models in the model library can be divided into different versions according to different applicable tasks. For example, the model library may store video background control models for video background control tasks, video object control models for video object control tasks, and video action control models for video action control tasks.

[0208] The task model identifier refers to a unique or specific label used to distinguish models applicable to different tasks. For example, the task model identifier may be "video background control". Based on the task model identifier, a video background control model applicable to the video background control task can be found from the model library.

[0209] By applying the solution of the embodiments of this specification, the process of acquiring the video generation model is made more flexible, efficient and standardized through pre-defined task scenario templates, task model identifiers and model library resources.

[0210] In another optional embodiment of the present specification, in addition to selecting a video generation model from a model library, the video generation model in the model library may be trained according to the sample video in the request information and the sample generation data corresponding to the sample video to obtain the video generation model, that is, the request information includes the sample video of the target video task and the sample generation data corresponding to the sample video; the above-mentioned acquisition of the video generation model based on the request information may include the following steps:

[0211] Based on the sample video and the sample generation data corresponding to the sample video, a video generation model corresponding to the target video task is trained to obtain a trained video generation model.

[0212] It should be noted that the implementation method of "training the video generation model corresponding to the target video task based on the sample video and the sample generation data corresponding to the sample video to obtain the trained video generation model" is the same as the training method of the above-mentioned "video generation model", and will not be repeated in the embodiments of this specification.

[0213] By applying the solution of the embodiments of this specification, a video generation model corresponding to a target video task is trained based on sample videos and sample generation data corresponding to the sample videos to obtain a trained video generation model. While ensuring the accuracy of the video generation model, the video generation model is made more in line with the task scenario of the target video task.

[0214] Step 806: Generate task information based on the video generation model, where the task information is used to perform the target video task.

[0215] Specifically, the task information includes the model configuration and processing flow required to perform the target video task. The terminal device or other server-side components can correctly use the video generation model to process the target video task based on the task information. The target video tasks include but are not limited to video background control characters, video action control tasks, video object control tasks, and video generation tasks.

[0216] It should be noted that when generating task information based on the video generation model, the model parameters of the video generation model can be directly packaged to obtain the task information. Other model information of the video generation model can also be obtained to construct the task information based on the other model information, where the other model information includes the processing method of the model input data, the specifications of the expected output results, and the intermediate steps and other auxiliary information that may be involved.

[0217] By applying the solution of the embodiments of this specification, by generating task information of the target video task, it is possible to reduce system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provide users with convenient and efficient target video task processing services.

[0218] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a task platform provided by an embodiment of the present specification is shown, the task platform includes a request interface 902 and a response unit 904;

[0219] A request interface 902, used to receive a task generation request, wherein the task generation request includes request information;

[0220] A response unit 904 is used for the response unit, and is used to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in the time dimension; based on the video generation model, task information is generated, wherein the task information is used to execute the target video task.

[0221] In an optional embodiment of the present specification, the task platform further includes a model library, wherein the model library stores a plurality of video generation models;

[0222] The response unit is specifically used to determine a target scene template from multiple preset scene templates based on a task scene identifier, and to search a video generation model from a model library based on the target scene template, wherein the request information includes the task scene identifier of the target video task; or, to search a video generation model from a model library based on a task model identifier, wherein the request information includes the task model identifier of the target video task.

[0223] By applying the solution of the embodiments of this specification, by generating task information of the target video task, it is possible to reduce system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provide users with convenient and efficient target video task processing services.

[0224] The above is a schematic scheme of a task platform of this embodiment. The technical scheme of the task platform and the technical scheme of the information processing method based on the video generation model are of the same concept. For details not described in detail in the technical scheme of the task platform, please refer to the description of the technical scheme of the information processing method based on the video generation model.

[0225] Corresponding to the above-mentioned video generation method embodiment, this specification also provides a video generation device embodiment, Fig.10 FIG. 2 shows a schematic diagram of the structure of a video generation device provided by an embodiment of the present specification. Fig.10 As shown, the device comprises:

[0226] A first acquisition module 1002 is configured to acquire video generation data of a target object;

[0227] The first input module 1004 is configured to input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0228] Optionally, the video generation data includes a reference action sequence, a reference background sequence and an object image, different types of coding units include image coding units, object coding units, action coding units and background coding units, data coding features include object coding features, action coding sequences and background coding sequences, and the video generation model also includes a decoding unit; the first input module 1004 is further configured to input the object image into the image coding unit to obtain image coding features, and input the image coding features into the object coding unit to obtain object coding features; input the reference action sequence into the action coding unit to obtain an action coding sequence; input the reference background sequence into the background coding unit to obtain a background coding sequence; input the object coding features, action coding sequence and background coding sequence into the generation unit to obtain video coding features; input the video coding features into the decoding unit to obtain a target video of the target object.

[0229] Optionally, the object coding unit includes multiple coding blocks; the first input module 1004 is further configured to input the image coding feature into the first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is the first coding block among the multiple coding blocks; input the coding feature output by the previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks; determine the object coding feature based on the coding features respectively output by the multiple coding blocks.

[0230] Optionally, the generation unit includes multiple generation blocks, the object coding unit includes multiple coding blocks, and the generation blocks and the coding blocks correspond one to one; the first input module 1004 is further configured to determine the coding features corresponding to the multiple generation blocks respectively according to the correspondence between the multiple generation blocks and the multiple coding blocks; the action coding sequence, the background coding sequence and the coding features corresponding to the first generation block are input into the first generation block to obtain the first generation feature output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks; the generation feature output by the previous generation block of the second generation block and the coding feature corresponding to the second generation block are input into the second generation block to obtain the second generation feature output by the second generation block, wherein the second generation block is any generation block other than the first generation block among the multiple generation blocks; when the second generation block is the last generation block among the multiple generation blocks, the second generation feature output by the second generation block is determined as the video coding feature.

[0231] Optionally, the second generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit; the first input module 1004 is further configured to input the generation feature output by the previous generation block of the second generation block into the self-attention unit to obtain a self-attention feature; input the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain a cross-attention feature; input the cross-attention feature into the temporal attention unit to obtain a second generation feature.

[0232] Optionally, the device also includes: a first training module, configured to obtain a sample video and sample generation data corresponding to the sample video; input the sample generation data into a video generation model to obtain a first prediction feature output by a generation unit in the video generation model; input the sample video into an encoding unit to obtain a first sample feature; and adjust unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0233] Optionally, the sample video includes multiple sample video frames; the device also includes: a second training module, configured to determine, for a first sample video frame, first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from multiple sample video frames; input the first sample generation data into an initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model; input the first sample video frame into an encoding unit to obtain a second sample feature; adjust unit parameters of the encoding unit and the generation unit according to the second prediction feature and the second sample feature to obtain a pre-trained generation model; construct a temporal attention unit in the generation unit of the pre-trained generation model to obtain a video generation model.

[0234] Optionally, the video generation data includes a reference action sequence, a reference background sequence and an object image; the first acquisition module 1002 is further configured to receive a reference video, background requirement information and an object image of a target object sent by a client; input the reference video into an action extraction model to obtain a reference action sequence; and determine a reference background sequence based on the background requirement information.

[0235] Optionally, the device also includes: a sending module configured to send the target video to the client; receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client on the target video; constructing model optimization data based on the result feedback information; and adjusting parameters of the video generation model using the model optimization data.

[0236] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, the video generation model can learn temporal information, thereby generating a target video with stable timing and more realistic and natural, thereby improving the quality of the target video.

[0237] The above is a schematic scheme of a video generation device of this embodiment. It should be noted that the technical scheme of the video generation device and the technical scheme of the above video generation method belong to the same concept, and the details not described in detail in the technical scheme of the video generation device can be referred to the description of the technical scheme of the above video generation method.

[0238] Corresponding to the above-mentioned method embodiment for generating a motion video of a virtual object, this specification also provides an embodiment of a device for generating a motion video of a virtual object. Fig.11 FIG. 1 is a schematic diagram showing a structure of a motion video generation device for a virtual object provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0239] The second acquisition module 1102 is configured to acquire motion video generation data of the virtual object;

[0240] The second input module 1104 is configured to input the motion video generation data into the video generation model to obtain the target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the motion video generation data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0241] By applying the solution of the embodiments of this specification, since the video generation model includes a temporal attention unit, the video generation model can learn temporal information to generate a temporally stable and natural target motion video.

[0242] The above is a schematic scheme of a motion video generation device for a virtual object of this embodiment. It should be noted that the technical scheme of the motion video generation device for a virtual object and the technical scheme of the motion video generation method for a virtual object are of the same concept, and the details not described in detail in the technical scheme of the motion video generation device for a virtual object can be referred to the description of the technical scheme of the motion video generation method for a virtual object.

[0243] Corresponding to the above-mentioned video editing method embodiment, this specification also provides a video editing device embodiment, Fig.12 FIG. 2 shows a schematic diagram of the structure of a video editing device provided by an embodiment of the present specification. Fig.12 As shown, the device comprises:

[0244] The third acquisition module 1202 is configured to acquire the original video and the target editing data;

[0245] The parsing module 1204 is configured to parse the original video and determine the original video data in the original video;

[0246] The third input module 1206 is configured to input the target editing data and the original video data into the video generation model to obtain the target edited video, wherein the video generation model includes a generation unit and different types of encoding units, and the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features. The generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features using the attention mechanism of the time dimension.

[0247] By applying the solution of the embodiments of this specification, the editing of the original video is achieved by inputting the target editing data and the original video data into the video generation model. In addition, since the video generation model includes a temporal attention unit, the video generation model can learn the temporal information to generate a target edited video with stable and natural timing.

[0248] The above is a schematic scheme of a video editing device of this embodiment. It should be noted that the technical scheme of the video editing device and the technical scheme of the above video editing method belong to the same concept, and the details not described in detail in the technical scheme of the video editing device can be referred to the description of the technical scheme of the above video editing method.

[0249] Corresponding to the above-mentioned video generation model training method embodiment, this specification also provides a video generation model training device embodiment, Fig.13 FIG. 1 is a schematic diagram showing the structure of a video generation model training device provided by an embodiment of the present specification. Fig.13 As shown, the device comprises:

[0250] The fourth acquisition module 1302 is configured to acquire a sample video and sample generation data corresponding to the sample video;

[0251] The fourth input module 1304 is configured to input the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain the sample encoding feature, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the sample encoding feature using an attention mechanism in a time dimension;

[0252] A fifth input module 1306 is configured to input the sample video into the encoding unit to obtain a first sample feature;

[0253] The adjustment module 1308 is configured to adjust the unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature to obtain a trained video generation model.

[0254] By applying the solution of the embodiments of this specification, an additional temporal attention unit is added to each generation block included in the generation unit of the pre-trained generation model to obtain a video generation model, so that the video generation model can learn temporal information, thereby generating a target video with stable timing and natural fusion of foreground and background, making the target video more realistic and natural.

[0255] The above is a schematic scheme of a video generation model training device of this embodiment. It should be noted that the technical scheme of the video generation model training device and the technical scheme of the above-mentioned video generation model training method belong to the same concept, and the details not described in detail in the technical scheme of the video generation model training device can be referred to the description of the technical scheme of the above-mentioned video generation model training method.

[0256] Corresponding to the above-mentioned information processing method embodiment based on the video generation model, this specification also provides an information processing device embodiment based on the video generation model, Fig.14 FIG. 1 shows a schematic diagram of the structure of an information processing device based on a video generation model provided by an embodiment of the present specification. Fig.14 As shown, the device comprises:

[0257] The receiving module 1402 is configured to receive a task generation request, wherein the task generation request includes request information;

[0258] A fifth acquisition module 1404 is configured to acquire a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by different types of encoding units using an attention mechanism in a time dimension;

[0259] The generation module 1406 is configured to generate task information based on the video generation model, wherein the task information is used to perform the target video task.

[0260] Optionally, the fifth acquisition module 1404 is further configured to determine a target scene template from multiple preset scene templates based on the task scene identifier, and search for a video generation model from a model library based on the target scene template, wherein the model library stores multiple video generation models and the request information includes the task scene identifier of the target video task; or, search for a video generation model from the model library based on the task model identifier, wherein the request information includes the task model identifier of the target video task.

[0261] Optionally, the request information includes a sample video of the target video task and sample generation data corresponding to the sample video; the fifth acquisition module 1404 is further configured to train a video generation model corresponding to the target video task based on the sample video and the sample generation data corresponding to the sample video to obtain a trained video generation model.

[0262] By applying the solution of the embodiments of this specification, by generating task information of the target video task, the system deployment and operation and maintenance costs are reduced while ensuring the processing quality and efficiency of the target video task, providing users with convenient and efficient task processing services.

[0263] The above is a schematic scheme of an information processing device based on a video generation model of this embodiment. It should be noted that the technical scheme of the information processing device based on the video generation model and the technical scheme of the information processing method based on the video generation model belong to the same concept, and the details not described in detail in the technical scheme of the information processing device based on the video generation model can be referred to the description of the technical scheme of the information processing method based on the video generation model.

[0264] Fig.15 The block diagram of a computing device provided by one embodiment of the present specification is shown. The components of the computing device 1500 include but are not limited to a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 via a bus 1530, and the database 1550 is used to store data.

[0265] The computing device 1500 also includes an access device 1540 that enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1540 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0266] In one embodiment of the present specification, the above components of the computing device 1500 and Fig.15 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.15 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0267] The computing device 1500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1500 may also be a mobile or stationary server.

[0268] Among them, processor 1520 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned video generation method or virtual object motion video generation method or video editing method or video generation model training method or information processing method based on video generation model.

[0269] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model belong to the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0270] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0271] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model belong to the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0272] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0273] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model are of the same concept. For details not described in detail in the technical scheme of the computer program product, please refer to the description of the technical scheme of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0274] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0275] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0276] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0277] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0278] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A video generation method, include: Obtaining video generation data of a target object; The video generation data is input into a video generation model to obtain a target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the data encoding features using an attention mechanism in the time dimension.

2. The method according to claim 1, wherein the video generation data comprises a reference action sequence, a reference background sequence and an object image, the different types of encoding units comprise an image encoding unit, an object encoding unit, an action encoding unit and a background encoding unit, the data encoding features comprise an object encoding feature, an action encoding sequence and a background encoding sequence, and the video generation model further comprises a decoding unit; The video generation data is input into a video generation model to obtain a target video of the target object, include: Inputting the object image into the image encoding unit to obtain image encoding features, and inputting the image encoding features into the object encoding unit to obtain the object encoding features; Inputting the reference action sequence into the action encoding unit to obtain the action encoding sequence; Inputting the reference background sequence into the background coding unit to obtain the background coding sequence; Inputting the object coding feature, the action coding sequence and the background coding sequence into the generation unit to obtain a video coding feature; The video coding feature is input into the decoding unit to obtain the target video of the target object.

3. The method according to claim 2, wherein the object coding unit comprises a plurality of coding blocks; The image coding feature is input into the object coding unit to obtain the object coding feature, include: Inputting the image coding feature into a first coding block to obtain a first coding feature output by the first coding block, wherein the first coding block is a first coding block among the multiple coding blocks; Inputting a coding feature output by a previous coding block of the second coding block into the second coding block to obtain a second coding feature output by the second coding block, wherein the second coding block is any coding block other than the first coding block among the multiple coding blocks; The object coding feature is determined according to the coding features respectively output by the multiple coding blocks.

4. The method according to claim 2, wherein the generation unit comprises a plurality of generation blocks, the object coding unit comprises a plurality of coding blocks, and the generation blocks correspond to the coding blocks one by one; The step of inputting the object coding feature, the action coding sequence and the background coding sequence into the generating unit to obtain the video coding feature comprises: Determine, according to the correspondence between the multiple generation blocks and the multiple encoding blocks, encoding features corresponding to the multiple generation blocks respectively; Inputting the action coding sequence, the background coding sequence and the coding features corresponding to the first generation block into the first generation block, obtaining a first generation feature output by the first generation block, wherein the first generation block is the first generation block among the multiple generation blocks; Inputting a generated feature output by a previous generation block of the second generation block and a coding feature corresponding to the second generation block into the second generation block, to obtain a second generated feature output by the second generation block, wherein the second generation block is any generation block other than the first generation block among the multiple generation blocks; In a case where the second generation block is the last generation block among the multiple generation blocks, a second generation feature output by the second generation block is determined as the video encoding feature.

5. The method according to claim 4, wherein the second generation block comprises a self-attention unit, a cross-attention unit, and a temporal attention unit; The generating feature output by the previous generating block of the second generating block and the encoding feature corresponding to the second generating block are input into the second generating block to obtain the second generating feature output by the second generating block, include: Inputting the generated features output by the previous generation block of the second generation block into the self-attention unit to obtain the self-attention features; Inputting the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain a cross-attention feature; The cross-attention feature is input into the temporal attention unit to obtain the second generated feature.

6. The method according to claim 1, before inputting the video generation data into the video generation model to obtain the target video of the target object, include: Obtaining a sample video and sample generation data corresponding to the sample video; Inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model; Inputting the sample video into the encoding unit to obtain a first sample feature; According to the first prediction feature and the first sample feature, the unit parameters of the encoding unit and the temporal attention unit are adjusted to obtain the trained video generation model.

7. The method according to claim 6, wherein the sample video comprises a plurality of sample video frames; Before inputting the sample generation data into the video generation model to obtain the first prediction feature output by the generation unit in the video generation model, include: For a first sample video frame, determining first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from the multiple sample video frames; Inputting the first sample generation data into an initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model; Inputting the first sample video frame into the encoding unit to obtain a second sample feature; According to the second prediction feature and the second sample feature, adjusting unit parameters of the encoding unit and the generating unit to obtain a pre-trained generating model; In the generation unit of the pre-trained generation model, a temporal attention unit is constructed to obtain the video generation model.

8. The method according to claim 1, wherein the video generation data comprises a reference action sequence, a reference background sequence and an object image; The step of obtaining the video generated data of the target object includes: Receive a reference video, background requirement information, and an object image of a target object sent by a client; Inputting the reference video into an action extraction model to obtain the reference action sequence; The reference background sequence is determined according to the background requirement information.

9. The method according to any one of claims 1 to 8, wherein after inputting the video generation data into a video generation model to obtain a target video of the target object, include: Sending the target video to the client; Receiving result feedback information sent by the client, wherein the result feedback information is information provided by the client regarding the target video; According to the result feedback information, a model is constructed to optimize the data; The model optimization data is used to adjust parameters of the video generation model.

10. A method for generating motion video of a virtual object, include: Obtaining motion video generation data of a virtual object; The motion video generation data is input into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the data encoding features using an attention mechanism in the time dimension.

11. A video editing method, include: Get the original video and target editing data; Parsing the original video to determine original video data in the original video; The target editing data and the original video data are input into a video generation model to obtain a target editing video, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the data encoding features using an attention mechanism in the time dimension.

12. A video generation model training method, include: Obtaining a sample video and sample generation data corresponding to the sample video; Inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the sample encoding features using an attention mechanism in a time dimension; Inputting the sample video into the encoding unit to obtain a first sample feature; According to the first prediction feature and the first sample feature, the unit parameters of the encoding unit and the temporal attention unit are adjusted to obtain the trained video generation model.

13. An information processing method based on a video generation model, include: receiving a task generation request, wherein the task generation request includes request information; Based on the request information, a video generation model is obtained, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by the different types of encoding units using an attention mechanism in a time dimension; Based on the video generation model, task information is generated, wherein the task information is used to perform a target video task.

14. The method according to claim 13, wherein the video generation model is obtained based on the request information. include: Based on the task scene identifier, determine a target scene template from a plurality of preset scene templates, and based on the target scene template, search for a video generation model from a model library, wherein the model library stores a plurality of video generation models, and the request information includes the task scene identifier of the target video task; or Based on the task model identifier, a video generation model is searched from the model library, wherein the request information includes the task model identifier of the target video task.

15. The method according to claim 13, wherein the request information includes a sample video of the target video task and sample generation data corresponding to the sample video; Based on the request information, a video generation model is obtained, include: Based on the sample video and the sample generation data corresponding to the sample video, the video generation model corresponding to the target video task is trained to obtain a trained video generation model.

16. A task platform, comprising a request interface and a response unit; The request interface is used to receive a task generation request, wherein: The task generation request includes request information; The response unit is used to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by the different types of encoding units using an attention mechanism in the time dimension; based on the video generation model, task information is generated, wherein the task information is used to execute a target video task.

17. The task platform according to claim 16, further comprising a model library, in, The model library stores a plurality of video generation models; The response unit is specifically used to determine a target scene template from multiple preset scene templates based on a task scene identifier, and to search for a video generation model from the model library based on the target scene template, wherein the request information includes the task scene identifier of the target video task; or, to search for a video generation model from the model library based on a task model identifier, wherein the request information includes the task model identifier of the target video task.

18. A computing device, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15 are implemented.

19. A computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15.

20. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15.

Citation Information

Cited By

  • Model optimization method and device

    CN120373391A

  • Video generation method, motion video generation method for virtual object, video editing method, video generation model training method and video generation model-based information processing method

    WO2026001219A1