Video generation method, motion video generation method for virtual object, video editing method, video generation model training method and video generation model-based information processing method

Abstract

Provided in the embodiments of the present disclosure are a video generation method, a motion video generation method for a virtual object, a video editing method, a video generation model training method and a video generation model-based information processing method. The video generation method comprises: acquiring video generation data of a target object; and inputting the video generation data into a video generation model so as to obtain a target video of the target object, wherein the video generation model comprises a generation unit and different types of coding units, the different types of coding units are used for coding the video generation data to obtain data coding features, the generation unit comprises a temporal attention unit, and the temporal attention unit is used for associating the data coding features by using an attention mechanism in a time dimension. Since the video generation model comprises the temporal attention unit, the video generation model can learn temporal information, so as to generate target videos that are temporally stable and more realistic and natural, thus improving the quality of target videos.

Description

Video generation, motion video generation of virtual object, video editing, video generation model training and information processing method based on video generation model

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410833799.6, filed on June 25, 2024, with the Chinese Patent Office, entitled "Video generation, motion video generation of virtual object, video editing, video generation model training and information processing method based on video generation model", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the field of computer technology, in particular to a video generation method, a motion video generation method of a virtual object, a video editing method, a video generation model training method, and an information processing method based on a video generation model. BACKGROUND

[0003] With the development of computer technology, video generation has gradually become an important research content of multi-modal and vision in generative artificial intelligence (AIGC, AI Generated Content), and has broad prospects and application value in digital human, e-commerce content, film and television production, etc.

[0004] In the video generation scene, due to the large amount of data used for generating videos, the generated videos may have problems of time sequence disorder, which further leads to poor quality of the generated videos. Therefore, there is an urgent need for a video generation scheme with stable time sequence. SUMMARY

[0005] Therefore, embodiments of the present disclosure provide a video generation method. One or more embodiments of the present disclosure also provide a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0006] According to a first aspect of embodiments of the present disclosure, a video generation method is provided, comprising:

[0007] obtaining video generation data of a target object;

[0008] The video generation data is input into a video generation model to obtain a target video of the target object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, and the time sequence attention unit is used to associate the data encoding features by using an attention mechanism in a time dimension.

[0009] According to a second aspect of the embodiments of the present disclosure, a motion video generation method of a virtual object is provided, comprising:

[0010] Obtaining motion video generation data of a virtual object;

[0011] Inputting the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, and the time sequence attention unit is used to associate the data encoding features by using an attention mechanism in a time dimension.

[0012] According to a third aspect of the embodiments of the present disclosure, a video editing method is provided, comprising:

[0013] Obtaining an original video and target editing data;

[0014] Parsing the original video to determine original video data in the original video;

[0015] Inputting the target editing data and the original video data into a video generation model to obtain a target editing video, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, and the time sequence attention unit is used to associate the data encoding features by using an attention mechanism in a time dimension.

[0016] According to a fourth aspect of the embodiments of the present disclosure, a video generation model training method is provided, comprising:

[0017] Obtaining a sample video and sample generation data corresponding to the sample video;

[0018] Inputting the sample generation data into a video generation model to obtain first prediction features output by a generation unit in the video generation model, wherein the video generation model comprises the generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, and the generation unit comprises a time sequence attention unit, and the time sequence attention unit is used to associate the sample encoding features by using an attention mechanism in a time dimension.

[0019] The sample video is input into an encoding unit to obtain first sample features;

[0020] According to the first prediction features and the first sample features, the unit parameters of the encoding unit and the temporal attention unit are adjusted to obtain a trained video generation model.

[0021] According to a fifth aspect of embodiments of the present disclosure, a method for processing information based on a video generation model is provided, including:

[0022] receiving a task generation request, wherein the task generation request includes request information;

[0023] based on the request information, obtaining a video generation model, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism;

[0024] based on the video generation model, generating task information, wherein the task information is used to perform a target video task.

[0025] According to a sixth aspect of embodiments of the present disclosure, a task platform is provided, including a request interface and a response unit;

[0026] The request interface is configured to receive a task generation request, wherein the task generation request includes request information;

[0027] The response unit is configured to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism; based on the video generation model, generating task information, wherein the task information is used to perform a target video task.

[0028] According to a seventh aspect of embodiments of the present disclosure, a video generation device is provided, including:

[0029] The first obtaining module is configured to obtain video generation data of a target object;

[0030] The first input module is configured to input the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, the generation unit includes a temporal attention unit, and the temporal attention unit is used to associate the data encoding features by using a time dimension attention mechanism.

[0031] According to an eighth aspect of embodiments of the present disclosure, a virtual object motion video generation apparatus is provided, comprising:

[0032] A second obtaining module configured to obtain virtual object motion video generation data;

[0033] A second input module configured to input the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, which is used to associate the data encoding features by using a time dimension attention mechanism.

[0034] According to a ninth aspect of embodiments of the present disclosure, a video editing apparatus is provided, comprising:

[0035] A third obtaining module configured to obtain an original video and target editing data;

[0036] A parsing module configured to parse the original video to determine original video data in the original video;

[0037] A third input module configured to input the target editing data and the original video data into a video generation model to obtain a target editing video, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, which is used to associate the data encoding features by using a time dimension attention mechanism.

[0038] According to a tenth aspect of embodiments of the present disclosure, a video generation model training apparatus is provided, comprising:

[0039] A fourth obtaining module configured to obtain a sample video and sample generation data corresponding to the sample video;

[0040] A fourth input module configured to input the sample generation data into a video generation model to obtain first prediction features output by a generation unit in the video generation model, wherein the video generation model comprises the generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, and the generation unit comprises a time sequence attention unit, which is used to associate the sample encoding features by using a time dimension attention mechanism.

[0041] A fifth input module configured to input the sample video into the encoding unit to obtain first sample features;

[0042] The adjusting module is configured to adjust the cell parameters of the encoding unit and the timing attention unit according to the first predicted feature and the first sample feature, and obtain the trained video generation model.

[0043] According to a first aspect of the embodiments of the present disclosure, a video generation model based information processing method is provided, including:

[0044] The receiving module is configured to receive a task generation request, wherein the task generation request includes request information.

[0045] The fifth obtaining module is configured to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, and the generation unit includes a timing attention unit, which is used to associate data coding features output by the different types of encoding units by using a time dimension attention mechanism.

[0046] The generating module is configured to generate task information based on the video generation model, wherein the task information is used to execute a target video task.

[0047] According to a twelfth aspect of the embodiments of the present disclosure, a computing device is provided, including a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which realize the steps of the method provided in the first aspect or the second aspect or the third aspect or the fourth aspect or the fifth aspect when executed by the processor.

[0048] According to a thirteenth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer programs / instructions, which realize the steps of the method provided in the first aspect or the second aspect or the third aspect or the fourth aspect or the fifth aspect when executed by the processor.

[0049] According to a fourteenth aspect of the embodiments of the present disclosure, a computer program product is provided, including computer programs / instructions, which realize the steps of the method provided in the first aspect or the second aspect or the third aspect or the fourth aspect or the fifth aspect when executed by the processor.

[0050] This disclosure provides a video generation method according to one embodiment, comprising: acquiring video generation data of a target object; inputting the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units being used to encode the video generation data to obtain data encoding features, and the generation unit including a temporal attention unit, the temporal attention unit being used to associate the data encoding features using a time-dimensional attention mechanism. Because the video generation model includes a temporal attention unit, the video generation model can learn temporal information, thereby generating a temporally stable and more realistic and natural target video, improving the quality of the target video. Attached Figure Description

[0051] Figure 1 is an architecture diagram of a video generation system provided in an embodiment of this disclosure;

[0052] Figure 2 is a flowchart of a video generation method provided in an embodiment of this disclosure;

[0053] Figure 3 is a flowchart of the processing procedure of a video generation model training method provided in an embodiment of this disclosure;

[0054] Figure 4 is a flowchart of the processing procedure of a pre-trained generative model training method provided in an embodiment of this disclosure;

[0055] Figure 5 is a flowchart of a method for generating motion video of a virtual object according to an embodiment of this disclosure;

[0056] Figure 6 is a flowchart of a video editing method provided in an embodiment of this disclosure;

[0057] Figure 7 is a flowchart of a video generation model training method provided in an embodiment of this disclosure;

[0058] Figure 8 is a flowchart of an information processing method based on a video generation model provided in an embodiment of this disclosure;

[0059] Figure 9 is a schematic diagram of the structure of a task platform provided in an embodiment of this disclosure;

[0060] Figure 10 is a schematic diagram of the structure of a video generation device provided in an embodiment of this disclosure;

[0061] Figure 11 is a schematic diagram of the structure of a motion video generation device for virtual objects provided in an embodiment of this disclosure;

[0062] Figure 12 is a schematic diagram of a video editing device provided in an embodiment of this disclosure;

[0063] Figure 13 is a schematic diagram of a video generation model training device provided in an embodiment of this disclosure;

[0064] FIG. 14 is a structural schematic diagram of an information processing apparatus based on a video generation model according to an embodiment of the present disclosure;

[0065] FIG. 15 is a structural block diagram of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure can be practiced without the specific details, and it is understood that the present disclosure is not limited to the embodiments described herein. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present disclosure.

[0067] The terminology used in one or more embodiments of the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present disclosure. As used in one or more embodiments of the present disclosure and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present disclosure, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0068] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "comprise," "comprises," and / or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is to be understood that the terms "including," "comprising," "consisting" and "involving" or any variation thereof, are not

[0069] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0070] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, billions, tens of billions, hundreds of billions, or even tens of billions of model parameters. The large model can also be referred to as a foundation model. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLM) and multi-modal pre-training models.

[0071] In practical applications, a large model only needs a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP) and computer vision fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0072] First, the technical terms related to one or more embodiments of the present disclosure are explained.

[0073] Cross Attention: A type of attention mechanism used to handle dependencies between two different data sources or sequences, such as injecting data from different modalities into image / video generation networks. Cross Attention allows the model to selectively focus on relevant parts of the input data when generating output, thus more effectively utilizing contextual information.

[0074] Transformer Model: A network architecture based on multi-head self-attention mechanism modules, primarily used for processing sequential data. The Transformer model consists of repeatedly stacked encoder and decoder units. This design allows the Transformer to efficiently learn long-term dependencies, making it suitable for a variety of natural language processing tasks, including machine translation, text summarization, and question answering systems.

[0075] Stable Diffusion: A generative model based on generative adversarial networks, which gradually increases the noise disturbance to make the generated images gradually clear and realistic.

[0076] Lora (Low-Rank Adaptation of Large Language Models): A low-rank adaptation expression in large models, which can use smaller parameter quantities to increase additional parameters and functions in large models.

[0077] Text-to-Video (T2V): Automatically converting written text content into video content containing visual and auditory elements.

[0078] Relighting: A computer graphics and computer vision technique that aims to modify or recalculate the lighting effects in existing images or three-dimensional scenes, generating consistent foreground and background lighting pictures or video results. This process allows users to adjust the light source position, color, intensity, etc. without changing the original scene's geometry and material properties, thus achieving a different lighting effect.

[0079] Convolutional Neural Network (CNN): A deep learning model suitable for processing data with grid structures, such as images, videos, and speech signals.

[0080] In recent years, text-to-video models have emerged that can generate relatively stable and realistic videos based on text input. Open-source text-to-image and text-to-video generation models such as Stable Diffusion and Video Stable Diffusion have greatly promoted the progress of the visual generation field. However, current image / video generation models have poor controllability, and many text-to-image models often need to run multiple times to get a relatively satisfactory result. Ensuring the effect and strong controllability is a problem that needs to be solved in the actual landing of the generation model. Therefore, controllable image / video generation has gradually become a research focus. Taking the field of human video generation as an example, not only does it need to focus on the consistency of the role and controllability of the action of the foreground character, but also needs to focus on the controllability of the background generation.

[0081] Based on this, the embodiment of the disclosure proposes a video generation scheme using a video generation model, wherein the video generation model comprises a time sequence attention unit. Through the time sequence attention unit, the video generation model can learn the light consistency of the foreground and the background, so that the generated video can not only meet the controllability requirement of the background, but also the foreground and the background of the whole video are consistent in light, the foreground can be harmoniously and naturally integrated into the given background, so that the generated video is real and natural.

[0082] In the disclosure, a video generation method is provided. The disclosure also relates to a motion video generation method of a virtual object, a video editing method, a video generation model training method, an information processing method based on a video generation model, a task platform, a video generation device, a motion video generation device of a virtual object, a video editing device, a video generation model training device, an information processing device based on a video generation model, a computing device, a computer-readable storage medium, and a computer program product. The above-mentioned embodiments are described in detail one by one.

[0083] Referring to FIG. 1, FIG. 1 shows an architecture diagram of a video generation system provided by an embodiment of the disclosure. The video generation system can include a client 100 and a server 200.

[0084] The client 100 is configured to send video generation data of a target object to the server 200.

[0085] The server 200 is configured to input the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are configured to encode the video generation data to obtain data encoding features, the generation unit comprises a time sequence attention unit, the time sequence attention unit is configured to associate the data encoding features by using a time dimension attention mechanism; and the server 200 is configured to send the target video to the client 100.

[0086] The client 100 is further configured to receive the target video sent by the server 200.

[0087] By using the scheme of the embodiment of the disclosure, since the video generation model comprises a time sequence attention unit, the video generation model can learn time sequence information, thereby generating a target video that is stable in time sequence and more real and natural, and the quality of the target video is improved.

[0088] In practical applications, the video generation system can include a plurality of clients 100. The plurality of clients 100 can establish a communication connection through the server 200. In the video generation scenario, the server 200 is used to provide a video generation service between the plurality of clients 100. The plurality of clients 100 can respectively act as a sending end or a receiving end to realize communication through the server 200.

[0089] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100 or send data to other clients 100, and the like. In the video generation scenario, the user can publish a data stream to the server 200 through the client 100. The server 200 generates a target video according to the data stream and pushes the target video to other clients that establish a communication. The client 100 and the server 200 establish a connection through a network. The network provides a medium for a communication link between the client 100 and the server 200. The network can include various connection types, such as a wired, wireless communication link, or a fiber cable, and the like. The data transmitted by the client 100 can need to be processed through encoding, transcoding, compression, and the like before being published to the server 200.

[0090] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5) application, or a light application (also known as a small program, a lightweight application program), or a cloud application, and the like. The client 100 can be developed based on a software development kit (SDK) provided by the server 200 according to a corresponding service, such as an RTC (Real Time Communication) SDK, and the like. The client 100 can be deployed in an electronic device and needs to depend on the device or some APP in the device to run, and the like. The electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant communication tools, mailbox clients, social platform software, and the like.

[0091] The server 200 can include a server providing various services, for example, a server providing a communication service for a plurality of clients, for example, a server for background training supporting a model used on a client, for example, a server processing data sent by a client, and the like. It should be noted that the server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server of a cloud service, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, and the like basic cloud computing services, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0092] It should be noted that the video generation method provided in the embodiments of the present disclosure is generally executed by the server, but in other embodiments of the present disclosure, the client can also have similar functions as the server, so as to execute the video generation method provided in the embodiments of the present disclosure. In other embodiments, the video generation method provided in the embodiments of the present disclosure can also be executed by the client and the server together. Next, taking the server executing the video generation method proposed in the embodiments of the present disclosure as an example, the video generation method is described in detail.

[0093] Referring to FIG. 2, FIG. 2 shows a flowchart of a video generation method according to an embodiment of the present disclosure, which specifically includes the following steps:

[0094] Step 202: Obtain video generation data of a target object.

[0095] In one or more embodiments of the present disclosure, when generating a video, video generation data can be obtained, and a target video meeting the requirements can be generated based on the video generation data.

[0096] Specifically, the video generation data refers to a data set used to generate a new target video. The video generation data can be data of different video tasks, including but not limited to a video generation task, a video background replacement task, a video object replacement task, and a video action replacement task. The target object refers to an object appearing in the target video, which can be a person, an animal, a virtual character, or the like. The video generation data can include a reference action sequence, a reference background sequence, and an object image. The object image refers to an image of the target object, which can also be understood as a video foreground image. For example, the object image can be an image of a person standing. The number of object images can be one or multiple. The reference action sequence includes multiple reference action images, each of which contains multiple action keypoints. Through the reference action sequence, the action or posture (such as standing or lying) to be demonstrated by the target object in the target video can be defined. The reference background sequence includes multiple reference background images. The multiple reference background images can be a sequence of static images, and the background in the generated target video is constant (such as grassland). The multiple reference background images can also be a sequence of dynamic images, and the background in the generated target video can change over time (such as a moving crowd).

[0097] In actual applications, there are various ways to obtain video generation data, which can be selected according to actual conditions, and the present disclosure does not make any limitation on this. In the first possible implementation manner of the present disclosure, the video generation data sent by the user through the client can be received. In the second possible implementation manner of the present disclosure, since it is difficult for the user to directly construct the reference action sequence and the reference background sequence, the user can be provided with multiple candidate action sequences and candidate background sequences, the reference action sequence selected by the user from the multiple candidate action sequences, and the reference background sequence selected by the user from the multiple candidate background sequences. In the third possible implementation manner of the present disclosure, the user can send a reference video and an object image through the client, and the server can parse the reference action sequence and / or the reference background sequence from the reference video.

[0098] In an optional embodiment of the present disclosure, the video generation data includes a reference action sequence, a reference background sequence, and an object image. The above method for obtaining the video generation data of the target object can include the following steps:

[0099] receiving a reference video of the target object, background requirement information, and an object image sent by the client;

[0100] inputting the reference video into an action extraction model to obtain a reference action sequence;

[0101] determining a reference background sequence according to the background requirement information.

[0102] Specifically, the background requirement information is used to describe the requirement for the video background in the video generation process. For example, the background requirement information can be "indoor" or "outdoor". The action extraction model is used to extract the action sequence in the input video. The action extraction model can be a deep learning model trained based on a CNN model. For example, the action extraction model can be trained based on a sample video and a sample action sequence carried by the sample video.

[0103] It should be noted that the reference action sequence can be parsed from the reference video, and the reference action sequence can also be parsed. Therefore, the reference video can include the target object or can not include the target object. For example, the target object is object A, and the reference video can be a dance video of object A, or a dance video of object B, and can also be a landscape video that does not include any object.

[0104] In actual application, there are various ways to determine the reference background sequence according to the background requirement information, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. In a possible implementation manner of the present disclosure, a plurality of candidate background sequences can be obtained, and each candidate background sequence carries background description information. The background requirement information and the background description information are matched, and the candidate background sequence with the same background description information as the background requirement information is determined as the reference background sequence. In another possible implementation manner of the present disclosure, the background requirement information can be input into an image generation model to obtain the reference background sequence.

[0105] By applying the scheme of the embodiments of the present disclosure, the user only needs to provide the reference video, the background requirement information and the object image of the target object, and the service end determines the reference action sequence based on the reference video and determines the reference background sequence based on the background requirement information, thereby reducing the difficulty of data provision of the user and reducing the threshold of video generation.

[0106] Step 204: input the video generation data into the video generation model to obtain the target video of the target object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit includes a time sequence attention unit, the time sequence attention unit is used to associate the data encoding features by using the attention mechanism in the time dimension.

[0107] In one or more embodiments of the present disclosure, after obtaining the video generation data of the target object, the video generation data can be further input into the video generation model to obtain the target video of the target object.

[0108] Specifically, the target video refers to a video in which the video data conforms to the video generation data. For example, the target video can be a video in which the action of the target object conforms to the reference action sequence and the video background conforms to the reference background sequence. The training manners of the video generation model and the pre-trained generation model are both supervised training.

[0109] By applying the scheme of the embodiments of the present disclosure, since the time sequence attention unit is included in the video generation model, the video generation model can learn time sequence information, thereby generating a target video that is stable in time sequence and more realistic and natural, and improving the quality of the target video.

[0110] In an optional embodiment of the present disclosure, the video generation data includes a reference action sequence, a reference background sequence and an object image, different types of encoding units include an image encoding unit, an object encoding unit, an action encoding unit and a background encoding unit, the data encoding features include object encoding features, an action encoding sequence and a background encoding sequence, and the video generation model further includes a decoding unit. The above inputting the video generation data into the video generation model to obtain a target video of a target object can include the following steps:

[0111] inputting the object image into the image encoding unit to obtain image encoding features, and inputting the image encoding features into the object encoding unit to obtain object encoding features;

[0112] inputting the reference action sequence into the action encoding unit to obtain an action encoding sequence;

[0113] inputting the reference background sequence into the background encoding unit to obtain a background encoding sequence;

[0114] inputting the object encoding features, the action encoding sequence and the background encoding sequence into the generation unit to obtain video encoding features;

[0115] inputting the video encoding features into the decoding unit to obtain a target video of a target object.

[0116] It should be noted that the image encoding unit is connected with the object encoding unit, the object encoding unit is connected with the generation unit, the action encoding unit is connected with the generation unit, the background encoding unit is connected with the generation unit, and the generation unit is connected with the decoding unit. The image encoding unit, the action encoding unit and the background encoding unit are composed of CNN network layers. The object encoding unit and the generation unit are composed of Transformer network modules. Since the background is pixel-level alignment, the background encoding sequence can be directly input into the generation unit. The generation unit can be constructed based on the UNet network of Stable Diffusion or based on Video Stable Diffusion.

[0117] According to the scheme of the embodiment of the present disclosure, after the image encoding unit, an object encoding unit is connected, and object control is performed on the video encoding features, so that the objects in each frame of the generated target video are consistent with the target objects.

[0118] In an optional embodiment of the present disclosure, the structure of the object encoding unit is described. That is, the object encoding unit includes a plurality of encoding blocks; and the step of inputting the image encoding features into the object encoding unit to obtain the object encoding features can include the following steps:

[0119] The image encoding features are input into the first encoding block to obtain first encoding features output by the first encoding block, wherein the first encoding block is the first encoding block in the plurality of encoding blocks;

[0120] The encoding features output by the previous encoding block of the second encoding block are input into the second encoding block to obtain second encoding features output by the second encoding block, wherein the second encoding block is any encoding block in the plurality of encoding blocks except the first encoding block;

[0121] The object encoding features are determined according to the encoding features output by the plurality of encoding blocks respectively.

[0122] Specifically, the encoding block includes a self-attention unit and a cross-attention unit. When the image encoding features are input into the first encoding block to obtain first encoding features output by the first encoding block, the image encoding features can be input into the self-attention unit in the encoding block to obtain image self-attention features, and the image self-attention features are input into the cross-attention unit in the encoding block to obtain the first encoding features.

[0123] For example, it is assumed that the object encoding unit includes three encoding blocks connected in sequence, which are encoding block 1, encoding block 2 and encoding block 3. The process of inputting the image encoding features into the object encoding unit can include: first, inputting the image encoding features into the encoding block 1 to obtain encoding features 1; then, inputting the encoding features 1 into the encoding block 2 to obtain encoding features 2; finally, inputting the encoding features 2 into the encoding block 3 to obtain encoding features 3. The encoding features 1, the encoding features 2 and the encoding features 3 jointly constitute the object encoding features.

[0124] According to the scheme of the embodiment of the present disclosure, by setting a plurality of encoding blocks in the encoding unit, the respective generation blocks in the generation unit are facilitated to be corresponded, and the operation of the subsequent cross-attention mechanism is prepared.

[0125] In an optional embodiment of the present disclosure, the structure of the generation unit is described. That is, the generation unit includes a plurality of generation blocks, the object encoding unit includes a plurality of encoding blocks, and the generation blocks and the encoding blocks correspond one by one; and the step of inputting the object encoding features, the action encoding sequence and the background encoding sequence into the generation unit to obtain the video encoding features can include the following steps:

[0126] determine the encoding features corresponding to the plurality of generated blocks respectively according to the correspondence between the plurality of generated blocks and the plurality of encoding blocks;

[0127] input the action encoding sequence, the background encoding sequence and the encoding features corresponding to the first generated block into the first generated block to obtain first generated features output by the first generated block, wherein the first generated block is the first generated block in the plurality of generated blocks;

[0128] input the generated features output by the previous generated block of the second generated block and the encoding features corresponding to the second generated block into the second generated block to obtain second generated features output by the second generated block, wherein the second generated block is any generated block in the plurality of generated blocks except the first generated block;

[0129] in a case where the second generated block is the last generated block in the plurality of generated blocks, determine the second generated features output by the second generated block as the video encoding features.

[0130] It should be noted that the features of the object images of different layers are extracted by the object encoding unit, and the object features are fused into the generated unit in the same layer of the generated unit using the cross-attention mechanism, so that the objects in each frame of the generated target video are consistent with the target object.

[0131] For example, it is assumed that the object encoding unit includes three encoding blocks connected in sequence, which are encoding block 1, encoding block 2 and encoding block 3. The generated unit includes three generated blocks connected in sequence, which are generated block 1, generated block 2 and generated block 3. The image encoding features are input into the object encoding unit, and the encoding features 1 output by the encoding block 1, the encoding features 2 output by the encoding block 2 and the encoding features 3 output by the encoding block 3 can be obtained. After the encoding block 1 outputs the encoding features 1, the cross-attention mechanism can be used to fuse the encoding features 1 into the input of the generated block 1. For the generated block 1, the action encoding sequence, the background encoding sequence and the encoding features 1 are input into the generated block 1, and the generated block 1 outputs the generated features 1. After the encoding block 2 outputs the encoding features 2, the cross-attention mechanism can be used to fuse the encoding features 2 into the input of the generated block 2. For the generated block 2, the generated features 1 and the encoding features 2 are input into the generated block 2, and the generated block 2 outputs the generated features 2. After the encoding block 3 outputs the encoding features 3, the cross-attention mechanism can be used to fuse the encoding features 3 into the input of the generated block 3. For the generated block 3, the generated features 2 and the encoding features 3 are input into the generated block 3, and the generated block 3 outputs the generated features 3. Since the generated block 3 is the last generated block, the generated features 3 are determined as the video encoding features.

[0132] According to the scheme of the embodiment of the present disclosure, the encoding features in the input generation block are determined according to the correspondence between the plurality of generation blocks and the plurality of encoding blocks, so that the generation process of each generation block has the encoding features output by the corresponding encoding block as a constraint, preparing for the subsequent operation of the cross-attention mechanism.

[0133] In an optional embodiment of the present disclosure, the structure of the generation block in the generation unit is described. Each generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit. Taking the second generation block as an example, that is, the second generation block includes a self-attention unit, a cross-attention unit and a temporal attention unit; the above inputting the generation features output by the previous generation block of the second generation block and the encoding features corresponding to the second generation block into the second generation block to obtain the second generation features output by the second generation block can include the following steps:

[0134] Inputting the generation features output by the previous generation block of the second generation block into the self-attention unit to obtain self-attention features;

[0135] Inputting the self-attention features and the encoding features corresponding to the second generation block into the cross-attention unit to obtain cross-attention features;

[0136] Inputting the cross-attention features into the temporal attention unit to obtain the second generation features.

[0137] Specifically, the self-attention unit is connected with the cross-attention unit, and the cross-attention unit is connected with the temporal attention unit. For example, in the second generation block, the temporal attention unit is after the cross-attention unit, and the self-attention unit is before the cross-attention unit. The self-attention unit is used for data processing by using the self-attention mechanism. The self-attention mechanism allows the self-attention unit to consider how each position in the input data (such as different regions of an image or different parts of a feature map) depends on all other positions. The self-attention mechanism can help the model identify and utilize the correlation between distant pixels. The cross-attention unit is used for data processing by using the cross-attention mechanism. The cross-attention mechanism can help the model guide the generation process of the video according to the conditional information (such as the background feature sequence and the action feature sequence). The cross-attention mechanism can make the model align the conditional information with the image features, ensuring that the generated image meets the given conditions. The temporal attention unit is used for data processing by using the attention mechanism in the time dimension, so that the temporal sequences of the features are related to each other, and the generated video result is more stable.

[0138] By applying the scheme of the embodiments of the present disclosure, the self-attention mechanism is used for data processing in the self-attention unit, the correlation between distant pixels is recognized and utilized, the cross-attention mechanism is used for data processing in the cross-attention unit, the model aligns the conditional information with the image features, ensures that the generated image meets the given conditions, the attention mechanism of the time dimension is used for data processing in the timing attention unit, the timing of the features is associated with each other, and the generated video result is more stable.

[0139] In an optional embodiment of the present disclosure, after the video generation data is input into the video generation model to obtain the target video of the target object, the following steps can be further included:

[0140] sending the target video to the client;

[0141] receiving result feedback information sent by the client, wherein the result feedback information is information fed back by the client to the target video;

[0142] constructing model optimization data according to the result feedback information;

[0143] adjusting parameters of the video generation model by using the model optimization data.

[0144] Specifically, the result feedback information can be information fed back to the content, quality and completion degree of the target video, reflecting the real feelings and expectations of the client to the target video. The result feedback information includes but is not limited to quality evaluation information, corrected accurate target video, and optimization field of the model. The model optimization data refers to accurate optimization sample video used for optimizing the video generation model.

[0145] In actual application, there are various ways to construct the model optimization data according to the result feedback information, which is selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation. In a possible implementation manner of the present disclosure, the model optimization data can be automatically constructed according to the result feedback information. In another possible implementation manner of the present disclosure, optimization prompt information can be generated based on the result feedback information, and the model optimization data sent by the client based on the optimization prompt information is received.

[0146] Exemplarily, when the optimization prompt information is generated based on the result feedback information, in a possible implementation of the present disclosure, the preset prompt information can be directly obtained, and the result feedback information can be added in the preset prompt information to obtain the optimization prompt information. For example, the preset prompt information is "I am very sorry to bring you inaccurate video. Please point out where is not accurate enough or provide accurate video, and I will correct and optimize my answer as soon as possible to better serve you". The result feedback information is "video background is not accurate", and the optimization prompt information is "I am very sorry to bring you inaccurate video according to the problem of video background not being accurate. Please point out where is not accurate enough or provide accurate video, and I will correct and optimize my answer as soon as possible to better serve you". In another possible implementation of the present disclosure, the result feedback information can be identified by type, the information type of the result feedback information can be determined, the information type can be further matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type as the information type can be determined as the optimization prompt information.

[0147] Further, when the model optimization data is constructed directly according to the result feedback information, if the result feedback information is the corrected accurate target video, the corrected accurate target video can be determined as the model optimization data. If the result feedback information is the optimization field of the model, such as a certain field, a plurality of sample videos in the certain field can be obtained, and the plurality of sample videos can be determined as the model optimization data.

[0148] By collecting and utilizing the result feedback information, the performance of the video generation model can be continuously optimized to more accurately meet the actual needs of the client and improve the quality and accuracy of the final target video by applying the scheme of the embodiments of the present disclosure.

[0149] In an optional embodiment of the present disclosure, the training method of the video generation model is described, that is, before the video generation data is input into the video generation model to obtain the target video of the target object, the following steps can be further included:

[0150] The sample video and the sample generation data corresponding to the sample video are obtained;

[0151] The sample generation data is input into the video generation model to obtain the first predicted feature output by the generation unit in the video generation model;

[0152] The sample video is input into the encoding unit to obtain the first sample feature;

[0153] The unit parameters of the encoding unit and the time sequence attention unit are adjusted according to the first predicted feature and the first sample feature, and the trained video generation model is obtained.

[0154] Specifically, the sample generation data includes a sample object image, a sample action sequence, and a sample background sequence. The sample generation data can be parsed from a sample video. The first sample feature is a generation target of a generation unit in the video generation model, and is used to guide the training process of the video generation model. The video generation model is obtained by additionally adding a time attention unit in each generation block of the generation unit of the pre-trained generation model, that is, the video generation model includes different types of encoding units, generation units, and decoding units; the different types of encoding units include an image encoding unit, an object encoding unit, an action encoding unit, and a background encoding unit; the object encoding unit includes a plurality of encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and the generation block includes a self-attention unit, a cross-attention unit, and a time attention unit.

[0155] It should be noted that there are many ways to obtain sample videos, which are selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation. In one possible implementation of the present disclosure, a plurality of sample videos sent by a user through a client can be received. In another possible implementation of the present disclosure, a plurality of sample videos can be read from other data acquisition devices or databases. When the sample video is input into the encoding unit to obtain the first sample feature, the sample video can be input into the image encoding unit to obtain the first sample feature. When the unit parameters of the encoding unit and the time attention unit are adjusted according to the first prediction feature and the first sample feature, only the unit parameters of the background encoding unit and the time attention unit can be adjusted.

[0156] Further, after obtaining the sample video, for the sample object image: any video frame including the sample object can be randomly selected from the sample video as the sample object image, or a video frame including the sample object with higher quality can be selected as the sample object image. For the sample action sequence: the sample video can be input into an action extraction model to obtain the sample action sequence, or the sample video can be sampled (such as sampling at 30FPS), and for each sample video frame, the sample video frame can be input into a key point recognition model to obtain sample action key points, and a sample action image including object bones can be drawn. The sample action image corresponding to each sample video frame can constitute the sample background sequence. For the sample background sequence: the sample video can be sampled, and for each sample video frame, a segmentation model can be used to obtain masks of the foreground character and the background, and then a filling model can be used to complete the background according to the known part of the background for the foreground part to generate a sample background image. The sample background image corresponding to each sample video frame can constitute the sample background sequence.

[0157] In actual application, when the unit parameters of the encoding unit and the temporal attention unit are adjusted according to the first predicted feature and the first sample feature to obtain the trained video generation model, the loss value can be calculated according to the first predicted feature and the first sample feature, and the unit parameters of the encoding unit and the temporal attention unit are adjusted according to the loss value in a low-rank adaptive manner until the training process meets the preset stopping condition, and the trained video generation model is obtained. The function of calculating the loss value has many, such as cross-entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of the present disclosure do not make any limitation. The preset stopping condition includes but is not limited to that the loss value is less than or equal to the preset threshold, and the iteration number reaches the preset iteration number. The preset threshold and the preset iteration number are selected according to the actual situation, and the embodiments of the present disclosure do not make any limitation.

[0158] In a possible implementation of the present disclosure, after the loss value is calculated, the loss value is compared with the preset threshold. Specifically, if the loss value is greater than the preset threshold, it means that the difference between the first predicted feature and the first sample feature is large, and the prediction ability of the video generation model for the video feature is poor. At this time, the unit parameters of the encoding unit and the temporal attention unit can be adjusted, and the step of inputting the sample generation data into the video generation model to obtain the first predicted feature output by the generation unit in the video generation model is returned to continue training the video generation model until the loss value is less than or equal to the preset threshold, which means that the difference between the target first predicted feature and the first sample feature is small, and the preset stopping condition is reached to obtain the trained video generation model.

[0159] In another possible implementation of the present disclosure, in addition to comparing the size relationship between the loss value and the preset threshold, the iteration number can also be combined to determine whether the current video generation model is trained. Specifically, if the loss value is greater than the preset threshold, the unit parameters of the encoding unit and the temporal attention unit are adjusted, and the step of inputting the sample generation data into the video generation model to obtain the first predicted feature output by the generation unit in the video generation model is returned to continue training the video generation model until the preset iteration number is reached, and the iteration is stopped to obtain the trained video generation model.

[0160] By applying the scheme of the embodiments of the present disclosure, since the video generation model includes the temporal attention unit, the target video with stable time sequence and natural fusion of foreground and background can be generated, so that the target video is more real and natural.

[0161] Referring to FIG. 3, FIG. 3 shows a process flow diagram of a video generation model training method provided by an embodiment of the present disclosure. The training stage of the video generation model can be referred to as a video generation training stage. In this stage, the training target of the model is to generate a time-stable video segment while being able to control the generation of the background. Therefore, a time attention unit is introduced into the pre-trained generation model to obtain a video generation model. By increasing the time attention unit, the correlation between different frames in the time dimension can be learned, and the video generation model can further learn time information, such as a flowing hem, a continuously stable time-varying feature. That is, the video generation model includes an image encoding unit, an object encoding unit, a motion encoding unit, a background encoding unit, a generation unit, and a decoding unit. The object encoding unit includes a plurality of encoding blocks, and each encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and each generation block includes a self-attention unit, a cross-attention unit, and a time attention unit. In the training process, only the unit parameters of the background encoding unit and the time attention unit are adjusted, and the unit parameters of other units except the background encoding unit and the time attention unit are fixed.

[0162] Next, the training process of the video generation model is described. As shown in FIG. 3, a sample video is obtained; the sample video is parsed to obtain a sample object image, a sample motion sequence, and a sample background sequence; the sample object image is input into the image encoding unit of the video generation model to obtain sample image encoding features, and the image encoding features are input into the object encoding unit of the video generation model to obtain sample object encoding features; the sample motion sequence is input into the motion encoding unit of the video generation model to obtain a sample motion encoding sequence; the sample background sequence is input into the background encoding unit of the video generation model to obtain a sample background encoding sequence; the sample object encoding features, the sample motion encoding sequence, and the sample background encoding sequence are input into the generation unit of the video generation model to obtain first prediction features; the first prediction features are input into the decoding unit of the video generation model to obtain a predicted video. The sample video is input into the image encoding unit of the video generation model to obtain first sample features; the unit parameters of the background encoding unit and the time attention unit of the video generation model are adjusted according to the first prediction features and the first sample features to obtain a trained video generation model.

[0163] By applying the scheme of the embodiments of the present disclosure, when training the video generation model, the newly added time attention unit and the background encoding unit are trained at the same time, so that the generation result is time-stable and can learn the consistency of the foreground and background illumination, making the entire video result more realistic and natural. Furthermore, through the above model training, the final obtained video generation model can generate a video segment with consistent objects and consistent with the input motion sequence and background sequence.

[0164] In an optional embodiment of the present disclosure, the training manner of the pre-training generation model is described, that is, the sample video includes a plurality of sample video frames; before the sample generation data is input into the video generation model to obtain the first predicted feature output by the generation unit in the video generation model, the following steps can be further included:

[0165] For the first sample video frame, determine the first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from the plurality of sample video frames;

[0166] Input the first sample generation data into the initial generation model to obtain the second predicted feature output by the generation unit in the initial generation model;

[0167] Input the first sample video frame into the encoding unit to obtain the second sample feature;

[0168] Adjust the unit parameters of the encoding unit and the generation unit according to the second predicted feature and the second sample feature to obtain the pre-training generation model;

[0169] In the generation unit of the pre-training generation model, a time sequence attention unit is constructed to obtain the video generation model.

[0170] Specifically, the first sample generation data includes a first sample object image, a first sample action image, and a first sample background image. The second sample feature is the generation target of the generation unit in the initial generation model, which is used to guide the training process of the initial generation model. Since the initial generation model is trained based on a single video frame and does not include time sequence information, the time sequence attention unit is not included in the generation unit of the initial generation model, that is, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a background encoding unit, a generation unit, and a decoding unit; the object encoding unit includes a plurality of encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and the generation block includes a self-attention unit and a cross-attention unit.

[0171] It should be noted that when the first sample video frame is sampled from the plurality of sample video frames, in one possible implementation, all sample video frames can be determined as the first sample video frame. In another possible implementation, since the similarity between adjacent frames is high, the first sample video frame can be extracted every preset time length. When the first sample video frame is input into the encoding unit to obtain the second sample feature, the first sample video frame can be input into the image encoding unit to obtain the second sample feature. When the unit parameters of the encoding unit and the generation unit are adjusted according to the second predicted feature and the second sample feature, the unit parameters of the object encoding unit, the action encoding unit, the background encoding unit, and the generation unit can be adjusted to obtain the pre-training generation model.

[0172] In actual applications, the implementation manner of "adjusting the unit parameters of the encoding unit and the generation unit according to the second prediction feature and the second sample feature to obtain the pre-trained generation model" is the same as the implementation manner of "adjusting the unit parameters of the encoding unit and the time sequence attention unit according to the first prediction feature and the first sample feature to obtain the trained video generation model", and thus the present disclosure embodiment will not be described in detail.

[0173] Further, after adjusting the unit parameters of the encoding unit and the generation unit in the initial generation model according to the second prediction feature and the second sample feature to obtain the pre-trained generation model, the pre-trained generation model can be used to process an image generation task, that is, the object image, the action image and the background image of the target object are input into the pre-trained generation model to obtain a target image, the target image includes the target object, the action of the target object is consistent with the action in the action image, and the background of the target image is consistent with the background image.

[0174] By applying the scheme of the present disclosure embodiment, an additional time sequence attention unit is added in each generation block of the generation unit in the pre-trained generation model to obtain a video generation model, so that the video generation model can learn time sequence information, the unit parameters of the encoding unit and the generation unit in the initial generation model are adjusted according to the second prediction feature and the second sample feature, so that the pre-trained generation model can generate an image consistent in object, action and background, and the generation ability of the pre-trained generation model is ensured.

[0175] Referring to FIG. 4, FIG. 4 shows a processing process flowchart of a pre-trained generation model training method provided by an embodiment of the present disclosure. The pre-trained generation model is obtained by adjusting the unit parameters of the object encoding unit, the action encoding unit, the background encoding unit and the generation unit in the initial generation model based on sample videos. This training stage can be referred to as an image generation training stage. In this stage, the model training target is to generate an image consistent in object, target action and target background. The initial generation model can reuse the weights of Stable Diffusion and increase the action encoding unit, the object encoding unit and the background encoding unit based on Stable Diffusion. That is, as shown in FIG. 4, the initial generation model includes an image encoding unit, an object encoding unit, an action encoding unit, a background encoding unit, a generation unit and a decoding unit; the object encoding unit includes a plurality of encoding blocks; the encoding block includes a self-attention unit and a cross-attention unit. The generation unit includes a plurality of generation blocks, and the generation block includes a self-attention unit and a cross-attention unit. In the training process, only the unit parameters of the object encoding unit, the action encoding unit, the background encoding unit and the generation unit are adjusted, and the unit parameters of the image encoding unit and the decoding unit are fixed and unchanged.

[0176] Next, the processing procedure of the initial generation model is described. As shown in FIG. 4, a sample video is obtained, wherein the sample video includes a plurality of sample video frames; for a first sample video frame, a first sample object image, a first sample action image and a first sample background image corresponding to the first sample video frame are determined, wherein the first sample video frame is sampled from the plurality of sample video frames; the first sample object image, the first sample action image and the first sample background image are input into the initial generation model to obtain a first predicted video frame output by the initial generation model. The first sample video frame is input into an image encoding unit in the initial generation model to obtain second sample features; based on the second sample features and second predicted features output by a generation unit in the initial generation model, unit parameters of an object encoding unit, an action encoding unit and a background encoding unit in the initial generation model are adjusted to obtain a pre-training generation model.

[0177] The video generation method provided in the disclosure is further described below by taking the application of the video generation method in a virtual object motion video generation scenario as an example with reference to FIG. 5. FIG. 5 is a flowchart of a virtual object motion video generation method according to an embodiment of the disclosure, which specifically includes the following steps:

[0178] In step 502, motion video generation data of a virtual object is obtained.

[0179] In step 504, the motion video generation data is input into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit includes a time sequence attention unit, which is used to associate the data encoding features by using a time dimension attention mechanism.

[0180] It should be noted that the implementation manners of steps 502 to 504 are the same as those of steps 202 to 204, and thus the disclosure will not be described in detail.

[0181] In actual applications, in the virtual object motion video generation scenario, a virtual object image of a virtual object, an action sequence of a certain sports item and a background sequence of a sports site can be input into the video generation model to obtain a target motion video of the virtual object performing the sports item in the sports site.

[0182] By applying the scheme of the embodiment of the disclosure, since the video generation model includes the time sequence attention unit, the video generation model can learn time sequence information, thereby generating a target motion video that is stable in time sequence and has a natural fusion of foreground and background, so that the target motion video is more real and natural.

[0183] The video generation method provided in the disclosure is further described below by taking the application of the video generation method in the video editing (such as replacement, modification, and relighting) scenario as an example in combination with FIG. 6. FIG. 6 is a flowchart of a video editing method according to an embodiment of the disclosure, and specifically includes the following steps:

[0184] Step 602: Obtain an original video and target editing data.

[0185] Step 604: Analyze the original video to determine original video data in the original video.

[0186] Step 606: Input the target editing data and the original video data into a video generation model to obtain a target edited video, wherein the video generation model includes a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit includes a time attention unit, the time attention unit is used to associate the data encoding features by using a time dimension attention mechanism.

[0187] It should be noted that the implementation manners of steps 602 to 606 are the same as those of steps 202 to 204, and thus the disclosure will not be described in detail.

[0188] In actual applications, when the original video is analyzed to determine the original video data in the original video, the original video can be analyzed based on the target editing data to determine the original video data in the original video. Specifically, if the video editing task is a video background editing task, the target editing data is a target background sequence, and analyzing the original video to determine the original video data in the original video includes an original action sequence and an original object image. If the video editing task is a video object editing task, the target editing data is a target object image, and analyzing the original video to determine the original video data in the original video includes an original background sequence and an original action sequence. If the video editing task is a video action editing task, the target editing data is a target action sequence, and analyzing the original video to determine the original video data in the original video includes an original background sequence and an original object image.

[0189] By inputting the target editing data and the original video data into the video generation model, the action, background, and object of the original video are edited, and the video generation model can learn time sequence information to generate a target edited video that is stable in time sequence and has natural fusion of foreground and background, so that the target edited video is more real and natural.

[0190] Referring to FIG. 7, FIG. 7 is a flowchart of a video generation model training method according to an embodiment of the disclosure, and specifically includes the following steps:

[0191] Step 702: Obtain a sample video and sample generation data corresponding to the sample video.

[0192] Step 704: Input the sample generation data into the video generation model to obtain first predicted features output by a generation unit in the video generation model, wherein the video generation model comprises the generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, and the generation unit comprises a time attention unit, and the time attention unit is used to associate the sample encoding features by using a time dimension attention mechanism.

[0193] Step 706: Input the sample video into the encoding unit to obtain first sample features.

[0194] Step 708: Adjust unit parameters of the encoding unit and the time attention unit according to the first predicted features and the first sample features to obtain a trained video generation model.

[0195] It should be noted that the implementation manners of steps 702 to 708 are the same as the training manner of the video generation model in the video generation method provided in FIG. 2, and thus the embodiments of the present disclosure will not be described in detail.

[0196] By applying the scheme of the embodiments of the present disclosure, a time attention unit is additionally added in each generation block included in the generation unit of the pre-trained generation model to obtain the video generation model, so that the video generation model can learn time sequence information, thereby generating a target video with stable time sequence and natural fusion of foreground and background, and making the target video more real and natural.

[0197] Referring to FIG. 8, FIG. 8 shows a flowchart of an information processing method based on a video generation model according to an embodiment of the present disclosure, which specifically comprises the following steps:

[0198] Step 802: Receive a task generation request, wherein the task generation request comprises request information.

[0199] Specifically, the information processing method based on the video generation model can be applied to a task platform, and the task platform can be deployed on an end-side device or a cloud-side device. The task generation request is used to request generation of task information of a target video task. The task generation request usually contains a task type, an expected output format, and request information. The request information refers to parameters or description information related to the target video task carried in the task generation request. The request information includes but is not limited to a task scene identifier of the target video task, a task model identifier, and a plurality of sample videos corresponding to the target video task.

[0200] Step 804: based on the request information, obtaining a video generation model, wherein the video generation model comprises a generation unit and different types of encoding units, the generation unit comprises a time attention unit, and the time attention unit is configured to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism.

[0201] In actual applications, there are various ways to obtain the video generation model based on the request information, which can be selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation in this regard.

[0202] In an optional embodiment of the present disclosure, the above-mentioned obtaining the video generation model based on the request information can comprise the following steps:

[0203] Based on the task scene identifier, a target scene template is determined from a plurality of preset scene templates, and based on the target scene template, a video generation model is searched from a model library, wherein the model library stores a plurality of video generation models, and the request information comprises a task scene identifier of a target video task; or

[0204] Based on the task model identifier, a video generation model is searched from the model library, wherein the request information comprises a task model identifier of a target video task.

[0205] Specifically, the task scene identifier refers to a unique or specific label used to distinguish different task application scenarios. In the embodiments of the present disclosure, the task scene identifier is part of the request information. Through the task scene identifier, a target scene template matching the request information can be selected from a series of preset scene templates to generate task information. For example, the task scene identifier is "video background control", which means that the user wants to control and replace the background of the uploaded original video, and then a video background control scene template can be selected from a plurality of preset scene templates according to the task scene identifier.

[0206] The preset scene template is a standard configuration scene template defined in advance for different target video task application scenarios, and each template contains model information and task processing flow information matching the task application scenario. Through a series of preset scene templates, different scene task generation requests can be quickly responded. Different preset scene templates correspond to different task types, model information and processing flow. For example, there can be a video action control scene template in the preset scene template, which is specially designed for video action control tasks, and contains model information and processing flow of a trained video action control model.

[0207] The target scene template refers to a scene template matched with the task scene identifier. When parsing the task generation request, the corresponding target scene template can be located based on the task scene identifier, and the corresponding video generation model and other related configuration information can be selected from the model library according to the model information included in the target scene template. For example, when the task scene identifier is "video object control", the target scene template is a template including model information of the video object control model and related configuration parameters.

[0208] The model library is a resource library that centrally stores video generation models, which are trained and optimized to solve different target video tasks. Moreover, the video generation models in the model library can be divided into different versions according to different applicable tasks, for example, the model library can store a video background control model for a video background control task, a video object control model for a video object control task, and a video action control model for a video action control task.

[0209] The task model identifier refers to a unique or specific label used to distinguish models applicable to different tasks. For example, the task model identifier can be "video background control", based on which a video background control model applicable to the video background control task can be found in the model library.

[0210] By using the scheme of the embodiments of the present disclosure, the pre-defined task scene template, task model identifier, and model library resource make the video generation model acquisition process more flexible, efficient, and standard.

[0211] In another optional embodiment of the present disclosure, in addition to selecting a video generation model from the model library, the video generation model in the model library can also be trained according to the sample video in the request information and the sample generation data corresponding to the sample video, to obtain a video generation model, that is, the request information includes a sample video of a target video task and sample generation data corresponding to the sample video; the above step of acquiring a video generation model based on the request information can include the following steps:

[0212] The video generation model corresponding to the target video task is trained based on the sample video and the sample generation data corresponding to the sample video, to obtain a trained video generation model.

[0213] It should be noted that the implementation of "training the video generation model corresponding to the target video task based on the sample video and the sample generation data corresponding to the sample video to obtain a trained video generation model" is the same as the above training method of the video generation model, and the embodiments of the present disclosure will not be described again.

[0214] By applying the scheme of the embodiment of the present disclosure, the video generation model corresponding to the target video task is trained based on the sample video and the sample generation data corresponding to the sample video, and the trained video generation model is obtained, so as to make the video generation model more consistent with the task scene of the target video task on the basis of ensuring the accuracy of the video generation model.

[0215] In step 806, task information is generated based on the video generation model, and the task information is used to execute the target video task.

[0216] Specifically, the task information includes model configuration and processing flow required for executing the target video task. The terminal device or other server components can correctly use the video generation model to process the target video task based on the task information. The target video task includes but is not limited to video background control characters, video action control tasks, video object control tasks, and video generation tasks.

[0217] It should be noted that when the task information is generated based on the video generation model, the model parameters of the video generation model can be directly packaged to obtain the task information. Other model information of the video generation model can also be obtained, and the task information is constructed based on the other model information, wherein the other model information includes processing mode of model input data, specification of expected output result, and possible intermediate steps and other auxiliary information.

[0218] By applying the scheme of the embodiment of the present disclosure, the task information of the target video task is generated, so as to reduce the system deployment and operation and maintenance cost while ensuring the processing quality and efficiency of the target video task, and provide convenient and efficient target video task processing service for users.

[0219] Referring to FIG. 9, FIG. 9 shows a structural schematic diagram of a task platform provided by an embodiment of the present disclosure. The task platform includes a request interface 902 and a response unit 904.

[0220] The request interface 902 is configured to receive a task generation request, wherein the task generation request includes request information.

[0221] The response unit 904 is configured to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a time attention unit, and the time attention unit is configured to associate data coding features output by the different types of encoding units by using a time dimension attention mechanism; and generate task information based on the video generation model, wherein the task information is used to execute the target video task.

[0222] In an optional embodiment of the present disclosure, the task platform further includes a model library, wherein the model library stores a plurality of video generation models.

[0223] The response unit is specifically configured to determine a target scene template from a plurality of preset scene templates based on the task scene identification, and find a video generation model from a model library based on the target scene template, wherein the request information comprises a task scene identification of the target video task; or find a video generation model from the model library based on the task model identification, wherein the request information comprises a task model identification of the target video task.

[0224] By generating the task information of the target video task, the scheme of the embodiments of the present disclosure can reduce the system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provide convenient and efficient target video task processing services for users.

[0225] The above is a schematic scheme of a task platform according to the embodiments. The technical scheme of the task platform belongs to the same concept as the technical scheme of the information processing method based on the video generation model described above, and the details of the technical scheme of the task platform that are not described in detail can be referred to the description of the technical scheme of the information processing method based on the video generation model described above.

[0226] Corresponding to the video generation method embodiments described above, the present disclosure also provides video generation device embodiments. FIG. 10 shows a structural schematic diagram of a video generation device according to an embodiment of the present disclosure. As shown in FIG. 10, the device comprises:

[0227] The first acquisition module 1002 is configured to acquire video generation data of a target object.

[0228] The first input module 1004 is configured to input the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, which is used to associate the data encoding features by using a time dimension attention mechanism.

[0229] Optionally, the video generation data comprises a reference action sequence, a reference background sequence and an object image, different types of encoding units comprise an image encoding unit, an object encoding unit, an action encoding unit and a background encoding unit, the data encoding features comprise object encoding features, an action encoding sequence and a background encoding sequence, the video generation model further comprises a decoding unit; the first input module 1004 is further configured to input the object image into the image encoding unit, obtain image encoding features, and input the image encoding features into the object encoding unit to obtain object encoding features; input the reference action sequence into the action encoding unit to obtain an action encoding sequence; input the reference background sequence into the background encoding unit to obtain a background encoding sequence; input the object encoding features, the action encoding sequence and the background encoding sequence into the generation unit to obtain video encoding features; and input the video encoding features into the decoding unit to obtain a target video of a target object.

[0230] Optionally, the object encoding unit comprises a plurality of encoding blocks; the first input module 1004 is further configured to input the image encoding features into a first encoding block to obtain first encoding features output by the first encoding block, wherein the first encoding block is a first encoding block in the plurality of encoding blocks; input the encoding features output by a previous encoding block of a second encoding block into the second encoding block to obtain second encoding features output by the second encoding block, wherein the second encoding block is any encoding block other than the first encoding block in the plurality of encoding blocks; and determine the object encoding features according to the encoding features output by the plurality of encoding blocks respectively.

[0231] Optionally, the generation unit comprises a plurality of generation blocks, the object encoding unit comprises a plurality of encoding blocks, and the generation blocks and the encoding blocks correspond to each other one by one; the first input module 1004 is further configured to determine the encoding features corresponding to the plurality of generation blocks respectively according to the correspondence between the plurality of generation blocks and the plurality of encoding blocks; input the action encoding sequence, the background encoding sequence and the encoding features corresponding to a first generation block of the first generation block into the first generation block to obtain first generation features output by the first generation block, wherein the first generation block is a first generation block in the plurality of generation blocks; input the generation features output by a previous generation block of a second generation block and the encoding features corresponding to the second generation block into the second generation block to obtain second generation features output by the second generation block, wherein the second generation block is any generation block other than the first generation block in the plurality of generation blocks; and in a case where the second generation block is a last generation block in the plurality of generation blocks, determine the second generation features output by the second generation block as the video encoding features.

[0232] Optionally, the second generation block includes a self-attention unit, a cross-attention unit, and a temporal attention unit; the first input module 1004 is further configured to input the generated feature output by the previous generation block of the second generation block into the self-attention unit to obtain a self-attention feature; input the self-attention feature and the corresponding encoded feature of the second generation block into the cross-attention unit to obtain a cross-attention feature; and input the cross-attention feature into the temporal attention unit to obtain the second generated feature.

[0233] Optionally, the apparatus further includes a first training module configured to obtain a sample video and sample generation data corresponding to the sample video; input the sample generation data into the video generation model to obtain first predicted features output by the generation unit in the video generation model; input the sample video into the encoding unit to obtain first sample features; and adjust the unit parameters of the encoding unit and the temporal attention unit according to the first predicted features and the first sample features to obtain the trained video generation model.

[0234] Optionally, the sample video includes a plurality of sample video frames; and the apparatus further includes a second training module configured to, for a first sample video frame, determine first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from the plurality of sample video frames; input the first sample generation data into the initial generation model to obtain second predicted features output by the generation unit in the initial generation model; input the first sample video frame into the encoding unit to obtain second sample features; adjust the unit parameters of the encoding unit and the generation unit according to the second predicted features and the second sample features to obtain the pre-trained generation model; and construct the temporal attention unit in the generation unit of the pre-trained generation model to obtain the video generation model.

[0235] Optionally, the video generation data includes a reference action sequence, a reference background sequence, and an object image; the first obtaining module 1002 is further configured to receive a reference video of a target object, background demand information, and an object image sent by a client; input the reference video into an action extraction model to obtain the reference action sequence; and determine the reference background sequence according to the background demand information.

[0236] Optionally, the apparatus further includes a sending module configured to send the target video to the client; and a receiving module configured to receive result feedback information sent by the client, wherein the result feedback information is information fed back by the client on the target video; construct model optimization data according to the result feedback information; and adjust parameters of the video generation model by using the model optimization data.

[0237] By applying the scheme of the embodiments of the present disclosure, since the video generation model includes the temporal attention unit, the video generation model can learn the temporal information, thereby generating a target video that is stable in time sequence and more real and natural, and improving the quality of the target video.

[0238] The above is a schematic scheme of the video generation device of the embodiment. It should be noted that the technical scheme of the video generation device belongs to the same concept as the technical scheme of the video generation method described above, and the details of the technical scheme of the video generation device that are not described in detail can be referred to the description of the technical scheme of the video generation method.

[0239] Corresponding to the motion video generation method embodiment of the virtual object described above, the disclosure also provides a motion video generation device embodiment of the virtual object, and FIG. 11 shows a structural schematic diagram of a motion video generation device of a virtual object according to an embodiment of the disclosure. As shown in FIG. 11, the device comprises:

[0240] The second acquisition module 1102 is configured to acquire motion video generation data of the virtual object.

[0241] The second input module 1104 is configured to input the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, which is used to associate the data encoding features by using a time dimension attention mechanism.

[0242] By applying the scheme of the embodiment of the disclosure, because the time sequence attention unit is included in the video generation model, the video generation model can learn the time sequence information, thereby generating a target motion video that is stable and natural in time sequence.

[0243] The above is a schematic scheme of the motion video generation device of the virtual object of the embodiment. It should be noted that the technical scheme of the motion video generation device of the virtual object belongs to the same concept as the technical scheme of the motion video generation method of the virtual object described above, and the details of the technical scheme of the motion video generation device of the virtual object that are not described in detail can be referred to the description of the technical scheme of the motion video generation method of the virtual object.

[0244] Corresponding to the video editing method embodiment described above, the disclosure also provides a video editing device embodiment, and FIG. 12 shows a structural schematic diagram of a video editing device according to an embodiment of the disclosure. As shown in FIG. 12, the device comprises:

[0245] The third acquisition module 1202 is configured to acquire an original video and target editing data.

[0246] The analysis module 1204 is configured to analyze the original video to determine original video data in the original video.

[0247] The third input module 1206 is configured to input the target editing data and the original video data into the video generation model to obtain a target editing video, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit comprises a time attention unit, and the time attention unit is used to associate the data encoding features by using a time dimension attention mechanism.

[0248] By inputting the target editing data and the original video data into the video generation model, the editing of the original video is realized, and the video generation model can learn time sequence information to generate a target editing video that is stable and natural in time sequence.

[0249] The above is a schematic solution of the video editing device according to the embodiment. It should be noted that the technical solution of the video editing device belongs to the same concept as the technical solution of the video editing method described above, and the details of the technical solution of the video editing device that are not described in detail can be referred to the description of the technical solution of the video editing method.

[0250] Corresponding to the video generation model training method embodiment described above, the present disclosure further provides a video generation model training device embodiment. FIG. 13 shows a structural schematic diagram of a video generation model training device according to an embodiment of the present disclosure. As shown in FIG. 13, the device comprises:

[0251] The fourth acquisition module 1302 is configured to acquire a sample video and sample generation data corresponding to the sample video;

[0252] The fourth input module 1304 is configured to input the sample generation data into the video generation model to obtain first prediction features output by the generation unit in the video generation model, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, the generation unit comprises a time attention unit, and the time attention unit is used to associate the sample encoding features by using a time dimension attention mechanism;

[0253] The fifth input module 1306 is configured to input the sample video into the encoding unit to obtain first sample features;

[0254] The adjustment module 1308 is configured to adjust the unit parameters of the encoding unit and the time attention unit according to the first prediction features and the first sample features to obtain a trained video generation model.

[0255] By applying the scheme of the embodiment of the present disclosure, a time sequence attention unit is additionally added in each generation block included in the generation unit of the pre-trained generation model to obtain a video generation model, so that the video generation model can learn time sequence information, thereby generating a target video that is stable in time sequence and has natural fusion of foreground and background, and making the target video more real and natural.

[0256] The above is a schematic scheme of a video generation model training device according to an embodiment of the present disclosure. It should be noted that the technical scheme of the video generation model training device belongs to the same concept as the technical scheme of the video generation model training method described above, and the details of the technical scheme of the video generation model training device that are not described in detail can be referred to the description of the technical scheme of the video generation model training method.

[0257] Corresponding to the above-mentioned embodiment of the information processing method based on the video generation model, the present disclosure also provides an embodiment of an information processing device based on the video generation model. FIG. 14 shows a structural schematic diagram of an information processing device based on a video generation model according to an embodiment of the present disclosure. As shown in FIG. 14, the device includes:

[0258] The receiving module 1402 is configured to receive a task generation request, wherein the task generation request includes request information;

[0259] The fifth obtaining module 1404 is configured to obtain a video generation model based on the request information, wherein the video generation model includes a generation unit and different types of encoding units, the generation unit includes a time sequence attention unit, and the time sequence attention unit is used to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism;

[0260] The generation module 1406 is configured to generate task information based on the video generation model, wherein the task information is used to execute a target video task.

[0261] Optionally, the fifth obtaining module 1404 is further configured to determine a target scene template from a plurality of preset scene templates based on a task scene identifier, and find the video generation model from a model library based on the target scene template, wherein the model library stores a plurality of video generation models, and the request information includes a task scene identifier of the target video task; or find the video generation model from the model library based on a task model identifier, wherein the request information includes a task model identifier of the target video task.

[0262] Optionally, the request information includes a sample video of the target video task and sample generation data corresponding to the sample video; and the fifth obtaining module 1404 is further configured to train the video generation model corresponding to the target video task based on the sample video and the sample generation data corresponding to the sample video, to obtain a trained video generation model.

[0263] By generating the task information of the target video task, the scheme of the embodiment of the present disclosure reduces the system deployment and operation and maintenance costs while ensuring the processing quality and efficiency of the target video task, and provides convenient and efficient task processing services for users.

[0264] The above is a schematic scheme of the information processing device based on the video generation model according to the present embodiment. It should be noted that the technical scheme of the information processing device based on the video generation model belongs to the same concept as the technical scheme of the information processing method based on the video generation model described above, and the details of the technical scheme of the information processing device based on the video generation model that are not described in detail can be referred to the description of the technical scheme of the information processing method based on the video generation model.

[0265] FIG. 15 shows a structural block diagram of a computing device according to an embodiment of the present disclosure. The components of the computing device 1500 include, but are not limited to, a memory 1510 and a processor 1520. The processor 1520 is connected to the memory 1510 through a bus 1530, and a database 1550 is used to save data.

[0266] The computing device 1500 also includes an access device 1540, which enables the computing device 1500 to communicate via one or more networks 1560. Examples of these networks include the public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1540 can include one or more of any type of network interface (e.g., network interface card (NIC)) such as a IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, etc.

[0267] In one embodiment of the present disclosure, the above-mentioned components of the computing device 1500 and other components not shown in FIG. 15 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 15 is only for the purpose of example, and is not a limitation on the scope of the present disclosure. Those skilled in the art can add or replace other components as needed.

[0268] The computing device 1500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1500 can also be a mobile or stationary server.

[0269] The processor 1520 is configured to execute computer programs / instructions that implement the steps of the above-mentioned video generation method or the virtual object motion video generation method or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0270] The above is a schematic scheme of a computing device according to the present embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical schemes of the above-mentioned video generation method, virtual object motion video generation method, video editing method, video generation model training method, and information processing method based on the video generation model. For details of the technical scheme of the computing device that are not described in detail, please refer to the description of the technical scheme of the above-mentioned video generation method or the virtual object motion video generation method or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0271] The present embodiment also provides a computer-readable storage medium storing computer programs / instructions that implement the steps of the above-mentioned video generation method or the virtual object motion video generation method or the video editing method or the video generation model training method or the information processing method based on the video generation model when executed by a processor.

[0272] The above describes a schematic solution of the computer readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solutions of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model. Details of the technical solution of the storage medium that are not described in detail can be found in the descriptions of the technical solutions of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0273] An embodiment of the present disclosure further provides a computer program product, including computer programs / instructions, which, when executed by a processor, implement the steps of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0274] The above describes a schematic solution of the computer program product of the embodiment. It should be noted that the technical solution of the computer program product belongs to the same concept as the technical solutions of the video generation method, the motion video generation method of a virtual object, the video editing method, the video generation model training method, and the information processing method based on the video generation model. Details of the technical solution of the computer program product that are not described in detail can be found in the descriptions of the technical solutions of the video generation method or the motion video generation method of a virtual object or the video editing method or the video generation model training method or the information processing method based on the video generation model.

[0275] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0276] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0277] It should be noted that, for the foregoing method embodiments, in order to facilitate description, each is described as a combination of a series of acts, but those skilled in the art should appreciate that the embodiments of the present disclosure are not limited by the order of the described acts, because, according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should appreciate that the embodiments described in the specification are all preferred embodiments, and the acts and modules involved are not necessarily essential to the embodiments of the present disclosure.

[0278] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0279] The preferred embodiments of the present disclosure disclosed above are only used to help explain the present disclosure. The alternative embodiments do not describe all the details and do not limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present disclosure, many modifications and changes can be made. The present disclosure selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can well understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

A video generation method, comprising: obtaining video generation data of a target object; inputting the video generation data into a video generation model to obtain a target video of the target object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the video generation data to obtain data encoding features, and the generation unit comprises a time attention unit used to associate the data encoding features by using a time dimension attention mechanism. According to the method of claim 1, the video generation data comprises a reference action sequence, a reference background sequence and an object image, the different types of encoding units comprise an image encoding unit, an object encoding unit, an action encoding unit and a background encoding unit, the data encoding features comprise object encoding features, an action encoding sequence and a background encoding sequence, and the video generation model further comprises a decoding unit; the inputting the video generation data into a video generation model to obtain a target video of the target object comprises: inputting the object image into the image encoding unit to obtain image encoding features, and inputting the image encoding features into the object encoding unit to obtain the object encoding features; inputting the reference action sequence into the action encoding unit to obtain the action encoding sequence; inputting the reference background sequence into the background encoding unit to obtain the background encoding sequence; inputting the object encoding features, the action encoding sequence and the background encoding sequence into the generation unit to obtain video encoding features; inputting the video encoding features into the decoding unit to obtain the target video of the target object. According to the method of claim 2, the object encoding unit comprises a plurality of encoding blocks; the inputting the image encoding features into the object encoding unit to obtain the object encoding features comprises: inputting the image encoding features into a first encoding block to obtain first encoding features output by the first encoding block, wherein the first encoding block is a first encoding block in the plurality of encoding blocks; inputting encoding features output by a previous encoding block of a second encoding block into the second encoding block to obtain second encoding features output by the second encoding block, wherein the second encoding block is any encoding block other than the first encoding block in the plurality of encoding blocks; determining the object encoding features according to the encoding features output by the plurality of encoding blocks respectively. According to the method of claim 2 or 3, the generation unit comprises a plurality of generation blocks, the object encoding unit comprises a plurality of encoding blocks, and the generation blocks and the encoding blocks correspond to each other one by one; the inputting the object encoding features, the action encoding sequence and the background encoding sequence into the generation unit to obtain video encoding features comprises: determining encoding features corresponding to the plurality of generation blocks respectively according to a correspondence relationship between the plurality of generation blocks and the plurality of encoding blocks; and input the action encoding sequence, the background encoding sequence and an encoding feature corresponding to a first generation block of the plurality of generation blocks into the first generation block to obtain a first generation feature output by the first generation block, wherein the first generation block is a first generation block in the plurality of generation blocks; input a generation feature output by a previous generation block of a second generation block and an encoding feature corresponding to the second generation block into the second generation block to obtain a second generation feature output by the second generation block, wherein the second generation block is any generation block in the plurality of generation blocks except the first generation block; in a case where the second generation block is a last generation block in the plurality of generation blocks, determine the second generation feature output by the second generation block as the video encoding feature. The method according to claim 4, wherein the second generation block comprises a self-attention unit, a cross-attention unit and a temporal attention unit. The inputting a generation feature output by a previous generation block of a second generation block and an encoding feature corresponding to the second generation block into the second generation block to obtain a second generation feature output by the second generation block comprises: inputting the generation feature output by the previous generation block of the second generation block into the self-attention unit to obtain a self-attention feature; inputting the self-attention feature and the encoding feature corresponding to the second generation block into the cross-attention unit to obtain a cross-attention feature; inputting the cross-attention feature into the temporal attention unit to obtain the second generation feature. The method according to any one of claims 1-5, and before the inputting the video generation data into a video generation model to obtain a target video of a target object, further comprising: obtaining a sample video and sample generation data corresponding to the sample video; inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model; inputting the sample video into the encoding unit to obtain a first sample feature; adjusting unit parameters of the encoding unit and the temporal attention unit according to the first prediction feature and the first sample feature to obtain the video generation model after training. The method according to claim 6, wherein the sample video comprises a plurality of sample video frames. Before the inputting the sample generation data into the video generation model to obtain a first prediction feature output by a generation unit in the video generation model, further comprising: for a first sample video frame, determining first sample generation data corresponding to the first sample video frame, wherein the first sample video frame is sampled from the plurality of sample video frames; inputting the first sample generation data into an initial generation model to obtain a second prediction feature output by a generation unit in the initial generation model; inputting the first sample video frame into the encoding unit to obtain a second sample feature; adjusting unit parameters of the encoding unit and the generation unit according to the second prediction feature and the second sample feature to obtain a pre-training generation model; constructing a temporal attention unit in the generation unit of the pre-training generation model to obtain the video generation model. The method according to any one of claims 1-7, wherein the video generation data comprises a reference action sequence, a reference background sequence and an object image; The method according to any one of claims 1-7, wherein the video generation data of the target object comprises: receiving a reference video of the target object, background requirement information and an object image sent by a client; inputting the reference video into an action extraction model to obtain the reference action sequence; determining the reference background sequence according to the background requirement information. The method according to any one of claims 1-8, wherein after the video generation data is inputted into the video generation model to obtain the target video of the target object, the method further comprises: sending the target video to the client; receiving result feedback information sent by the client, wherein the result feedback information is information fed back by the client to the target video; constructing model optimization data according to the result feedback information; and adjusting parameters of the video generation model by using the model optimization data. A method for generating a motion video of a virtual object, comprising: obtaining motion video generation data of a virtual object; inputting the motion video generation data into a video generation model to obtain a target motion video of the virtual object, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the motion video generation data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, the time sequence attention unit is used to associate the data encoding features by using a time dimension attention mechanism. A video editing method, comprising: obtaining an original video and target editing data; parsing the original video to determine original video data in the original video; inputting the target editing data and the original video data into a video generation model to obtain a target editing video, wherein the video generation model comprises a generation unit and different types of encoding units, the different types of encoding units are used to encode the target editing data and the original video data to obtain data encoding features, and the generation unit comprises a time sequence attention unit, the time sequence attention unit is used to associate the data encoding features by using a time dimension attention mechanism. A video generation model training method, comprising: obtaining a sample video and sample generation data corresponding to the sample video; inputting the sample generation data into the video generation model to obtain first prediction features output by a generation unit in the video generation model, wherein the video generation model comprises the generation unit and different types of encoding units, the different types of encoding units are used to encode the sample generation data to obtain sample encoding features, and the generation unit comprises a time sequence attention unit, the time sequence attention unit is used to associate the sample encoding features by using a time dimension attention mechanism; inputting the sample video into the encoding unit to obtain first sample features; adjusting unit parameters of the encoding unit and the time sequence attention unit according to the first prediction features and the first sample features to obtain a trained video generation model. An information processing method based on a video generation model, comprising: receiving a task generation request, wherein the task generation request comprises request information; obtaining a video generation model based on the request information, wherein the video generation model comprises a generation unit and different types of encoding units, the generation unit comprises a time attention unit, and the time attention unit is used to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism; generating task information based on the video generation model, wherein the task information is used to execute a target video task. The method according to claim 13, wherein the obtaining the video generation model based on the request information comprises: determining a target scene template from a plurality of preset scene templates based on a task scene identifier, and searching for the video generation model from a model library based on the target scene template, wherein the model library stores a plurality of video generation models, and the request information comprises the task scene identifier of the target video task; or searching for the video generation model from the model library based on a task model identifier, wherein the request information comprises the task model identifier of the target video task. The method according to claim 13, wherein the request information comprises a sample video of a target video task and sample generation data corresponding to the sample video; The obtaining the video generation model based on the request information comprises: training a video generation model corresponding to the target video task based on the sample video and the sample generation data corresponding to the sample video, to obtain a trained video generation model. A task platform, comprising a request interface and a response unit; The request interface is configured to receive a task generation request, wherein the task generation request comprises request information; The response unit is configured to obtain a video generation model based on the request information, wherein the video generation model comprises a generation unit and different types of encoding units, the generation unit comprises a time attention unit, and the time attention unit is used to associate data encoding features output by the different types of encoding units by using a time dimension attention mechanism; and generate task information based on the video generation model, wherein the task information is used to execute a target video task. The task platform of claim 16, further comprising a model library, wherein, The model library stores a plurality of video generation models; The response unit is specifically configured to determine a target scene template from a plurality of preset scene templates based on a task scene identifier, and search for the video generation model from the model library based on the target scene template, wherein the request information comprises the task scene identifier of the target video task; or search for the video generation model from the model library based on a task model identifier, wherein the request information comprises the task model identifier of the target video task. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15. A computer readable storage medium storing computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15. A computer program product comprising computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9 or claim 10 or claim 11 or claim 12 or any one of claims 13 to 15.

Images