Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first video clip and text description information; processing the first video clip based on a second adapter model obtained by jointly training a first adapter model and a video model to obtain a first vector; and based on the video model, processing the text description information and the first vector to obtain a second video clip. According to the embodiment of the invention, the video clip which is highly matched with the text description information and the first video clip representation information and can control the moving track of the target object can be generated.

Description

Technical Field

[0001] The present disclosure relates to Vincent video technology, and in particular to a video generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of video generation model technology, video generation models have shined in the field of artificial intelligence content generation (AIGC) technology.

[0003] In the related technology, natural language processing is performed on the text description information input by the user to obtain the corresponding feature information, and the feature information is processed using a video generation model to generate a video corresponding to the text description information. When generating a video, it is a very important requirement to control the motion trajectory of objects in the video. For example, in the post-production of a movie or TV series, this technology can be used to generate or modify specific scenes, such as adding or adjusting the motion of an object, without reshooting. This can save a lot of cost and time, while improving production efficiency and quality.

[0004] How to generate videos that are highly consistent with text description information and can control the motion trajectory of objects is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present disclosure provide a video generation method, device, electronic device and storage medium to solve the above problems.

[0006] According to a first aspect of the present disclosure, a video generation method is provided, including: Acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first target object, a pixel value of each video frame of the first video clip is a preset pixel value, and each video frame of the first video clip has a target frame of the position of the first target object; Processing the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector; Based on the video model, the text description information and the first vector are processed to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first target object based on the target frame.

[0007] In some embodiments of the present disclosure, obtaining the first video clip includes: Based on a preset video frame rate, resolution, and video duration, a third video segment is acquired, wherein a pixel value of each video frame of the third video segment is the preset pixel value; Based on preset object position information, determining the position of the first target object in each video frame of the third video segment; Based on the position of the first target object in each video frame of the third video segment, a target frame in each video frame of the third video segment is determined to obtain the first video segment.

[0008] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the second adapter model and the input layer of the video model both include multiple downsampling layers; The step of processing the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector includes: processing the first video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model, wherein the first vector includes the output vector of each downsampling layer of the second adapter model; The text description information and the first vector are processed based on the video model to obtain a second video clip, including: inputting the text description information and the first Gaussian noise into the video model; at each time step, weightedly calculating the output vectors of each downsampling layer of the two adapter models with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0009] In some embodiments of the present disclosure, the step of performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model to obtain weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next sampling layer corresponding to the video model includes: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model; Performing weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model; Using the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth input vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained by weighted calculation based on the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0010] In some embodiments of the present disclosure, before the first video clip is processed based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain the first vector, the method further includes: Acquire a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second target object, and each video frame of the sample video clip has a target frame indicating a position of the second target object; Fixing the parameters of the video model, and jointly training the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model; Acquire a fourth video segment having the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same; The parameters of the video model are fixed, and based on the fourth video clip and the sample text description information, the third adapter model and the video model are jointly trained to obtain the second adapter model.

[0011] In some embodiments of the present disclosure, the parameters of the video model are fixed, and the first adapter model and the video model are jointly trained based on the sample video clip and the sample text description information to obtain a third adapter model, including: Processing the sample video clip based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; Based on the video model, the sample text description information, the output vectors of each intermediate layer of the first adapter model, and the second Gaussian noise matrix are processed to obtain a fifth video clip; The parameters of the video model are fixed, and based on the difference between the fifth video segment and the sample video segment, the parameters of the first adapter model are adjusted to obtain the third adapter model.

[0012] In some embodiments of the present disclosure, fixing the parameters of the video model, and jointly training the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model includes: Processing the fourth video clip based on the third adapter model to obtain output vectors of each downsampling layer of the third adapter model; Based on the video model, the sample text description information, the output vectors of each intermediate layer of the third adapter model, and the third Gaussian noise matrix are processed to obtain a sixth video clip; The parameters of the video model are fixed, and based on the difference between the sixth video segment and the sample video segment, the parameters of the third adapter model are adjusted to obtain the second adapter model.

[0013] According to a second aspect of the embodiments of the present disclosure, a video generating device is provided, including: an acquisition module, configured to acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first target object, a pixel value of each video frame of the first video clip is a preset pixel value, and each video frame of the first video clip has a target frame of the position of the first target object; an adapter model processing module, configured to process the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector; A video model processing module is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first target object based on the target frame.

[0014] In some embodiments of the present disclosure, the acquisition module is used to acquire a third video segment based on a preset video frame rate, resolution, and video duration, wherein a pixel value of each video frame of the third video segment is the preset pixel value; The acquisition module is further used to determine the position of the first target object in each video frame of the third video segment based on preset object position information; The acquisition module is further used to determine a target frame in each video frame of the third video segment based on a position of the first target object in each video frame of the third video segment to obtain the first video segment.

[0015] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the second adapter model and the input layer of the video model both include multiple downsampling layers; The adapter model processing module is used to process the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes output vectors of each downsampling layer of the second adapter model; The video model processing module is used to input the text description information and the first Gaussian noise into the video model; at each time step, the output vectors of each downsampling layer of the two adapter models are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and the weighted calculation results of the output vectors of each downsampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0016] In some embodiments of the present disclosure, the video model processing module is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model; The video model processing module is also used to perform weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model; The video model processing module is further used to use the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The video model processing module is also used to use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained based on the weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0017] In some embodiments of the present disclosure, the device further includes: A model training module, used for obtaining a sample video segment and sample text description information, wherein the sample text description information at least includes attribute information of a second target object, and each video frame of the sample video segment has a target frame indicating a position of the second target object; The model training module is further used to fix the parameters of the video model, and to jointly train the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model; The model training module is also used to obtain a fourth video segment with the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same; The model training module is also used to fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0018] In some embodiments of the present disclosure, the model training module is used to process the sample video clip based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; The model training module is further used to process the sample text description information, the output vectors of each intermediate layer of the first adapter model, and the second Gaussian noise matrix based on the video model to obtain a fifth video clip; The model training module is also used to fix the parameters of the video model, and based on the difference between the fifth video segment and the sample video segment, adjust the parameters of the first adapter model to obtain the third adapter model.

[0019] In some embodiments of the present disclosure, the model training module is used to process the fourth video clip based on the third adapter model to obtain output vectors of each downsampling layer of the third adapter model; The model training module is also used to process the sample text description information, the output vectors of each intermediate layer of the third adapter model, and the third Gaussian noise matrix based on the video model to obtain a sixth video clip; The model training module is also used to fix the parameters of the video model, and based on the difference between the sixth video segment and the sample video segment, adjust the parameters of the third adapter model to obtain the second adapter model.

[0020] According to a third aspect of the present disclosure, an electronic device is provided, including: a memory for storing a computer program product; The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0021] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0022] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the method described in the first aspect.

[0023] The video generation method, device, electronic device and storage medium of the embodiments of the present invention use the text description information as the basic condition for generating the video, and use the first video segment with the target object position change information as the auxiliary limiting condition for generating the video, because the adapter model cooperates with the video model to generate a video segment that is highly matched with the text description information and the first video segment representation information and can control the movement trajectory of the target object.

[0024] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 A schematic diagram of a process of generating a video in an embodiment of the present disclosure; Figure 2 This is a partial flow chart of step S1 in some embodiments of the present disclosure; Figure 3 is a flowchart of step S3 in some embodiments of the present disclosure; Figure 4 is a flowchart of step S0 in some embodiments of the present disclosure; Figure 5 is a structural block diagram of a video generating device in some embodiments of the present disclosure; Figure 6 is a structural block diagram of a video generation device in another embodiment of the present disclosure; Figure 7 The structure block diagram of an electronic device in one embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0027] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0028] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0029] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0030] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0031] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0032] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0035] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0036] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0037] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0038] Figure 1 FIG. 1 is a flow chart of a video generation method in one embodiment of the present disclosure. Figure 1 As shown, the video generation method comprises the following steps: S1: Obtain a first video clip and text description information.

[0039] A first video clip and text description information input by a user are received.

[0040] The pixel value of each video frame of the first video segment is a preset pixel value. The preset pixel value may be a pixel value corresponding to a preset color. Exemplarily, all pixel values ​​in each video frame of the first video segment are pixel values ​​RGB (0, 0, 0) corresponding to black.

[0041] The text description information at least includes the attribute information of the first target object, for example, it may include the type of the first target object and other attribute information of the first target object. The type of the first target object may include: a person type, an animal type, or other object type (for example, an object that needs to be moved in a movie or game scene). Other attribute information of the first target object may include color information, structure information (for example, when the first target object is a person or an animal, it may include the body proportions of the first target object) and other information of the first target object.

[0042] Each video frame of the first video clip has a target frame representing the position of the first target object in the video frame. When the first target object includes multiple objects, each video frame of the first video clip may include multiple target frames, and the multiple objects correspond to each target frame. The target frames representing the same object in each video frame of the first video clip may have the same or different sizes. For example, when the same object in each video frame of the first video clip moves at the same visual distance, the size of the target frame corresponding to the object in each video frame may be the same; when the same object in each video frame of the first video clip moves at different visual distances, the size of the target frame corresponding to the object in each video frame may be different.

[0043] S2: Based on the second adapter model obtained by jointly training the first adapter model and the video model, the first video clip is processed to obtain a first vector.

[0044] Among them, the first adapter model is an untrained adapter model, which can also be called the initial adapter model. The second adapter model is an adapter model obtained through joint training, which can also be called the final adapter model. Optionally, in the process of obtaining the second adapter model based on the joint training of the first adapter model and the video model, the first adapter model and the video model are first obtained, wherein the network layer structure of the first adapter model is the same as the network structure of the input layer of the video model, and each network layer of the first adapter model is mapped one-to-one with the network layer of the input layer of the video model, that is, the number of network layers of the first adapter model is the same as the number of network layers of the input layer of the video model, and each network layer of the first adapter model has the same network parameters as the input layer corresponding to the video model. Secondly, the network parameters of each network layer of the first adapter model are adjusted through the loss function of the video model to obtain the second adapter model.

[0045] In an embodiment of the present invention, by obtaining a second adapter model based on joint training of the first adapter model and the video model, it is not necessary to directly adjust the parameters of the video model (the magnitude of the parameters of the video model is generally large, such as billions or tens of billions), which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up the training speed. At the same time, when the video model performs a specific task, the output vector of the input layer of the video model is fine-tuned through the trained second adapter model, thereby achieving the specific task that can be completed efficiently and accurately without large-scale overall training of the video model. It can be seen that the second adapter model is obtained by joint training of the first adapter model and the video model, and it is realized that based on AIGC technology, the second video clip with the target object position change information can be quickly generated through the first video clip and text description information, and the versatility of the video model will not be affected.

[0046] The video model may be a neural network model based on a diffusion model, and is used to generate a corresponding video according to input data information, wherein the data information includes text information, image information, or voice information, etc., wherein the voice information may be converted into text information through voice information recognition and input into the video model. The video model may be pre-trained, for example, it may be trained according to input samples including input text descriptions and input videos.

[0047] Furthermore, the diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a gradual process of adding noise and denoising, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In the disclosed embodiment, the diffusion model is a pre-trained model. After the conditional information is mapped to the latent space expression by the encoder, under the guidance of the latent space expression of the conditional information, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the final latent space expression of the second video clip, and then the latent space expression of the target video is further mapped to the pixel space by the decoder to obtain the final second video clip. Among them, the latent space expression of the conditional information is the latent space expression obtained after encoding the conditional information using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (LatentSpace) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some implementations, the encoder and decoder used in the diffusion model of the disclosed embodiments may both be variational autoencoders (VAEs). The latent space of VAEs is continuous and suitable for generating continuous data, i.e., the generated target videos or images are correlated with each other.

[0048] The adapter model is a model based on neural network model technology, which is used to fine-tune the parameters of the generated video based on the video model according to the video generation task. It achieves this goal by adding small, trainable modules without retraining the video model. It can not only quickly and cost-effectively meet the video generation needs of the corresponding application scenarios, but also does not affect the versatility of the video model.

[0049] The first video segment is processed using the second adapter model, and a first vector corresponding to the first video segment is output.

[0050] S3: Based on the video model, the text description information and the first vector are processed, and the second video clip is obtained.

[0051] The second video segment includes a motion trajectory of the first target object. The motion trajectory of the first target object is determined based on the change in the position of the target frame in each video frame of the first video segment, that is, the position of the first target object in each video frame of the second video segment is the same as the position of the first target object in the corresponding video frame of the first video segment.

[0052] The text description information and the first vector are used as inputs of the video model, and a second video segment corresponding to the text description information is output. Each video frame of the second video segment has a first target object, and the position of the first target object in each video frame of the second video segment corresponds one-to-one to the position of the target frame in each video frame of the first video segment.

[0053] In this embodiment, since the adapter model cooperates with the video model to generate a video clip that is highly matched with the text description information, the text description information is used as the basic condition for generating the video, and the first video clip with the target object position change information is used as the auxiliary limiting condition for generating the video. The adapter model is used in conjunction with the video model to generate a video clip that is highly matched with the text description information and the first video clip representation information and can control the movement trajectory of the target object.

[0054] Figure 2 is a partial flow chart of step S1 in some embodiments of the present disclosure. Figure 2 As shown, step S1 may include the following steps: S1-1: Based on a preset video frame rate, resolution, and video duration, a third video segment is obtained.

[0055] For example, if the preset video frame rate is B frames / second, the resolution is L×W, and the video duration is A seconds, then the duration of the third video segment is A seconds, the frame rate is B frames per second, the resolution is L×W, and all pixel values ​​of each video frame are preset pixel values ​​(e.g., RGB (0,0,0)). A, B, L, and W are all integers greater than 1.

[0056] S1-2: Determine the position of the first target object in each video frame of the third video segment based on preset object position information.

[0057] Among them, the preset object position information may include: the pixel coordinate range of the target box in each video frame of the third video clip, or the pixel coordinate values ​​of the four vertices of the target box (when it is a rectangular target box) in each video frame of the third video clip, or the pixel coordinates of the center point of the first target object in each video frame of the third video clip.

[0058] According to the preset object position information, the position of the first target object in each video frame of the third video segment can be accurately determined.

[0059] S1-3: Based on the position of the first target object in each video frame of the third video segment, generate a target frame in each video frame of the third video segment to obtain the first video segment.

[0060] When the preset object position information includes the pixel coordinate range of the target box in each video frame of the third video segment, the target box in each video frame can be generated according to the pixel coordinate range of the target box in each video frame, thereby obtaining a second video segment in which each video frame has a target box representing the position of the first target object.

[0061] When the preset object position information includes the pixel coordinate values ​​of the four vertices of the target box in each video frame of the third video clip, the four edges of the target box can be formed by connecting the four vertices in each video frame, and then the target box in each video frame is generated, thereby obtaining a second video clip in which each video frame has a target box representing the position of the first target object.

[0062] When the preset object position information includes the pixel coordinates of the center point of the first target object in each video frame of the third video segment, combined with the pixel size information of the first target object in each video frame of the third video segment, a target frame in each video frame can be generated, thereby obtaining a second video segment in which each video frame has a target frame representing the position of the first target object.

[0063] In this embodiment, the third video segment is determined according to the preset video frame rate, resolution and video length, and combined with the position of the first target object in each video frame of the third video segment, a first video segment in which each video frame has a target box representing the position of the first target object can be quickly obtained.

[0064] In some embodiments of the present disclosure, step S1-1 may include: obtaining a target video segment having the same frame rate, resolution and duration as the third video segment; adjusting the pixel value of each video frame in the target video segment to a preset pixel value to obtain a third video segment.

[0065] According to the frame rate, resolution and length of the third video clip, a target video clip (such as an advertisement clip, a movie clip or an animation clip, etc.) with the same frame rate, resolution and length as the third video clip and with video content can be captured from a network video resource or a local video library, and the pixel value of each video frame in the target video clip can be adjusted to a preset pixel value to obtain the third video clip.

[0066] In an example of the present disclosure, when the third video segment needs to have a duration of A seconds, a frame rate of B frames / second, a resolution of L×W, and all pixel values ​​of each video frame are RGB (0,0,0), a target video segment with a duration of A seconds, a frame rate of B frames / second, a resolution of L×W, and video content can be intercepted from a network video resource or a local video resource library, and then all pixel values ​​in all video frames of the target video segment are adjusted to RGB (0,0,0) to obtain the third video segment.

[0067] In this embodiment, since the required length, frame rate and resolution of the third video clip are known, a target video clip with the same length, frame rate and resolution as the third video clip can be captured from a network resource or a local resource library, and then the required third video clip can be quickly obtained by adjusting the pixel values ​​of each video frame of the target video clip.

[0068] In some embodiments of the present disclosure, the input layer structures of the second adaptor model and the video model are the same. The second adaptability model input layers and the video model input layers have the same number and the same network structure, and feature vectors of the same dimension can be extracted for the input information of the model through the same number and the same network structure of the input layers.

[0069] Accordingly, step S2 includes: processing the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes the output vectors of each downsampling layer of the second adapter model. For example, when the second adapter model includes N+1 downsampling layers and N is an integer greater than 1, the output vectors of the 1st downsampling layer to the output vectors of the N+1th downsampling layer are stored in the order from the 1st layer to the N+1th layer, and are recorded as vectors VA1 to VA N+1 .

[0070] Correspondingly, step S3 includes: inputting the text description information and the first Gaussian noise into the video model; at each time step, performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model, respectively, to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next downsampling layer corresponding to the video model to obtain the second video clip. Among them, the first Gaussian noise matrix can be a randomly generated Gaussian distribution matrix used to add noise to the video model. The text description information, the output vectors of each downsampling layer of the second adapter model, and the first Gaussian noise matrix are used as the input of the video model, and denoising is performed step by step through each downsampling layer of the video model, and finally the second video clip is output.

[0071] In each denoising process, the output vectors of each downsampling layer of the first adapter model are weighted with the output vectors of the corresponding downsampling layers of the video model to obtain the output vector of the video model input layer. The video model includes an input layer and an output layer, the input layer of the video model includes multiple downsampling layers, the multiple downsampling layers of the video model input layer are mapped one-to-one with the multiple downsampling layers of the first adapter model, and the structure of each group of corresponding downsampling layers is the same. The output vector of each downsampling layer of the first adapter model is used for weighted calculation with the output vector of the mapped downsampling layer of the video model to serve as the input vector of the next downsampling layer of the video model.

[0072] It can be understood that the input layer and output layer of the above video model perform one round of calculation for noise prediction of one time step, that is, denoising processing of one time step. The video model can obtain the final target video by performing T rounds of iterative denoising. After the output layer of the video model outputs the output vector of the first round, it is input into the input layer of the video model again as the initial input vector of the second round. Among them, T is an integer greater than 1. After the denoising is completed after T rounds of loop execution according to the above process, the final output vector of the output layer of the video model is decoded by the decoder, and it is mapped from the latent space to the pixel space to obtain the second video clip.

[0073] In this embodiment, since the input layer structures of the second adapter model and the video model are the same, feature vectors of the same dimension can be extracted for the input information of each model, and the video model performs step-by-step denoising on the first Gaussian noise matrix, thereby obtaining a second video clip that has a high degree of match with the text description information.

[0074] Figure 3 FIG. 1 is a partial flow chart of step S3 in some embodiments of the present disclosure. Figure 3 As shown, step S3 may further include the following steps: S3-1: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector.

[0075] By extracting features from the text description information and the first Gaussian noise matrix through the input layer of the video model, a vector of a predetermined dimension can be obtained as an input vector of the first downsampling layer of the video model. After processing the input vector through the first downsampling layer of the video model, an output vector of the first downsampling layer of the video model can be obtained, which is recorded as the first video output vector and represented by vector VB1.

[0076] S3-2: Perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first input vector.

[0077] From the output vectors of each downsampling layer of the second adapter model, the output vector of the first downsampling layer of the second adapter model (ie, vector VA1) is extracted and recorded as the first adapter output vector. The vector VA1 and the vector VB1 are added to obtain the first input vector.

[0078] S3-3: Use the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector.

[0079] The first input vector is used as the input of the second downsampling layer of the video model to obtain the output vector of the second downsampling layer of the video model, which is recorded as the second video output vector VB2. The first downsampling layer of the video model is adjacent to the second downsampling layer of the video model, that is, the second downsampling layer of the video model is the next downsampling layer adjacent to the first downsampling layer of the video model.

[0080] S3-4: Use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model.

[0081] In a manner similar to step S3-3, the 2nd downsampling layer to the N+1th downsampling layer of the video model are processed to obtain the output vectors of the 2nd downsampling layer to the Nth downsampling layer of the video model.

[0082] In this embodiment, since the input layer structures of the second adapter model and the video model are the same, and the dimensions of the corresponding downsampling layer input vectors of the second adapter model and the video model are the same, the first Gaussian noise matrix is ​​input into the video model, and the output vectors of each downsampling layer of the first adapter model and the video model are weighted to obtain a vector as the input of the next downsampling layer of the video model, thereby achieving denoising within a single time step, and finally obtaining a second video clip that highly matches the guidance video generation conditions through step-by-step denoising.

[0083] In some embodiments of the present disclosure, before step S2, the following steps may also be included: Step S0: Based on the sample video clips and the sample text description information, the first adapter model and the video model are jointly trained to obtain a second adapter model.

[0084] Figure 4 FIG. 1 is a flow chart of step S0 in some embodiments of the present disclosure. Figure 4 As shown, step S0 may include the following steps: S0-1: Obtain sample video clips and sample text description information.

[0085] The sample text description information at least includes attribute information of the second target object, for example, it may include the type of the second target object and other attribute information of the second target object. The pixel value of each video frame of the sample video clip is a preset pixel value. Each video frame of the sample video clip has a target frame representing the position of the second target object. The second target object may include: a target person, a target animal or other object. The second target object may include only one object or multiple objects. When the second target object includes multiple objects, each video frame of the sample video clip may include multiple target frames, and the multiple objects correspond to the multiple target frames. The target frames representing the same object in each video frame of the sample video clip may be of the same or different sizes.

[0086] Sample text description information can be obtained by manually annotating sample video clips.

[0087] S0-2: Fix the parameters of the video model, and jointly train the first adapter model and the video model based on the sample video clips and the sample text description information to obtain the third adapter model.

[0088] When the first adapter model and the video model are jointly trained, the parameters of the video model are fixed, sample video clips and sample text description information are used as inputs for the joint training, the video model is used to output the video clips, and the parameters of the first adapter model are adjusted according to the differences between the sample video clips and the video clips output by the video model to obtain a third adapter model.

[0089] The third adapter model combined with the video model can generate a corresponding video segment according to the input text description information and the video segment, but the generated video segment has a target box indicating the position of the target object.

[0090] S0-3: Obtain a fourth video segment with the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being a preset pixel value and the target frame position of each video frame being the same.

[0091] According to the frame rate, resolution and duration of the sample video clip, a video clip having the same frame rate, resolution and duration as the sample video clip can be captured from a network video resource or a local video resource library.

[0092] After adjusting all pixel values ​​of each video frame of the intercepted video clip to preset pixel values, a target frame with the same target frame position is generated in each video frame of the intercepted video clip according to the position of the target frame in each video frame of the sample video clip, thereby obtaining a fourth video clip.

[0093] S0-4: Fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0094] When the third adapter model and the video model are jointly trained, the parameters of the video model are fixed, the fourth video clip and the sample text description information are used as inputs for the joint training, the video model is used to output the video clip, and the parameters of the third adapter model are adjusted according to the difference between the sample video clip and the video clip output by the video model to obtain the second adapter model.

[0095] The second adapter model combined with the video model can generate a video segment that highly matches the input text information based on the input text description information and the video segment in which each video frame has a target box representing the position of the target object.

[0096] In this embodiment, since the model accuracy obtained by directly jointly training the first adapter model and the video model using video clips with sample text description information and target pixel values ​​of each video frame as preset pixel values ​​is insufficient, the first adapter model and the video model are jointly trained in a first stage using sample video clips and sample text description information to obtain a third adapter model, and the third adapter model and the video model are jointly trained in a second stage using video clips with sample text description information and target pixel values ​​of each video frame as preset pixel values ​​to obtain a second adapter model, which helps the second adapter model to cooperate with the video model to generate video clips that are highly matched with the input text description information.

[0097] In some embodiments of the present disclosure, step S0-2 may include: processing the sample video clip based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model; based on the video model, processing the sample text description information, the output vectors of each downsampling layer of the first adapter model, and the second Gaussian noise matrix to obtain a fifth video clip; fixing the parameters of the video model, and adjusting the parameters of the first adapter model based on the difference between the fifth video clip and the sample video clip to obtain a third adapter model.

[0098] The sample text description information and the sample video clip are used as inputs for joint training, and a fifth video clip is obtained by a processing method similar to step S2 and step S3. The parameters of the video model are fixed, and the parameters of the first adapter model are adjusted by back propagation according to the difference between the fifth video clip and the sample video clip, so as to obtain a third adapter model.

[0099] In this embodiment, without adjusting the parameters of the video model, the text description information is used as the basic condition for generating the video, and the sample video clip is used as the auxiliary limiting condition for generating the video. The first adapter model and the video model are jointly trained to obtain the third adapter model, which helps the third adapter model to combine with the video model to generate a video clip that has a certain degree of match with the text description information based on the input text description information and the video clip, but the generated video clip has a target box representing the position of the target object.

[0100] In some embodiments of the present disclosure, step S0-4 may include: processing the fourth video clip based on the third adapter model to obtain the output vectors of each downsampling layer of the third adapter model; based on the video model, processing the sample text description information, the output vectors of each downsampling layer of the third adapter model, and the third Gaussian noise matrix to obtain the sixth video clip; fixing the parameters of the video model, and adjusting the parameters of the third adapter model based on the difference between the sixth video clip and the sample video clip to obtain the second adapter model.

[0101] The sample text description information and the fourth video clip are used as inputs for joint training, and a sixth video clip is obtained by a processing method similar to step S2 and step S3. The parameters of the video model are fixed, and the parameters of the third adapter model are adjusted by back propagation according to the difference between the sixth video clip and the sample video clip, so as to obtain a second adapter model.

[0102] In this embodiment, without adjusting the parameters of the video model, the text description information is used as the basic condition for generating the video, and the fourth video clip is used as the auxiliary limiting condition for generating the video. The third adapter model and the video model are jointly trained to obtain the second adapter model, which helps the second adapter model to combine with the video model to generate a video clip that is highly matched with the input text description information and can control the movement trajectory of the target object, and the target box representing the position of the target object may not be displayed in each video frame of the generated video clip.

[0103] Figure 5 FIG. 1 is a structural block diagram of a video generating device in some embodiments of the present disclosure. Figure 5 As shown, the video generating device comprises: An acquisition module 100 is used to acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first target object, a pixel value of each video frame of the first video clip is a preset pixel value, and each video frame of the first video clip has a target frame of a position of the first target object; The adapter model processing module 200 is used to process the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector; The video model processing module 300 is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first target object based on the target frame.

[0104] In some embodiments of the present disclosure, the acquisition module 100 is used to acquire a third video segment based on a preset video frame rate, resolution, and video duration, wherein a pixel value of each video frame of the third video segment is a preset pixel value; The acquisition module 100 is further used to determine the position of the first target object in each video frame of the third video segment based on the preset object position information; The acquisition module 100 is further configured to determine a target frame in each video frame of the third video segment based on the position of the first target object in each video frame of the third video segment to obtain the first video segment.

[0105] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and both the input layer of the second adapter model and the input layer of the video model include multiple downsampling layers; The adapter model processing module 200 is used to process the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes the output vectors of each downsampling layer of the second adapter model; The video model processing module 300 is used to input text description information and the first Gaussian noise into the video model; at each time step, the output vectors of each downsampling layer of the two adapter models are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and the weighted calculation results of the output vectors of each downsampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0106] In some embodiments of the present disclosure, the video model processing module 300 is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model; The video model processing module 300 is also used to perform weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model; The video model processing module 300 is further used to use the first input vector as an input of a second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is an output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The video model processing module 300 is also used to use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained based on a weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0107] Figure 6 FIG. 1 is a structural block diagram of a video generation device in another embodiment of the present disclosure. Figure 6 As shown, the video generating device may further include: The model training module 400 is used to obtain a sample video segment and sample text description information, wherein the sample text description information at least includes attribute information of the second target object, and each video frame of the sample video segment has a target frame indicating the position of the second target object; The model training module 400 is also used to fix the parameters of the video model, and jointly train the first adapter model and the video model based on the sample video clips and the sample text description information to obtain a third adapter model; The model training module 400 is also used to obtain a fourth video segment having the same frame rate, resolution, duration, pixel values ​​of each video frame being preset pixel values, and target frame positions of each video frame as the sample video segment; The model training module 400 is also used to fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0108] In some embodiments of the present disclosure, the model training module 400 is used to process the sample video clip based on the first adapter model to obtain the output vectors of each downsampling layer of the first adapter model; The model training module 400 is also used to process the sample text description information, the output vectors of each intermediate layer of the first adapter model, and the second Gaussian noise matrix based on the video model to obtain a fifth video clip; The model training module 400 is also used to fix the parameters of the video model, and based on the difference between the fifth video segment and the sample video segment, adjust the parameters of the first adapter model to obtain the third adapter model.

[0109] In some embodiments of the present disclosure, the model training module 400 is used to process the fourth video clip based on the third adapter model to obtain output vectors of each downsampling layer of the third adapter model; The model training module 400 is also used to process the sample text description information, the output vectors of each intermediate layer of the third adapter model, and the third Gaussian noise matrix based on the video model to obtain a sixth video clip; The model training module 400 is also used to fix the parameters of the video model, and based on the difference between the sixth video segment and the sample video segment, adjust the parameters of the third adapter model to obtain the second adapter model.

[0110] It should be noted that the specific implementation of the video generating device of the embodiment of the present disclosure is similar to the specific implementation of the video generating method of the embodiment of the present disclosure, and the technical effect of the video generating device of the embodiment of the present disclosure is similar to the technical effect of the video generating method of the embodiment of the present disclosure. Please refer to the description of the video generating method part for details. In order to reduce redundancy, it is not repeated here.

[0111] In addition, an embodiment of the present disclosure further provides an electronic device, including: Memory for storing computer programs; The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video generation method described in any of the above embodiments of the present disclosure is implemented.

[0112] Below, reference Figure 7 To describe the electronic device according to the embodiment of the present disclosure. Figure 7 As shown, the electronic device includes one or more processors and memory.

[0113] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0114] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0115] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0116] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0117] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0118] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0119] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0120] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0121] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0122] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0123] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0124] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0126] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0127] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0128] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0129] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video generation method, characterized in that: include: Acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first target object, a pixel value of each video frame of the first video clip is a preset pixel value, and each video frame of the first video clip has a target frame of the position of the first target object; Processing the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector; Based on the video model, the text description information and the first vector are processed to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first target object.

2. The method according to claim 1, characterized in that The obtaining of the first video clip comprises: Based on a preset video frame rate, resolution, and video duration, a third video segment is acquired, wherein a pixel value of each video frame of the third video segment is the preset pixel value; Based on preset object position information, determining the position of the first target object in each video frame of the third video segment; Based on the position of the first target object in each video frame of the third video segment, a target frame in each video frame of the third video segment is generated to obtain the first video segment.

3. The method according to claim 1 or 2, characterized in that: The input layer structures of the second adapter model and the video model are the same, and both the input layer of the second adapter model and the input layer of the video model include multiple downsampling layers; The step of processing the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector includes: processing the first video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model, wherein the first vector includes the output vector of each downsampling layer of the second adapter model; The text description information and the first vector are processed based on the video model to obtain a second video clip, including: inputting the text description information and the first Gaussian noise into the video model; at each time step, weightedly calculating the output vectors of each downsampling layer of the second adapter model with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next downsampling layer corresponding to the video model to obtain the second video clip.

4. The method according to claim 3, characterized in that The step of performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model to obtain weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next sampling layer corresponding to the video model, comprises: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model; Performing weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model; Using the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth input vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained by weighted calculation based on the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

5. The method according to claim 1 or 2, characterized in that: Before the second adapter model obtained by jointly training the first adapter model and the video model is used to process the first video clip to obtain the first vector, the method further includes: Acquire a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second target object, and each video frame of the sample video clip has a target frame indicating a position of the second target object; Fixing the parameters of the video model, and jointly training the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model; Acquire a fourth video segment having the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same; The parameters of the video model are fixed, and based on the fourth video clip and the sample text description information, the third adapter model and the video model are jointly trained to obtain the second adapter model.

6. The method according to claim 5, characterized in that The method of fixing the parameters of the video model and jointly training the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model includes: Processing the sample video clip based on the first adapter model to obtain output vectors of each downsampling layer of the first adapter model; Based on the video model, the sample text description information, the output vectors of each intermediate layer of the first adapter model, and the second Gaussian noise matrix are processed to obtain a fifth video clip; The parameters of the video model are fixed, and based on the difference between the fifth video segment and the sample video segment, the parameters of the first adapter model are adjusted to obtain the third adapter model.

7. The method according to claim 5, characterized in that The method of fixing the parameters of the video model and jointly training the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model includes: Processing the fourth video clip based on the third adapter model to obtain output vectors of each downsampling layer of the third adapter model; Based on the video model, the sample text description information, the output vectors of each intermediate layer of the third adapter model, and the third Gaussian noise matrix are processed to obtain a sixth video clip; The parameters of the video model are fixed, and based on the difference between the sixth video segment and the sample video segment, the parameters of the third adapter model are adjusted to obtain the second adapter model.

8. A video generating device, characterized in that: include: an acquisition module, configured to acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first target object, a pixel value of each video frame of the first video clip is a preset pixel value, and each video frame of the first video clip has a target frame of the position of the first target object; an adapter model processing module, configured to process the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector; A video model processing module is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first target object based on the target frame.

9. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images