Video generation method, device, electronic device, storage medium and program product

Abstract

The disclosed embodiment discloses a video generation method, device, electronic device, storage medium and program product, wherein the method comprises: obtaining a first video clip and text description information; processing the first video clip based on a second adapter model obtained by jointly training a first adapter model and a video model to obtain a first vector; processing the text description information and the first vector based on the video model to obtain a second video clip. The disclosed embodiment can generate a video clip that is highly matched with the text description information and can control the motion trajectory of the object.

Description

Technical Field

[0001] The present disclosure relates to Vincent video technology, and in particular to a video generation method, device, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of video generation model technology, video generation models have shined in the field of artificial intelligence content generation (AIGC) technology.

[0003] In the related technology, natural language processing is performed on the text description information input by the user to obtain the corresponding feature information, and the feature information is processed using a video generation model to generate a video corresponding to the text description information. When generating a video, it is a very important requirement to control the motion trajectory of objects in the video. For example, in the post-production of a movie or TV series, this technology can be used to generate or modify specific scenes, such as adding or adjusting the motion of an object, without reshooting. This can save a lot of cost and time, while improving production efficiency and quality.

[0004] How to generate videos that are highly consistent with text description information and can control the motion trajectory of objects is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present disclosure provide a video generation method, device, electronic device, storage medium and program product to solve the above problems.

[0006] According to a first aspect of the present disclosure, a video generation method is provided, including:

[0007] Acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first object, a pixel value of each video frame of the first video clip is a preset pixel value, at least two video frames of the first video clip have target frames indicating positions of the first object in the at least two video frames respectively, and the first video clip includes a video frame without the target frame;

[0008] Processing the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector;

[0009] Based on the video model, the text description information and the first vector are processed to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first object.

[0010] In some embodiments of the present disclosure, obtaining the first video clip includes:

[0011] Based on a preset video frame rate, resolution, and video duration, a third video segment is acquired, wherein a pixel value of each video frame of the third video segment is the preset pixel value;

[0012] Based on the motion trajectory information of the first object, determining the position of the first object in at least two target video frames of the third video segment;

[0013] Based on the position of the first object in at least two target video frames of the third video segment, a target frame is generated in the at least two target video frames to obtain the first video segment, wherein the at least two video frames correspond one-to-one to the at least two target video frames.

[0014] In some embodiments of the present disclosure, determining the position of the first object in at least two target video frames of the third video segment based on the motion trajectory information of the first object includes:

[0015] Determining the position of the first object at multiple time points based on the motion trajectory information of the first object;

[0016] Determining the at least two target video frames based on the multiple time points and the time points of the video frames of the third video clip;

[0017] Based on the position of the first object at each of the multiple time points, the position of the first object in each of the at least two target video frames is determined.

[0018] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the second adapter model and the input layer of the video model both include multiple downsampling layers;

[0019] The step of processing the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector includes: processing the first video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model, wherein the first vector includes the output vector of each downsampling layer of the second adapter model;

[0020] The text description information and the first vector are processed based on the video model to obtain a second video clip, including: inputting the text description information and the first Gaussian noise into the video model; at each time step, weightedly calculating the output vectors of each downsampling layer of the second adapter model with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next downsampling layer corresponding to the video model to obtain the second video clip.

[0021] In some embodiments of the present disclosure, the step of performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model to obtain weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next sampling layer corresponding to the video model includes:

[0022] Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model;

[0023] Performing weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model;

[0024] Using the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model;

[0025] The Nth input vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained by weighted calculation based on the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0026] In some embodiments of the present disclosure, before the first video clip is processed based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain the first vector, the method further includes:

[0027] Acquire a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second object, a pixel value of each video frame of the sample video clip is a preset pixel value, each of at least two video frames of the sample video clip has a target frame indicating a position of the second object, and the sample video clip includes a video frame without the target frame;

[0028] Fixing the parameters of the video model, and jointly training the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model;

[0029] Acquire a fourth video segment having the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same;

[0030] The parameters of the video model are fixed, and based on the fourth video clip and the sample text description information, the third adapter model and the video model are jointly trained to obtain the second adapter model.

[0031] According to a second aspect of the embodiments of the present disclosure, a video generating device is provided, including:

[0032] an acquisition module, configured to acquire a first video segment and text description information, wherein the text description information at least includes attribute information of a first object, a pixel value of each video frame of the first video segment is a preset pixel value, at least two video frames of the first video segment have target frames indicating positions of the first object in the at least two video frames respectively, and the first video segment includes a video frame without the target frame;

[0033] an adapter model processing module, configured to process the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector;

[0034] A video model processing module is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes the motion trajectory of the first object.

[0035] In some embodiments of the present disclosure, the acquisition module is used to acquire a third video segment based on a preset video frame rate, resolution, and video duration, wherein a pixel value of each video frame of the third video segment is the preset pixel value;

[0036] The acquisition module is further used to determine the position of the first object in at least two target video frames of the third video segment based on the motion trajectory information of the first object;

[0037] The acquisition module is also used to generate a target frame in at least two target video frames of the third video segment based on the position of the first object in the at least two target video frames to obtain the first video segment, wherein the at least two video frames correspond one-to-one to the at least two target video frames.

[0038] In some embodiments of the present disclosure, the acquisition module is used to determine the position of the first object at multiple time points based on the motion trajectory information of the first object;

[0039] The acquisition module is further used to determine the at least two target video frames based on the multiple time points and the time points of each video frame of the third video segment;

[0040] The acquisition module is further configured to determine a position of the first object in each of the at least two target video frames based on the position of the first object at each of the multiple time points.

[0041] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and the input layer of the second adapter model and the input layer of the video model both include multiple downsampling layers;

[0042] The adapter model processing module is used to process the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes output vectors of each downsampling layer of the second adapter model;

[0043] The video model processing module is used to input the text description information and the first Gaussian noise into the video model; at each time step, the output vectors of each downsampling layer of the two adapter models are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and the weighted calculation results of the output vectors of each downsampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0044] In some embodiments of the present disclosure, the video model processing module is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model;

[0045] The video model processing module is also used to perform weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model;

[0046] The video model processing module is further used to use the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model;

[0047] The video model processing module is also used to use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained based on the weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0048] In some embodiments of the present disclosure, the device further includes:

[0049] A model training module, configured to obtain a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second object, a pixel value of each video frame of the sample video clip is a preset pixel value, each of at least two video frames of the sample video clip has a target frame indicating a position of the second object, and the sample video clip includes a video frame without the target frame;

[0050] The model training module is further used to fix the parameters of the video model, and to jointly train the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model;

[0051] The model training module is also used to obtain a fourth video segment with the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same;

[0052] The model training module is also used to fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0053] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0054] a memory for storing a computer program product;

[0055] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0056] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0057] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the method described in the first aspect.

[0058] The video generation method, device, electronic device, storage medium and program product of the embodiments of the present disclosure use the text description information as the basic condition for generating the video, and use the first video segment with the partial motion trajectory of the object as the auxiliary limiting condition for generating the video. The adapter model is used in combination with the video model to generate a video segment that is highly matched with the text description information and can control the motion trajectory of the object.

[0059] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0061] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0062] Figure 1 A schematic diagram of a process of generating a video in an embodiment of the present disclosure;

[0063] Figure 2 Schematic diagram of the process of step S1 in some embodiments of the present disclosure;

[0064] Figure 3 is a partial flow chart of step S1-2 in some embodiments of the present disclosure;

[0065] Figure 4 is a flowchart of step S3 in some embodiments of the present disclosure;

[0066] Figure 5 is a flowchart of step S0 in some embodiments of the present disclosure;

[0067] Figure 6 is a structural block diagram of a video generating device in some embodiments of the present disclosure;

[0068] Figure 7 is a structural block diagram of a video generation device in another embodiment of the present disclosure;

[0069] Figure 8 The structure block diagram of an electronic device in one embodiment of the present disclosure is shown in FIG. DETAILED DESCRIPTION

[0070] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0071] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0072] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0073] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0074] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0075] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0076] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0077] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0078] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0079] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0080] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0081] Figure 1 FIG. 1 is a flow chart of a video generation method in one embodiment of the present disclosure. Figure 1 As shown, the video generation method comprises the following steps:

[0082] S1: Obtain a first video clip and text description information.

[0083] A first video clip and text description information input by a user are received.

[0084] The pixel value of each video frame of the first video segment is a preset pixel value. The preset pixel value may be a pixel value corresponding to a preset color. Exemplarily, all pixel values ​​in each video frame of the first video segment are pixel values ​​RGB (0, 0, 0) corresponding to black.

[0085] The text description information at least includes the attribute information of the first target object, for example, it may include the type of the first target object and other attribute information of the first target object. The type of the first target object may include: a person type, an animal type, or other object type (for example, an object that needs to be moved in a movie or game scene). Other attribute information of the first target object may include color information, structure information (for example, when the first target object is a person or an animal, it may include the body proportions of the first target object) and other information of the first target object.

[0086] At least two video frames of the first video clip have a target frame indicating the position of the first target object in the video frame, and the first video clip includes a video frame without a target frame (i.e., the position of the first object in the video frame cannot be known). When the first object includes multiple objects, each video frame of the first video clip may include multiple target frames, and the multiple objects correspond to each target frame. The target frames representing the same object in each video frame of the first video clip may have the same or different sizes. For example, when the same object in each video frame of the first video clip moves at the same visual distance, the size of the target frame corresponding to the object in each video frame may be the same; when the same object in each video frame of the first video clip moves at different visual distances, the size of the target frame corresponding to the object in each video frame may be different.

[0087] For the remaining video frames except the at least two video frames in the first video segment: no processing may be performed, or all pixel values ​​of each of the remaining video frames may be adjusted to a specified pixel value, such as RGB (255, 255, 255).

[0088] S2: Based on the second adapter model obtained by jointly training the first adapter model and the video model, the first video clip is processed to obtain a first vector.

[0089] Among them, the first adapter model is an untrained adapter model, which can also be called the initial adapter model. The second adapter model is an adapter model obtained through joint training, which can also be called the final adapter model. Optionally, in the process of obtaining the second adapter model based on the joint training of the first adapter model and the video model, the first adapter model and the video model are first obtained, wherein the network layer structure of the first adapter model is the same as the network structure of the input layer of the video model, and each network layer of the first adapter model is mapped one-to-one with the network layer of the input layer of the video model, that is, the number of network layers of the first adapter model is the same as the number of network layers of the input layer of the video model, and each network layer of the first adapter model has the same network parameters as the input layer corresponding to the video model. Secondly, the network parameters of each network layer of the first adapter model are adjusted through the loss function of the video model to obtain the second adapter model.

[0090] In an embodiment of the present invention, by obtaining a second adapter model based on joint training of the first adapter model and the video model, it is not necessary to directly adjust the parameters of the video model (the magnitude of the parameters of the video model is generally large, such as billions or tens of billions), which can not only reduce the number of parameters required for fine-tuning, but also reduce the demand for computing resources during training, thereby speeding up the training speed. At the same time, when the video model performs a specific task, the output vector of the input layer of the video model is fine-tuned through the trained second adapter model, thereby achieving the specific task that can be completed efficiently and accurately without large-scale overall training of the video model. It can be seen that the second adapter model is obtained by joint training of the first adapter model and the video model, and it is realized that based on AIGC technology, the second video clip with the target object position change information can be quickly generated through the first video clip and text description information, and the versatility of the video model will not be affected.

[0091] The video model may be a neural network model based on a diffusion model, and is used to generate a corresponding video according to input data information, wherein the data information includes text information, image information, or voice information, etc., wherein the voice information may be converted into text information through voice information recognition and input into the video model. The video model may be pre-trained, for example, it may be trained according to input samples including input text descriptions and input videos.

[0092] Furthermore, the diffusion model is a generative model based on a deep learning algorithm, which generates high-quality content through a gradual process of adding noise and denoising, performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generating tasks. In the disclosed embodiment, the diffusion model is a pre-trained model. After the conditional information is mapped to the latent space expression by the encoder, under the guidance of the latent space expression of the conditional information, the random noise is denoised for T time steps, that is, downsampling and upsampling are performed at each time step to obtain the final latent space expression of the second video clip, and then the latent space expression of the target video is further mapped to the pixel space by the decoder to obtain the final second video clip. Among them, the latent space expression of the conditional information is the latent space expression obtained after encoding the conditional information using the encoder. The encoder is responsible for mapping the data from the original data space to the latent space (LatentSpace) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the video from this latent space expression and map the generation result of the diffusion model from the latent space to the pixel space. In some implementations, the encoder and decoder used in the diffusion model of the disclosed embodiments may both be variational autoencoders (VAEs). The latent space of VAEs is continuous and suitable for generating continuous data, i.e., the generated target videos or images are correlated with each other.

[0093] The adapter model is a model based on neural network model technology, which is used to fine-tune the parameters of the generated video based on the video model according to the video generation task. It achieves this goal by adding small, trainable modules without retraining the video model. It can not only quickly and cost-effectively meet the video generation needs of the corresponding application scenarios, but also does not affect the versatility of the video model.

[0094] The first video segment is processed using the second adapter model, and a first vector corresponding to the first video segment is output.

[0095] S3: Based on the video model, the text description information and the first vector are processed to obtain a second video clip.

[0096] The second video segment includes a motion trajectory of the first object. The motion trajectory of the first object is determined based on a change in a position of a target frame in at least two video frames of the first video segment, and a position of the first target object in a corresponding video frame of the first video segment is the same as a position of the first target object in a corresponding video frame of the second video segment.

[0097] The text description information and the first vector are used as inputs of the video model, and a second video clip corresponding to the text description information is output. Each video frame of the second video clip has a first target object, and the position of the first target object in each video frame of the second video clip corresponds to the position of the target frame in each video frame of the first video clip. In other words, by simply expressing the motion trajectory of the object to be moved in the first video clip, a second video clip with the same duration and consistent video content expression as the first video clip can be obtained based on the second adapter model and the video model. In addition, by first obtaining the pixel value of each video frame as a preset pixel value and at least two video frames having a target frame representing the position of the first object in the at least two video frames, it is not only convenient for the second adapter model and the video model to understand the input content, but also based on the sparse distribution of the target frame in the first video clip (that is, not all video frames of the first video clip have a target frame), the application of the scheme disclosed in the present invention is more universal.

[0098] In this embodiment, since the adapter model cooperates with the video model to generate a video clip that is highly matched with the text description information, the text description information is used as the basic condition for generating the video, and the first video clip with the partial motion trajectory of the object is used as the auxiliary limiting condition for generating the video. The adapter model is used in combination with the video model to generate a video clip that is highly matched with the text description information and can control the motion trajectory of the object.

[0099] Figure 2 is a flow chart of step S1 in some embodiments of the present disclosure. Figure 2 As shown, step S1 may include the following steps:

[0100] S1-1: Based on a preset video frame rate, resolution, and video duration, a third video segment is obtained.

[0101] For example, if the preset video frame rate is B frames / second, the resolution is L×W, and the video duration is A seconds, then the duration of the third video segment is A seconds, the frame rate is B frames per second, the resolution is L×W, and all pixel values ​​of each video frame are preset pixel values ​​(e.g., RGB (0,0,0)). A, B, L, and W are all integers greater than 1.

[0102] S1-2: Determine a position of the first object in at least two target video frames of a third video segment based on the motion trajectory information of the first object.

[0103] The positions of the first object at multiple time points are extracted from the motion trajectory information of the first object, and the multiple time points are mapped one-to-one with at least two video frames of the third video segment to obtain the positions of the first object in the at least two video frames of the third video segment.

[0104] S1-3: Based on the position of the first object in at least two target video frames of the third video segment, a target frame is generated in the at least two target video frames to obtain the first video segment.

[0105] Among them, at least two video frames with target frames in the first target video correspond to at least two target video frames in the third video segment. When the position of the first object in the at least two target video frames of the third video segment includes the pixel coordinates of the center point of the object, according to the pixel coordinates of the center point of the first object in the at least two target video frames of the third video segment, combined with the size information of the first object in the pixel coordinate system, a target frame representing the position of the first object is generated in the at least two target video frames.

[0106] When the position of the first object in the at least two target video frames of the third video segment includes the pixel coordinate range of the first object, a target frame surrounding the pixel coordinate range of the first object is generated in the at least two target video frames of the third video segment.

[0107] In this embodiment, the third video segment is determined according to the preset video frame rate, resolution and video duration, and combined with the motion trajectory information of the first object, a first video segment in which some video frames have a target box representing the position of the first target object can be quickly obtained.

[0108] Figure 3 is a partial flow chart of step S1-2 in some embodiments of the present disclosure. Figure 3 As shown, step S1-2 may include the following steps:

[0109] S1-2-1: Determine positions of the first object at multiple time points based on motion trajectory information of the first object.

[0110] The position of the first object at multiple time points can be extracted from the motion trajectory information of the first object. The position of the first object can be represented by the coordinates of the first object in the world coordinate system.

[0111] S1-2-2: Determine at least two target video frames based on multiple time points and the time points of each video frame of the third video clip.

[0112] In an example of the present disclosure, the motion trajectory information of the first object includes at least: the position z1 of the first object at time point t1, the position z2 of the first object at time point t2, the position z3 of the first object at time point t3, and the position z4 of the first object at time point t4. The time point t1, the time point t2, the time point t3, and the time point t4 are mapped one by one to the target video frame p1, the target video frame p2, the target video frame p3, and the target video frame p4 of the third video clip, respectively, to obtain a mapping relationship between the time points and the target video frames.

[0113] S1-2-3: Determine a position of the first object in each target video frame of at least two target video frames based on the position of the first object at each time point in a plurality of time points.

[0114] Following the example of step S1-2-2, a coordinate transformation relationship is constructed between the space where the motion trajectory of the first object is located and the pixel coordinate system of the video frame of the third video clip, and then the positions of position z1, position z2, position z3 and position z4 in the target video frame p1, target video frame p2, target video frame p3 and target video frame p4 are determined based on the mapping relationship and coordinate transformation relationship between the time point and the target video frame.

[0115] In this embodiment, the position of the first object at multiple time points is extracted from the motion trajectory information of the first object, and the position of the first object in each target video frame of at least two target video frames can be quickly and accurately determined by combining the relationship between the time points in the motion trajectory of the first object and the time points of the video frames.

[0116] In some embodiments of the present disclosure, the input layer structures of the second adaptor model and the video model are the same. The second adaptability model input layers and the video model input layers have the same number and the same network structure, and feature vectors of the same dimension can be extracted for the input information of the model through the same number and the same network structure of the input layers.

[0117] Accordingly, step S2 includes: processing the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes the output vectors of each downsampling layer of the second adapter model. For example, when the second adapter model includes N+1 downsampling layers and N is an integer greater than 1, the output vectors of the 1st downsampling layer to the output vectors of the N+1th downsampling layer are stored in the order from the 1st layer to the N+1th layer, and are recorded as vectors VA1 to VA N+1 .

[0118] Correspondingly, step S3 includes: inputting the text description information and the first Gaussian noise into the video model; at each time step, performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model, respectively, to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next downsampling layer corresponding to the video model to obtain the second video clip. Among them, the first Gaussian noise matrix can be a randomly generated Gaussian distribution matrix used to add noise to the video model. The text description information, the output vectors of each downsampling layer of the second adapter model, and the first Gaussian noise matrix are used as the input of the video model, and denoising is performed step by step through each downsampling layer of the video model, and finally the second video clip is output.

[0119] In each denoising process, the output vectors of each downsampling layer of the first adapter model are weighted with the output vectors of the corresponding downsampling layers of the video model to obtain the output vector of the video model input layer. The video model includes an input layer and an output layer, the input layer of the video model includes multiple downsampling layers, the multiple downsampling layers of the video model input layer are mapped one-to-one with the multiple downsampling layers of the first adapter model, and the structure of each group of corresponding downsampling layers is the same. The output vector of each downsampling layer of the first adapter model is used for weighted calculation with the output vector of the mapped downsampling layer of the video model to serve as the input vector of the next downsampling layer of the video model.

[0120] It can be understood that the input layer and output layer of the above video model perform one round of calculation for noise prediction of one time step, that is, denoising processing of one time step. The video model can obtain the final target video by performing T rounds of iterative denoising. After the output layer of the video model outputs the output vector of the first round, it is input into the input layer of the video model again as the initial input vector of the second round. Among them, T is an integer greater than 1. After the denoising is completed after T rounds of loop execution according to the above process, the final output vector of the output layer of the video model is decoded by the decoder, and it is mapped from the latent space to the pixel space to obtain the second video clip.

[0121] In this embodiment, since the input layer structures of the second adapter model and the video model are the same, feature vectors of the same dimension can be extracted for the input information of each model, and the video model performs step-by-step denoising on the first Gaussian noise matrix, thereby obtaining a second video clip that has a high degree of match with the text description information.

[0122] Figure 4 FIG. 1 is a flow chart of step S3 in some embodiments of the present disclosure. Figure 4 As shown, step S3 may further include the following steps:

[0123] S3-1: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector.

[0124] By extracting features from the text description information and the first Gaussian noise matrix through the input layer of the video model, a vector of a predetermined dimension can be obtained as an input vector of the first downsampling layer of the video model. After processing the input vector through the first downsampling layer of the video model, an output vector of the first downsampling layer of the video model can be obtained, which is recorded as the first video output vector and represented by vector VB1.

[0125] S3-2: Perform weighted calculation on the first video output vector and the first adapter output vector to obtain a first input vector.

[0126] From the output vectors of each downsampling layer of the second adapter model, the output vector of the first downsampling layer of the second adapter model (ie, vector VA1) is extracted and recorded as the first adapter output vector. The vector VA1 and the vector VB1 are added to obtain the first input vector.

[0127] S3-3: Use the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector.

[0128] The first input vector is used as the input of the second downsampling layer of the video model to obtain the output vector of the second downsampling layer of the video model, which is recorded as the second video output vector VB2. The first downsampling layer of the video model is adjacent to the second downsampling layer of the video model, that is, the second downsampling layer of the video model is the next downsampling layer adjacent to the first downsampling layer of the video model.

[0129] S3-4: Use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model.

[0130] In a manner similar to step S3-3, the 2nd downsampling layer to the N+1th downsampling layer of the video model are processed to obtain the output vectors of the 2nd downsampling layer to the Nth downsampling layer of the video model.

[0131] In this embodiment, since the input layer structures of the second adapter model and the video model are the same, and the dimensions of the corresponding downsampling layer input vectors of the second adapter model and the video model are the same, the first Gaussian noise matrix is ​​input into the video model, and the output vectors of each downsampling layer of the first adapter model and the video model are weighted to obtain a vector as the input of the next downsampling layer of the video model, thereby achieving denoising within a single time step, and finally obtaining a second video clip that highly matches the guidance video generation conditions through step-by-step denoising.

[0132] In some embodiments of the present disclosure, before step S2, the following steps may also be included:

[0133] Step S0: Based on the sample video clips and the sample text description information, the first adapter model and the video model are jointly trained to obtain a second adapter model.

[0134] Figure 5 FIG. 1 is a flow chart of step S0 in some embodiments of the present disclosure. Figure 5 As shown, step S0 may include the following steps:

[0135] S0-1: Obtain sample video clips and sample text description information.

[0136] The sample text description information includes at least attribute information of the second object, for example, it may include the type of the second target object and other attribute information of the second target object. The pixel value of each video frame of the sample video clip is a preset pixel value. Each video frame of at least two video frames of the sample video clip has a target frame indicating the position of the second object, and the sample video clip includes at least a video frame without a target frame. The second object may include: a person, an animal or other object. The second object may include only one object or multiple objects. When the second object includes multiple objects, at least two video frames of the sample video clip may include multiple target frames, and the multiple objects correspond to the multiple target frames.

[0137] S0-2: Fix the parameters of the video model, and jointly train the first adapter model and the video model based on the sample video clips and the sample text description information to obtain the third adapter model.

[0138] When the first adapter model and the video model are jointly trained, the parameters of the video model are fixed, sample video clips and sample text description information are used as inputs for the joint training, the video model is used to output the video clips, and the parameters of the first adapter model are adjusted according to the differences between the sample video clips and the video clips output by the video model to obtain a third adapter model.

[0139] The third adapter model combined with the video model can generate a corresponding video segment according to the input text description information and the video segment, but the generated video segment has a target box indicating the position of the target object.

[0140] S0-3: Obtain a fourth video segment with the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being a preset pixel value and the target frame position of each video frame being the same.

[0141] According to the frame rate, resolution and duration of the sample video clip, a video clip having the same frame rate, resolution and duration as the sample video clip can be captured from a network video resource or a local video resource library.

[0142] After adjusting all pixel values ​​of each video frame of the intercepted video clip to preset pixel values, a target frame with the same target frame position is generated in each video frame of the intercepted video clip according to the position of the target frame in each video frame of the sample video clip, thereby obtaining a fourth video clip.

[0143] S0-4: Fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0144] When the third adapter model and the video model are jointly trained, the parameters of the video model are fixed, the fourth video clip and the sample text description information are used as inputs for the joint training, the video model is used to output the video clip, and the parameters of the third adapter model are adjusted according to the difference between the sample video clip and the video clip output by the video model to obtain the second adapter model.

[0145] The second adapter model combined with the video model can generate a video segment that highly matches the input text information based on the input text description information and the video segment in which each video frame has a target box representing the position of the target object.

[0146] In this embodiment, since the model accuracy obtained by directly jointly training the first adapter model and the video model using video clips with sample text description information and target pixel values ​​of each video frame as preset pixel values ​​is insufficient, the first adapter model and the video model are jointly trained in a first stage using sample video clips and sample text description information to obtain a third adapter model, and the third adapter model and the video model are jointly trained in a second stage using video clips with sample text description information and target pixel values ​​of each video frame as preset pixel values ​​to obtain a second adapter model, which helps the second adapter model to cooperate with the video model to generate video clips that are highly matched with the input text description information.

[0147] Figure 6 FIG. 1 is a structural block diagram of a video generating device in some embodiments of the present disclosure. Figure 6 As shown, the video generating device comprises:

[0148] The acquisition module 100 is used to acquire a first video clip and text description information, wherein the text description information at least includes attribute information of the first object, the pixel value of each video frame of the first video clip is a preset pixel value, at least two video frames of the first video clip have target frames indicating the positions of the first object in the at least two video frames respectively, and the first video clip includes a video frame without a target frame;

[0149] The adapter model processing module 200 is used to process the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector;

[0150] The video model processing module 300 is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes the motion trajectory of the first object.

[0151] In some embodiments of the present disclosure, the acquisition module 100 is used to acquire a third video segment based on a preset video frame rate, resolution, and video duration, wherein a pixel value of each video frame of the third video segment is a preset pixel value;

[0152] The acquisition module 100 is further used to determine the position of the first object in at least two target video frames of the third video segment based on the motion trajectory information of the first object;

[0153] The acquisition module 100 is also used to generate a target frame in at least two target video frames of the third video segment based on the position of the first object in at least two target video frames to obtain the first video segment, wherein the at least two video frames correspond one-to-one to the at least two target video frames.

[0154] In some embodiments of the present disclosure, the acquisition module 100 is used to determine the position of the first object at multiple time points based on the motion trajectory information of the first object;

[0155] The acquisition module 100 is further used to determine at least two target video frames based on the multiple time points and the time points of each video frame of the third video segment;

[0156] The acquisition module 100 is further configured to determine a position of the first object in each target video frame of at least two target video frames based on the position of the first object at each time point in a plurality of time points.

[0157] In some embodiments of the present disclosure, the input layer structures of the second adapter model and the video model are the same, and both the input layer of the second adapter model and the input layer of the video model include multiple downsampling layers;

[0158] The adapter model processing module 200 is used to process the first video clip based on the second adapter model to obtain output vectors of each downsampling layer of the second adapter model, wherein the first vector includes the output vectors of each downsampling layer of the second adapter model;

[0159] The video model processing module 200 is used to input text description information and the first Gaussian noise into the video model; at each time step, the output vectors of each downsampling layer of the two adapter models are weightedly calculated with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and the weighted calculation results of the output vectors of each downsampling layer are used as the input of the next sampling layer corresponding to the video model to obtain the second video clip.

[0160] In some embodiments of the present disclosure, the video model processing module 300 is used to process the text description information and the first Gaussian noise matrix based on the video model to obtain a first video output vector, wherein the first video output vector is an output vector of the first downsampling layer of the video model;

[0161] The video model processing module 300 is also used to perform weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model;

[0162] The video model processing module 300 is further used to use the first input vector as an input of a second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is an output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model;

[0163] The video model processing module 300 is also used to use the Nth input vector as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained based on a weighted calculation of the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

[0164] Figure 7 FIG. 1 is a structural block diagram of a video generation device in another embodiment of the present disclosure. Figure 7 As shown, the video generating device may further include:

[0165] A model training module 400 is used to obtain a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second object, a pixel value of each video frame of the sample video clip is a preset pixel value, each of at least two video frames of the sample video clip has a target frame indicating a position of the second object, and the sample video clip includes a video frame without a target frame;

[0166] The model training module 400 is also used to fix the parameters of the video model, and to jointly train the first adapter model and the video model based on the sample video clips and the sample text description information to obtain a third adapter model;

[0167] The model training module 400 is also used to obtain a fourth video segment having the same frame rate, resolution, duration, pixel values ​​of each video frame being preset pixel values, and target frame positions of each video frame as the sample video segment;

[0168] The model training module 400 is also used to fix the parameters of the video model, and jointly train the third adapter model and the video model based on the fourth video clip and the sample text description information to obtain the second adapter model.

[0169] It should be noted that the specific implementation of the video generating device of the embodiment of the present disclosure is similar to the specific implementation of the video generating method of the embodiment of the present disclosure, and the technical effect of the video generating device of the embodiment of the present disclosure is similar to the technical effect of the video generating method of the embodiment of the present disclosure. Please refer to the description of the video generating method part for details. In order to reduce redundancy, it is not repeated here.

[0170] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0171] Memory for storing computer programs;

[0172] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the video generation method described in any of the above embodiments of the present disclosure is implemented.

[0173] Below, reference Figure 8 To describe the electronic device according to the embodiment of the present disclosure. Figure 8 As shown, the electronic device includes one or more processors and memory.

[0174] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0175] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the video generation method of each embodiment of the present disclosure described above and / or other desired functions.

[0176] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0177] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0178] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0179] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0180] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0181] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the video generation method according to various embodiments of the present disclosure described in the above part of this specification.

[0183] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0184] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0185] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0187] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0188] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0189] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0190] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video generation method, characterized in that: include: Acquire a first video clip and text description information, wherein the text description information at least includes attribute information of a first object, a pixel value of each video frame of the first video clip is a preset pixel value, at least two video frames of the first video clip have target frames indicating positions of the first object in the at least two video frames respectively, and the first video clip includes a video frame without the target frame; Based on the second adapter model obtained by jointly training the first adapter model and the video model, the first video clip is processed to obtain a first vector, wherein the joint training of the first adapter model and the video model includes: obtaining the first adapter model and the video model, the network layer structure of the first adapter model is the same as the network structure of the input layer of the video model, each network layer of the first adapter model is mapped one-to-one with the network layer of the input layer of the video model, and each network layer of the first adapter model has the same network parameters as the input layer corresponding to the video model; adjusting the network parameters of each network layer of the first adapter model through the loss function of the video model to obtain the second adapter model; Based on the video model, the text description information and the first vector are processed to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first object, and the motion trajectory of the first object is determined based on a position change of a target box in at least two video frames of the first video clip.

2. The method according to claim 1, characterized in that The obtaining of the first video clip comprises: Based on a preset video frame rate, resolution, and video duration, a third video segment is acquired, wherein a pixel value of each video frame of the third video segment is the preset pixel value; Based on the motion trajectory information of the first object, determining the position of the first object in at least two target video frames of the third video segment; Based on the position of the first object in at least two target video frames of the third video segment, a target frame is generated in the at least two target video frames to obtain the first video segment, wherein the at least two video frames correspond one-to-one to the at least two target video frames.

3. The method according to claim 2, characterized in that The determining, based on the motion trajectory information of the first object, a position of the first object in at least two target video frames of the third video segment includes: Determining the position of the first object at multiple time points based on the motion trajectory information of the first object; Determining the at least two target video frames based on the multiple time points and the time points of the video frames of the third video clip; Based on the position of the first object at each of the multiple time points, the position of the first object in each of the at least two target video frames is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The input layer structures of the second adapter model and the video model are the same, and both the input layer of the second adapter model and the input layer of the video model include multiple downsampling layers; The step of processing the first video clip based on the second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector includes: processing the first video clip based on the second adapter model to obtain an output vector of each downsampling layer of the second adapter model, wherein the first vector includes the output vector of each downsampling layer of the second adapter model; The text description information and the first vector are processed based on the video model to obtain a second video clip, including: inputting the text description information and the first Gaussian noise into the video model; at each time step, weightedly calculating the output vectors of each downsampling layer of the second adapter model with the output vectors of the downsampling layer corresponding to the video model to obtain the weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next downsampling layer corresponding to the video model to obtain the second video clip.

5. The method according to claim 4, characterized in that The step of performing weighted calculation on the output vectors of each downsampling layer of the second adapter model and the output vectors of the downsampling layer corresponding to the video model to obtain weighted calculation results of the output vectors of each downsampling layer, and using the weighted calculation results of the output vectors of each downsampling layer as the input of the next sampling layer corresponding to the video model, comprises: Based on the video model, the text description information and the first Gaussian noise matrix are processed to obtain a first video output vector, wherein the first video output vector is an output vector of a first downsampling layer of the video model; Performing weighted calculation on the first video output vector and the output vector of the first adapter corresponding layer to obtain a first input vector, wherein the output vector of the first adapter corresponding layer of the first video output vector is the output vector of the first downsampling layer of the first adapter model; Using the first input vector as the input of the second downsampling layer of the video model to obtain a second video output vector, wherein the second video output vector is the output vector of the second downsampling layer of the video model, and the first downsampling layer of the video model is adjacent to the second downsampling layer of the video model; The Nth input vector is used as the input of the N+1th downsampling layer of the video model to obtain the output vector of the N+1th downsampling layer of the video model, wherein the Nth input vector is obtained by weighted calculation based on the output vector of the Nth downsampling layer of the video model and the output vector of the Nth downsampling layer of the second adapter model, and N is an integer greater than 1.

6. The method according to any one of claims 1 to 3, characterized in that: Before the second adapter model obtained by jointly training the first adapter model and the video model is used to process the first video clip to obtain the first vector, the method further includes: Acquire a sample video clip and sample text description information, wherein the sample text description information at least includes attribute information of a second object, a pixel value of each video frame of the sample video clip is a preset pixel value, each video frame of at least two video frames of the sample video clip has a target frame indicating a position of the second object, and the sample video clip at least includes a video frame without the target frame; Fixing the parameters of the video model, and jointly training the first adapter model and the video model based on the sample video clip and the sample text description information to obtain a third adapter model; Acquire a fourth video segment having the same frame rate, resolution, and duration as the sample video segment, with the pixel value of each video frame being the preset pixel value and the target frame position of each video frame being the same; The parameters of the video model are fixed, and based on the fourth video clip and the sample text description information, the third adapter model and the video model are jointly trained to obtain the second adapter model.

7. A video generating device, characterized in that: include: an acquisition module, configured to acquire a first video segment and text description information, wherein the text description information at least includes attribute information of a first object, a pixel value of each video frame of the first video segment is a preset pixel value, at least two video frames of the first video segment have target frames indicating positions of the first object in the at least two video frames respectively, and the first video segment includes a video frame without the target frame; an adapter model processing module, for processing the first video clip based on a second adapter model obtained by jointly training the first adapter model and the video model to obtain a first vector, wherein the joint training of the first adapter model and the video model comprises: obtaining the first adapter model and the video model, the network layer structure of the first adapter model is the same as the network structure of the input layer of the video model, each network layer of the first adapter model is mapped one-to-one with the network layer of the input layer of the video model, and each network layer of the first adapter model has the same network parameters as the input layer corresponding to the video model; adjusting the network parameters of each network layer of the first adapter model through the loss function of the video model to obtain the second adapter model; A video model processing module is used to process the text description information and the first vector based on the video model to obtain a second video clip, wherein the second video clip includes a motion trajectory of the first object, and the motion trajectory of the first object is determined based on a target box position change in at least two video frames of the first video clip.

8. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the method according to any one of claims 1 to 6.

Images