Training of a video processing model and video processing method, apparatus, device, and medium

Abstract

The application discloses a video processing model training method and a video processing method, device, equipment and medium, the video processing model training method comprises the following steps: obtaining a video sample set matched with multiple video task types; generating a latent variable corresponding to each video sample, and determining a task mask of each sample in the latent variable scale according to each video task type, and generating a spliced latent variable corresponding to each video sample; and pre-training a diffusion model pre-constructed by adding noise and denoising according to the spliced latent variable of each video sample, to obtain a unified video processing model suitable for multiple video task types. By using the mask corresponding to the task type in combination with the latent variable, the diffusion model is pre-trained, multi-task collaborative pre-training is realized, the calculation and storage resource overhead is reduced, the training and deployment efficiency is improved, and the generalization ability and adaptability of the video processing model in the multi-task scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video generation, and in particular to a video processing model training method and a video processing method, device, equipment and medium. BACKGROUND

[0002] With the remarkable achievements of diffusion models in the field of image generation, video, as a sequence of images in time series, naturally extends the success of this technology to the field of video generation.

[0003] In the prior art, a two-stage method is mainly used in the field of video generation: first, a text-to-video generation model is trained, and then fine-tuning is performed on the basis to adapt to specific needs. Due to the need to train, deploy and infer two models, double consumption of computing and storage resources is caused, which increases the deployment cost and reduces the deployment efficiency. By fine-tuning the model for a specific task, the model tends to be specific to the task, and there is a lack of sharing and unification between models trained for each specific task, which reduces the generalization ability and adaptability of the video processing model in a multi-task scenario. SUMMARY

[0004] The present application provides a video processing model training method and a video processing method, device, equipment and medium to solve the problems of large model training computing and storage resource consumption, low model training and deployment efficiency, low generalization ability and poor adaptability of the model in a multi-task scenario.

[0005] According to an aspect of an embodiment of the present application, a video processing model training method is provided, comprising:

[0006] obtaining a set of video samples matched with a plurality of video task types; wherein one video sample corresponds to one video task type;

[0007] generating a latent variable corresponding to each video sample respectively, and determining a task mask of each video sample in the dimension of the latent variable to which the video sample belongs according to the video task type of each video sample;

[0008] generating a spliced latent variable corresponding to each video sample respectively according to the latent variable and the task mask of each video sample;

[0009] pre-training a diffusion model pre-constructed by adding noise and denoising according to the spliced latent variables of the video samples, to obtain a unified video processing model adapted to the plurality of video task types.

[0010] According to another aspect of an embodiment of the present application, a video processing method is also provided, comprising:

[0011] obtain a current video processing task, and generate a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask;

[0012] generate a current spliced latent variable according to the current latent variable and the current task mask;

[0013] input the current spliced latent variable into a unified video processing model trained by a method according to any one of embodiments of the present application to obtain a target processing video matched with the current video processing task.

[0014] According to another aspect of an embodiment of the present application, a training device of a video processing model is also provided, comprising:

[0015] a sample set obtaining module configured to obtain a video sample set matched with a plurality of video task types; wherein one video sample corresponds to one video task type;

[0016] a task mask determining module configured to generate a latent variable corresponding to each video sample respectively, and determine a task mask of each video sample in a latent variable scale of the video sample according to a video task type of the video sample;

[0017] a latent variable obtaining module configured to generate a spliced latent variable corresponding to each video sample respectively according to the latent variable and the task mask of each video sample;

[0018] a processing model generating module configured to perform pre-training of adding noise and removing noise on a pre-constructed diffusion model according to the spliced latent variable of each video sample to obtain a unified video processing model adapted to the plurality of video task types.

[0019] According to another aspect of an embodiment of the present application, a video processing device is also provided, comprising:

[0020] a task obtaining module configured to obtain a current video processing task, and generate a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask;

[0021] a latent variable generating module configured to generate a current spliced latent variable according to the current latent variable and the current task mask;

[0022] a video obtaining module configured to input the current spliced latent variable into a unified video processing model trained by a method according to any one of embodiments of the present application to obtain a target processing video matched with the current video processing task.

[0023] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising:

[0024] at least one processor; and

[0025] a memory in communication with the at least one processor; wherein

[0026] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the training method of the video processing model or the video processing method according to any one of the embodiments of the present application.

[0027] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which stores computer instructions for enabling a processor to implement the training method of the video processing model or the video processing method according to any one of the embodiments of the present application when executed by the processor.

[0028] According to another aspect of the embodiments of the present application, a computer program product is also provided, which comprises a computer program for enabling a processor to implement the steps of the training method of the video processing model or the video processing method according to any one of the embodiments of the present application when executed by the processor.

[0029] The technical solution of the embodiments of the present application obtains a set of video samples matched with a plurality of video task types; generates a latent variable corresponding to each video sample respectively, and determines a task mask of each video sample in the dimension of the latent variable to which the video sample belongs according to the video task type of each video sample; generates a spliced latent variable corresponding to each video sample according to the latent variable and the task mask of each video sample; and pre-trains a diffusion model pre-constructed by adding noise and removing noise according to the spliced latent variable of each video sample, to obtain a unified video processing model adaptive to the plurality of video task types. By using the mask corresponding to the task type in combination with the latent variable, the diffusion model pre-constructed is pre-trained, multi-task collaborative pre-training is realized, the overhead of computing and storage resources is reduced, the training and deployment efficiency is improved, and the generalization ability and adaptability of the video processing model in the multi-task scenario are improved.

[0030] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0032] Figure 1 is a flow chart of a video processing model training method according to an embodiment of the present application;

[0033] Figure 2 is a flow chart of a video processing method according to an embodiment of the present application;

[0034] Figure 3 is a flow chart of a video processing model pre-training method according to an embodiment of the present application;

[0035] Figure 4 is a structural schematic diagram of a video processing model training device according to an embodiment of the present application;

[0036] Figure 5 is a structural schematic diagram of a video processing device according to an embodiment of the present application;

[0037] Figure 6 is a structural schematic diagram of an electronic device implementing a video processing model training method or a video processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] Embodiment one

[0041] Figure 1A flowchart of a training method of a video processing model is provided for Embodiment One of the present application. The present embodiment can be applied to the case of training a video processing model according to a video sample set matched with multiple video task types. The method can be executed by a training device of a video processing model, which can be realized in the form of hardware and / or software and generally configured in an electronic device. As shown in FIG. 9, the method comprises: Figure 1

[0042] S110, obtaining a video sample set matched with multiple video task types.

[0043] In the present embodiment, one video sample corresponds to one video task type.

[0044] In the present embodiment, the video task type can be understood as a specific application scenario for processing and analyzing video content, which can include video generation tasks, video continuation tasks, and video completion tasks, etc.

[0045] Specifically, the video sample set matched with multiple video task types is obtained, and the video sample set contains video samples of multiple video task types, and each video sample corresponds to one video task type.

[0046] S120, generating a latent variable corresponding to each video sample respectively, and determining a task mask of each video sample in the latent variable scale according to the video task type of each video sample.

[0047] In the present embodiment, the latent variable can be understood as a result of feature extraction for explaining the relationship between observable variables, representing the potential representation or features of data. The task mask can be understood as a key information for distinguishing different tasks, which can realize task sharing and task differentiation in the model training process. In the training of a multi-task video processing model, the task mask is related to the latent variable to which each video sample belongs, and is used to indicate the data part that the model should focus on in a specific task. The number of variable data contained in the latent variable corresponds to the number of data bits in the mask one by one, and both correspond to the frames in the video. For example, the first three numbers of variable data contained in the latent variable correspond to the first three data in the mask, and both correspond to the first frame in the video.

[0048] ​Specifically, the neural network can be used to generate the latent variable corresponding to each video sample according to the video sample set. The neural network can specifically include an autoencoder and a temporal consistency transformer, etc. For example, the encoder of the autoencoder is used to map the video data to a probability distribution (such as a multivariate Gaussian distribution) to generate the latent variable, and the decoder can subsequently reconstruct the video data from the latent variable; or the convolutional encoder of the temporal consistency transformer is used to encode the current frame and the previous frame to compress the data into discrete latent variable representation, then perform nearest neighbor search in the codebook, and finally quantize the latent variable, which can be reconstructed by the decoder to regenerate the video.

[0049] The latent variable can be expressed as a set of potential representations or features (collectively referred to as variable data in this embodiment) of the data. The latent variable scale can be understood as the total number of variable data contained in the latent variable. The task mask of each video sample under the corresponding latent variable scale can be understood as a mask sequence composed of 0 and / or 1, and the data length of the mask sequence is consistent with the total number of variable data contained in the latent variable.

[0050] Then, according to the video task type of each video sample, the task mask of each video sample under the corresponding latent variable scale is determined. For example, for the video generation task, because the generated video needs to be constructed from scratch, the mask can cover the entire video sequence, and the part of the video sequence that needs to be focused on, i.e., the mask of the entire video sequence, is set to 0 or 1; for the video continuation task, because the model needs to generate new video frames to continue the story line or action while maintaining the first half of the content unchanged, the mask can be focused on the second half of the video sequence, and the part of the video sequence that needs to be focused on, i.e., the mask of the part that needs to be continued, is set to 0 or 1, and the mask of the other part of the video is set to another number for distinction; for the video completion task, the mask can be specifically for the area that needs to be completed in the video, while keeping other areas unchanged, and the part of the video that needs to be focused on, i.e., the mask of the part that needs to be completed, is set to 0 or 1, and the mask of the other part of the video is set to another number for distinction. Thus, the model can focus on restoring or generating the missing video content while keeping other parts of the video unchanged.

[0051] S130, generating a spliced latent variable corresponding to each video sample according to the latent variable and the task mask of each video sample.

[0052] In the embodiment of the application, the spliced latent variable can be understood as follows: the masked latent variable and the latent variable are spliced and combined in the channel dimension to form a complete input. The channel dimension refers to the number of features, and each channel corresponds to a different feature.

[0053] Specifically, according to the latent variable and the task mask of each video sample, the latent variable is multiplied, added or exponentially operated with the mask generated by the task to obtain a masked latent variable. For example, the mask is multiplied with the latent variable element by element to obtain the masked latent variable; the mask is added to the latent variable element by element to obtain the masked latent variable; the mask is exponentially operated with the latent variable, which not only encodes the mask into the latent variable, but also adjusts the magnification or reduction of the index size control information in the model, and introduces the sensitivity to the mask, i.e. different task types. In this way, the mask effectively encodes task information into the latent variable, further improving the task adaptability of the model. The masked latent variable is spliced and combined with the latent variable in the channel dimension to generate a spliced latent variable corresponding to each video sample, and the two features are connected along the depth direction of the feature map, and the combined latent variable has a channel dimension of the sum of the two input feature channel numbers, which enables the network to process multiple features at the same time. This combination method not only provides more rich feature input, but also provides more robust information for model training.

[0054] In S140, the diffusion model is pre-trained by adding noise and denoising according to the spliced latent variable of each video sample, to obtain a unified video processing model suitable for multiple video task types.

[0055] In the embodiment of the application, the diffusion model refers to a generative model based on a probability diffusion process, which converts data into a noise distribution by gradually adding noise to the data, and then gradually restores the data from the noise through an inverse process to obtain a predicted output of the target video. The noise corresponds to the forward diffusion process in the diffusion model, which involves gradually adding noise to the data until the data completely becomes noise, i.e. by introducing randomness into the data, the data gradually loses its original structure and information, and finally becomes a high-noise-level distribution. The denoising corresponds to the reverse diffusion process in the diffusion model, which involves gradually restoring the original data or generating new data samples from the noise, i.e. extracting feature information from the noise.

[0056] Specifically, according to the spliced latent variable of each video sample, pre-training of the pre-constructed diffusion model is performed for noise adding and noise removing, for example, pre-constructed Gaussian noise or Poisson noise adding strategies can be used for noise adding, and pre-constructed attention transformer (Transformer) or U-shaped network (U-Net) can be used to predict noise at each step and recover clear video frames from the noise, that is, at each sampling time interval, the model receives the current noisy video sample and predicts the noise added at the sampling time interval. Then, the model uses this prediction to recover the clear video frames from the noisy video, obtaining the predicted output of the target video. By using the spliced latent variables of the video samples corresponding to multiple video task types for pre-training, a unified video processing model that can adapt to multiple video task types can be obtained.

[0057] The technical scheme of the embodiment of the application obtains a video sample set matched with multiple video task types; generates a latent variable corresponding to each video sample respectively, determines a task mask of each video sample in the latent variable scale corresponding to the video sample according to the video task type of each video sample; generates a spliced latent variable corresponding to each video sample respectively according to the latent variable and the task mask of each video sample; and pre-trains a pre-constructed diffusion model for noise adding and noise removing according to the spliced latent variables of the video samples, to obtain a unified video processing model that is adapted to multiple video task types. By using the mask corresponding to the task type in combination with the latent variable, the pre-constructed diffusion model is pre-trained, multi-task collaborative pre-training is realized, the computational and storage resource overhead is reduced, the training and deployment efficiency is improved, and the generalization ability and adaptability of the video processing model in a multi-task scenario are improved.

[0058] Optionally, on the basis of each of the above embodiments, the latent variable corresponding to each video sample is generated, and the task mask of each video sample in the latent variable scale corresponding to the video sample is determined according to the video task type of each video sample, including:

[0059] The target video sample currently processed is encoded by the autoencoder to obtain a target latent variable, wherein the target latent variable carries the bounding information of each video frame in the target video sample;

[0060] If the video task type of the target video sample is a video generation task, a task mask of all 1s is generated according to the data scale of the target latent variable, and the video generation task includes text-to-video or image-to-video tasks;

[0061] If the video task type of the target video sample is a video continuation task, each continuation frame in the target video sample is determined, and a first data position matching each continuation frame in the target latent variable is identified; according to the data scale of the target latent variable, a task mask is generated, in which the positions matching the first data positions are 0, and the remaining positions are 1.

[0062] If the video task type of the target video sample is a video completion task, each completion frame in the target video sample is determined, and a second data position matching each completion frame in the target latent variable is identified; according to the data scale of the target latent variable, a task mask is generated, in which the positions matching the second data positions are 0, and the remaining positions are 1.

[0063] In the embodiment of the application, the target latent variable can be understood as feature information obtained by encoding the target video sample currently processed by the autoencoder. The number of variable data contained in the target latent variable corresponds to the number of data bits in the mask, and both correspond to the frames in the video, that is, the feature information carried in the target latent variable corresponds to the position information of each video frame in the target video sample, which can specifically include frame rate, resolution, code rate and frame type (such as I frame, P frame and B frame) and the like. That is, by combining the above position information, the variable data at the set position in the target latent variable can be determined, which specifically corresponds to the video frame at the position in the target video sample.

[0064] The first data position can be understood as the data position of the variable data matching each continuation frame in the target video sample in the target latent variable when the video task type of the target video sample is a video continuation task. The second data position can be understood as the data position of the variable data matching each completion frame in the target video sample in the target latent variable when the video task type of the target video sample is a video completion task. The data scale of the target latent variable can be understood as the number of specific variable data contained in the target latent variable.

[0065] Specifically, the target latent variable is obtained by encoding the target video sample currently processed by the autoencoder. When the video task type of the target video sample is a video generation task, a task mask with all 1s is generated according to the data scale of the target latent variable. The region with 1 in the mask represents the position that needs to be paid attention to in the video generation task, that is, the position where the video needs to be generated. The video generation task includes a text-to-video task or a picture-to-video task, that is, the video content is generated according to the text key information extracted from the text description or the video content is generated according to the image key information extracted from one or a series of static images.

[0066] When the video task type of the target video sample is a video continuation task, each continuation frame in the target video sample is determined, and a first data position matched with each continuation frame is identified in the target latent variable; a task mask is generated according to the data dimension of the target latent variable, the task mask being 0 at a mask position matched with the first data position and 1 at other positions, a region with the mask being 1 representing that there is content at the video frame and the content needs to be retained, and a region with the mask being 0 representing that the content at the video frame is empty, i.e., a position that needs to be continued.

[0067] When the video task type of the target video sample is a video completion task, each completion frame in the target video sample is determined, and a second data position matched with each completion frame is identified in the target latent variable; a task mask is generated according to the data dimension of the target latent variable, the task mask being 0 at a mask position matched with the second data position and 1 at other positions, a region with the mask being 1 representing that there is content at the video frame and the content needs to be retained, and a region with the mask being 0 representing that the content at the video frame is empty, i.e., a position that needs to be completed. By setting the mask, the model can focus on important regions in the video content during the training process, improving the learning efficiency and prediction accuracy and enhancing the generalization ability of the model to specific features.

[0068] Optionally, based on the latent variable and the task mask of each video sample, a spliced latent variable corresponding to each video sample is generated, including:

[0069] The latent variable and the task mask corresponding to each video sample are multiplied to obtain a mask latent variable corresponding to each video sample;

[0070] The mask latent variable and the latent variable corresponding to each video sample are channel spliced to obtain a spliced latent variable corresponding to each video sample.

[0071] Specifically, the mask and the latent variable are multiplied element by element to obtain a latent variable with a mask corresponding to each video sample, i.e., a mask latent variable. By the element-by-element multiplication operation, the information in the latent variable can be filtered, the 0 value in the mask will completely shield the information at the corresponding position in the latent variable, and the 1 value will allow the information to pass through, enhancing the representation of the target feature and improving the robustness of the model feature learning, thereby improving the generalization ability of the model. The mask latent variable and the latent variable corresponding to each video sample are channel spliced to obtain a spliced latent variable corresponding to each video sample. In multi-task learning, the mask latent variable feature and the latent variable feature can be combined by channel splicing to provide the model with richer feature input, and the model can adapt to different task requirements, improving the flexibility and generalization ability of the model.

[0072] Optionally, on the basis of each of the above embodiments, according to the splicing latent variable of each video sample, pre-training of the diffusion model constructed in advance is performed, including:

[0073] diffusion processing is performed on the target splicing latent variable currently processed, to obtain a noise latent variable corresponding to the target splicing latent variable;

[0074] the noise latent variable is input into the diffusion model, to obtain a denoising prediction value output by the diffusion model for the noise latent variable;

[0075] a loss value is calculated according to the denoising prediction value and the target splicing latent variable, and the diffusion model is optimized in terms of model parameters according to the calculated loss value.

[0076] Specifically, diffusion processing is performed on the target splicing latent variable currently processed, that is, noise is gradually added in the target splicing latent variable data, the original data is converted into high-noise data, the data is gradually destroyed, and finally the high-noise data obtained is the noise latent variable corresponding to the target splicing latent variable. The noise latent variable is input into the diffusion model, the model gradually predicts and removes noise from the noise data, restores the video frame, and reconstructs the original data, and finally obtains the denoising prediction value output by the diffusion model for the noise latent variable, that is, the prediction output of the target video.

[0077] A loss value is calculated according to the denoising prediction value and the target splicing latent variable, and the diffusion model is optimized in terms of model parameters according to the calculated loss value, such as the number of network layers, the number of neurons in each layer, the size of the convolution kernel, and the time step parameter, etc. The weights of the model can be updated using methods such as stochastic gradient descent or batch gradient descent to reduce the value of the loss function. Through multiple iterations, each iteration makes the model more accurately predict the original data, thereby improving the generalization ability and adaptability of the video processing model in the multi-task scenario.

[0078] Embodiment Two

[0079] Figure 2 A flowchart of a video processing method provided for Embodiment Two of the present application. This embodiment can be applied to the case where a video processing model is trained to generate a video according to a video processing task. The method can be performed by a video processing device, which can be implemented in the form of hardware and / or software, and is generally configured in an electronic device. As shown in FIG. 2, the method includes: Figure 2

[0080] ​S210, acquire a current video processing task, and generate a current latent variable associated with a target processing video expected by the current video processing task and a current task mask.

[0081] In the embodiments of the present application, the target processing video can be understood as a video predicted and output by the unified video processing model trained according to the current video processing task. The current latent variable can be understood as a latent variable associated with the target processing video generated according to the current video processing task. The current task mask can be understood as a mask associated with the target processing video generated according to the current video processing task.

[0082] The target processing video is a video expected by the current video processing task, for example, a video of a set time length based on a text and an image, a video obtained by performing a set time length video continuation on an original video, or a video obtained by inserting one or more video frames at one or more positions in the original video. It can be understood that the number of video frames in the target processing video is known, and thus the data scale of the current latent variable and the current task mask associated with the target processing video is known, and they are consistent.

[0083] Specifically, the current video processing task is acquired, and the current latent variable associated with the target processing video expected by the current video processing task and the current task mask are generated. The current latent variable can be obtained by encoding and processing known text, image or video through a self-encoder to extract features. The current task mask sets the mask value according to the data scale of the current latent variable, for example, the mask value corresponding to the known frame can be set to 0, and the mask value corresponding to the frame position whose video content needs to be predicted can be set to 1. By setting the mask, the model can focus on the important area in the video content during the inference and prediction process, and the prediction accuracy is improved.

[0084] S220, generate a current spliced latent variable according to the current latent variable and the current task mask.

[0085] Specifically, the current latent variable and the current task mask are multiplied to obtain a corresponding target mask latent variable, and the target mask latent variable and the corresponding current latent variable are channel spliced to generate the current spliced latent variable.

[0086] S230, input the current spliced latent variable into the unified video processing model trained by the method of any one of the embodiments of the present application to obtain a target processing video matched with the current video processing task.

[0087] Specifically, the current spliced latent variable is input into the unified video processing model trained by the method according to any one of the embodiments of the application, and after model inference prediction, a target processing video matching the current video processing task is obtained.

[0088] The technical scheme of the embodiments of the application obtains a current video processing task, generates a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask, generates a current spliced latent variable according to the current latent variable and the current task mask, inputs the current spliced latent variable into the unified video processing model trained, and obtains a target processing video matching the current video processing task. By using the mask corresponding to the task type in combination with the latent variable, the target processing video is obtained through the unified video processing model, the model is simultaneously supported for multiple video processing tasks in a unified framework, the inference process of the model is simplified, and the inference efficiency, generalization capability and adaptability of the video processing model in a multi-task scenario are improved.

[0089] Optionally, on the basis of each of the above embodiments, the current latent variable associated with the target processing video expected to be obtained by the current video processing task and the current task mask are generated, including:

[0090] If the current video processing task is a video generation task, a current latent variable of all empty is generated according to the expected video frame number of the target processing video, and a current task mask of all 1 consistent with the data scale of the current latent variable is generated.

[0091] If the current video processing task is a video continuation task, a first current latent variable is generated according to the first original video in the current video processing task and the first video position of each expected continuation frame, and a first current data position matching the first video position is identified in the first current latent variable; a current task mask in which the mask position matching the first current data position is 0 and the remaining positions are 1 is generated according to the data scale of the first current latent variable.

[0092] If the current video processing task is a video completion task, a second current latent variable is generated according to the second original video in the current video processing task and the second video position of each expected completion frame, and a second current data position matching the second video position is identified in the second current latent variable; a current task mask in which the mask position matching the second current data position is 0 and the remaining positions are 1 is generated according to the data scale of the current latent variable.

[0093] In the embodiments of the present application, the first original video can be specifically understood as: when the current video processing task is a video continuation task, the original input video before video continuation. The first video position can be specifically understood as: the video position where the expected continuation frame is located, for example, it can be the tail position of the last video frame in the original input video. The first current latent variable can be specifically understood as: the latent variable obtained by encoding the first original video through the autoencoder, which is processed into a latent variable consistent with the data scale of the latent variable corresponding to the target processed video. For example, in the latent variable, the variable data corresponding to the video frame to be continued in the first original video is supplemented with 0 or reserved space, etc. The first current data position can be specifically understood as: the position of the data in the first current latent variable that matches the first video position.

[0094] The second original video can be specifically understood as: when the current video processing task is a video completion task, the original input video before video completion. The second video position can be specifically understood as: the video position where the expected completion frame is located, for example, between the 100th frame and the 101st frame of the second original video, and 3 consecutive frames are expected to be completed. The second current latent variable can be specifically understood as: the latent variable obtained by encoding the second original video through the autoencoder, which is processed into a latent variable consistent with the data scale of the latent variable corresponding to the target processed video. For example, in the latent variable, the variable data corresponding to the video frame to be completed in the second original video is supplemented with 0 or reserved space, etc. The second current data position can be specifically understood as: the position of the data in the second current latent variable that matches the second video position.

[0095] Specifically, if the current video processing task is a video generation task, such as a text-to-video or image-to-video task, a full-empty current latent variable can be generated according to the expected number of video frames of the target processed video, and a full-1 current task mask consistent with the data scale of the current latent variable can be generated.

[0096] If the current video processing task is a video continuation task, the first current latent variable can be generated through the autoencoder according to the first original video in the current video processing task and the first video positions of the expected continuation frames, and the first current data position matching the first video position can be identified in the first current latent variable. The first current latent variable can be used as the input of the model to predict the content of the subsequent frame. According to the data scale of the first current latent variable, a current task mask is generated, which is 0 at the mask position matching the first current data position and 1 at the remaining positions.

[0097] If the current video processing task is a video completion task, a second current latent variable can be generated according to a second original video in the current video processing task and a second video position of each expected completion frame, and a second current data position matched with the second video position is identified in the second current latent variable. The second current latent variable can be used as an input of the model to predict the content of the completion frame. According to the data scale of the current latent variable, a current task mask in which the positions matched with the second current data position are 0 and the other positions are 1 is generated. Through the representation of 0 and 1 of the mask, filtering of the latent variable features is realized, and the representation of the features related to the current video processing task is enhanced.

[0098] Specific application scenarios

[0099] In recent years, diffusion models have achieved remarkable results in the field of image generation. As a sequence of images with time sequence, the application of diffusion models naturally extends to the field of video generation. However, the video generation process is more complex and needs to consider the continuity and consistency in the time dimension. The existing two-stage video generation method, which first trains a model to generate videos from text and then fine-tunes it, has the problems of large consumption of model training calculation and storage resources, low efficiency of model training and deployment, low generalization ability and poor adaptability of the model in multi-task scenarios. To solve the above problems, the present embodiment proposes a training method of a video processing model and a video processing method, Figure 3 is a flowchart of a pre-training method of a video processing model applicable to the present embodiment. As shown in Figure 3 , the method can specifically include:

[0100] 1. Dataset input and latent variable generation

[0101] The video data in the dataset is input into the autoencoder one by one, and the feature extraction and dimension reduction processing of the encoder are performed to obtain the latent variable representation corresponding to each video. The latent variable can preserve the main information of the video while significantly reducing the data dimension, providing a concise representation for subsequent processing.

[0102] 2. Task classification and mask generation

[0103] According to the preset task category (such as video generation task, video continuation task and video completion task, etc.), the corresponding mask type is determined. According to the requirements of different tasks, the system generates a unique mask template to adapt to specific pre-training tasks. Each task category corresponds to a customized mask to ensure that the target features can be extracted most effectively in the task.

[0104] For example, if the task is video generation, a task mask of all 1s is generated according to the data dimension of the latent variable, and the video generation task includes text-to-video or image-to-video tasks; if the task is video continuation, each continuation frame is determined in the video, and the data position matching each continuation frame is identified in the latent variable; a task mask is generated according to the data dimension of the latent variable, which is 0 at the mask position matching the data position and 1 at the remaining positions; if the task is video completion, each completion frame is determined in the video, and the data position matching each completion frame is identified in the latent variable; a task mask is generated according to the data dimension of the latent variable, which is 0 at the mask position matching the data position and 1 at the remaining positions.

[0105] 3. Mask application and mask latent variable calculation

[0106] The obtained latent variable representation is multiplied by the mask generated by the task to obtain a masked latent variable. In this way, the mask effectively encodes task information into the latent variable, further improving the task adaptability of the model.

[0107] 4. Latent variable splicing and synthesis

[0108] The masked latent variable and the latent variable are spliced and combined in the channel dimension to obtain a spliced latent variable to form a complete input. This combination not only provides richer feature input, but also provides more robust information for the diffusion model.

[0109] 5. Noise latent variable generation in diffusion process

[0110] The spliced latent variable is subjected to a diffusion process operation, and noise is introduced to generate a noisy latent variable, preparing for the training of the diffusion model for noise addition and noise reduction.

[0111] 6. Diffusion model input and predicted value calculation of noise latent variable

[0112] The noise latent variable is input into the diffusion model, and the predicted value is output through the processing of the diffusion model. The diffusion model here is equivalent to a denoising process, which gradually restores the original video information contained in the latent variable to obtain the predicted output of the target video.

[0113] 7. Loss calculation

[0114] The loss value is calculated by comparing the predicted value output by the diffusion model with the original noise latent variable. The loss value can reflect the current model adaptation to the task, and provide targeted optimization basis for the model.

[0115] 8. Model pre-training optimization

[0116] The loss value is fed back to the diffusion model to pre-train and adjust the parameters of the model. In this way, the model is gradually optimized in multiple iterations, and its generalization ability in different video generation tasks is enhanced.

[0117] The video processing method provided by the embodiment of the present application can be specifically:

[0118] 1. Current task mask and current latent variable generation

[0119] The current video processing task is obtained, and the current latent variable associated with the target processing video expected to be obtained by the current video processing task and the current task mask are generated.

[0120] If the current video processing task is a video generation task, the current latent variable is generated according to the expected video frame number of the target processing video, and the current task mask is generated according to the data scale of the current latent variable. If the current video processing task is a video continuation task, the current latent variable is generated according to the original video in the current video processing task and the video position of each expected continuation frame, and the current data position matched with the video position is identified in the current latent variable. According to the data scale of the current latent variable, the current task mask is generated, which is 0 at the mask position matched with the current data position and 1 at the remaining positions. If the current video processing task is a video completion task, the current latent variable is generated according to the original video in the current video processing task and the video position of each expected completion frame, and the current data position matched with the video position is identified in the current latent variable. According to the data scale of the current latent variable, the current task mask is generated, which is 0 at the mask position matched with the current data position and 1 at the remaining positions.

[0121] 2. Current splicing latent variable generation

[0122] The current splicing latent variable is generated according to the current latent variable and the current task mask.

[0123] 3. Target processing video prediction generation

[0124] The current splicing latent variable is input into the unified video processing model trained to obtain the target processing video matched with the current video processing task.

[0125] The embodiment of the application provides a video processing model training and video processing method, which adopts a mask generation method based on task types to convert different task categories into independent mask representations. By using masks corresponding to the task types and combining the latent variables, a pre-training paradigm is formed to realize multi-task collaborative pre-training. Not only is the prediction ability of the model in different video processing tasks improved, but also various different tasks such as video generation (graph video and text video), video continuation and video completion are unified into the same model, avoiding the problems of multi-model training and deployment, improving the resource utilization efficiency, reducing the computational resource overhead in large-scale pre-training; by training the model in the unified pre-training framework, it is ensured that the model can still retain the general video generation ability learned before after specific task fine-tuning, the generalization performance of the model is enhanced, and the catastrophic forgetting problem is effectively alleviated; based on the unified pre-training task, a channel mask driven method is introduced, the model can learn more rich and general features, so that better results can be achieved in each task, and the scalability and adaptability of the model in different task scenarios are improved.

[0126] Embodiment three

[0127] Figure 4 A structural schematic diagram of a video processing model training device provided by the embodiment three of the application is shown in the figure. Figure 4 As shown in the figure, the device comprises a sample set acquisition module 410, a task mask determination module 420, a latent variable acquisition module 430 and a processing model generation module 440.

[0128] The sample set acquisition module 410 is configured to acquire a video sample set matched with a plurality of video task types; wherein one video sample corresponds to one video task type;

[0129] The task mask determination module 420 is configured to generate a latent variable corresponding to each video sample respectively, and determine a task mask of each video sample in the latent variable scale of the video sample according to the video task type of the video sample;

[0130] The latent variable acquisition module 430 is configured to generate a spliced latent variable corresponding to each video sample according to the latent variable and the task mask of each video sample;

[0131] The processing model generation module 440 is configured to perform pre-training of noise adding and de-noising on a pre-constructed diffusion model according to the spliced latent variable of each video sample, to obtain a unified video processing model suitable for a plurality of video task types.

[0132] The technical scheme of the embodiment of the application comprises the following steps: obtaining a video sample set matched with multiple video task types; generating a latent variable corresponding to each video sample respectively, and determining a task mask of each video sample in the dimension of the latent variable to which the video sample belongs according to the video task type of each video sample; generating a spliced latent variable corresponding to each video sample respectively according to the latent variable and the task mask of each video sample; and pre-training a diffusion model pre-constructed by adding noise and removing noise according to the spliced latent variable of each video sample, so as to obtain a unified video processing model adapted to multiple video task types. By using the mask corresponding to the task type in combination with the latent variable, the diffusion model pre-constructed is pre-trained, multi-task collaborative pre-training is realized, the calculation and storage resource overheads are reduced, the training and deployment efficiency is improved, and the generalization capability and adaptability of the video processing model in a multi-task scenario are improved.

[0133] On the basis of the above embodiments, the task mask determination module 420 is specifically configured to:

[0134] The target video sample currently processed is encoded by using the autoencoder to obtain a target latent variable, wherein the target latent variable carries the limiting information of each video frame in the target video sample.

[0135] If the video task type of the target video sample is a video generation task, a task mask of all 1s is generated according to the data dimension of the target latent variable, and the video generation task includes a text-to-video or a picture-to-video task.

[0136] If the video task type of the target video sample is a video continuation task, each continuation frame in the target video sample is determined, and a first data position matched with each continuation frame is identified in the target latent variable; a task mask in which the mask positions matched with the first data positions are 0 and the remaining positions are 1 is generated according to the data dimension of the target latent variable.

[0137] If the video task type of the target video sample is a video completion task, each completion frame in the target video sample is determined, and a second data position matched with each completion frame is identified in the target latent variable; a task mask in which the mask positions matched with the second data positions are 0 and the remaining positions are 1 is generated according to the data dimension of the target latent variable.

[0138] On the basis of the above embodiments, the latent variable acquisition module 430 is specifically configured to:

[0139] The latent variable corresponding to each video sample respectively and the task mask are multiplied correspondingly to obtain a mask latent variable corresponding to each video sample respectively.

[0140] The mask latent variable corresponding to each video sample respectively and the latent variable are channel spliced to obtain a spliced latent variable corresponding to each video sample respectively.

[0141] On the basis of the above-mentioned embodiments, the processing model generation module 440 is specifically configured to:

[0142] diffuse the target splicing latent variable of the current processing to obtain a noise latent variable corresponding to the target splicing latent variable;

[0143] input the noise latent variable into the diffusion model to obtain a denoising prediction value output by the diffusion model for the noise latent variable;

[0144] calculate a loss value according to the denoising prediction value and the target splicing latent variable, and optimize the model parameters of the diffusion model according to the calculated loss value.

[0145] The training device for the video processing model provided in the embodiments of the present application can execute the training method for the video processing model provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0146] Embodiment Four

[0147] Figure 5 A structural schematic diagram of a video processing device provided in Embodiment Four of the present application is shown in FIG. 5. Figure 5 As shown in the figure, the device includes a task acquisition module 510, a latent variable generation module 520, and a video acquisition module 530.

[0148] The task acquisition module 510 is configured to acquire a current video processing task, and generate a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask.

[0149] The latent variable generation module 520 is configured to generate a current splicing latent variable according to the current latent variable and the current task mask.

[0150] The video acquisition module 530 is configured to input the current splicing latent variable into a unified video processing model trained by the method described in any of the embodiments of the present application to obtain a target processing video matched with the current video processing task.

[0151] The technical scheme of the embodiment of the present application obtains a current video processing task, and generates a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask; generates a current splicing latent variable according to the current latent variable and the current task mask; inputs the current splicing latent variable into a unified video processing model obtained by training, to obtain a target processing video matched with the current video processing task. By using the mask corresponding to the task type in combination with the latent variable, the target processing video is obtained through the unified video processing model, so that the model supports multiple video processing tasks in a unified framework, simplifies the inference process of the model, and improves the inference efficiency, generalization ability and adaptability of the video processing model in a multi-task scenario.

[0152] On the basis of the above-mentioned embodiments, the task acquisition module 510 is specifically configured to:

[0153] If the current video processing task is a video generation task, a current latent variable of all empty is generated according to the expected video frame number of the target processing video, and a current task mask of all 1 consistent with the data scale of the current latent variable is generated;

[0154] If the current video processing task is a video continuation task, a first current latent variable is generated according to the first original video in the current video processing task and the first video position of each expected continuation frame, and a first current data position matched with the first video position is identified in the first current latent variable; a current task mask in which the mask position matched with the first current data position is 0 and the remaining positions are 1 is generated according to the data scale of the first current latent variable;

[0155] If the current video processing task is a video completion task, a second current latent variable is generated according to the second original video in the current video processing task and the second video position of each expected completion frame, and a second current data position matched with the second video position is identified in the second current latent variable; a current task mask in which the mask position matched with the second current data position is 0 and the remaining positions are 1 is generated according to the data scale of the current latent variable.

[0156] The video processing device provided in the embodiment of the present application can execute the video processing method provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0157] In the technical scheme of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical scheme comply with relevant laws and regulations and do not violate public order and good customs.

[0158] Embodiment five

[0159] Figure 6A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0160] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0161] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0162] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the training methods for video processing models, i.e.:

[0163] Obtain a set of video samples that match multiple video task types; where one video sample corresponds to one video task type.

[0164] generate a latent variable corresponding to each video sample respectively, and determine a task mask of each video sample in a dimension of the latent variable to which the video sample belongs according to a video task type of each video sample;

[0165] generate a spliced latent variable corresponding to each video sample respectively according to the latent variable and the task mask of each video sample;

[0166] perform pre-training of noise adding and de-noising on the diffusion model pre-constructed according to the spliced latent variable of each video sample, to obtain a unified video processing model adaptive to multiple video task types.

[0167] Alternatively, the video processing method, namely:

[0168] obtain a current video processing task, and generate a current latent variable associated with a target processing video expected to be obtained by the current video processing task and a current task mask;

[0169] generate a current spliced latent variable according to the current latent variable and the current task mask;

[0170] input the current spliced latent variable into the unified video processing model trained by the method according to any one of the embodiments of the present application, to obtain a target processing video matched with the current video processing task.

[0171] In some embodiments, the training method of the video processing model or the video processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the video processing model or the video processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the training method of the video processing model or the video processing method by any other appropriate means, for example, by means of firmware.

[0172] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0173] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0174] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0176] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0177] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0178] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0179] The above detailed description does not limit the scope of the present disclosure. It is understood that various modifications, combinations, sub-combinations, and alternatives can be made to the detailed disclosure without departing from the spirit and principles of the present disclosure. Any modifications, equivalent substitutions, improvements, and the like that are made within the spirit and principles of the present disclosure are included in the scope of the present disclosure.

Claims

1. A training method for a video processing model, characterized in that, include: Obtain a set of video samples that match multiple video task types; where one video sample corresponds to one video task type. Generate latent variables corresponding to each video sample, and determine the task mask of each video sample at the scale of its respective latent variable based on the video task type of each video sample; Based on the latent variables and task mask of each video sample, generate concatenated latent variables corresponding to each video sample; Based on the latent variables of each video sample, the pre-built diffusion model is pre-trained for adding and denoising, resulting in a unified video processing model that is compatible with multiple video task types. Specifically, based on the latent variables and task mask of each video sample, concatenated latent variables corresponding to each video sample are generated, including: Multiply the latent variable and task mask corresponding to each video sample respectively to obtain the mask latent variable corresponding to each video sample; The latent variables corresponding to each video sample are concatenated with the mask latent variables to obtain the concatenated latent variables corresponding to each video sample.

2. The method according to claim 1, characterized in that, Generate latent variables corresponding to each video sample, and determine the task mask for each video sample at the scale of its respective latent variable based on the video task type of each video sample, including: The target video sample is encoded by an autoencoder to obtain target latent variables, which carry the limiting information of each video frame in the target video sample. If the video task type of the target video sample is a video generation task, then generate a task mask of all 1s according to the data scale of the target latent variable. Video generation tasks include text-generated or image-generated video tasks. If the video task type of the target video sample is a video continuation task, then each continuation frame is determined in the target video sample, and the first data position matching each continuation frame is identified in the target latent variable; according to the data scale of the target latent variable, a task mask is generated where the mask position matching the first data position is 0 and the other positions are 1. If the video task type of the target video sample is a video completion task, then each completion frame is determined in the target video sample, and the second data position matching each completion frame is identified in the target latent variable; according to the data scale of the target latent variable, a task mask is generated where the mask position matching the second data position is 0 and the rest positions are 1.

3. The method according to claim 1, characterized in that, Based on the concatenated latent variables of each video sample, the pre-built diffusion model is pre-trained for adding and denoising, including: The target splicing latent variable being processed is diffused to obtain the noise latent variable corresponding to the target splicing latent variable; Input the noise latent variable into the diffusion model and obtain the denoised prediction value output by the diffusion model for the noise latent variable; The loss value is calculated based on the denoised predicted value and the target spliced ​​latent variable, and the model parameters of the diffusion model are optimized based on the calculated loss value.

4. A video processing method, characterized in that, include: Obtain the current video processing task and generate the current latent variables and the current task mask associated with the target video to be processed by the current video processing task. Generate the current concatenation latent variable based on the current latent variable and the current task mask; The current latent variables are input into the unified video processing model trained by the method described in any one of claims 1-3 to obtain the target processing video that matches the current video processing task.

5. The method according to claim 4, characterized in that, Generate current latent variables and a current task mask associated with the target video to be processed as desired by the current video processing task, including: If the current video processing task is a video generation task, then generate an empty current latent variable based on the expected number of video frames of the target video to be processed, and generate a current task mask with all 1s that is consistent with the data scale of the current latent variable. If the current video processing task is a video continuation task, then based on the first original video in the current video processing task and the first video position of each expected continuation frame, a first current latent variable is generated, and the first current data position matching the first video position is identified in the first current latent variable; based on the data scale of the first current latent variable, a current task mask is generated with 0 at the mask position matching the first current data position and 1 at the other positions. If the current video processing task is a video completion task, then based on the second original video and the second video position of each expected completion frame in the current video processing task, a second current latent variable is generated, and the second current data position matching the second video position is identified in the second current latent variable; based on the data scale of the current latent variable, a current task mask is generated with 0 at the mask position matching the second current data position and 1 at the other positions.

6. A training device for a video processing model, characterized in that, include: The sample set acquisition module is used to acquire video sample sets that match multiple video task types; where one video sample corresponds to one video task type. The task mask determination module is used to generate latent variables corresponding to each video sample, and determine the task mask of each video sample at the scale of the latent variables according to the video task type of each video sample. The latent variable acquisition module is used to generate concatenated latent variables corresponding to each video sample based on the latent variables and task mask of each video sample; The processing model generation module is used to pre-train a pre-built diffusion model by adding and denoising based on the splicing latent variables of each video sample, so as to obtain a unified video processing model that is compatible with multiple video task types. The latent variable acquisition module is specifically used for: Multiply the latent variable and task mask corresponding to each video sample respectively to obtain the mask latent variable corresponding to each video sample; The latent variables corresponding to each video sample are concatenated with the mask latent variables to obtain the concatenated latent variables corresponding to each video sample.

7. A video processing apparatus, characterized in that, include: The task acquisition module is used to acquire the current video processing task and generate the current latent variables and the current task mask associated with the target video to be processed by the current video processing task. The latent variable generation module is used to generate the current concatenated latent variable based on the current latent variable and the current task mask; The video acquisition module is used to input the current splicing latent variables into the unified video processing model trained by the method described in any one of claims 1-3, so as to obtain the target processing video that matches the current video processing task.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the video processing model according to any one of claims 1-3 or the video processing method according to any one of claims 4-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the training method of the video processing model according to any one of claims 1-3 or the video processing method according to any one of claims 4-5.

Images