Video processing method and device, medium and program product

Abstract

The invention discloses a video processing method and device, a medium and a program product, and relates to the technical field of computers. According to the method, the diffusion model can sense the adjacent frame optical flow motion information in the real initial video, and the adjacent frame optical flow motion information corresponds to the time sequence motion related information in the video, so that the diffusion model edits the initial video into a new video according to the editing information and the adjacent frame optical flow motion information; according to the scheme, the diffusion model has the editing capability of a real video, the video editing precision is improved by means of time sequence motion related information in the video, model parameters do not need to be adjusted, calculation resources and model reasoning time are saved, and therefore the video editing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, device, medium and program product. Background Art

[0002] The current video editing model has limited editing capabilities for real videos. Although the model parameters of the video editing model can be fine-tuned using real samples, each fine-tuning consumes a certain amount of computing resources, which not only easily causes a waste of computing power, but also prolongs the model inference time. Among them, fine-tuning refers to further training on a dataset for a specific task based on the use of a pre-trained model to adjust the model parameters to better adapt to the target task. During the fine-tuning process, most of the layers of the pre-trained model are usually frozen, and only the newly added layers are trained or a small number of key layers are adjusted. This can not only retain the useful features learned by the pre-trained model, but also quickly adapt to the specific needs of the new task. In addition, choosing the appropriate learning rate and training rounds is also the key to successful fine-tuning. It can be seen that fine-tuning refers to conducting small-scale training on the basis of the pre-trained model for specific task objectives and task data, achieving minor adjustments to the parameters of the pre-trained model, and finally obtaining a model adapted to specific tasks and data.

[0003] Therefore, how to improve the editing efficiency and accuracy of the video editing model is a problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a video processing method, device, medium and program product to improve the editing efficiency and accuracy of the video editing model.

[0005] In a first aspect, the present application provides a video processing method, comprising:

[0006] Get initial video and editing information;

[0007] Performing forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video;

[0008] Using the optical flow network to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video;

[0009] The initial video is edited into a new video by using a preset diffusion model according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the backward optical flow video.

[0010] In a second aspect, the present application provides a video processing device, including:

[0011] An acquisition module, used to acquire the initial video and edit information;

[0012] A forward optical flow processing module, used to perform forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video;

[0013] A reverse optical flow processing module, used for performing reverse optical flow processing on the initial video using the optical flow network to obtain a reverse optical flow video;

[0014] The editing module is used to edit the initial video into a new video by using a preset diffusion model according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the backward optical flow video.

[0015] In a third aspect, the present application provides an electronic device, including:

[0016] Memory for storing computer programs;

[0017] The processor is used to execute the computer program to implement the aforementioned disclosed video processing method.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed video processing method when executed by a processor.

[0019] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which implements the steps of the aforementioned disclosed video processing method when executed by a processor.

[0020] Through the present application, the diffusion model (i.e., the video editing model) can perceive the optical flow motion information of adjacent frames in the real initial video, and the optical flow motion information of adjacent frames corresponds to the temporal motion related information in the video. Then, the diffusion model edits the initial video into a new video according to the editing information and the optical flow motion information of adjacent frames, so that the diffusion model not only has the editing ability of the real video, but also improves the video editing accuracy with the help of the temporal motion related information in the video. This solution does not need to adjust the model parameters, saves computing resources and model inference time, and thus also improves the video editing efficiency of the diffusion model.

[0021] Correspondingly, a video processing device, medium and program product provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A flow chart of a video processing method disclosed in this application;

[0024] Figure 2 A comparison diagram of an optical flow map and an original video frame disclosed in this application;

[0025] Figure 3 A schematic diagram of a processing flow of a video description model disclosed in this application;

[0026] Figure 4 A schematic diagram of a processing flow of another video description model disclosed in this application;

[0027] Figure 5 A schematic diagram of a training process of a video reconstruction model disclosed in this application;

[0028] Figure 6 A schematic diagram of a processing flow of a trained diffusion model disclosed in this application;

[0029] Figure 7 A server structure diagram provided for this application;

[0030] Figure 8 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0033] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] See also Figure 1 As shown, the embodiment of the present application discloses a video processing method, including:

[0035] S101: Obtain an initial video and editing information.

[0036] In this embodiment, the initial video may be a video recorded in the real world, or an animated cartoon video, an AI synthesized video, or an edited video. The editing information may include textual descriptions of the corresponding modifications to the video's tone, style, characters, animals, etc. For example, the editing information may include: controlling the video within 3 minutes, increasing the screen brightness to 20, adding subtitles, exporting in MP4 format, and exporting in 1080P resolution.

[0037] S102: Perform forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video.

[0038] S103, using the optical flow network to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video.

[0039] In this embodiment, the optical flow network can be implemented based on a structure such as a convolutional neural network. The optical flow network can calculate the optical flow value between the current frame and the next frame for each frame in the initial video, and then convert the optical flow value into an optical flow map according to the amplitude. Based on all the optical flow maps, the corresponding optical flow video can be generated. Figure 2 The optical flow map obtained by converting the optical flow value between the first frame image and the second frame image in the initial video is Figure 2 B and A in the figure are the first frame images. The Munsell hue system is used here. The forward optical flow video generates and transforms the optical flow map according to the positive order of each video frame in the initial video; the reverse optical flow video generates and transforms the optical flow map according to the reverse order of each video frame in the initial video; that is, the generation ideas of the forward optical flow video and the reverse optical flow video are similar, except that the reverse optical flow video needs to arrange the video frames in reverse order before generating and transforming the optical flow map.

[0040] In one embodiment, a preset optical flow network is used to perform forward optical flow processing on an initial video to obtain a forward optical flow video, including: using the optical flow network to calculate the optical flow values ​​between adjacent frames in the direction from the first frame to the last frame in the initial video, and converting the optical flow values ​​into corresponding optical flow maps, and aggregating the optical flow maps to obtain a forward optical flow video; correspondingly, the optical flow network is used to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video, including: using the optical flow network to calculate the optical flow values ​​between adjacent frames in the direction from the last frame to the first frame in the initial video, and converting the optical flow values ​​into corresponding optical flow maps, and aggregating the optical flow maps to obtain a reverse optical flow video.

[0041] S104: Using a preset diffusion model, the initial video is edited into a new video according to the editing information and the optical flow motion information of adjacent frames carried in the forward optical flow video and the backward optical flow video.

[0042] It should be noted that the diffusion model includes two main processes: the forward diffusion process (i.e., the noise training process) and the reverse diffusion process (i.e., the denoising inference process). In the forward diffusion process, the model starts with the original data and gradually adds Gaussian noise to it until the data completely becomes pure Gaussian noise. The reverse diffusion process is the inverse operation of the forward process, that is, starting from pure Gaussian noise, gradually removing the noise, and finally restoring the original data. Specifically, model training can be achieved through a parameterized neural network (such as a noise predictor), which learns how to predict and remove the noise added at each step.

[0043] This embodiment uses a diffusion model to perceive the optical flow motion information of adjacent frames in a real initial video. The optical flow motion information of adjacent frames corresponds to the temporal motion related information in the video. Then, the diffusion model edits the initial video into a new video according to the editing information and the optical flow motion information of adjacent frames. This enables the diffusion model to have the ability to edit real videos and improve the video editing accuracy with the help of the temporal motion related information in the video. This solution does not require adjustment of model parameters, saves computing resources and model inference time, and thus improves the efficiency of video editing by the diffusion model.

[0044] Among them, the adjacent frame optical flow motion information carried in the forward optical flow video and the reverse optical flow video can be determined by referring to the following process. In one embodiment, before editing the initial video into a new video using a preset diffusion model according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the reverse optical flow video, it also includes: generating forward optical flow features according to the first frame editing frame and the forward optical flow video; the first frame editing frame is obtained by editing the first frame of the initial video according to the editing information; generating reverse optical flow features according to the last frame editing frame and the reverse optical flow video; the last frame editing frame is obtained by editing the last frame of the initial video according to the editing information; the forward optical flow features and the reverse optical flow features are spliced ​​to obtain the adjacent frame optical flow motion information, and it can be seen that the adjacent frame optical flow motion information includes optical flow motion information in both the forward and reverse directions; the editing information, the encoding features of the initial video, the adjacent frame optical flow motion information and the random noise are input into the preset diffusion model. The random noise is the pure Gaussian noise that needs to be added after the preset diffusion model training is completed.

[0045] In order to make the forward optical flow features and the reverse optical flow features consistent in time sequence, in one embodiment, the forward optical flow features and the reverse optical flow features are spliced, including: arranging the reverse optical flow features in reverse order so that the forward optical flow features and the reverse optical flow features are consistent in time sequence, and then splicing the forward optical flow features with the reverse optical flow features arranged in reverse order.

[0046] It should be noted that, in order to ensure the effect of the new video obtained by editing, the first frame and the last frame of the initial video can be edited respectively, and then the first frame edited frame and the last frame edited frame are displayed to the user. If the user is satisfied with the effect of the first frame edited frame and the last frame edited frame, then the subsequent steps of this embodiment are executed to edit the entire initial video, thereby avoiding unnecessary waste of resources. Any image editing model can be used here to edit the first frame and the last frame of the initial video respectively. In one embodiment, the first frame and the last frame of the initial video can be edited according to the editing information using an editing model (which can be any image editing model) to obtain a first frame editing frame and a last frame editing frame; the first frame editing frame and the last frame editing frame are displayed; after the first frame editing frame and the last frame editing frame are confirmed, the initial video is processed by a preset optical flow network to obtain a forward optical flow video; the initial video is processed by an optical flow network to obtain a reverse optical flow video; and the initial video is edited into a new video using a preset diffusion model according to the editing information and the optical flow motion information of adjacent frames carried in the forward optical flow video and the reverse optical flow video. After receiving a denial instruction for the first frame editing frame and / or a denial instruction for the last frame editing frame, the editing information and / or the editing model are changed, and the first frame and the last frame of the initial video are re-edited, and the re-edited first frame editing frame and the last frame editing frame are displayed, so that the user is satisfied with the effects of the first frame editing frame and the last frame editing frame, and then the entire initial video is edited.

[0047] The aforementioned editing model can be an image editing model based on a generative adversarial network, an image editing model based on a variational autoencoder, or an image editing model based on a diffusion model.

[0048] In one example, the training process of the preset diffusion model includes: constructing multiple triplets including the original video, the editing instruction (that is, the editing information) and the edited video; using the multiple triplets to train the video editing capability of the initial diffusion model to obtain the preset diffusion model. Among them, a triplet is a training data. In one example, a triplet may include: O, P, Q, O represents the original video, P represents the editing instruction for the original video, and Q represents the edited video obtained after the original video O is modified according to the editing instruction P.

[0049] In one embodiment, a plurality of triplets including original videos, editing instructions, and edited videos are constructed, including: using a video description model to output corresponding text descriptions for a plurality of original videos respectively to obtain a plurality of text descriptions; after optimizing the plurality of text descriptions respectively, generating corresponding editing instructions according to the optimized text descriptions to obtain a plurality of editing instructions; using a video reconstruction model to respectively determine static features and time-domain dynamic features in each original video; constructing a plurality of data groups including original videos, corresponding editing instructions, corresponding static features, and corresponding time-domain dynamic features; for each data group, editing the static features in the current data group according to the editing instructions in the current data group to obtain static editing features (any image editing model can be used here to implement: editing the static features in the current data group according to the editing instructions); mapping the time-domain dynamic features in the current data group to the static editing features to obtain edited videos corresponding to the original videos in the current data group; constructing a plurality of triplets including original videos, corresponding editing instructions, and corresponding edited videos. If there are 10 original videos, then the video description model is used to output 10 text descriptions accordingly, and then the natural language model can be used to optimize these 10 text descriptions, and the natural language model is used to generate corresponding 10 editing instructions according to the optimized 10 text descriptions, and then the video reconstruction model is used to determine the static features and time-domain dynamic features of the 10 original videos; then the static features of the current original video are edited according to the editing instructions of each original video to obtain the static editing features of the current original video, and the time-domain dynamic features of the current original video are mapped to the static editing features of the current original video to obtain the edited video corresponding to the current original video; then 10 triplets including the original video, the corresponding editing instructions and the corresponding edited video can be constructed. In one embodiment, optimizing multiple text descriptions separately includes: inputting the current text description and the corresponding merged description generated by the current text description into the natural language model, so that the natural language model outputs the optimization result of the current text description.

[0050] It should be noted that the video description model may include: a video encoder and a natural language model. The video encoder may implement functions such as image coding, vector alignment, temporal position coding, video adaptation and feature mapping to extract video features and align the video features with the input data dimension of the natural language model. The natural language model here outputs, clusters and merges the segment description for the input data, further merges the corresponding video segments according to the merging of the segment descriptions, and selects the target frame to generate the text description of the entire video frame. In one embodiment, the process of outputting the corresponding text description for any original video using the video description model includes: outputting the corresponding segment description for the k video segments in the current original video respectively, obtaining k segment descriptions, where k is a preset natural number; clustering the k segment descriptions to obtain a clustering result, for example, obtaining 8 categories; merging the k video segments and merging the k segment descriptions according to the clustering result to obtain S merged video segments and the corresponding merged description; generating the corresponding text description of the current original video according to the target frame in each merged video segment. If 8 categories are obtained, then S is equal to 8, and then the target frames are selected from the 8 merged video segments to generate the text description of the entire video frame. The target frame may be a video frame at the middle position of each merged video segment (i.e., the median of the merged video segment), or a video frame at one-third position or other positions. In short, the target frame is one or more video frames determined from each merged video segment that best represent the merged video segment.

[0051] In one embodiment, generating a text description corresponding to the current original video according to the target frames in each merged video segment includes: selecting at least one target frame in each merged video segment; encoding the image features and time sequence features of the selected target frame as fusion features; and obtaining a text description corresponding to the current original video based on the fusion features. For example, if one target frame is selected in each of the eight merged video segments, eight target frames are obtained, and a text description corresponding to the current original video can be obtained based on the time sequence of the eight target frames and the image features of the eight target frames.

[0052] It should be noted that the video reconstruction model includes two parts: a deformable network and a static network. The static network is mainly used to extract static features, and the deformable network is mainly used to extract time-domain dynamic features. In one embodiment, the process of processing each original video using the video reconstruction model includes: extracting corresponding static features from each original video; splicing the static features of any original video with the time vector of the current original video, and deforming and mapping the splicing result according to the optical flow features of the current original video to obtain the time-domain dynamic features of the current original video.

[0053] It can be seen that this embodiment trains the diffusion model to perceive the temporal motion related information in the real initial video, which not only enables the diffusion model to have the ability to edit real videos, but also improves the video editing accuracy with the help of the temporal motion related information in the video. This solution does not require adjustment of model parameters, saves computing resources and model inference time, and thus improves the efficiency of video editing by the diffusion model.

[0054] Referring to the above, the training data set of the diffusion model is the triple {(x s ,x0,c)},x s represents the original video, x0 represents the edited video obtained by editing the original video, and c represents the content of the editing information.

[0055] In order to construct this triplet, we can crawl real videos on the Internet to form a real video data set S, and then use the video description model to predict the video description text for each video in the set, and we can get the following output: . Where s i represents the i-th sample in the set S, represents the text description of the kth video clip in the i-th sample predicted by the video description model, Indicates that the kth video segment is in s i See Figure 3 The video description model includes: a video encoder and a natural language model (i.e., a large language model). The video encoder can encode the visual features of the input video and then input them into the large language model, so that the large language model outputs the text descriptions corresponding to the preset number S (the preset number of clustering categories) of segments and the text description for the entire video. The preset number S of segments and their corresponding text descriptions can be referred to Figure 3 The output result in the format of: time period + text.

[0056] in, There is no limit to the method of performing semantic similarity clustering on the text description corresponding to each segment in , for example, using a text encoder to obtain the text features of the text description corresponding to each segment, and then using kmeans clustering on these text features to merge repeated text content, and obtain the class where S text descriptions belong. Each text description in a class can be summarized into one, so it can be considered that S merged text descriptions are obtained. Then, according to the clustering results, the k video clips of each video are merged into S clips. With the help of the large language model, the middle frames in the S clips are input into the large language model to obtain the video text description of the entire video. The text description of the entire video can be found at Figure 4 The output results in .

[0057] like Figure 4 As shown. After the image encoder in the video encoder encodes the image features, the query vector randomizes the initial parameter to q, which is used to determine the time dimension of the S intermediate frames to obtain the solidified vector h. For example, if the time dimension is 10, the time dimension of each intermediate frame is 10. In other words, the time length of the video frames taken from the S fragments cannot exceed q. Of course, it is not necessary to take only the intermediate frames, and any frame between the 1 / 3 position and the 1 / 2 position can also be taken. If the time dimension is 10, the frame vector B×T×D is mapped to B×10×D through the queried q. B represents the batch size, D represents the feature extraction vector of each frame, and T represents the time length of the frame vector. When querying the network, a transformer structure is used to perform a cross-attention operation with the initialized query vector using the frame vector as the conditional vector to obtain a solidified vector h of a certain dimension. Among them, the video adapter, query network and feature mapping network can be trained by contrastive learning. The entire video is converted into a unified dimension after passing through the image encoder, query network, video adapter and feature mapping, such as B×10×d, where B represents batchsize, 10 represents the time dimension, and d represents the dimension of the input character encoding of the large language model, such as 4096. Then, contrastive loss learning is performed with the original text of the video to update the model parameters of the video adapter, query network and feature mapping network.

[0058] Among them, the image features of the middle frame in the video clip corresponding to the cluster sub-description and the query vector q are used to obtain a feature vector f with a fixed dimension through the query network, and the temporal position encoding is added to f to obtain the fused feature fuse to strengthen the temporal order of the features; fuse is mapped to a dimension acceptable to the large language model through a video adapter and feature mapping, and finally a specific description of the entire video is obtained through the large language model.

[0059] The following is the optimization of the video text description: and Combined and fed into any natural language model to output a description of the complete content of the entire video i The prompt words input to the natural language model may be: "Now you are a text summarization expert. You are given a series of sub-segment texts and a complete segment text. Please summarize the sub-segment and complete segment texts and output a complete segment description. Sub-segments: {sub-segment 1}{sub-segment 2}{sub-segment 3}{sub-segment 4}{sub-segment 5}{sub-segment 6}{sub-segment 7}{sub-segment 8}. Complete segment: {complete segment text description}". This step uses the natural language model to optimize the language logic of "a series of sub-segment texts and a complete segment text" to obtain a new text description for the entire video. i .

[0060] Then t i Input it into any natural language model again to get the editing instructions The prompt words input to the natural language model in this step can be: "Now you are given a piece of text. You need to randomly replace the entities, styles, and behaviors in the text. The number of replacements should not exceed three times, and output the replaced text. Text description: {given text description}", using the natural language model based on text description Output the replaced text, which is an editing instruction for the entire video .

[0061] Below is the original video and edit instructions , using the video reconstruction model and the editing model to determine the corresponding edited video Among them, the video reconstruction model is used to determine the static features and time-domain dynamic features in the original video; the editing model is used to edit the static features of the current original video according to the editing instructions of the original video to obtain the static editing features of the current original video, and the time-domain dynamic features of the current original video are mapped to the static editing features of the current original video to obtain the edited video corresponding to the current original video.

[0062] See also Figure 5 ,The training process of the video reconstruction model includes: inputting the original video Here, the original video is regarded as a three-dimensional structure of X, Y, T, where X represents the width of the video, Y represents the height of the video, and T represents the number of video frames in the video. There are a total of X×Y×T data sampling time and space points in such a video. The deformation network and static network in the video reconstruction model are simple multi-layer perceptron networks. Sampling time and space points Input the deformation network to get the deformation vector ; If there are three scale perceptrons in the deformation network, that is , then the feature encoding formula of each perceptron is: , if the output dimension of the perceptron is 256, then the resulting deformation vector The output features of the three scales are concatenated, and the deformation vector of each sampling space-time point is The dimension is 768. The static network is for sampling points Output space grid vector (i.e. static features) and the time vector , splicing the two together, and then passing through the static network to obtain the dynamic and static combined features of the entire video , this feature is used to predict the RGB three-channel color value of the corresponding sampling point. Then, the dynamic and static combined features output by the static network and the color value of the original sampling time and space point are used to predict the RGB three-channel color value of the corresponding sampling point. Calculate the color reconstruction loss. The color reconstruction loss formula is: , L2 represents the second norm.

[0063] In order to enhance the accuracy of dynamic feature extraction, the optical flow between two adjacent frames is calculated and the previously sampled spatiotemporal points are According to the optical flow mapping to , to determine the deformation of the point from the tth frame to the t+1th frame. The mapped sampling spatiotemporal points are passed through the deformation network to obtain the mapping deformation vector (i.e., time-domain dynamic features), and then calculate and The similarity loss of , represents the bi-norm of two deformation vectors, To calculate the confidence probability of each pixel displacement when calculating the optical flow of two adjacent frames, that is Any optical flow algorithm can be used here. In this scheme, the dynamic and static features of the video can be separated through the color reconstruction loss and the similarity loss of the deformation vector, and the self-supervised reconstruction task of the video can be completed. The deformation vector represents the pixel deformation from the static content to each frame of the image.

[0064] According to the above, multiple triples {(x s ,x0,c)},x s represents the original video, x0 represents the edited video obtained by editing the original video, and c represents the content of the editing information. s ,x0,c)} to train the diffusion model.

[0065] The initial diffusion model targets the input triple {(x s ,x0,c)}, first generate a series of Markov chain hidden vectors , this series of latent vectors is gradually transferred to the edited video Add Gaussian noise. The formula for adding Gaussian noise is as follows: , N here represents Gaussian distribution, represents the noise scheduling parameter, which is a value between 0 and 1 indicating the variance of the noise added at time step t. t Usually it increases with the increase of t, indicating the gradual increase of noise intensity; q represents the conditional probability distribution, indicating the change from arrive The model learns the noise that needs to be added and finally obtains a trained diffusion model.

[0066] The trained diffusion model is based on the latent vector , the specific formula of denoising reasoning process is as follows: , is the mean of the model predictions, is the variance of the model prediction (usually simplified to a fixed value). The model learns to predict the noise at each step, gradually removes the noise, and finally recovers .

[0067] Specifically in the process of video content editing, the initial From Gaussian noise Random sampling, combined with the initial input original video , gradually reducing the noise to obtain the modified video In practice, the deviation of the predicted noise is usually used as the specific optimization training function. The specific function is as follows: , E means to find the expectation of the subsequent input function when the subscript is satisfied; Indicates the distribution of data The original video sampled ; represents the noise ϵ sampled from the standard Gaussian distribution N(0,I); represents a time step t randomly sampled from the time step range [1,T]; where is the original input video, Is editing information c, original input video , the deviation (i.e., loss) obtained by neural network inference of the noisy image at input time step t and time t, are the parameters learned by the neural network.

[0068] See also Figure 6 ,The training process of the trained diffusion model includes:

[0069] 1. Parse the video content to get the initial frame and end frame .

[0070] 2. Edit command c and The edited image is obtained through the image editing model . Edit instruction c is: A giraffe is walking on the moon in a space suit.

[0071] 3. Edit command c and The edited image is obtained through the image editing model Note that steps 2 and 3 do not restrict which image editing model is used, but in order to maintain the consistency of the generated data, the same image editing model is generally used.

[0072] 4. Calculate the forward optical flow of the original video to obtain the forward optical flow video .

[0073] 5. Reverse the order of the video frames of the original video, calculate the reverse optical flow, and get the reverse optical flow video .

[0074] 6. and Pass through the encoder to get the encoding vector and , represents the encoder network. and The encoder used is the same, which plays a role in feature compression, for example, converting the original resolution H×W to H / 8×W / 8.

[0075] 7. Encode the vector and Splice to get the forward branch vector .

[0076] 8. Yes and Execute steps 6 and 7 to get the backward branch vector .

[0077] 9. The original video is passed through the encoder to obtain the video vector .

[0078] 10. Perform conditional vector splicing according to the following formula to obtain the conditional splicing vector : ; express Figure 6 The timing reverse operation in Arrange in reverse order in the time dimension.

[0079] 11. It is spliced ​​with random noise, and the editing instruction c is input as the text condition, which is input into the diffusion model for denoising, and finally a denoised vector is obtained. The denoised vector is passed through the decoder to obtain the final edited video.

[0080] It can be seen that the diffusion model obtained in this way is a general video content generation and editing model. The model can directly perform video editing operations on the video data and editing instructions passed in by the user without additional training on the user-input data. In order to keep the model spatial semantic changes while maintaining temporal consistency. This scheme designs a three-branch input. First, the spatial content of the first and last frames of the original video is initially edited. The result of this step can also be added to the user interaction experience in the actual application process. This step will produce the initial spatial content editing. If the user is not satisfied with the spatial content editing, the input conditions can be modified or the image editing model can be replaced. This can greatly improve the user experience and avoid the useless resource consumption caused by the excessive error of the initial spatial content editing. After the user is satisfied with the spatial content editing of the first and last frames, the temporal content consistency is enhanced. The forward optical flow video and the reverse optical flow video are calculated for the original video. The forward optical flow video and the edited frame of the initial frame are compressed to the latent vector by the encoder, and then the vector splicing is completed to obtain the forward branch vector. At the same time, the backward optical flow video and the edited frame of the ending frame are compressed to the latent vector by the encoder, and then the vector splicing is completed to obtain the backward branch vector. The backward branch vector (i.e., reverse optical flow feature) is added to the forward branch vector (i.e., forward optical flow feature) after time reversal to obtain the motion splicing vector (i.e., optical flow motion information of adjacent frames). These two branches enhance the consistency of temporal content enhancement from the two dimensions of the video, forward and backward.

[0081] In order to maintain the temporal and spatial consistency of the overall video, the third branch inputs the original video and is also encoded into the latent vector space by the encoder. After the motion splicing vector, the original video vector and the vector of random noise are spliced, the editing instruction is used as a text condition, and then input into the diffusion model to complete the denoising of the corresponding noise, and finally decoded by the decoder to obtain the final edited video.

[0082] It can be seen that on the one hand, this solution introduces video construction training data based on real-world videos, and on the other hand, it designs a branched general video content generation and editing model to complete the training of the general model based on the high-quality construction data samples in the previous step. After the training is completed, the user does not need to perform further model fine-tuning training, which greatly improves the model reasoning efficiency.

[0083] A video processing device provided in an embodiment of the present application is introduced below. The video processing device described below can be referenced to other embodiments described in this document.

[0084] The embodiment of the present application discloses a video processing device, including:

[0085] An acquisition module, used to acquire the initial video and edit information;

[0086] A forward optical flow processing module is used to perform forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video;

[0087] A reverse optical flow processing module is used to perform reverse optical flow processing on the initial video using an optical flow network to obtain a reverse optical flow video;

[0088] The editing module is used to edit the initial video into a new video by using a preset diffusion model according to the editing information and the optical flow motion information of adjacent frames carried in the forward optical flow video and the backward optical flow video.

[0089] In one implementation, the forward optical flow processing module is specifically used to:

[0090] The optical flow network is used to calculate the optical flow values ​​between adjacent frames from the first frame to the last frame in the initial video, and the optical flow values ​​are converted into corresponding optical flow maps, and the optical flow maps are summarized to obtain the forward optical flow video;

[0091] Accordingly, the reverse optical flow processing module is specifically used for:

[0092] The optical flow network is used to calculate the optical flow values ​​between adjacent frames in the direction from the last frame to the first frame in the initial video, and the optical flow values ​​are converted into corresponding optical flow maps. The optical flow maps are summarized to obtain the reverse optical flow video.

[0093] In one embodiment, it further includes:

[0094] The feature processing module is used to generate forward optical flow features according to the first frame editing frame and the forward optical flow video; the first frame editing frame is obtained by editing the first frame of the initial video according to the editing information; the reverse optical flow features are generated according to the last frame editing frame and the reverse optical flow video; the last frame editing frame is obtained by editing the last frame of the initial video according to the editing information; the forward optical flow features and the reverse optical flow features are spliced ​​to obtain the optical flow motion information of adjacent frames; the editing information, the coding features of the initial video, the optical flow motion information of adjacent frames and random noise are input into a preset diffusion model.

[0095] In one embodiment, the feature processing module is specifically used to:

[0096] The reverse optical flow features are arranged in reverse order, and the forward optical flow features and the reverse optical flow features after reverse order are concatenated.

[0097] In one embodiment, it further includes:

[0098] A trial editing module is used to use the editing model to edit the first frame and the last frame of the initial video according to the editing information to obtain the first frame editing frame and the last frame editing frame; display the first frame editing frame and the last frame editing frame; after the first frame editing frame and the last frame editing frame are confirmed, perform forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video; perform reverse optical flow processing on the initial video using the optical flow network to obtain a reverse optical flow video; use a preset diffusion model to edit the initial video into a new video according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the reverse optical flow video.

[0099] In one embodiment, the trial editing module is further used to:

[0100] After receiving a first frame editing denial instruction and / or a last frame editing denial instruction, the editing information and / or the editing model are changed, and the first frame and the last frame of the initial video are re-edited.

[0101] In one embodiment, the training process of the preset diffusion model includes:

[0102] Constructing multiple triplets including original videos, editing instructions, and edited videos;

[0103] The video editing capability of the initial diffusion model is trained using multiple triplets to obtain a preset diffusion model.

[0104] In one implementation, constructing a plurality of triplets including an original video, an editing instruction, and an edited video includes:

[0105] Using the video description model, outputting corresponding text descriptions for the multiple original videos respectively to obtain multiple text descriptions;

[0106] After optimizing the multiple text descriptions respectively, corresponding editing instructions are generated according to the optimized text descriptions to obtain multiple editing instructions;

[0107] The video reconstruction model is used to respectively determine the static features and the temporal dynamic features in each original video;

[0108] Constructing a plurality of data groups including original videos, corresponding editing instructions, corresponding static features, and corresponding temporal dynamic features;

[0109] For each data group, the static features in the current data group are edited according to the editing instructions in the current data group to obtain the static editing features; the time domain dynamic features in the current data group are mapped to the static editing features to obtain the edited video corresponding to the original video in the current data group;

[0110] Construct multiple triplets including original videos, corresponding editing instructions and corresponding edited videos.

[0111] In one embodiment, the process of outputting a corresponding text description for any original video using the video description model includes:

[0112] Output corresponding segment descriptions for k video segments in the current original video respectively, and obtain k segment descriptions;

[0113] Cluster k fragment descriptions to obtain clustering results;

[0114] Merge k video clips and k clip descriptions according to the clustering results to obtain S merged video clips and corresponding merged descriptions;

[0115] Generate a text description corresponding to the current original video according to the target frames in each merged video segment.

[0116] In one implementation, generating a text description corresponding to the current original video according to the target frame in each merged video segment includes:

[0117] Selecting at least one target frame in each merged video segment;

[0118] Encode the image features and time sequence features of the selected target frame into fusion features;

[0119] Based on the fused features, the corresponding text description of the current original video is obtained.

[0120] In one implementation, optimizing multiple text descriptions separately includes:

[0121] The current text description and the corresponding merged description generated by the current text description are input into the natural language model, so that the natural language model outputs the optimization result of the current text description.

[0122] In one implementation, the process of processing each original video using the video reconstruction model includes:

[0123] Extract corresponding static features from each original video;

[0124] The static features of any original video are spliced ​​with the time vector of the current original video, and the splicing result is deformed and mapped according to the optical flow features of the current original video to obtain the time domain dynamic features of the current original video.

[0125] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0126] It can be seen that this embodiment provides a video processing device, which trains the diffusion model to perceive the temporal motion related information in the real initial video. It not only enables the diffusion model to have the ability to edit real videos, but also improves the video editing accuracy with the help of the temporal motion related information in the video. This solution does not require adjusting the model parameters, saves computing resources and model inference time, and thus improves the efficiency of the diffusion model in editing videos.

[0127] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below can be referenced to other embodiments described in this document.

[0128] The present application discloses an electronic device, including:

[0129] Memory, used to store computer programs;

[0130] A processor is used to execute the computer program to implement the method disclosed in any of the above embodiments.

[0131] Furthermore, the present application also provides an electronic device. The electronic device can be Figure 7 The server shown can also be Figure 8 Terminal shown. Figure 7 and Figure 8 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0132] Figure 7 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the relevant steps in the video processing disclosed in any of the aforementioned embodiments.

[0133] In this embodiment, the power supply is used to provide working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0134] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer programs and data, etc. The storage method can be temporary storage or permanent storage.

[0135] The operating system is used to manage and control the hardware devices and computer programs on the server to realize the operation and processing of the data in the memory by the processor, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the video processing method disclosed in any of the above embodiments, the computer program can also further include computer programs that can be used to complete other specific tasks. In addition to data such as application update information, the data can also include data such as application developer information.

[0136] Figure 8 A schematic diagram of the structure of a terminal provided in an embodiment of the present application, the terminal may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer, etc.

[0137] Generally, the terminal in this embodiment includes: a processor and a memory.

[0138] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0139] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is at least used to store the following computer program, wherein, after the computer program is loaded and executed by the processor, it can implement the relevant steps in the video processing method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, update information of the application.

[0140] In some embodiments, the terminal may also include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0141] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than those shown in the figure.

[0142] A computer-readable storage medium provided in an embodiment of the present application is introduced below. The computer-readable storage medium described below can be cross-referenced with other embodiments described herein.

[0143] A computer-readable storage medium is used to store a computer program, wherein the computer program, when executed by a processor, implements the video processing method disclosed in the above-mentioned embodiment.

[0144] Among them, the non-volatile storage medium is a computer-readable non-volatile storage medium, which, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system, computer programs and data, etc., and the storage method can be temporary storage or permanent storage. In an exemplary embodiment, the above-mentioned computer-readable storage medium can include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0145] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be cross-referenced with other embodiments described in this document.

[0146] A computer program product comprises a computer program / instruction, which implements the steps of the above-disclosed video processing method when executed by a processor.

[0147] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned video processing methods is implemented.

[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0149] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of non-volatile storage medium known in the art.

[0150] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: include: Get initial video and editing information; Performing forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video; Using the optical flow network to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video; The initial video is edited into a new video by using a preset diffusion model according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the backward optical flow video.

2. The method according to claim 1, characterized in that Performing forward optical flow processing on the initial video using a preset optical flow network to obtain a forward optical flow video, including: Calculating the optical flow values ​​between adjacent frames in the direction from the first frame to the last frame in the initial video using the optical flow network, converting the optical flow values ​​into corresponding optical flow graphs, and summarizing the optical flow graphs to obtain the forward optical flow video; Accordingly, the optical flow network is used to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video, including: The optical flow network is used to calculate the optical flow values ​​between adjacent frames in the direction from the last frame to the first frame in the initial video, and the optical flow values ​​are converted into corresponding optical flow maps, and the optical flow maps are summarized to obtain the reverse optical flow video.

3. The method according to claim 2, characterized in that Before editing the initial video into a new video by using a preset diffusion model and according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the backward optical flow video, the method further includes: Generate a forward optical flow feature according to a first frame editing frame and the forward optical flow video; the first frame editing frame is obtained by editing the first frame of the initial video according to the editing information; Generate a reverse optical flow feature according to a tail frame editing frame and the reverse optical flow video; the tail frame editing frame is obtained by editing the tail frame of the initial video according to the editing information; The forward optical flow feature and the reverse optical flow feature are concatenated to obtain the adjacent frame optical flow motion information; The editing information, the encoding features of the initial video, the adjacent frame optical flow motion information and random noise are input into the preset diffusion model.

4. The method according to claim 3, characterized in that The forward optical flow feature and the reverse optical flow feature are concatenated, including: The reverse optical flow features are arranged in reverse order, and the forward optical flow features and the reverse optical flow features after the reverse order arrangement are spliced.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Using the editing model, the first frame and the last frame of the initial video are edited according to the editing information to obtain a first frame editing frame and a last frame editing frame; Display the first frame editing frame and the last frame editing frame; After the first edited frame and the last edited frame are confirmed, forward optical flow processing is performed on the initial video using a preset optical flow network to obtain a forward optical flow video; Using the optical flow network to perform reverse optical flow processing on the initial video to obtain a reverse optical flow video; The step of editing the initial video into a new video by using a preset diffusion model according to the editing information and the adjacent frame optical flow motion information carried in the forward optical flow video and the backward optical flow video.

6. The method according to claim 5, characterized in that Also includes: After receiving the denial instruction of the first frame editing frame and / or the denial instruction of the last frame editing frame, the editing information and / or the editing model are changed, and the first frame and the last frame of the initial video are re-edited.

7. The method according to any one of claims 1 to 4, characterized in that: The training process of the preset diffusion model includes: Constructing multiple triplets including original videos, editing instructions, and edited videos; The video editing capability of the initial diffusion model is trained using the multiple triplets to obtain the preset diffusion model.

8. The method according to claim 7, characterized in that Construct multiple triplets including original video, editing instructions and edited video, including: Using the video description model, outputting corresponding text descriptions for the multiple original videos respectively to obtain multiple text descriptions; After respectively optimizing the multiple text descriptions, corresponding editing instructions are generated according to the optimized text descriptions to obtain multiple editing instructions; The video reconstruction model is used to respectively determine the static features and the temporal dynamic features in each original video; Constructing a plurality of data groups including original videos, corresponding editing instructions, corresponding static features, and corresponding temporal dynamic features; For each data group, the static features in the current data group are edited according to the editing instructions in the current data group to obtain the static editing features; the time domain dynamic features in the current data group are mapped to the static editing features to obtain the edited video corresponding to the original video in the current data group; Construct multiple triplets including original videos, corresponding editing instructions and corresponding edited videos.

9. The method according to claim 8, characterized in that The process of outputting a corresponding text description for any original video using the video description model includes: Output corresponding segment descriptions for k video segments in the current original video respectively, and obtain k segment descriptions; Clustering the k fragment descriptions to obtain a clustering result; Merging k video segments and k segment descriptions according to the clustering result to obtain S merged video segments and corresponding merged descriptions; Generate a text description corresponding to the current original video according to the target frames in each merged video segment.

10. The method according to claim 9, characterized in that Generate a text description of the current original video based on the target frames in each merged video segment, including: Selecting at least one target frame in each merged video segment; Encode the image features and time sequence features of the selected target frame into fusion features; A text description corresponding to the current original video is obtained based on the fusion features.

11. The method according to claim 9, characterized in that Optimizing the multiple text descriptions respectively includes: The current text description and the corresponding merged description generated by generating the current text description are input into a natural language model, so that the natural language model outputs an optimization result of the current text description.

12. The method according to claim 8, characterized in that The process of processing each original video using the video reconstruction model includes: Extract corresponding static features from each original video; The static features of any original video are spliced ​​with the time vector of the current original video, and the splicing result is deformed and mapped according to the optical flow features of the current original video to obtain the time domain dynamic features of the current original video.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images