Automatic video mixing and cutting method based on generative model
By applying the sequence-sequence generation model of the generative model in video mixing and cutting, combining the multimodal image-text model and segmented training method, the semantic understanding and model training problems in the existing technology are solved, and high quality is achieved. Stable video automatic mixing effect.
Patent Information
- Application Number
- CN202411049550.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-20
AI Technical Summary
The existing automatic mixed video clipping method has problems such as inaccurate semantic understanding, inconsistent style, low quality of generated videos, and difficult to converge model training.
Using a sequence-sequence generation model based on a generative model, by establishing a video library, training the beginning model and continuing the writing model, using multimodal image-text model and autoregression method to splice video clips and audio files, combined with segmentation Training and joint training methods solve the problems of semantic understanding and model training errors.
It realizes video mixed clipping with accurate semantics and consistent style, reduces training errors, improves the stability and quality of video generation, and meets users' video needs.
Smart Images
Figure CN119172589A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video remixing, and in particular to a method for automatically remixing videos based on a generative model. Background Art
[0002] Recently, remixed videos have emerged as a new force and are widely popular on various online platforms. Video remixes require authors to use video editing tools to reassemble existing video materials based on their own creative ideas, so as to express their ideas.
[0003] Ordinary remixing methods need to be done manually, which often consumes a lot of manpower and material resources and takes a long time; sometimes the quality of the remixed video is not high due to the limited materials available to the author; as the scale of video materials expands and audience demand increases, manual remixing can no longer meet the demand.
[0004] In recent years, a variety of automatic video remixing methods have been proposed to address the shortcomings of manual video remixing. However, many of these methods require manual intervention and adjustment after generating preview results; or they fail to understand semantics and misunderstand references and modifications in language; or the generated videos are not stable enough and have large style differences.
[0005] Generative models are a technology that has attracted much attention in the field of artificial intelligence in recent years. They are widely used in fields such as speech synthesis and image generation. They can learn the data generation process and generate new data samples by learning the probability distribution of input data.
[0006] In view of the defects of the existing methods, the present invention applies a generative model to video remixing, allowing the model itself to understand the semantics and style, thereby generating a remixed video with accurate semantics and consistent style.
[0007] Compared with the above methods, the method proposed in the present invention has the following characteristics:
[0008] 1. The generative model is used to understand semantics more accurately, and the generated mixed-cut videos meet user expectations;
[0009] 2 The input and output of the generative model are feature sequences, and ordinary model training methods are difficult to converge. The present invention solves the difficulty of model training. Summary of the invention
[0010] The purpose of the present invention is to provide a method for automatically mixing videos based on a generative model, which has the advantages of using a sequence-sequence generative model to complete automatic video mixing and solves the problems raised in the above background.
[0011] To achieve the above object, the present invention provides the following technical solution: a method for automatically mixing videos based on a generative model, the method comprising the following steps:
[0012] S1: Establish a video library. First, collect a large number of original videos, decode the original videos, calculate the image similarity, and use the image features as the representative features of the video;
[0013] S2: training model, including the start model and the continuation model, using the sequence-sequence segmentation method to train the model;
[0014] S3: Synthesize videos based on the model. After the model training is completed, the model is used to stitch together the video clips and audio files to synthesize the final video.
[0015] Preferably, in step S1, the original video includes any one of a movie, a short video, a stage performance, and news.
[0016] Preferably, in step S1, the specific method of establishing the video library is as follows:
[0017] S1.1: The images obtained by video decoding are input into the multimodal image-text model in chronological order. The model outputs a vector, i.e., the features corresponding to each image, which is highly related to the semantics.
[0018] S1.2: Calculate the similarity of corresponding features of adjacent images: If the similarity is less than a threshold, segment the video at the current position, and the adjacent frames belong to different video segments;
[0019] S1.3: For each video clip, the features of the image at its temporal midpoint are used as the representative features of this video clip, so that a video library can be established, which includes a large number of video clips with independent semantics and the representative features of each video clip.
[0020] Preferably, in step S2, the specific process of training the model is as follows:
[0021] S2.1: Prepare data. First, collect a large amount of original videos and perform speech recognition on the videos to obtain the content and start and end time of all sentences in the videos. Then, calculate the midpoint of time based on the start and end time of each sentence and obtain the image at this time point. At this time, you have a text sequence and an image sequence. Finally, use a multimodal image-text model to extract features from each sentence and each picture in the text sequence and image sequence to obtain a text feature sequence and an image feature sequence. This is the data used for model training.
[0022] S2.2: Use segmented training to train the model, use a sequence-sequence generative model, use text feature sequences and image feature sequences for training, and use autoregressive method to generate results. During the initial training, the training error will continue to decrease. After multiple rounds of training, the training error will no longer decrease.
[0023] S2.3: Divide into two model trainings, by training the autoregressive generative model, that is, the output of the previous step is attached to the input of the previous step to form the current input. Since there is an error in the output of the previous step, this method will gradually amplify the error. After the model training of step S2.2 is completed, the model is verified on the test set;
[0024] S2.4: Run forward inference during training and jointly train the two models. The generative model behaves differently during training and prediction.
[0025] Preferably, in step S2.2, segmented training is adopted, that is, in the text feature sequence and image feature sequence of the training data, the first two segments of data are first intercepted to train the model; when the training error is small enough, the first three segments of data are intercepted to train the model, and so on; however, as the length of the data sequence increases, the training time becomes longer and longer, and the training error decreases more and more slowly; finally, the first N segments of data are used to train the model, and the training error is significantly reduced compared with the initial method.
[0026] Preferably, in step S2.3, this problem is solved by training two models, namely, a starting model and a continuation model. The model trained in step S2.2 is used as the starting model, which is only responsible for processing the first N segments of data, that is, the input data is N segments of text features, and the output N segments of results are used to calculate the error with the N segments of image features. A new continuation model is trained, and M segments are randomly intercepted from each training data. The M segments of text features and M-1 segments of image features are used as model inputs, and the output 1 segment of results is used to calculate the error with the Mth segment of image features. For example, when N=10 and M=8, the starting model is responsible for the case where the input text length is less than 10. When the text length is greater than 10, the continuation model predicts the next feature based on the 8 segments of text features and the last 7 segments of features predicted by the model. This process is repeated until the text ends and the error is calculated.
[0027] Preferably, in step S2.4, the reason for the behavioral difference is that the inputs used in inference and training are different. During training, each input comes from a real sample, and during inference, the current input uses the previous output, resulting in a very low error during training, but a high error during forward inference, that is, training can no longer improve the model effect, and may even reduce the model effect.
[0028] Preferably, in step S3, the specific process of using the model to synthesize the video is as follows:
[0029] S3.1: First, when the user inputs N sentences of text, they are converted into corresponding N audio files. At the same time, N text features are obtained through the multimodal image-text model, and the N text features are input into the model to obtain N outputs.
[0030] S3.3: Using each output, search the video library to get the nearest K video clips;
[0031] S3.3: For each sentence, select a number of video clips according to the length of the corresponding audio file, so that the total length of the video clips of each sentence is greater than the length of the audio file;
[0032] S3.4: Finally, the video clips and audio files are spliced together to synthesize the final video.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] 1. The present invention uses a sequence-sequence generative model to complete automatic video mixing. The original sequence-sequence generative model has an input and output that is a sequence composed of serial numbers. When this model is applied, the input and output are converted into a sequence composed of features according to the characteristics of the video mixing itself, that is, the input and output of the model are an N-dimensional text feature sequence and a corresponding N-dimensional video feature sequence. After training, this model can obtain the intrinsic connection between text and video, and between video clips, so as to accurately understand the semantics.
[0035] 2. The present invention uses a new training method, the error is reduced, and a converged model is obtained, which can effectively solve the problem of semantic understanding. In the existing methods, the semantics of the context are often not accurately understood. When demonstrative pronouns such as "it" and "these" appear in the sentence or the subject is omitted, it is often unclear what the subject of the indication is. The sequence-sequence generative model refers to all inputs when predicting the results, and can accurately understand the semantics. A sequence-sequence based generative model is proposed for video remixing, and a large number of text features and video feature sequences are used for training, which better solves this problem.
[0036] 3. The present invention adopts generative model training. Since the input and output of the model are N-dimensional sequences, the following problems arise and are solved during the training process:
[0037] (1) Training error is difficult to reduce
[0038] Directly using the full length of the text feature sequence-video feature sequence for training, the training error remains at a high level for a long time and is difficult to reduce. To address the problem that the model training error is high and difficult to reduce, the present invention uses a segmented training method, first training a shorter length sequence during training; after a period of training, when the training error is no longer significantly reduced, the sequence length is increased for training; through this training method, the error is significantly reduced, and a usable model is obtained;
[0039] (2) Unbalanced distribution of training errors
[0040] During training, the error of the segments at the front of the sequence is lower, while the error of the segments at the back is higher; and as the length of the sequence increases, the error will become larger and larger. To address this problem, the present invention uses two models to predict different parts of the sequence. The beginning model is responsible for predicting based on the beginning of the input text, and the continuation model predicts subsequent results based on the text and the existing prediction results. This process is repeated until the end of the text. The division of labor between the two models can eliminate the imbalance of errors.
[0041] (3) Low training error and large forward inference difference
[0042] The input data text feature sequence - video feature sequence has a low error during training; however, during forward inference, the same text feature sequence is used, and the predicted result is quite different from the real video feature sequence. During training, the next result is predicted based on the real data, and during forward inference, the next prediction is made based on the existing prediction results. Such a difference leads to inconsistent errors. The present invention solves this problem by running forward during training and jointly training two models. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a flow chart of the automatic video mixing method of the present invention;
[0044] Figure 2 Input model diagram for data text_fea0, text_fea1, text_fea2, BOI, img_fea0 of the present invention;
[0045] Figure 3 It is a flow chart of img_fea1_pred1_new predicted by the model of the present invention;
[0046] Figure 4 This is a diagram showing the difference between the input data during training and the input data during forward inference when the present invention predicts img_fea1. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Embodiment 1:
[0049] A method for automatically mixing videos based on a generative model, the method comprising the following steps:
[0050] S1: Establish a video library. First, collect a large number of original videos, decode the original videos, calculate the image similarity, and use the image features as the representative features of the video;
[0051] In this embodiment, the original video in step S1 includes any one of a movie, a short video, a stage performance, and news.
[0052] In this embodiment, the specific method of establishing the video library in step S1 is as follows:
[0053] S1.1: The images obtained by video decoding are input into the multimodal image-text model in chronological order. The model outputs a vector, i.e., the features corresponding to each image, which is highly related to the semantics.
[0054] S1.2: Calculate the similarity of corresponding features of adjacent images: If the similarity is less than a threshold, segment the video at the current position, and the adjacent frames belong to different video segments;
[0055] S1.3: For each video clip, the features of the image at its temporal midpoint are used as the representative features of this video clip, so that a video library can be established, which includes a large number of video clips with independent semantics and the representative features of each video clip.
[0056] S2: training model, including the start model and the continuation model, using the sequence-sequence segmentation method to train the model;
[0057] In this embodiment, the specific process of step S2 training the model is as follows:
[0058] S2.1: Prepare data. First, collect a large amount of original videos and perform speech recognition on the videos to obtain the content and start and end time of all sentences in the video. Then, calculate the midpoint of time based on the start and end time of each sentence and obtain the image at this time point. At this time, you have a text sequence and an image sequence. Finally, use a multimodal image-text model to extract features from each sentence and each picture in the text sequence and image sequence to obtain a text feature sequence and an image feature sequence. This is the data used for model training.
[0059] S2.2: Use a segmented approach to train the model, use a sequence-sequence generative model, use text feature sequences and image feature sequences for training, and use autoregressive methods to generate results. During the initial training, the training error will continue to decrease, and after multiple rounds of training, the training error will no longer decrease.
[0060] The training is conducted in a segmented manner, that is, the first two segments of data in the text feature sequence and image feature sequence of the training data are first extracted to train the model; when the training error is small enough, the first three segments of data are extracted to train the model, and so on; however, as the length of the data sequence increases, the training time becomes longer and longer, and the training error decreases more and more slowly; finally, the first N segments of data are used to train the model, and the training error is significantly reduced compared with the initial method.
[0061] S2.3: Divide into two model trainings. By training the autoregressive generative model, that is, the output of the previous step is attached to the input of the previous step to form the current input. Since there is an error in the output of the previous step, this method will gradually amplify the error. After the model training of step S2.2 is completed, the model is verified on the test set.
[0062] This problem is solved by training two models, namely the starting model and the continuation model. The model trained in step S2.2 is used as the starting model, which is only responsible for processing the first N segments of data, that is, the input data is N segments of text features, and the output N segments of results are used to calculate the error with the N segments of image features. A new continuation model is trained, and M segments are randomly intercepted from each training data. The M segments of text features and M-1 segments of image features are used as model input, and the 1 segment of output results are used to calculate the error with the Mth segment of image features. For example, when N=10 and M=8, the starting model is responsible for the case where the input text length is less than 10. When the text length is greater than 10, the continuation model predicts the next feature based on the 8 segments of text features and the last 7 segments of features predicted by the model. This process is repeated until the end of the text and the error is calculated.
[0063] S2.4: Run forward inference during training and jointly train the two models. The generative model behaves differently during training and prediction.
[0064] The reason for the behavioral difference is that different inputs are used during inference and training. During training, each input comes from a real sample, while during inference, the current input uses the previous output. This results in a very low error during training, but a high error during forward inference. That is, training can no longer improve the model effect and may even reduce the model effect.
[0065] S3: Synthesize videos based on the model. After the model training is completed, the model is used to stitch together the video clips and audio files to synthesize the final video.
[0066] In this embodiment, the specific process of using the model to synthesize the video in step S3 is as follows:
[0067] S3.1: First, when the user inputs N sentences of text, they are converted into corresponding N audio files. At the same time, N text features are obtained through the multimodal image-text model, and the N text features are input into the model to obtain N outputs.
[0068] S3.3: Using each output, search the video library to get the nearest K video clips;
[0069] S3.3: For each sentence, select a number of video clips according to the length of the corresponding audio file, so that the total length of the video clips of each sentence is greater than the length of the audio file;
[0070] S3.4: Finally, the video clips and audio files are spliced together to synthesize the final video.
[0071] Embodiment 2:
[0072] In step S2.4 of this embodiment, during training, data text_fea0, text_fea1, text_fea2, BOI, and img_fea0 are input into the model, and img_fea1_pred is output as the predicted value of img_fea1, where text_fea0, text_fea1, and text_fea2 represent three text features, BOI is a flag, and img_fea0 is an image feature;
[0073] In this embodiment, the model predicts the process of img_fea1_pred1_new;
[0074] First, the data text_fea0, text_fea1, text_fea2, and BOI are input into the model to predict img_fea0_pred;
[0075] Then, img_fea0_pred is appended to the input data to form new input data: text_fea0, text_fea1, text_fea2, BOI, img_fea0_pred;
[0076] Finally, it is fed into the model and outputs img_fea1_pred1_new as the predicted value of img_fea1.
[0077] In this embodiment, when predicting img_fea1, the input data during training is text_fea0, text_fea1, text_fea2, BOI, and img_fea0; the input data during forward inference is text_fea0, text_fea1, text_fea2, BOI, and img_fea0_pred, and there are differences between the two.
[0078] In order to solve the above problems, forward inference is run during training to simulate the real forward process to obtain real input, eliminate the differences between the training stage and the prediction stage, and use the starting model and the continuation model for joint training to reduce errors more quickly.
[0079] In summary, the present invention uses a sequence-sequence generative model to complete automatic video mixing. The original sequence-sequence generative model has an input and output that is a sequence composed of serial numbers. When this model is applied, the input and output are converted into a sequence composed of features according to the characteristics of the video mixing itself, that is, the input and output of the model are an N-dimensional text feature sequence and a corresponding N-dimensional video feature sequence. After training, this model can obtain the intrinsic connection between text and video, and between video clips, so as to accurately understand the semantics.
[0080] The present invention uses a new training method, the error is reduced, and a converged model is obtained, which can effectively solve the problem of semantic understanding. In the existing methods, the semantics of the context are often not accurately understood. When demonstrative pronouns such as "it" and "these" appear in a sentence or the subject is omitted, it is often unclear what the subject of the indication is. The sequence-sequence generative model refers to all inputs when predicting the results, and can accurately understand the semantics. A sequence-sequence based generative model is proposed for video remixing, and a large number of text features and video feature sequences are used for training, which better solves this problem.
[0081] The present invention adopts generative model training. Since the input and output of the model are N-dimensional sequences, the following problems arise and are solved during the training process:
[0082] (1) Training error is difficult to reduce
[0083] Directly using the full length of the text feature sequence-video feature sequence for training, the training error remains at a high level for a long time and is difficult to reduce. To address the problem that the model training error is high and difficult to reduce, the present invention uses a segmented training method, first training a shorter length sequence during training; after a period of training, when the training error is no longer significantly reduced, the sequence length is increased for training; through this training method, the error is significantly reduced, and a usable model is obtained;
[0084] (2) Unbalanced distribution of training errors
[0085] During training, the error of the segments at the front of the sequence is lower, while the error of the segments at the back is higher; and as the length of the sequence increases, the error will become larger and larger. To address this problem, the present invention uses two models to predict different parts of the sequence. The beginning model is responsible for predicting based on the beginning of the input text, and the continuation model predicts subsequent results based on the text and the existing prediction results. This process is repeated until the end of the text. The division of labor between the two models can eliminate the imbalance of errors.
[0086] (3) Low training error and large forward inference difference
[0087] The input data text feature sequence - video feature sequence has a low error during training; however, during forward inference, the same text feature sequence is used, and the predicted result is quite different from the real video feature sequence. During training, the next result is predicted based on the real data, and during forward inference, the next prediction is made based on the existing prediction results. Such a difference leads to inconsistent errors. The present invention solves this problem by running forward during training and jointly training two models.
[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0089] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for automatically mixing videos based on a generative model. Features: The method comprises the following steps: S1: Establish a video library. First, collect a large number of original videos, decode the original videos, calculate the image similarity, and use the image features as the representative features of the video; S2: training model, including the start model and the continuation model, using the sequence-sequence segmentation method to train the model; S3: Synthesize videos based on the model. After the model training is completed, the model is used to stitch together the video clips and audio files to synthesize the final video.
2. The method for automatically mixing videos based on a generative model according to claim 1, Features: In step S1, the original video includes any one of a movie, a short video, a stage performance, and news.
3. The method for automatically mixing videos based on a generative model according to claim 1, Features: In step S1, the specific method of establishing the video library is as follows: S1.1: The images obtained by video decoding are input into the multimodal image-text model in chronological order. The model outputs a vector, i.e., the features corresponding to each image, which is highly related to the semantics. S1.2: Calculate the similarity of corresponding features of adjacent images: If the similarity is less than a threshold, segment the video at the current position, and the adjacent frames belong to different video segments; S1.3: For each video clip, the features of the image at its temporal midpoint are used as the representative features of this video clip, so that a video library can be established, which includes a large number of video clips with independent semantics and the representative features of each video clip.
4. The method for automatically mixing videos based on a generative model according to claim 1, Features: In step S2, the specific process of training the model is as follows: S2.1: Prepare data. First, collect a large amount of original videos and perform speech recognition on the videos to obtain the content and start and end time of all sentences in the videos. Then, calculate the midpoint of time based on the start and end time of each sentence and obtain the image at this time point. At this time, you have a text sequence and an image sequence. Finally, use a multimodal image-text model to extract features from each sentence and each picture in the text sequence and image sequence to obtain a text feature sequence and an image feature sequence. This is the data used for model training. S2.2: Use segmented training to train the model, use a sequence-sequence generative model, use text feature sequences and image feature sequences for training, and use autoregressive method to generate results. During the initial training, the training error will continue to decrease. After multiple rounds of training, the training error will no longer decrease. S2.3: Divide into two model trainings, by training the autoregressive generative model, that is, the output of the previous step is attached to the input of the previous step to form the current input. Since there is an error in the output of the previous step, this method will gradually amplify the error. After the model training of step S2.2 is completed, the model is verified on the test set; S2.4: Run forward inference during training and jointly train the two models. The generative model behaves differently during training and prediction.
5. The method for automatically mixing videos based on a generative model according to claim 4, Features: In the step S2.2, segmented training is adopted, that is, in the text feature sequence and image feature sequence of the training data, the first two segments of data are first intercepted to train the model; when the training error is small enough, the first three segments of data are intercepted to train the model, and so on; however, as the length of the data sequence increases, the training time becomes longer and longer, and the training error decreases more and more slowly; finally, the first N segments of data are used to train the model, and the training error is significantly reduced compared with the initial method.
6. The method for automatically mixing videos based on a generative model according to claim 4, Features: In step S2.3, this problem is solved by training two models, namely, the starting model and the continuation model. The model trained in step S2.2 is used as the starting model, which is only responsible for processing the first N segments of data, that is, the input data is N segments of text features, and the output N segments of results are used to calculate the error with the N segments of image features. A new continuation model is trained, and M segments are randomly intercepted from each training data. The M segments of text features and M-1 segments of image features are used as model inputs, and the 1 segment of results output is used to calculate the error with the Mth segment of image features. For example, when N=10 and M=8, the starting model is responsible for the case where the input text length is less than 10. When the text length is greater than 10, the continuation model predicts the next feature based on the 8 segments of text features and the last 7 segments of features predicted by the model. This process is repeated until the text ends and the error is calculated.
7. The method for automatically mixing videos based on a generative model according to claim 4, Features: The reason for the behavioral difference in step S2.4 is that the inputs used in inference and training are different. During training, each input comes from a real sample, while during inference, the current input uses the previous output. As a result, the error is very low during training, but the error is still high during forward inference. That is, training can no longer improve the model effect, and may even reduce the model effect.
8. The method for automatically mixing videos based on a generative model according to claim 1, Features: In step S3, the specific process of using the model to synthesize the video is as follows: S3.1: First, when the user inputs N sentences of text, they are converted into corresponding N audio files. At the same time, N text features are obtained through the multimodal image-text model, and the N text features are input into the model to obtain N outputs. S3.3: Using each output, search the video library to get the nearest K video clips; S3.3: For each sentence, select a number of video clips according to the length of the corresponding audio file, so that the total length of the video clips of each sentence is greater than the length of the audio file; S3.4: Finally, the video clips and audio files are spliced together to synthesize the final video.