Video generation method, electronic equipment, storage medium and computer program product

Abstract

The invention provides a video generation method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of video processing, and the method comprises the steps: obtaining an original video and guide information; processing the original video and the guide information based on a preset video generation model to generate a target video; wherein the preset video generation model is obtained by iteratively training an initial video generation model based on a difference value between a fusion feature of the sample original video and the sample guide information and a feature of a video corresponding to the fusion feature, and the corresponding video is obtained by processing the fusion feature by using the iterated initial video generation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a video generation method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] The strongest model for current video content generation is Sora, which combines ChatGPT and Defusion models and can automatically generate a new video according to user input text, pictures, videos and other information. Sora is the strongest AI model in the current video generation field, but it has a big limitation, that is, Sora can only generate short videos (no more than 10 minutes). Because the difficulty of data set annotation during Sora model training is great, which leads to insufficient training of Sora model, and the content range generated by Sora is small. SUMMARY

[0003] The present application provides a video generation method, an electronic device, a storage medium and a computer program product.

[0004] The technical solution of the present application is implemented as follows: The present application provides a video generation method, comprising: obtaining an original video and guide information; processing the original video and the guide information based on a preset video generation model to generate a target video; wherein the preset video generation model is obtained by iteratively training an initial video generation model based on the difference between the fusion features of sample original videos and sample guide information and the features of corresponding videos obtained by processing the fusion features using the iteratively trained initial video generation model.

[0005] In the above solution, before processing the original video and the guide information based on the preset video generation model to generate a target video, the method further comprises: obtaining a first sample original video and a first sample guide information; determining a first fusion feature corresponding to the features of the first sample original video and the first sample guide information, and processing the first fusion feature using the initial video generation model to obtain a first sample intermediate video; training the initial video generation model based on the difference between the first intermediate features of the first sample intermediate video and the first fusion features to obtain a first video generation model, and then training the first video generation model based on the difference between the second intermediate features and the second fusion features until the training stops when a predetermined training condition is reached, and determining the preset video generation model.

[0006] In the scheme, the determination of the first fusion feature corresponding to the first sample original video and the first sample guide information respectively, and the processing of the first fusion feature by the initial video generation model to obtain the first sample intermediate video, includes: determining the first sample feature based on the processing of the first sample original video by the video understanding model, and extracting the second sample feature corresponding to the first sample guide information; fusing the second sample feature and the first sample feature to determine the first fusion feature; determining the first sample intermediate video based on the processing of the first fusion feature by the initial video generation model.

[0007] In the scheme, the first video generation model is obtained by training the initial video generation model based on the difference between the first intermediate feature of the first sample intermediate video and the first fusion feature, and then the first video generation model is trained based on the difference between the second intermediate feature and the second fusion feature, and when the predetermined training condition is reached, the training stops, and the preset video generation model is determined, including: determining the first intermediate feature based on the processing of the first sample intermediate video by the video understanding model; adjusting the model parameters of the initial video generation model based on the difference between the first intermediate feature and the first fusion feature to obtain the first video generation model through back propagation; determining the corresponding second intermediate feature based on the processing of the second fusion feature by the first video generation model, and training the first video generation model based on the difference between the second intermediate feature and the second fusion feature to obtain the second video generation model, and when the difference between the Nth intermediate feature and the Nth fusion feature is less than a preset threshold, the training stops, and the N-1 video generation model and the video understanding model are combined as the preset video generation model.

[0008] In the scheme, the method further includes: training the original model to obtain the first original model by taking the first sample feature as input and taking the first sample original video as real label, and then training the first original model by taking the second sample feature as input and taking the second sample original video as real label until the predetermined training condition is reached, thereby obtaining the initial video generation model.

[0009] In the scheme, the method further includes: fusing to obtain a video understanding model based on a plurality of modal feature extraction models; wherein the data types processed by the feature extraction models of different modalities are different.

[0010] In the scheme, the feature extraction model based on multiple modalities is fused to obtain a video understanding model, which includes: Adjusting dimensions of the feature extraction matrices corresponding to the plurality of feature extraction models to the same dimension to obtain a plurality of same dimension matrices; Weighted fusion of the plurality of same dimension matrices to determine the video understanding model.

[0011] The embodiment of the present application also provides an electronic device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps in the above method when executing the computer program.

[0012] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0013] The embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0014] In the embodiment of the present application, the original video and the guide information are obtained; the original video and the guide information are processed based on a preset video generation model to generate a target video; wherein the preset video generation model is obtained by iterative training of an initial video generation model based on the difference between the fusion features of the sample original video and the sample guide information and the features of the corresponding video obtained by processing the fusion features by the initial video generation model after iteration. In this way, the corresponding video is obtained by processing the fusion features by the initial video generation model after iteration, which solves the problem of large difficulty in data set labeling in related technologies, can provide a large amount of training data with rich video content, so that the initial video generation model can be fully trained to obtain the preset video generation model, and the content range of the target video obtained by processing by the fully trained preset video generation model is also larger, and the video duration is also larger. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 Flowchart of the video generation method provided by the embodiment of the present application Figure One ; Figure 2 Flowchart of the video generation method provided by the embodiment of the present application Figure Two ; Figure 3 Flowchart of the video generation method provided by the embodiment of the present application Figure Three ; Figure 4 Flowchart of the video generation method provided by the embodiment of the present application Figure Four ; Figure 5A flowchart of a video generation method provided by an embodiment of the present application Figure Five ; Figure 6 An effect diagram of the video generation method provided by the embodiment of the present application Figure 7 A structural diagram of a video generation device provided by the embodiment of the present application Figure 8 A hardware entity diagram of an electronic device provided by the embodiment of the present application DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be further described in detail below in combination with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0017] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0018] If similar descriptions of "first / second" appear in the application file, the following description is added. In the following description, the terms "first\second\third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0020] The embodiments of the present application provide a video generation method, which is applied to a video generation device. The video generation device can include a server, a cloud server or a terminal having corresponding video data processing functions, text information processing functions and audio information processing functions. Please refer to Figure 1 A flowchart of a video generation method provided by an embodiment of the present application Figure One The steps shown in Figure 1 will be described: S101, obtaining an original video and guide information.

[0021] In the embodiments of the present application, the video generation apparatus obtains an original video and guide information from a client.

[0022] The original video is a basic video collected by a user for generating a target video, and the target video can be generated by extending and adjusting the original video.

[0023] The guide information is used to guide the content information, objects, actions of the objects, arrangement hierarchy between the objects, scene content information in the target video, and information such as format, length, canvas size of the target video.

[0024] For example, the content of the original video is that a person walks with a dog on the street. Then the guide information can include: change the dog into a tiger. When the original video and the guide information are input into a preset video generation model, a video of "a person walking with a tiger on the street" is output.

[0025] The guide information can also include a prompt, which can be additional text information, audio information, and image information.

[0026] In S102, the original video and the guide information are processed based on a preset video generation model to generate a target video. The preset video generation model is obtained by iteratively training an initial video generation model based on a difference between a fusion feature of sample original video and sample guide information and a feature of a corresponding video obtained by processing the fusion feature using the initial video generation model after iteration.

[0027] In the embodiments of the present application, the video generation apparatus inputs the original video and the guide information into a trained preset video generation model to generate a target video. Before generating the target video, an initial video generation model can be trained. The sample original video and the sample guide information are obtained, and the features of the sample original video and the sample guide information are extracted and fused to determine a fusion feature. The initial video generation model is used to process the fusion feature to obtain a corresponding video, and the feature of the corresponding video is extracted. The initial video generation model is trained once based on the difference between the fusion feature and the feature of the corresponding video. The fusion feature of the next set of sample original video and sample guide information is determined, the initial video generation model trained once is used to process the fusion feature to obtain a corresponding video, and then the initial video generation model is trained based on the difference between the fusion feature of this training and the feature of the video. The training is stopped when a predetermined training condition is reached, and the preset video generation model is obtained.

[0028] The sample original video can include any type and any format of video. The sample guide information is used to guide the sample original video, and guide the content information, objects, actions of the objects, arrangement levels between the objects, scene content information in the target video, and format, length, canvas size, and the like of the target video.

[0029] The predetermined training condition can include reaching a predetermined training number of times, or a difference between the fusion feature and the feature of the corresponding video being less than a preset threshold.

[0030] The initial video generation model can be trained based on a plurality of sample original videos and the corresponding features of each sample original video.

[0031] In the embodiments of the present application, the original video and the guide information are obtained; the original video and the guide information are processed based on a preset video generation model to generate a target video; the preset video generation model is obtained by iteratively training an initial video generation model based on a fusion feature of a sample original video and sample guide information, and a difference between the fusion feature and a feature of a corresponding video obtained by processing the fusion feature using the iteratively trained initial video generation model. In this way, the corresponding video is obtained by processing the fusion feature using the initial video generation model, which solves the problem of difficulty in labeling a large amount of data in related technologies, and provides a large amount of training data with rich video content, so that the initial video generation model can be fully trained to obtain the preset video generation model, and the content range of the target video obtained by processing using the fully trained preset video generation model is also large, and the video length is also large.

[0032] Please refer to Figure 2 The flowchart of the video generation method provided in the embodiments of the present application is shown in Figure Two The steps shown in Figure 2 will be described: S201, obtaining a first sample original video and first sample guide information.

[0033] In the embodiments of the present application, the video generation device obtains the first sample original video and the first sample guide information.

[0034] The first sample original video can include any type and any format of video. The first sample guide information can include information for guiding the objects, actions of the objects, arrangement levels between the objects, scene content information in the target video, and format, length, canvas size of the generated video content of the first sample original video S202, determining a first fusion feature corresponding to the features of the first sample original video and the first sample guide information, and processing the first fusion feature using the initial video generation model to obtain a first sample intermediate video.

[0035] In the embodiments of the present application, the features of the first sample original video can be extracted by using the multi-modal feature extraction model, and the features of the first sample guide information can be extracted by using the text feature extraction model. Then the two features are fused to determine the first fusion feature. Then the initial video generation model is used to process the first fusion feature to obtain the first sample intermediate video.

[0036] In the embodiments of the present application, the fusion feature can process the dimensions of the two features to the same dimension, and then perform weighted fusion.

[0037] Among them, the data types processed by different feature extraction models in the multi-modal feature extraction model are different. The multi-modal feature extraction model can include a fusion model of an image type feature extraction model, a text type feature extraction model and a sound type feature extraction model.

[0038] S203, training the initial video generation model based on the difference between the first intermediate feature of the first sample intermediate video and the first fusion feature to obtain a first video generation model, and then training the first video generation model based on the difference between the second intermediate feature and the second fusion feature until the training stops when a predetermined training condition is reached, and determining the preset video generation model.

[0039] In the embodiments of the present application, the video generation device can update the model parameters of the initial video generation model based on the difference between the first intermediate feature of the first sample intermediate video and the first fusion feature to obtain a first video generation model, and then use the second fusion feature corresponding to the features of the next group of sample original videos and sample guide information to process the first video generation model to determine the corresponding video, and then extract the second intermediate feature of the video, and use the difference between the second fusion feature and the second intermediate feature for training until a predetermined training condition is reached to stop to obtain a final video generation model. The final video generation model and the multi-modal feature extraction model are combined to obtain a preset video generation model.

[0040] In the embodiments of the present application, by iterating the initial video generation model, a preset video generation model capable of creatively generating new videos according to different prompts (guide information) of users can be trained in an unsupervised manner. The preset video generation model includes a trained video understanding model (multi-modal feature extraction model) and a final video generation model. The user inputs the original video and the prompt into the video understanding model, and the video understanding model extracts the features of the original video and the prompt respectively, and then inputs the fusion of the two features into the final video generation model to generate a target video.

[0041] In this embodiment, a first fusion feature is determined corresponding to the features of the original video and the guidance information of the first sample, respectively. The first fusion feature is then processed using an initial video generation model to obtain an intermediate video of the first sample. Based on the difference between the first intermediate feature and the first fusion feature of the intermediate video, the initial video generation model is trained to obtain a first video generation model. Then, the difference between the second intermediate feature and the second fusion feature is used to train the first video generation model until a predetermined training condition is met, at which point training stops, and a preset video generation model is determined. In this way, by using the initial video generation model to process the fusion feature to obtain the corresponding video, the problem of high dataset annotation difficulty in related technologies is solved. This provides a large amount of training data with rich video content, allowing the initial video generation model to be fully trained to obtain the preset video generation model. Consequently, the target video processed by the fully trained preset video generation model has a larger content range and a longer video duration.

[0042] Please see Figure 3 The above is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure Three , Figure 2 S202 to S203 can also be implemented through S301 to S306, combining Figure 3 The steps shown are explained below: S301. Based on the video understanding model, process the original video of the first sample to determine the features of the first sample, and extract the features of the second sample corresponding to the guidance information of the first sample.

[0043] In this embodiment of the application, the video generation device can use a multimodal video understanding model to process the original video of the first sample to determine the features of the first sample, and then use the video understanding model or the text feature extraction model to process the guidance information of the first sample to determine the features of the second sample.

[0044] In this embodiment of the application, the video generation device can pre-process each original video sample using a video understanding model to obtain the corresponding sample features, and then use each set of sample videos and the corresponding sample features to train an initial video generation model.

[0045] In this embodiment of the application, the video generation device can use the first sample features as input, use the original video of the first sample as the real label to train the original model to obtain the first original model, and then use the second sample features as input to the first original model, use the original video of the second sample as the real label to train the first original model, until the predetermined training conditions are met and then stop, to obtain the initial video generation model.

[0046] This invention yields a multimodal video understanding model M0, whose input is a video (video) and output is the video's features (E0). This allows us to obtain data pairs from video to E0. Correspondingly, these data pairs can be viewed as data pairs from E0 to video, i.e., a one-to-one correspondence between video and E0. Therefore, we can use E0 as the input to the original model and video as the ground truth label output from the original model's training to back-train an initial video generation model, denoted as M1. Both the original model and the video generation model are used to generate videos.

[0047] S302. Merge the second sample feature and the first sample feature to determine the first fused feature.

[0048] In this embodiment of the application, the first sample features and the second sample features can be weighted and fused, or transformed into fusion after the same dimension, to obtain the first fused feature.

[0049] S303. Based on the initial video generation model, process the first fusion feature to determine the intermediate video of the first sample.

[0050] In this embodiment of the application, the video generation device processes the first fusion feature based on the initial video generation model to determine the first sample intermediate video.

[0051] In this embodiment, although an initial video generation model M1 is trained, its training data is too fixed, consisting of data pairs corresponding one-to-one between video<->E0. Therefore, the model lacks creative ability. Thus, in this embodiment, a prompt is added to enhance the model's creativity. For example, the original sample video shows a person walking on the street, and a text prompt (sample guidance information) is added, denoted as the second sample feature Pt (prompt of text). The text information is: "A person is walking on the street with a dog." The problem is how to ensure that after obtaining Pt, M1 generates a video that is "a person is walking on the street with a dog," which is the biggest challenge in training the video generation model. Feature extraction is performed on the prompt, and the second sample feature Pt is added to the first sample feature E0 of the original sample video (using the previous example, video represents a video of a person walking on the street, and prompt is the text information: "A person is walking on the street with a dog"), generating a new video feature E1. Then, the video generation model uses E1 to generate a new video video' (the first sample intermediate video).

[0052] S304. Process the intermediate video of the first sample based on the video understanding model to determine the first intermediate feature.

[0053] In this embodiment of the application, the video generation device processes the intermediate video of the first sample based on a video understanding model to determine the first intermediate feature.

[0054] S305. Based on the difference between the first intermediate feature and the first fused feature, the model parameters of the initial video generation model are adjusted by backpropagation to obtain the first video generation model.

[0055] In this embodiment of the application, the video generation device can adjust the model parameters of the initial video generation model by backpropagation based on the magnitude of the difference to obtain the first video generation model.

[0056] In this embodiment, the video understanding model M0 extracts the video features of video' to obtain E0' (the first intermediate feature). In this case, video' should be "a person walking a dog on a road," and E0' should be as similar as possible to E1. E1 can be directly used to supervise E0', keeping M0 unchanged, and M1 is trained. This supervised training includes making the two vectors E1 and E0' as close as possible, that is, calculating the distance between the two vectors using methods such as L2 norm or cosine, and then adjusting the model parameters of model M1 through backpropagation so that the distance between the model output E1 and the "supervisor" E0' is as small as possible.

[0057] S306. Based on the first video generation model, process the second fusion feature to determine the corresponding second intermediate feature, and use the difference between the second intermediate feature and the second fusion feature to train the first video generation model to obtain the second video generation model. Training stops when the difference between the Nth intermediate feature and the Nth fusion feature is less than a preset threshold. Combine the N-1th video generation model and the video understanding model into the preset video generation model.

[0058] In this embodiment, the features of the original video of the second sample and the features of the guidance information of the second sample can be determined. These two features are fused to obtain a second fused feature. The second fused feature is then processed using a first video generation model to obtain an intermediate video of the second sample, and the corresponding second intermediate feature is determined. The difference between the second intermediate feature and the second fused feature is used to train the first video generation model to obtain a second video generation model. Training stops when the difference between the Nth intermediate feature and the Nth fused feature is less than a preset threshold. The (N-1)th video generation model and the video understanding model are then combined into the preset video generation model.

[0059] In this embodiment, the initial video generation model is used to process the fusion features corresponding to the original video and sample guidance information of each group of samples to obtain the corresponding intermediate video. This solves the problem of the difficulty in data annotation in related technologies and can provide a large amount of training data with rich video content. This allows the initial video generation model to be fully trained to obtain the preset video generation model. Furthermore, the target video obtained by the fully trained preset video generation model has a larger content range and a longer video duration.

[0060] Please see Figure 4 The above is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure Four , will combine Figure 4 The steps shown are explained below: S401. Based on feature extraction models of multiple modalities, a video understanding model is obtained by fusing them; wherein, the feature extraction models of different modalities process different types of data.

[0061] In this embodiment, multiple modal feature extraction models can be obtained. The different modal feature extraction models process different data types. The feature extraction parts of the multiple feature extraction models are fused to obtain a video understanding model.

[0062] Among them, the multimodal feature extraction model can include image feature extraction model, text feature extraction model and audio feature extraction model.

[0063] Image feature extraction models can include state-of-the-art open-source models such as Intervideo2, which outperforms all previous video understanding models across more than 10 datasets and 30 tasks. This patent uses Intervideo2 as an example, employing it as a large-scale video understanding model. Based on the Intervideo2 open-source model, this patent adds more video data relevant to the company's business scenarios for LoRa training. If the computing power is sufficient (the open-source Intervideo2 used 256 A100 chips for training; this application suggests 32 or more A100 Graphics Processing Unit (GPU) chips), the entire Intervideo2 model can be trained directly, resulting in a multimodal video understanding model.

[0064] Text feature extraction models can include large models such as Generative Pre-trained Transformer (GPT) or Baichuan. The capabilities of these large models are well-established.

[0065] Audio feature extraction models can include, for example, the BEATs model. Generally speaking, there are currently no dominant models or methods in audio understanding; most focus on specific domains, such as dialogue, atmosphere, or music. This patent considers dialogue as the primary audio element in the video, so the BEATs model can be used.

[0066] In this embodiment, a video understanding model is obtained by fusing feature extraction models from multiple modalities. Then, the data types corresponding to the advantageous data types of the feature extraction models from multiple modalities can be used to process the corresponding data types in the original video, thereby obtaining more accurate video features for the original video and improving the training accuracy of the model.

[0067] Please see Figure 5 The above is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure Four , Figure 4 S401 in the middle can also be implemented through S501 to S502, which will combine Figure 5 The steps shown are explained below: S501. Adjust the dimension of the feature extraction matrices corresponding to the multiple feature extraction models to the same dimension to obtain multiple matrices of the same dimension.

[0068] In this embodiment, the dimensions of the parameter matrices of the feature extraction parts corresponding to the multiple feature extraction models are inconsistent, so it is necessary to adjust the dimensions of the feature extraction matrices to the same dimension to obtain multiple matrices of the same dimension.

[0069] S502. Weighted fusion of multiple matrices of the same dimension to determine the video understanding model.

[0070] In this embodiment of the application, multiple matrices of the same dimension can be weighted and fused to determine the video understanding model.

[0071] The three modalities mentioned above are three distinct models. We combine the feature extraction components of these three models, sharing only a portion (30%) of the weights, and fuse the final output features of the three models. This fused feature is denoted as E0, forming a large multimodal video understanding model encompassing image, text, and audio, denoted as M0. The weights of the neural network model are simply a set of numerical matrices. Sharing 30% can be understood as the matrices of the three models overlapping, with the overlapping portion accounting for 30% of the total weights.

[0072] In this embodiment, the dimensions of the feature extraction matrices corresponding to the multiple feature extraction models are adjusted to the same dimension, resulting in multiple matrices of the same dimension. These matrices are then weighted and fused to determine the video understanding model. This results in a more accurate parameter matrix for the video understanding model obtained by fusing multiple matrices of the same dimension. Consequently, a more accurate multimodal video understanding model with a more precise parameter matrix can process various data types in the original video, thereby acquiring more accurate video features and improving the training accuracy of the model.

[0073] In this embodiment, addressing the challenges of dataset annotation and limited content range in existing video generation models, a method for training a large-scale multimodal video content generation model is provided. This method utilizes the video understanding model generated by the training process to generate video features, incorporating prompt information into these features for further training. This achieves the effect of obtaining a large-scale video generation model that is more creative and allows for user-defined video generation without requiring extensive dataset annotation. Figure 6 The innovation lies in: 1. Train the video understanding model M0, and use the video features output by the video understanding model as training data to train the video generation model, which reduces the need for dataset annotation; 2. Combining Figure 6 The new method adds a step to retrain the video generation model by incorporating the sample guidance information (prompt): Feature extraction (pt) is performed on the prompt; the trained video understanding model M0 generates video features E0 from the original sample video; the combined feature E1 of E0 and pt is input into the iterative initial video generation model M1 to obtain the generated video video'. Feature extraction (E0') is then performed on the video video' using the video understanding model M0; finally, the initial video generation model M1 is adjusted based on the distance between E1 and E0'. This enhances the creativity of the generated video model, enabling it to generate videos that meet user needs based on the user-input guidance information (prompt).

[0074] 3. The application method of the model is as follows: input the original video and the user's guidance information prompt, and use the video understanding model M0 to extract features from the original video and prompt respectively to obtain the video feature E1 and the prompt feature Ep. Then, after fusing these two features, input them into the trained M1 to generate a new video.

[0075] Please see Figure 7 This is a schematic diagram of the structure of the video generation device provided in the embodiments of this application.

[0076] In this embodiment of the application, a video generation apparatus 800 is provided, including an acquisition unit 801 and a generation unit 802.

[0077] The acquisition unit is used to acquire the original video and guidance information; The generation unit is used to process the original video and the guidance information based on a preset video generation model to generate a target video; The preset video generation model is obtained by iteratively training an initial video generation model based on the fusion features of the original sample video and the sample guidance information, and the difference between the features of the video corresponding to the fusion features. The corresponding video is obtained by processing the fusion features using the iterated initial video generation model.

[0078] In this embodiment of the application, the video generation device 800 is used to acquire the original video of the first sample and the guidance information of the first sample; First fusion features are determined to correspond to the features of the original video of the first sample and the guidance information of the first sample, respectively. The first fusion features are then processed using the initial video generation model to obtain the intermediate video of the first sample. The initial video generation model is trained based on the difference between the first intermediate feature and the first fusion feature of the first sample intermediate video to obtain the first video generation model. Then, the difference between the second intermediate feature and the second fusion feature is used to train the first video generation model until the predetermined training conditions are met, at which point the training stops, and the preset video generation model is determined.

[0079] In this embodiment of the application, the video generation device 800 is used to process the original video of the first sample based on the video understanding model to determine the first sample features, and extract the second sample features corresponding to the guidance information of the first sample. The first fused feature is determined by fusing the second sample feature and the first sample feature; The first fusion feature is processed based on the initial video generation model to determine the intermediate video of the first sample.

[0080] In this embodiment of the application, the video generation device 800 is used to process the intermediate video of the first sample based on a video understanding model to determine the first intermediate feature; Based on the difference between the first intermediate feature and the first fused feature, the model parameters of the initial video generation model are adjusted by backpropagation to obtain the first video generation model; The first video generation model processes the second fusion feature to determine the corresponding second intermediate feature, and uses the difference between the second intermediate feature and the second fusion feature to train the first video generation model to obtain the second video generation model. The training stops when the difference between the Nth intermediate feature and the Nth fusion feature is less than a preset threshold. The N-1th video generation model and the video understanding model are combined into the preset video generation model.

[0081] In this embodiment of the application, the video generation device 800 is used to take the first sample features as input, use the original video of the first sample as the real label to train the original model to obtain the first original model, then use the second sample features as input to the first original model, use the original video of the second sample as the real label to train the first original model, until a predetermined training condition is reached and then stops, to obtain the initial video generation model.

[0082] In this embodiment of the application, the video generation device 800 is used to fuse feature extraction models based on multiple modalities to obtain a video understanding model; wherein, the feature extraction models of different modalities process different types of data.

[0083] In this embodiment of the application, the video generation device 800 is used to adjust the dimension of the feature extraction matrices corresponding to the multiple feature extraction models to the same dimension, thereby obtaining multiple matrices of the same dimension. The video understanding model is determined by weighted fusion of multiple matrices of the same dimension.

[0084] It should be noted that, in the embodiments of this application, if the above-described video generation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a video generation device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0085] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the method on one side of the video generation apparatus 800.

[0086] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0087] It should be noted that, Figure 8 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 8As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein; Processor 901 typically controls the overall operation of electronic device 900.

[0088] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0089] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by the processor 901 of the electronic device 900 to complete the steps in the method of the video generation device 800.

[0090] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0093] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0096] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0097] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video generation method, characterized in that, include: Obtain the original video and guidance information; The original video and the guidance information are processed based on a preset video generation model to generate a target video. The preset video generation model is obtained by iteratively training an initial video generation model based on the fusion features of the original sample video and the sample guidance information, and the difference between the features of the video corresponding to the fusion features. The corresponding video is obtained by processing the fusion features using the iterated initial video generation model.

2. The video generation method according to claim 1, characterized in that, Before generating the target video by processing the original video and the guidance information based on a preset video generation model, the method further includes: Obtain the original video and guidance information of the first sample; First fusion features are determined to correspond to the features of the original video of the first sample and the guidance information of the first sample, respectively. The first fusion features are then processed using the initial video generation model to obtain the intermediate video of the first sample. The initial video generation model is trained based on the difference between the first intermediate feature and the first fusion feature of the first sample intermediate video to obtain the first video generation model. Then, the difference between the second intermediate feature and the second fusion feature is used to train the first video generation model until the predetermined training conditions are met, at which point the training stops, and the preset video generation model is determined.

3. The video generation method according to claim 2, characterized in that, The step of determining the first fusion feature corresponding to the features of the original video of the first sample and the guidance information of the first sample, and processing the first fusion feature using the initial video generation model to obtain the intermediate video of the first sample, includes: Based on the video understanding model, the original video of the first sample is processed to determine the features of the first sample, and the features of the second sample corresponding to the guidance information of the first sample are extracted. The first fused feature is determined by fusing the second sample feature and the first sample feature; The first fusion feature is processed based on the initial video generation model to determine the intermediate video of the first sample.

4. The video generation method according to claim 3, characterized in that, The process involves training an initial video generation model based on the difference between a first intermediate feature and a first fused feature from the first sample intermediate video to obtain a first video generation model, then training the first video generation model using the difference between a second intermediate feature and a second fused feature, until training stops when a predetermined training condition is met, thus determining the preset video generation model. This includes: The first intermediate feature is determined by processing the intermediate video of the first sample based on a video understanding model. Based on the difference between the first intermediate feature and the first fused feature, the model parameters of the initial video generation model are adjusted by backpropagation to obtain the first video generation model; The first video generation model processes the second fusion feature to determine the corresponding second intermediate feature, and uses the difference between the second intermediate feature and the second fusion feature to train the first video generation model to obtain the second video generation model. The training stops when the difference between the Nth intermediate feature and the Nth fusion feature is less than a preset threshold. The N-1th video generation model and the video understanding model are combined into the preset video generation model.

5. The video generation method according to claim 3, characterized in that, The method further includes: The first sample features are used as input, and the original video of the first sample is used as the real label to train the original model to obtain the first original model. Then, the second sample features are used as input to the first original model, and the original video of the second sample is used as the real label to train the first original model. The process continues until a predetermined training condition is met, and then the initial video generation model is obtained.

6. The video generation method according to any one of claims 1 to 5, characterized in that, The method further includes: A video understanding model is obtained by fusing feature extraction models based on multiple modalities; wherein the feature extraction models of different modalities process different types of data.

7. The video generation method according to claim 6, characterized in that, The feature extraction model based on multiple modalities is fused to obtain a video understanding model, including: The dimensions of the feature extraction matrices corresponding to the multiple feature extraction models are adjusted to the same dimension to obtain multiple matrices of the same dimension; The video understanding model is determined by weighted fusion of multiple matrices of the same dimension.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images