Video data generation method, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a video data generation method, electronic equipment, a storage medium and a program product, and the method comprises the steps: respectively converting a cross-language description and a target video clip in a sample pair into a cross-language feature vector and a video feature vector; based on a to-be-trained embedding alignment layer, converting the cross-language feature vector into an alignment embedding vector; based on the aligned embedding vector and the video feature vector, performing parameter optimization on the to-be-trained text video model to obtain a target text video model; aligning the embedded vector and the video feature vector in a semantic space; and inputting the target cross-language description into the target text video model to generate corresponding video data according to the target cross-language description based on the target text video model. By obtaining the alignment embedding vector corresponding to the cross-language feature vector, the understanding ability of the text video model for cross-language semantics is remarkably improved, and thus the accuracy of a video data generation result of the text video model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video data generation method, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, Wensheng video technology has gradually become a hot topic in research and application. Wensheng video technology refers to the process of automatically converting text descriptions into video content using computer programs.

[0003] In the related art, a text description is usually input into a text-based video model, so as to automatically generate video data based on the text-based video model according to the obtained text description.

[0004] However, current text-based video models generally generate video data based on English text descriptions. If the text description entered by the user is in Chinese, it needs to be translated into English first. The translation process may cause the meaning of the original text to be distorted or lost, reducing the accuracy of the video data generation results. Summary of the invention

[0005] The present application provides a video data generation method, electronic device, storage medium and program product to solve the defects of related technologies such as reducing the accuracy of video data generation results.

[0006] The first aspect of the present application provides a method for generating video data, comprising: Obtain a sample data set of a text-based video model to be trained; wherein the sample data set includes a plurality of sample pairs, and the sample pairs include a cross-language description and a target video clip; For any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively; Based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; Based on the aligned embedding vector and the video feature vector, the parameters of the to-be-trained Vincent video model are optimized to obtain a target Vincent video model; wherein the to-be-trained Vincent video model includes the to-be-trained embedding alignment layer, and the aligned embedding vector and the video feature vector are aligned in a semantic space; The target cross-language description is input into the target language video model to generate corresponding video data according to the target cross-language description based on the target language video model.

[0007] In an optional implementation, the step of obtaining a sample data set of a Vincent video model to be trained includes: Get the original video data; Preprocessing the original video data to obtain multiple target video clips; Inputting each of the target video clips into a visual language model to obtain a cross-language description corresponding to each of the target video clips; The target video clips and the cross-language descriptions corresponding to the target video clips are summarized to obtain a sample data set of the Wensheng video model to be trained.

[0008] In an optional implementation manner, the preprocessing of the original video data to obtain a plurality of target video segments includes: In the original video data, target video data whose duration satisfies a preset constraint condition is selected; Performing scene detection on the target video data to obtain a scene detection result of the target video data; According to the scene detection result, segmenting the target video data to obtain a plurality of original video segments, so that each original video segment corresponds to a single scene; The multiple original video segments are filtered to obtain multiple target video segments.

[0009] In an optional implementation manner, filtering the multiple original video segments to obtain multiple target video segments includes: Scoring each of the original video clips; Filtering the multiple original video clips according to the scores of the original video clips to obtain multiple target video clips; The score includes an aesthetic score and an optical flow score, wherein the aesthetic score is used to characterize the quality of the original video clip, and the optical flow score is used to characterize whether the original video clip is a static video clip.

[0010] In an optional implementation, for any sample pair in the sample data set, converting the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively includes: Dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage; For any sample pair in any sample sequence of the fine-tuning stage, the cross-language description and the target video segment in the sample pair are converted into a cross-language feature vector and a video feature vector respectively.

[0011] In an optional implementation, dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage includes: Classifying the sample pairs in the sample data set according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution; According to the model optimization requirements of each fine-tuning stage, determine the required resolution data ratio for each fine-tuning stage; For any of the fine-tuning stages, according to the resolution data ratio requirement of the fine-tuning stage, sample pairs of a corresponding proportion are divided from the sample pair set corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

[0012] In an optional implementation, converting the cross-language feature vector into an aligned embedding vector based on the embedding alignment layer to be trained includes: Inputting the cross-language feature vector into a first alignment block, so as to perform channel dimension compression on the cross-language feature vector based on the first alignment block to obtain a compressed cross-language feature vector; Inputting the compressed cross-language feature vector into a second alignment block, so as to perform deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector; Inputting the deep semantic feature vector into a third alignment block, so as to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block, and obtain an aligned embedding vector; The embedded alignment layer to be trained includes the first alignment block, the second alignment block and the third alignment block.

[0013] In an optional implementation, the step of optimizing parameters of the Wensheng video model to be trained based on the aligned embedding vector and the video feature vector includes: Inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned; Optimizing the parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned; The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

[0014] In an optional implementation, inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned includes: After the aligned embedding vector and the video feature vector are input into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector; The diffusion model to be fine-tuned determines the predicted noise by learning a mapping relationship between the noise vector and the aligned embedding vector.

[0015] In an optional embodiment, the method further includes: The error of the predicted noise output by the diffusion model to be fine-tuned is determined according to the error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

[0016] In an optional implementation, the step of optimizing parameters of the to-be-trained Vincent video model with the goal of reducing the error of the predicted noise output by the to-be-fine-tuned diffusion model includes: In the first fine-tuning stage, the parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; In the second fine-tuning stage and the third fine-tuning stage, the parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the to-be-trained Wensheng video model in the first fine-tuning stage reaches the first round number threshold, the second fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the second fine-tuning stage reaches the second round number threshold, the third fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the third fine-tuning stage reaches the third round number threshold, the current Wensheng video model to be trained is used as the target Wensheng video model.

[0017] In an optional implementation, inputting a target cross-language description into the target language video model to generate corresponding video data based on the target language video model and the target cross-language description includes: Inputting the target cross-lingual description into a text encoder to convert the target cross-lingual description into a target cross-lingual feature vector based on the text encoder; Inputting the target cross-language feature vector into a target embedding alignment layer to convert the target cross-language feature vector into a target aligned embedding vector based on the target embedding alignment layer; While inputting the target alignment embedding vector into the target diffusion model, target noise is input into the target diffusion model, so that the target diffusion model converts the target noise into target video feature encoding according to the target alignment embedding vector; Generate video data corresponding to the target cross-language description according to the target video feature encoding; The target text-generated video model includes the text encoder, the target embedding alignment layer and the target diffusion model.

[0018] A second aspect of the present application provides a video data generating device, comprising: An acquisition module is used to acquire a sample data set of a text video model to be trained; wherein the sample data set includes a plurality of sample pairs, and the sample pairs include a cross-language description and a target video clip; a conversion module, configured to convert, for any sample pair in the sample data set, a cross-language description and a target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively; An alignment module, configured to convert the cross-language feature vector into an aligned embedding vector based on an embedding alignment layer to be trained; An optimization module, configured to optimize parameters of the Vincent video model to be trained based on the aligned embedding vector and the video feature vector to obtain a target Vincent video model; wherein the Vincent video model to be trained includes the embedded alignment layer to be trained, and the aligned embedding vector and the video feature vector are aligned in a semantic space; The generating module is used to input the target cross-language description into the target language video model, so as to generate corresponding video data according to the target cross-language description based on the target language video model.

[0019] A third aspect of the present application provides an electronic device, comprising: at least one processor and a memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0021] A fifth aspect of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0022] The technical solution of this application has the following advantages: The present application provides a video data generation method, electronic device, storage medium and program product, the method comprising: obtaining a sample data set of a Vincent video model to be trained; wherein the sample data set comprises a plurality of sample pairs, the sample pairs comprising a cross-language description and a target video clip; for any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively; based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; based on the aligned embedding vector and the video feature vector, optimizing the parameters of the Vincent video model to be trained to obtain a target Vincent video model; wherein the Vincent video model to be trained comprises an embedding alignment layer to be trained, the aligned embedding vector and the video feature vector are aligned in a semantic space; inputting a target cross-language description into the target Vincent video model, so as to generate corresponding video data based on the target Vincent video model and the target cross-language description. The method provided by the above scheme significantly improves the Vincent video model's ability to understand cross-language semantics by performing semantic space alignment processing on the cross-language feature vector and the video feature vector, thereby improving the accuracy of the video data generation result of the Vincent video model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0024] Figure 1 A schematic diagram of the structure of a video data generation system based on an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a method for generating video data provided in an embodiment of the present application; Figure 3 A schematic diagram of the construction process of the sample data set provided in the embodiment of the present application; Figure 4 A schematic diagram of a classification strategy for an exemplary sample data set provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an exemplary sample sequence of each fine-tuning stage provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of the embedding alignment layer to be trained provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a Wensheng video model to be trained provided in an embodiment of the present application; Figure 8A schematic diagram of the structure of a video data generating device provided in an embodiment of the present application; Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] The above drawings have shown clear embodiments of the present application, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0028] With the rapid development of artificial intelligence technology, text-to-video generation has gradually become a hot topic in research and application. Text-to-video generation refers to the process of automatically converting text descriptions into video content using computer programs. It combines advanced technologies in the two fields of natural language processing (NLP) and computer vision (CV), showing strong application potential. The core task of this technology is to map the semantic information in the text description to the video frame, involving the visualization of multiple elements such as scenes, characters, and actions. This technology is of great significance to content creation, advertising, education and other fields, and can improve creation efficiency and reduce production costs. However, Chinese text-to-video generation technology faces many challenges, including accurate understanding of Chinese semantics, generation of high-quality video content, and high demand for computing resources. Existing technologies are mainly concentrated in the English field, and the application and research in the Chinese environment are relatively lagging behind, and there is a lack of benchmarking and optimization methods for large Chinese models.

[0029] As a cutting-edge technology in the field of artificial intelligence, Vincent video technology realizes the automatic conversion from text description to video content through advanced technologies such as deep learning, generative adversarial networks (GAN), variational autoencoders (VAE) and diffusion models (DM). This technology is of great significance to many fields such as content creation, advertising, and education, and can significantly improve creation efficiency and reduce production costs. However, related technologies still face many challenges in processing complex dynamic scenes in Chinese scenes, capturing attribute changes, generating the correct number of objects, simulating physical phenomena, and understanding cause and effect relationships. Video generation models may also have difficulties in accurately describing events that occur over time, and there are common sense errors in details. In addition, the scarcity of high-quality Chinese video-text data and the high cost of annotation, as well as the lack of unified evaluation standards, make the direction of model optimization unclear.

[0030] Although traditional video models (such as open-sora, open-sora-plan, cogvideox, etc.) have made technical innovations, there are still many mistakes in the generated videos, showing that the technology is not yet mature. Especially in terms of Chinese understanding, due to the use of English text encoders, Chinese cannot be directly input, and Chinese must be translated into English input or the Chinese input prompt word optimizer must be rewritten. These operations may cause the meaning of the original text to be misinterpreted or lost. Using the prompt word optimizer to rewrite Chinese prompt words requires the additional introduction of a large language model (LLM), which increases computing power resource consumption and limits the scale of model training and deployment applications.

[0031] In response to the above problems, an embodiment of the present application provides a video data generation method, an electronic device, a storage medium and a program product, the method comprising: obtaining a sample data set of a Vincent video model to be trained; wherein the sample data set comprises a plurality of sample pairs, the sample pairs comprising a cross-language description and a target video clip; for any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector, respectively; based on an embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; based on the aligned embedding vector and the video feature vector, performing parameter optimization on the Vincent video model to be trained to obtain a target Vincent video model; wherein the Vincent video model to be trained comprises an embedding alignment layer to be trained, the aligned embedding vector and the video feature vector are aligned in a semantic space; inputting a target cross-language description into the target Vincent video model to generate corresponding video data according to the target cross-language description based on the target Vincent video model. The method provided by the above scheme aligns the cross-language feature vector and the video feature vector in semantic space to obtain the aligned embedding vector corresponding to the cross-language feature vector, which significantly improves the Vincent video model's ability to understand cross-language semantics, thereby improving the accuracy of the video data generation results of the Vincent video model.

[0032] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0033] First, the structure of the video data generation system on which this application is based is described: The video data generation method, electronic device, storage medium and program product provided in the embodiments of the present application are suitable for fine-tuning an existing Vincent video model to be trained to improve the Vincent video model's ability to understand cross-language semantics. Figure 1 The figure is a schematic diagram of the structure of the video data generation system based on the embodiment of the present application, which mainly includes a data acquisition device, a model training device and a Vincent video model to be trained. The data acquisition device is used to collect a sample data set and send the obtained sample data set to the model training device. The model training device optimizes the parameters of the Vincent video model to be trained according to the obtained sample data set until the target Vincent video model is obtained.

[0034] The embodiment of the present application provides a video data generation method for fine-tuning an existing Vincent video model to be trained to improve the Vincent video model's ability to understand cross-language semantics. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used to optimize and run the Vincent video model.

[0035] like Figure 2 FIG. 1 is a flow chart of a method for generating video data according to an embodiment of the present application, the method comprising: Step 201, obtain a sample data set of a video model to be trained.

[0036] The sample data set includes multiple sample pairs, and the sample pairs include cross-language descriptions and target video clips. The language descriptions of the training samples used by the currently obtained Vincent video model to be trained in the previous training stage are different from the above cross-language descriptions. For example, if the obtained Vincent video model to be trained is a Vincent video model trained with English description samples, then the cross-language description is a non-English description, such as a Chinese description.

[0037] Specifically, the original video can be obtained from open video websites such as Mixkit, Pixabay and Pexel, and a sample data set of the to-be-trained video model can be constructed based on the obtained original video.

[0038] Step 202: for any sample pair in the sample data set, convert the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively.

[0039] Specifically, the text encoder can be used to capture the semantic information of the cross-language description, and then the corresponding cross-language feature vector can be determined based on the captured semantic information. The video encoder (VAE encoder) is used to convert the input video frame or image (target video clip) into a representation (video feature vector) in the latent space. This latent representation is a compressed, low-dimensional vector that captures the key features of the input video data.

[0040] Step 203: based on the embedding alignment layer to be trained, convert the cross-language feature vector into an aligned embedding vector.

[0041] Specifically, based on the embedding alignment layer to be trained, semantic alignment processing can be performed between the cross-language feature vector and the video feature vector in the semantic space to convert the cross-language feature vector into an aligned embedding vector that is semantically aligned with the video feature vector.

[0042] Step 204 , based on the aligned embedding vector and the video feature vector, the parameters of the to-be-trained video model are optimized to obtain a target video model.

[0043] The to-be-trained video model includes an embedding alignment layer to be trained, and the aligned embedding vector and the video feature vector are aligned in the semantic space.

[0044] It should be noted that the to-be-trained Vincent video model used in the embodiment of the present application is a traditional Vincent model with an embedded alignment layer to be trained added.

[0045] Specifically, based on the aligned embedding vector and the video feature vector, the parameters of the embedding alignment layer to be trained can be optimized to improve the accuracy of the aligned embedding vector provided by it. At the same time, the model parameters of the traditional language model in the language video model to be trained can be fine-tuned so that the final target language video model can accurately understand the semantics of cross-language descriptions.

[0046] Step 205 , input the target cross-language description into the target language video model, so as to generate corresponding video data according to the target cross-language description based on the target language video model.

[0047] Specifically, after obtaining the target language video model, the target cross-language description input by the user is received, and the target cross-language description is input into the target language video model to perform cross-language semantic analysis on the target cross-language description based on the target language video model and generate corresponding video data.

[0048] On the basis of the above embodiment, in order to ensure the reliability of the sample data set of the Vincent video model to be trained, as an implementable method, on the basis of the above embodiment, in one embodiment, obtaining the sample data set of the Vincent video model to be trained includes: Step 2011, obtaining original video data; Step 2012, preprocessing the original video data to obtain multiple target video clips; Step 2013, inputting each target video segment into a visual language model to obtain a cross-language description corresponding to each target video segment; Step 2014: summarize the target video clips and the cross-language descriptions corresponding to the target video clips to obtain a sample data set for the cultural video model to be trained.

[0049] It should be noted that the visual language model can adopt a large visual language model such as qwen-vl-7b-instruction. The large visual language model is used to label the target video clip with a title to obtain a cross-language description corresponding to the target video clip, ensuring that the final sample data set has strong semantic consistency.

[0050] Specifically, the prompt words and the target video clip can be input into the visual language model, where, taking the cross-language description as a Chinese description as an example, the prompt words are "Please describe the content of this video in as much detail as possible, including the objects, scenery, animals, characters and camera movements in the video. Please describe the content of the video and the changes that have occurred in chronological order. This description should be useful for AI to regenerate the video", and the visual language model will output a detailed description of the target video clip in Chinese. However, the visual language model tends to generate certain prefixes for video descriptions, such as "This video shows", "The video shows", "The video is", etc. Therefore, the embodiment of the present application compiles a prefix list that lists all irrelevant beginning character strings, deletes these prefixes in the title string, and finally obtains a sample data set of the Wensheng video model to be trained by summarizing the target video clips and the Chinese descriptions corresponding to each target video clip.

[0051] Specifically, in one embodiment, target video data whose duration satisfies preset constraints can be screened from the original video data; scene detection is performed on the target video data to obtain scene detection results of the target video data; based on the scene detection results, the target video data is segmented to obtain multiple original video segments, so that each original video segment corresponds to a single scene; and the multiple original video segments are filtered to obtain multiple target video segments.

[0052] Specifically, the original video data with a duration of not less than 1 second and not more than 10 seconds can be used as the target video data, that is, the preset constraint condition is that the duration is within 1 second to 10 seconds. The scene detection tool PySceneDetect is further used to detect the video, output the time periods of different scenes of the same video, and divide the original video into shorter segments according to the time periods, that is, to obtain multiple original video segments.

[0053] The scene in the video may change due to various factors, such as changes in the background, scene layout, shooting location, etc., or changes in the character's behavior, action, or combination of characters that appear.

[0054] Specifically, in one embodiment, each original video segment may be scored; and according to the score of each original video segment, multiple original video segments are filtered to obtain multiple target video segments.

[0055] The scores include aesthetic scores and optical flow scores. The aesthetic scores are used to characterize the quality of the original video clips, and the optical flow scores are used to characterize whether the original video clips are static video clips.

[0056] Specifically, Figure 3 As shown, it is a schematic diagram of the construction process of the sample data set provided in the embodiment of the present application. After completing the scene segmentation and obtaining multiple original video segments, the aesthetic score of the original video clips can be estimated, and low-quality video clips, including high blur, low bit rate, pixelation, black and white shots or abstract visual effects, etc., can be filtered out based on the aesthetic score. The optical flow score is estimated for the original video clip, and it is determined whether it is a static video based on the optical flow score. The motion value is evaluated based on the optical flow score, and the static video is discarded. Furthermore, the OCR model can also be used to detect whether there is text in the video and remove videos with too much text. After the final OCR filtering, multiple target video clips are obtained, and then the target video clips are annotated with a large model and the prefix content in the title is filtered to obtain the cross-language description (cross-language title) corresponding to each target video clip, thereby obtaining a sample data set of the Wensheng video model to be trained. Based on the following Figure 3 The video-text pairs (sample pairs) produced by the data processing pipeline shown have high aesthetic quality, large video motion, and strong semantic consistency. In addition, the use of pipeline data processing to annotate the dataset reduces the need for manual annotation and reduces costs. In addition, it makes the annotation content more cross-language semantic context, thereby enhancing the model's ability to understand cross-language semantics.

[0057] Based on the above embodiment, as an implementable manner, based on the above embodiment, in one embodiment, for any sample pair in the sample data set, the cross-language description and the target video clip in the sample pair are converted into a cross-language feature vector and a video feature vector respectively, including: Step 2021, dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage; Step 2022: for any sample pair in the sample sequence of any fine-tuning stage, convert the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively.

[0058] It should be noted that the embodiment of the present application adopts a phased approach to perform multi-stage parameter fine-tuning on the training Wensheng video model.

[0059] Specifically, in one embodiment, the sample pairs in the sample data set can be classified according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution; the resolution data ratio requirement of each fine-tuning stage is determined according to the model optimization requirement of each fine-tuning stage; for any fine-tuning stage, according to the resolution data ratio requirement of the fine-tuning stage, a corresponding proportion of sample pairs is divided from the set of sample pairs corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

[0060] Specifically, the bucket mechanism of Open-Sora can be used to classify the target video data according to resolution, number of frames and aspect ratio to form different "buckets" to adapt to inputs of different sizes, thereby supporting training requirements of multiple resolutions, different aspect ratios and frame rates. Before the start of each training cycle, the sample data set is reshuffled and assigned to the corresponding buckets according to the video characteristics to ensure that the number of samples in each bucket is controlled. The GPU load is balanced by searching for a good batch size for each bucket. The batch size is the number of samples input by a single GPU at one time. Specifically, the corresponding opposite batch size can be set for each bucket according to the resolution size. For example, a larger batch size is set for a bucket with a lower resolution in order to better balance computing resources and model learning efficiency in training at different resolutions.

[0061] Among them, Figure 4As shown, it is a schematic diagram of the classification strategy of an exemplary sample data set provided in an embodiment of the present application. In the embodiment of the present application, the target video clips are assigned to 5 buckets (video classification) with different attributes. The video resolutions represented by these 5 buckets are from low to high [256, 360p, 512, 480p, 720p]. Then all videos are cut to 51 frames, about 2 seconds. All videos are scaled to an aspect ratio of 1:1, and then the corresponding batch size is set according to the resolution size. The resolution of the staged training data is controlled by controlling the proportion of data falling into each bucket at different stages. Through this staged and bucketed training method, the model can gradually learn and adapt to video content of different resolutions and complexities, and ultimately achieve the generation of high-quality videos. This method not only improves the efficiency of training, but also ensures the diversity and quality of generated videos. The "p" in the resolution stands for progressive scan. It indicates the number of pixels in the vertical direction of the video, and also implies the number of pixels in the horizontal direction and the scanning method. For example, 720p means 720×1280 pixels, and 512 means 512×512 pixels.

[0062] Specifically, Figure 5 As shown, it is a schematic diagram of the structure of the sample sequence of each fine-tuning stage provided by the embodiment of the present application, taking the fine-tuning stages as the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage as an example. Among them, the model optimization requirement of the first fine-tuning stage is to achieve basic training, the model optimization requirement of the second fine-tuning stage is to achieve quality training, and the model optimization requirement of the third fine-tuning stage is to achieve detail training.

[0063] In the first fine-tuning stage, the video is scaled so that the target video clips are mainly distributed in the lower resolution 256-360p, and the proportion of resolution data of the five resolutions is 0.4, 0.3, 0.2, 0.1, and 0 respectively. In this stage, by using a larger batch size, a larger learning rate, and a larger number of rounds, this helps the model quickly learn the basic concepts and structures of the video content without being bothered by excessive details and complexity. At this stage, the model can learn basic video generation capabilities, including understanding the basic connection between text descriptions and video content, and the ability to generate low-resolution videos, laying a solid foundation for subsequent high-resolution training.

[0064] In the second fine-tuning stage, the video is scaled so that the target video clips are mainly distributed in the medium resolution 512, and the proportion of resolution data of the five resolutions is 0.1, 0.2, 0.4, 0.2, and 0.1 respectively. In this stage, the model improves the resolution and quality of the generated videos by using a medium batch size, a medium learning rate, and a medium number of rounds. The model is able to generate medium-resolution videos that are more consistent with the text description in content and style, while showing better visual quality and motion coherence.

[0065] The third fine-tuning stage scales the video so that the target video clips are mainly distributed in high-resolution 480p-720p, and the proportion of resolution data at the five resolutions is 0, 0, 0.1, 0.4, and 0.5 respectively. In this stage, the model further improves the resolution and details of the video by using a smaller batch size, a smaller learning rate, and a smaller number of rounds. The model is able to generate high-resolution videos that are not only highly consistent with the text description in terms of content, but also meet high standards in visual details and motion smoothness.

[0066] Through these three stages of fine-tuning, the model is able to gradually improve its ability to generate videos, from basic to advanced, and ultimately achieve the generation of high-quality cross-language videos. This phased training method helps the model focus on different learning goals at each stage, so that it can more effectively master the skills required to generate complex video content.

[0067] Based on the above embodiment, as an implementable manner, based on the above embodiment, as Figure 6 As shown, it is a schematic diagram of the structure of the embedding alignment layer to be trained provided in an embodiment of the present application. In one embodiment, based on the embedding alignment layer to be trained, the cross-language feature vector is converted into an aligned embedding vector, including: Step 2031, inputting the cross-language feature vector into the first alignment block, so as to perform channel dimension compression on the cross-language feature vector based on the first alignment block, and obtain a compressed cross-language feature vector; Step 2032, inputting the compressed cross-language feature vector into the second alignment block, so as to perform deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector; Step 2033: input the deep semantic feature vector into the third alignment block, so as to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block, and obtain the aligned embedding vector.

[0068] Among them, the embedded alignment layer to be trained includes a first alignment block, a second alignment block and a third alignment block. The embodiment of the present application mainly optimizes the parameters of the second alignment block in the embedded alignment layer to be trained.

[0069] Specifically, Figure 6 As shown, the first alignment block and the second alignment block both include a fully connected layer, an activation layer, a regularization layer (Dropout layer) and a normalization layer, and the third alignment block includes a fully connected layer and a regularization layer.

[0070] Specifically, the cross-language feature vector is first input into the first alignment block. After entering the first alignment block, the cross-language feature vector is linearly transformed through the fully connected layer to adjust its dimension, so as to compress the channel dimension to half of the original one. Then, the activation layer introduces nonlinear characteristics, the Dropout layer randomly discards a part of neurons to prevent overfitting, and the normalization layer normalizes the data to accelerate the convergence of the model. After these processes, the compressed cross-language feature vector is obtained. The compressed cross-language feature vector is then input into the second alignment block. In the second alignment block, the compressed cross-language feature vector is deeply semantically transformed through the processing of the fully connected layer, the activation layer, the Dropout layer and the normalization layer. Although the number of channels of the output feature remains unchanged after the processing of these layers, a deeper semantic transformation of the cross-language feature vector is achieved to obtain a deep semantic feature vector. The deep semantic feature vector is finally input into the third alignment block. In the third alignment block, the deep semantic feature vector is mapped back to the original channel of the cross-lingual feature vector, and finally an aligned embedding vector is generated, which realizes the conversion of the cross-lingual feature vector to the aligned embedding vector, ensures the consistency of text and video features in the semantic space, and helps the subsequent model learn the mapping relationship from text to video.

[0071] Based on the above embodiment, as an implementable manner, based on the above embodiment, as Figure 7 As shown, it is a schematic diagram of the structure of the Vincent video model to be trained provided in an embodiment of the present application, the solid line represents the training process, and the dotted line represents the reasoning process. In one embodiment, based on the alignment of the embedding vector and the video feature vector, the Vincent video model to be trained is optimized in parameters, including: Step 2041, inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned; Step 2042, with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned, the parameters of the Wensheng video model to be trained are optimized.

[0072] The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

[0073] It should be noted that the Wensheng video model to be trained also includes a VAE encoder and a text encoder that supports multiple languages. The text encoder can use a pre-trained model mT5 to convert the input cross-language description into an intrinsic embedding representation (cross-language feature vector) that can be understood by the large model. The text encoder can capture the semantic information of the text and provide guidance for subsequent video generation. The text encoder keeps the weights frozen during training and reasoning. Then, through the embedding alignment layer, the text features generated by the text encoder are aligned with the video features to obtain an aligned embedding vector. The consistency of text and video features in the semantic space is ensured, which helps the model learn the mapping relationship from text to video. Taking the English description as a sample training for the Wensheng video model to be trained as an example, due to the distinction between cross-language semantics and English semantics, the embodiment of the present application can mainly train the parameters of the embedding alignment layer during the fine-tuning process, and realize semantic migration without destroying the pre-training knowledge of the diffusion model DiT for image generation, so as to achieve good learning ability of the embedding alignment layer.

[0074] It should be further explained that the diffusion model DiT is a diffusion model combined with the Transformer architecture for image and video generation tasks. It gradually adds noise to the data by simulating the diffusion process, and then learns to reverse the process to construct the required data samples from the noise. The core idea of ​​DiT is to use Transformer as the backbone network of the diffusion model, rather than the traditional convolutional neural network (such as U-Net), to process the potential representation of the image. The DiT architecture is based on the Latent Diffusion Model (LDM) framework, uses Vision Transformer (ViT) as the backbone network, and constructs a scalable diffusion model by adjusting the normalization of ViT. The diffusion model to be fine-tuned used in the embodiment of the present application is an English language video pre-training model, which has been pre-trained on a large amount of English video data. The embodiment of the present application mainly transforms the text encoder and the embedding alignment layer part so that it can adapt to the cross-language semantic environment, that is, by optimizing the structure of the English language video pre-training model, the migration ability of the English language video pre-training model to the cross-language language video model is improved.

[0075] Specifically, in one embodiment, after the aligned embedding vector and the video feature vector are input into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector; the diffusion model to be fine-tuned determines the predicted noise by learning the mapping relationship between the noise vector and the aligned embedding vector.

[0076] Specifically, during the model training process, the diffusion model to be fine-tuned will simulate the forward diffusion process. When the video feature vector is input, Gaussian noise is gradually added to the video feature vector according to the forward diffusion rule. In the first training iteration, only a small amount of noise can be added. As the number of training iterations increases, more and more noise is added, and the video feature vector gradually becomes a noise vector. This process of gradually adding noise enables the model to learn the changes in the video feature vector under different degrees of noise, and then in the reverse denoising process (the process of the model generating video), it can accurately recover the video features corresponding to the cross-language description from the noise, and finally generate a video that meets the requirements.

[0077] That is, the diffusion model to be fine-tuned learns how to find the correspondence between this noise vector and the aligned embedding vector during the inverse denoising process. The goal of the diffusion model to be fine-tuned is to generate the corresponding video (related to the video feature vector) based on the cross-language description (aligned embedding vector). By learning the mapping relationship between the noise vector and the aligned embedding vector, the model can try to predict the noise in the process of adding noise, and then learn how to convert from text features to video features.

[0078] Specifically, in one embodiment, the error of the predicted noise output by the diffusion model to be fine-tuned may be determined according to the error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

[0079] Specifically, the mean squared error (MSE) can be used to measure the error between the predicted noise and the noise added to the video feature vector by the fine-tuned diffusion model. During the training process, by minimizing the MSE loss, the fine-tuned diffusion model learns how to more accurately reconstruct the original latent feature representation from the noise representation.

[0080] Furthermore, in one embodiment, in the first fine-tuning stage, parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; in the second fine-tuning stage and the third fine-tuning stage, parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned.

[0081] Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the Wensheng video model to be trained in the first fine-tuning stage reaches the first round number threshold, it enters the second fine-tuning stage. When the number of sample learning rounds of the Wensheng video model to be trained in the second fine-tuning stage reaches the second round number threshold, it enters the third fine-tuning stage. When the number of sample learning rounds of the Wensheng video model to be trained in the third fine-tuning stage reaches the third round number threshold, the current Wensheng video model to be trained is used as the target Wensheng video model.

[0082] Specifically, in the first fine-tuning stage, the main optimization goal is to optimize the parameters of the embedding alignment layer to be trained. Because in the early stage of training, the focus is to better align the text features (processed by the embedding alignment layer) and the video features in the semantic space. In the second and third fine-tuning stages, not only the parameters of the embedding alignment layer to be trained should be optimized, but also the parameters of the fine-tuning diffusion model should be optimized. Because as the training progresses, the diffusion model itself needs to better learn the mapping relationship between cross-language text and video, so as to more accurately generate video data that conforms to the text description from noise.

[0083] Among them, the performance of the fine-tuned model (target text-based video model) was evaluated, and the evaluation indicators included the matching degree between video content and text description, video realism, action coherence, etc. The experimental results show that after multi-stage fine-tuning and semantic enhancement annotation, the model's performance in cross-language video generation tasks has been significantly improved, especially in terms of the matching degree between video content and text description and video realism, which is better than the English text-based video model that relies on translation or prompt word optimizer. In addition, the main training embedding alignment layer can more friendly transfer the existing knowledge of the pre-trained model, which significantly accelerates the convergence of the model while improving computational efficiency.

[0084] Based on the above embodiment, as an implementable manner, based on the above embodiment, in one embodiment, a target cross-language description is input into a target language video model to generate corresponding video data based on the target language video model and the target cross-language description, including: Step 2051, inputting the target cross-language description into a text encoder, so as to convert the target cross-language description into a target cross-language feature vector based on the text encoder; Step 2052, inputting the target cross-language feature vector into a target embedding alignment layer, so as to convert the target cross-language feature vector into a target aligned embedding vector based on the target embedding alignment layer; Step 2053, while inputting the target alignment embedding vector into the target diffusion model, the target noise is also input into the target diffusion model, so that the target diffusion model converts the target noise into a target video feature code according to the target alignment embedding vector; Step 2054: Generate video data corresponding to the target cross-language description according to the target video feature encoding.

[0085] The target text-generated video model includes a text encoder, a target embedding alignment layer and a target diffusion model. The above steps 2051 to 2052 are the reasoning process of the target text-generated video model.

[0086] Specifically, the target cross-language description is first input into the text encoder to convert the target cross-language description into a target cross-language feature vector based on the text encoder, and the target cross-language feature vector includes the key features of the target cross-language description. Then the target cross-language feature vector is input into the target embedding alignment layer, and the target cross-language feature vector is converted into a target alignment embedding vector to ensure the consistency of text features and video features in the semantic space. The target alignment embedding vector is input into the target diffusion model by the target embedding alignment layer, and the target noise (Gaussian noise) is input into the target diffusion model at the same time. Based on the mapping from text to image learned during the training process, the target diffusion model converts the target noise into a target video feature encoding according to the target alignment embedding vector, that is, generates a latent representation of the video feature (target video feature encoding) from the noise and text feature information. The target diffusion model inputs the target video feature encoding into the VAE decoder, so as to use the VAE decoder to decode the representation of the latent space (target video feature encoding) into a video frame or image, and generate the final video output, that is, generate the video data corresponding to the target cross-language description.

[0087] In practical applications, based on the video data generation method provided in the embodiment of the present application, cross-language video data generation technology can be further developed to achieve seamless conversion and video content generation between different languages.

[0088] Specifically, in one embodiment, in order to further improve the reliability of the target text video model, a reinforcement learning mechanism can be introduced in the process of generating video data by the target text video model to optimize the generation strategy. Specifically, a reward function can be defined, which evaluates the generated video data based on factors such as the matching degree between the video data and the target cross-language description, visual quality, and action coherence. For example, a higher reward is given to videos with high matching degree between the video and the text description, good visual effects, and smooth actions. Then, in the process of generating the video, the model continuously adjusts the generation strategy according to the reward feedback, such as adjusting the parameters of the target embedding alignment layer or the target diffusion model, etc. The specific adjustment method is not limited in the embodiment of this application. Among them, reinforcement learning enables the model to continuously improve its behavior according to environmental feedback (reward) during the generation process. In this way, the model can better adapt to different text description requirements, generate high-quality videos that are more in line with user expectations, and improve the model's ability to generate high-quality videos in complex scenarios.

[0089] The video data generation method provided by the embodiment of the present application is obtained by obtaining a sample data set of a Vincent video model to be trained; wherein the sample data set includes multiple sample pairs, and the sample pairs include cross-language descriptions and target video clips; for any sample pair in the sample data set, the cross-language descriptions and target video clips in the sample pairs are converted into cross-language feature vectors and video feature vectors respectively; based on the embedding alignment layer to be trained, the cross-language feature vectors are converted into aligned embedding vectors; based on the aligned embedding vectors and video feature vectors, the parameters of the Vincent video model to be trained are optimized to obtain a target Vincent video model; wherein the Vincent video model to be trained includes an embedding alignment layer to be trained, and the aligned embedding vectors and video feature vectors are aligned in semantic space; the target cross-language description is input into the target Vincent video model to generate corresponding video data based on the target cross-language description based on the target Vincent video model. The method provided by the above scheme significantly improves the Vincent video model's ability to understand cross-language semantics by aligning the cross-language feature vectors with the video feature vectors in semantic space, thereby improving the accuracy of the video data generation result of the Vincent video model. In addition, by adopting multi-stage fine-tuning and semantic alignment technology, the computing resource consumption during model training and migration is optimized, the R&D cost is reduced, and the popularization and application of the model is facilitated.

[0090] An embodiment of the present application provides a video data generating device, which is used to execute the video data generating method provided by the above embodiment.

[0091] like Figure 8 , which is a schematic diagram of the structure of a video data generating device provided in an embodiment of the present application. The video data generating device 80 comprises: an acquisition module 801 , a conversion module 802 , an alignment module 803 , an optimization module 804 and a generation module 805 .

[0092] Among them, the acquisition module is used to obtain a sample data set of the Vincent video model to be trained; wherein the sample data set includes multiple sample pairs, and the sample pairs include cross-language descriptions and target video clips; the conversion module is used to convert the cross-language descriptions and target video clips in the sample pairs into cross-language feature vectors and video feature vectors respectively for any sample pair in the sample data set; the alignment module is used to convert the cross-language feature vectors into aligned embedding vectors based on the embedding alignment layer to be trained; the optimization module is used to optimize the parameters of the Vincent video model to be trained based on the aligned embedding vectors and the video feature vectors to obtain the target Vincent video model; wherein the Vincent video model to be trained includes the embedding alignment layer to be trained, and the aligned embedding vectors and the video feature vectors are aligned in the semantic space; the generation module is used to input the target cross-language description into the target Vincent video model to generate corresponding video data based on the target cross-language description based on the target Vincent video model.

[0093] Regarding the video data generating device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0094] The video data generating device provided in the embodiment of the present application is used to execute the video data generating method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0095] An embodiment of the present application provides an electronic device for executing the video data generating method provided in the above embodiment.

[0096] like Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 90 includes: at least one processor 91 and a memory 92.

[0097] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video data generating method provided in the above embodiment.

[0098] The electronic device provided in the embodiment of the present application is used to execute the video data generation method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0099] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video data generation method provided in any of the above embodiments is implemented.

[0100] The storage medium containing computer executable instructions provided in the embodiment of the present application can be used to store computer executable instructions of the video data generation method provided in the aforementioned embodiment. The implementation method and principle are the same and will not be repeated here.

[0101] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the video data generation method provided in the above-mentioned embodiment.

[0102] The embodiments of the present application provide a computer program product that can be used to execute computer instructions of the video data generation method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0103] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0106] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0107] A part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0108] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating video data, characterized in that: include: Obtain a sample data set of a text video model to be trained; wherein the sample data set includes a plurality of sample pairs, and the sample pairs include a cross-language description and a target video clip; For any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively; Based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; Based on the aligned embedding vector and the video feature vector, the parameters of the to-be-trained Vincent video model are optimized to obtain a target Vincent video model; wherein the to-be-trained Vincent video model includes the to-be-trained embedding alignment layer, and the aligned embedding vector and the video feature vector are aligned in a semantic space; The target cross-language description is input into the target language video model to generate corresponding video data according to the target cross-language description based on the target language video model.

2. The method according to claim 1, characterized in that The step of obtaining a sample data set of a Wensheng video model to be trained includes: Get the original video data; Preprocessing the original video data to obtain multiple target video clips; Inputting each of the target video clips into a visual language model to obtain a cross-language description corresponding to each of the target video clips; The target video clips and the cross-language descriptions corresponding to the target video clips are summarized to obtain a sample data set of the Wensheng video model to be trained.

3. The method according to claim 2, characterized in that The preprocessing of the original video data to obtain a plurality of target video segments includes: In the original video data, target video data whose duration satisfies a preset constraint condition is selected; Performing scene detection on the target video data to obtain a scene detection result of the target video data; According to the scene detection result, segmenting the target video data to obtain a plurality of original video segments, so that each original video segment corresponds to a single scene; The multiple original video clips are filtered to obtain multiple target video clips.

4. The method according to claim 3, characterized in that The filtering of the multiple original video clips to obtain multiple target video clips includes: Scoring each of the original video clips; Filtering the multiple original video clips according to the scores of the original video clips to obtain multiple target video clips; The score includes an aesthetic score and an optical flow score, wherein the aesthetic score is used to characterize the quality of the original video clip, and the optical flow score is used to characterize whether the original video clip is a static video clip.

5. The method according to claim 1, characterized in that For any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively includes: Dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage; For any sample pair in any sample sequence of the fine-tuning stage, the cross-language description and the target video segment in the sample pair are converted into a cross-language feature vector and a video feature vector respectively.

6. The method according to claim 5, characterized in that The step of dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage includes: Classifying the sample pairs in the sample data set according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution; According to the model optimization requirements of each fine-tuning stage, determine the required resolution data ratio for each fine-tuning stage; For any of the fine-tuning stages, according to the resolution data ratio requirement of the fine-tuning stage, sample pairs of corresponding proportions are divided from the sample pair set corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

7. The method according to claim 1, characterized in that The converting the cross-language feature vector into an aligned embedding vector based on the embedding alignment layer to be trained comprises: Inputting the cross-language feature vector into a first alignment block, so as to perform channel dimension compression on the cross-language feature vector based on the first alignment block to obtain a compressed cross-language feature vector; Inputting the compressed cross-language feature vector into a second alignment block, so as to perform a deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector; Inputting the deep semantic feature vector into a third alignment block, so as to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block, and obtain an aligned embedding vector; The embedded alignment layer to be trained includes the first alignment block, the second alignment block and the third alignment block.

8. The method according to claim 1, characterized in that The step of optimizing parameters of the Wensheng video model to be trained based on the aligned embedding vector and the video feature vector comprises: Inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned; Optimizing the parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned; The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

9. The method according to claim 8, characterized in that The step of inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned includes: After inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector; The diffusion model to be fine-tuned determines the predicted noise by learning a mapping relationship between the noise vector and the aligned embedding vector.

10. The method according to claim 9, characterized in that The method further comprises: The error of the predicted noise output by the diffusion model to be fine-tuned is determined according to the error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

11. The method according to claim 8, characterized in that The step of optimizing the parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned includes: In the first fine-tuning stage, the parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; In the second fine-tuning stage and the third fine-tuning stage, the parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the to-be-trained Wensheng video model in the first fine-tuning stage reaches the first round number threshold, the second fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the second fine-tuning stage reaches the second round number threshold, the third fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the third fine-tuning stage reaches the third round number threshold, the current Wensheng video model to be trained is used as the target Wensheng video model.

12. The method according to claim 1, characterized in that The step of inputting a target cross-language description into the target language video model to generate corresponding video data based on the target language video model and the target cross-language description includes: Inputting the target cross-language description into a text encoder to convert the target cross-language description into a target cross-language feature vector based on the text encoder; Inputting the target cross-language feature vector into a target embedding alignment layer to convert the target cross-language feature vector into a target aligned embedding vector based on the target embedding alignment layer; While inputting the target alignment embedding vector into the target diffusion model, target noise is input into the target diffusion model, so that the target diffusion model converts the target noise into target video feature encoding according to the target alignment embedding vector; Generate video data corresponding to the target cross-language description according to the target video feature encoding; The target text-generated video model includes the text encoder, the target embedding alignment layer and the target diffusion model.

13. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 12.

Images