Video data generation method, electronic device, storage medium and program product

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a video data generation method, electronic device, storage medium and program product, including: converting a cross-language description and a target video clip in a sample pair into a cross-language feature vector and a video feature vector respectively; converting the cross-language feature vector into an aligned embedding vector based on an embedding alignment layer to be trained; optimizing parameters of a Vincent video model to be trained based on the aligned embedding vector and the video feature vector to obtain a target Vincent video model; aligning the aligned embedding vector and the video feature vector in a semantic space; inputting a target cross-language description into the target Vincent video model to generate corresponding video data based on the target cross-language description based on the target Vincent video model. By obtaining the aligned embedding vector corresponding to the cross-language feature vector, the ability of the Vincent video model to understand cross-language semantics is significantly improved, thereby improving the accuracy of the video data generation result of the Vincent video model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video data generation method, electronic device, storage medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, Wensheng video technology has gradually become a hot topic in research and application. Wensheng video technology refers to the process of automatically converting text descriptions into video content using computer programs.

[0003] In the related art, a text description is usually input into a cultural video model, so that video data is automatically generated based on the cultural video model according to the obtained text description.

[0004] However, current text-based video models generally generate video data based on English text descriptions. If the text description entered by the user is in Chinese, it needs to be translated into English first. During the translation process, the meaning of the original text may be misinterpreted or lost, reducing the accuracy of the video data generation results. Summary of the Invention

[0005] The present application provides a video data generation method, electronic device, storage medium and program product to address the defects of related technologies such as reduced accuracy of video data generation results.

[0006] The first aspect of the present application provides a method for generating video data, comprising:

[0007] Obtain a sample dataset of a video model to be trained; wherein the sample dataset includes a plurality of sample pairs, each of which includes a cross-language description and a target video clip;

[0008] For any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively;

[0009] Based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector;

[0010] Based on the aligned embedding vector and the video feature vector, optimizing the parameters of the to-be-trained Vincent video model to obtain a target Vincent video model; wherein the to-be-trained Vincent video model includes the to-be-trained embedding alignment layer, and the aligned embedding vector and the video feature vector are aligned in a semantic space;

[0011] A target cross-language description is input into the target language video model to generate corresponding video data according to the target cross-language description based on the target language video model.

[0012] In an optional embodiment, obtaining a sample data set for a Vincent video model to be trained includes:

[0013] Get the original video data;

[0014] Preprocessing the original video data to obtain multiple target video clips;

[0015] Inputting each of the target video clips into a visual language model to obtain a cross-language description corresponding to each of the target video clips;

[0016] The target video clips and the cross-language descriptions corresponding to the target video clips are summarized to obtain a sample data set for the Wensheng video model to be trained.

[0017] In an optional implementation, the preprocessing of the original video data to obtain a plurality of target video segments includes:

[0018] Filtering target video data whose duration satisfies a preset constraint condition from the original video data;

[0019] Performing scene detection on the target video data to obtain a scene detection result of the target video data;

[0020] Segmenting the target video data according to the scene detection result to obtain a plurality of original video segments, so that each original video segment corresponds to a single scene;

[0021] The multiple original video clips are filtered to obtain multiple target video clips.

[0022] In an optional implementation, filtering the multiple original video segments to obtain multiple target video segments includes:

[0023] Scoring each of the original video clips;

[0024] Filtering the multiple original video clips according to the scores of the respective original video clips to obtain multiple target video clips;

[0025] The score includes an aesthetic score and an optical flow score. The aesthetic score is used to characterize the quality of the original video clip, and the optical flow score is used to characterize whether the original video clip is a static video clip.

[0026] In an optional embodiment, for any sample pair in the sample data set, converting the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively includes:

[0027] Dividing the sample data set into stages to obtain sample sequences for each fine-tuning stage;

[0028] For any sample pair in the sample sequence of any fine-tuning stage, the cross-language description and the target video segment in the sample pair are converted into a cross-language feature vector and a video feature vector respectively.

[0029] In an optional implementation, dividing the sample data set into stages to obtain sample sequences for each fine-tuning stage includes:

[0030] Classifying the sample pairs in the sample data set according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution;

[0031] Determine the required resolution data ratio for each fine-tuning stage based on the model optimization requirements at each fine-tuning stage;

[0032] For any of the fine-tuning stages, according to the resolution data ratio requirement of the fine-tuning stage, sample pairs of corresponding proportions are divided from the sample pair set corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

[0033] In an optional embodiment, converting the cross-language feature vector into an aligned embedding vector based on the embedding alignment layer to be trained includes:

[0034] Inputting the cross-language feature vector into a first alignment block, so as to perform channel dimension compression on the cross-language feature vector based on the first alignment block to obtain a compressed cross-language feature vector;

[0035] Inputting the compressed cross-language feature vector into a second alignment block, so as to perform a deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector;

[0036] Inputting the deep semantic feature vector into a third alignment block to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block to obtain an aligned embedding vector;

[0037] The embedding alignment layer to be trained includes the first alignment block, the second alignment block and the third alignment block.

[0038] In an optional embodiment, the performing parameter optimization on the to-be-trained Vincent video model based on the aligned embedding vector and the video feature vector includes:

[0039] Inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned;

[0040] Optimizing parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned;

[0041] The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

[0042] In an optional embodiment, inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned includes:

[0043] After inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector;

[0044] The diffusion model to be fine-tuned determines the predicted noise by learning a mapping relationship between the noise vector and the aligned embedding vector.

[0045] In an optional embodiment, the method further includes:

[0046] An error in the predicted noise output by the diffusion model to be fine-tuned is determined according to an error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

[0047] In an optional embodiment, the parameter optimization of the to-be-trained Vincent video model with the goal of reducing the error of the prediction noise output by the to-be-fine-tuned diffusion model includes:

[0048] In a first fine-tuning stage, the parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned;

[0049] In the second fine-tuning stage and the third fine-tuning stage, the parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned;

[0050] Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the Vincent video model to be trained in the first fine-tuning stage reaches the first round number threshold, it enters the second fine-tuning stage. When the number of sample learning rounds of the Vincent video model to be trained in the second fine-tuning stage reaches the second round number threshold, it enters the third fine-tuning stage. When the number of sample learning rounds of the Vincent video model to be trained in the third fine-tuning stage reaches the third round number threshold, the current Vincent video model to be trained is used as the target Vincent video model.

[0051] In an optional embodiment, inputting a target cross-language description into the target language video model to generate corresponding video data based on the target language video model and the target cross-language description includes:

[0052] Inputting the target cross-lingual description into a text encoder to convert the target cross-lingual description into a target cross-lingual feature vector based on the text encoder;

[0053] Inputting the target cross-lingual feature vector into a target embedding alignment layer to convert the target cross-lingual feature vector into a target aligned embedding vector based on the target embedding alignment layer;

[0054] Inputting the target alignment embedding vector into a target diffusion model and inputting target noise into the target diffusion model, so that the target diffusion model converts the target noise into a target video feature code according to the target alignment embedding vector;

[0055] Generating video data corresponding to the target cross-language description according to the target video feature code;

[0056] The target text-based video model includes the text encoder, the target embedding alignment layer and the target diffusion model.

[0057] A second aspect of the present application provides a video data generating device, comprising:

[0058] An acquisition module is used to acquire a sample data set for training a Wensheng video model; wherein the sample data set includes a plurality of sample pairs, each of which includes a cross-language description and a target video clip;

[0059] a conversion module, configured to convert, for any sample pair in the sample data set, the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively;

[0060] an alignment module, configured to convert the cross-lingual feature vector into an aligned embedding vector based on the embedding alignment layer to be trained;

[0061] an optimization module for optimizing parameters of a to-be-trained Vincent video model based on the aligned embedding vector and the video feature vector to obtain a target Vincent video model; wherein the to-be-trained Vincent video model includes the to-be-trained embedding alignment layer, and the aligned embedding vector and the video feature vector are aligned in a semantic space;

[0062] The generating module is configured to input a target cross-language description into the target language video model, so as to generate corresponding video data based on the target language video model and the target cross-language description.

[0063] A third aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0064] The memory stores computer-executable instructions;

[0065] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0066] The fourth aspect of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect is implemented.

[0067] A fifth aspect of the present application provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0068] The technical solution of this application has the following advantages:

[0069] The present application provides a video data generation method, electronic device, storage medium, and program product, the method comprising: obtaining a sample dataset of a to-be-trained Vincent video model; wherein the sample dataset comprises a plurality of sample pairs, each sample pair comprising a cross-language description and a target video clip; for any sample pair in the sample dataset, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector, respectively; converting the cross-language feature vector into an aligned embedding vector based on a to-be-trained embedding alignment layer; optimizing parameters of the to-be-trained Vincent video model based on the aligned embedding vector and the video feature vector to obtain a target Vincent video model; wherein the to-be-trained Vincent video model comprises a to-be-trained embedding alignment layer, wherein the aligned embedding vector and the video feature vector are aligned in semantic space; inputting a target cross-language description into the target Vincent video model, and generating corresponding video data based on the target cross-language description based on the target Vincent video model. The method provided by the above scheme significantly improves the Vincent video model's ability to understand cross-language semantics by performing semantic space alignment processing on the cross-language feature vector and the video feature vector to obtain an aligned embedding vector corresponding to the cross-language feature vector, thereby improving the accuracy of the video data generation results of the Vincent video model. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0071] Figure 1 This is a schematic diagram of the structure of the video data generation system based on the embodiments of the present application;

[0072] Figure 2 A flowchart of a method for generating video data according to an embodiment of the present invention;

[0073] Figure 3 A schematic diagram of the process of constructing a sample data set provided in an embodiment of the present application;

[0074] Figure 4 A schematic diagram of a classification strategy for an exemplary sample data set provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of the structure of a sample sequence in each fine-tuning stage provided as an example in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of the structure of the embedding alignment layer to be trained provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of the structure of a Wensheng video model to be trained provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of the structure of a video data generating device provided in an embodiment of the present application;

[0079] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0080] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0081] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] In addition, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. In the description of the following embodiments, "plurality" means more than two, unless otherwise specifically defined.

[0083] With the rapid development of artificial intelligence (AI), text-to-video generation (TTVG) has become a hot topic in research and application. TTVG refers to the process of automatically converting text descriptions into video content using computer programs. It combines advanced technologies from natural language processing (NLP) and computer vision (CV), demonstrating strong application potential. The core task of this technology is to map the semantic information in text descriptions onto video frames, involving the visualization of multiple elements such as scenes, characters, and actions. This technology holds significant significance for content creation, advertising, education, and other fields, improving creative efficiency and reducing production costs. However, Chinese TTVG technology faces numerous challenges, including accurately understanding Chinese semantics, generating high-quality video content, and requiring high computing resources. Existing technologies primarily focus on the English-language domain, while application and research in Chinese-language environments lags behind. Benchmarking and optimization methods for large Chinese models are also lacking.

[0084] As a cutting-edge technology in the field of artificial intelligence, Wensheng Video Technology achieves the automatic conversion from text descriptions to video content through advanced techniques such as deep learning, generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models (DMs). This technology has significant implications for content creation, advertising, education, and other fields, significantly improving creative efficiency and reducing production costs. However, related technologies still face many challenges in handling complex dynamic scenes in Chinese contexts, capturing attribute changes, generating the correct number of objects, simulating physical phenomena, and understanding causal relationships. Video generation models may also struggle to accurately describe events over time and may make common sense errors in details. Furthermore, the scarcity and high cost of labeling high-quality Chinese video-text data, as well as the lack of unified evaluation standards, make model optimization unclear.

[0085] While traditional text-based video models (such as open-sora, open-sora-plan, and cogvideox) have achieved technological innovation, they still exhibit numerous flaws in their generated videos, demonstrating their immaturity. This is particularly true for Chinese language understanding. Due to their use of an English text encoder, direct Chinese input is not possible. Instead, Chinese text must be translated into English or rewritten using a Chinese input prompt optimizer. These operations can distort or even lose the meaning of the original text. Using a prompt optimizer to rewrite Chinese prompts requires the use of a large language model (LLM), increasing computing power consumption and limiting model training scale and deployment.

[0086] In response to the above problems, embodiments of the present application provide a video data generation method, electronic device, storage medium, and program product, the method comprising: obtaining a sample data set of a to-be-trained cultural video model; wherein the sample data set comprises a plurality of sample pairs, each sample pair comprising a cross-language description and a target video clip; for any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector, respectively; based on an embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; based on the aligned embedding vector and the video feature vector, performing parameter optimization on the to-be-trained cultural video model to obtain a target cultural video model; wherein the to-be-trained cultural video model comprises an embedding alignment layer to be trained, and the aligned embedding vector and the video feature vector are aligned in a semantic space; and inputting a target cross-language description into the target cultural video model to generate corresponding video data according to the target cross-language description based on the target cultural video model. The method provided by the above scheme aligns the cross-language feature vector and the video feature vector in the semantic space to obtain the aligned embedding vector corresponding to the cross-language feature vector, which significantly improves the Vincent video model's ability to understand cross-language semantics, thereby improving the accuracy of the video data generation results of the Vincent video model.

[0087] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of the present invention in conjunction with the accompanying drawings.

[0088] First, the structure of the video data generation system on which this application is based is described:

[0089] The video data generation method, electronic device, storage medium and program product provided in the embodiments of the present application are suitable for fine-tuning an existing Vincent video model to be trained to improve the Vincent video model's ability to understand cross-language semantics. Figure 1Figure 2 shows a schematic diagram of the video data generation system used in the embodiments of the present application. The system primarily includes a data acquisition device, a model training device, and a Vincent video model to be trained. The data acquisition device is used to acquire a sample dataset and send the resulting sample dataset to the model training device. The model training device then optimizes the parameters of the Vincent video model to be trained based on the resulting sample dataset until a target Vincent video model is obtained.

[0090] This embodiment of the present application provides a video data generation method for fine-tuning an existing Vincent video model to be trained, thereby improving the Vincent video model's ability to understand cross-language semantics. This embodiment of the present application is performed by an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic device capable of optimizing and running the Vincent video model.

[0091] like Figure 2 FIG. 1 is a flow chart of a method for generating video data according to an embodiment of the present application, the method comprising:

[0092] Step 201: Obtain a sample data set for the Wensheng video model to be trained.

[0093] The sample dataset includes multiple sample pairs, each of which includes a cross-language description and a target video clip. If the language descriptions of the training samples used in the previous training phase of the currently acquired Vincent video model to be trained are different from the cross-language descriptions, for example, if the acquired Vincent video model to be trained is a Vincent video model trained using samples with English descriptions, then the cross-language description is a non-English description, such as a Chinese description.

[0094] Specifically, the original videos can be obtained from open video websites such as Mixkit, Pixabay, and Pexel, and a sample dataset for the to-be-trained video model can be constructed based on the obtained original videos.

[0095] Step 202 : for any sample pair in the sample data set, convert the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively.

[0096] Specifically, a text encoder can be used to capture the semantic information of cross-language descriptions, and then, based on this captured semantic information, a corresponding cross-language feature vector can be determined. A video encoder (VAE encoder) is used to convert the input video frame or image (target video clip) into a latent space representation (video feature vector). This latent representation is a compressed, low-dimensional vector that captures the key features of the input video data.

[0097] Step 203: Based on the embedding alignment layer to be trained, the cross-lingual feature vector is converted into an aligned embedding vector.

[0098] Specifically, based on the embedding alignment layer to be trained, semantic alignment processing can be performed between the cross-language feature vector and the video feature vector in the semantic space to convert the cross-language feature vector into an aligned embedding vector that is semantically aligned with the video feature vector.

[0099] Step 204 : Optimize the parameters of the training cultural video model based on the aligned embedding vector and the video feature vector to obtain a target cultural video model.

[0100] The to-be-trained video model includes an embedding alignment layer to be trained, and the aligned embedding vector and the video feature vector are aligned in the semantic space.

[0101] It should be noted that the Vincent video model to be trained adopted in the embodiment of the present application is a traditional Vincent model with an embedded alignment layer to be trained added.

[0102] Specifically, based on the aligned embedding vector and the video feature vector, the parameters of the embedding alignment layer to be trained can be optimized to improve the accuracy of the aligned embedding vector it provides. At the same time, the model parameters of the traditional language model in the language video model to be trained can be fine-tuned so that the final target language video model can accurately understand the semantics of cross-language descriptions.

[0103] Step 205 : Input the target cross-language description into the target language video model, so as to generate corresponding video data based on the target language video model and the target cross-language description.

[0104] Specifically, after obtaining the target language video model, the target cross-language description input by the user is received, and the target cross-language description is input into the target language video model to perform cross-language semantic analysis on the target cross-language description based on the target language video model and generate corresponding video data.

[0105] On the basis of the above embodiment, in order to ensure the reliability of the sample dataset of the Vincent video model to be trained, as an implementable approach, on the basis of the above embodiment, in one embodiment, obtaining the sample dataset of the Vincent video model to be trained includes:

[0106] Step 2011, obtaining original video data;

[0107] Step 2012: pre-processing the original video data to obtain multiple target video segments;

[0108] Step 2013: Input each target video segment into a visual language model to obtain a cross-language description corresponding to each target video segment;

[0109] In step 2014, each target video segment and the corresponding cross-language description of each target video segment are summarized to obtain a sample data set for training the Wensheng video model.

[0110] It should be noted that the visual language model can adopt a large visual language model such as qwen-vl-7b-instruction. The large visual language model is used to tag the target video clips to obtain the corresponding cross-language description of the target video clips, ensuring that the final sample data set has strong semantic consistency.

[0111] Specifically, the prompt words and the target video clip can be input into the visual language model, where, taking the cross-language description as a Chinese description as an example, the prompt words are "Please describe the content of this video as detailed as possible, including the objects, scenery, animals, characters and camera movements in the video. Please describe the content of the video and the changes that have occurred in chronological order. This description should be useful for AI to regenerate the video", and the visual language model will output a detailed description of the target video clip in Chinese. However, the visual language model tends to generate certain prefixes for video descriptions, such as "This video shows", "The video shows", "The video is", etc. Therefore, the embodiment of the present application compiles a prefix list that lists all irrelevant starting character strings, deletes these prefixes in the title string, and finally obtains a sample data set for the visual video model to be trained by summarizing each target video clip and the Chinese description corresponding to each target video clip.

[0112] Specifically, in one embodiment, target video data whose duration satisfies preset constraints can be screened from the original video data; scene detection is performed on the target video data to obtain a scene detection result of the target video data; based on the scene detection result, the target video data is segmented to obtain multiple original video segments, so that each original video segment corresponds to a single scene; and the multiple original video segments are filtered to obtain multiple target video segments.

[0113] Specifically, original video data with a duration of no less than 1 second and no more than 10 seconds can be used as target video data. That is, the preset constraint is that the duration is between 1 and 10 seconds. The video is further detected using the scene detection tool PySceneDetect, which outputs time periods for different scenes in the same video. The original video is then segmented into shorter segments based on the time periods, resulting in multiple original video segments.

[0114] The scene in the video may change due to various factors, such as changes in the background, scene layout, shooting location, etc., or changes in the character's behavior, actions, or the combination of characters that appear.

[0115] Specifically, in one embodiment, each original video segment may be scored; and based on the score of each original video segment, multiple original video segments may be filtered to obtain multiple target video segments.

[0116] The scores include aesthetic scores and optical flow scores. The aesthetic scores are used to characterize the quality of the original video clips, and the optical flow scores are used to characterize whether the original video clips are static video clips.

[0117] Specifically, if Figure 3 As shown, it is a schematic diagram of the construction process of the sample data set provided by the embodiment of the present application. After completing the scene segmentation and obtaining multiple original video segments, the aesthetic score of the original video clip can be estimated, and low-quality video clips, including high blur, low bit rate, pixelation, black and white shots or abstract visual effects, etc., can be filtered out based on the aesthetic score. The optical flow score is estimated for the original video clip, and whether it is a static video is determined based on the optical flow score. The motion value is evaluated based on the optical flow score, and the static video is discarded. Furthermore, the OCR model can also be used to detect whether there is text in the video and remove videos with too much text. After the final OCR filtering, multiple target video clips are obtained, and then the target video clips are annotated with a large model and the prefix content in the title is filtered to obtain the cross-language description (cross-language title) corresponding to each target video clip, thereby obtaining a sample data set for the Wensheng video model to be trained. Based on the following Figure 3 The data processing pipeline shown here produces video-text pairs (sample pairs) with high aesthetic quality, large video motion, and strong semantic consistency. Furthermore, using this pipeline to annotate the dataset reduces the need for manual annotation and lowers costs. Furthermore, it provides more cross-lingual semantic context for the annotations, thereby enhancing the model's ability to understand cross-lingual semantics.

[0118] Based on the above embodiment, as an implementable manner, based on the above embodiment, in one embodiment, for any sample pair in the sample dataset, the cross-language description and the target video segment in the sample pair are converted into a cross-language feature vector and a video feature vector respectively, including:

[0119] Step 2021: Divide the sample data set into stages to obtain sample sequences for each fine-tuning stage;

[0120] Step 2022 : For any sample pair in the sample sequence of any fine-tuning stage, convert the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector, respectively.

[0121] It should be noted that the embodiment of the present application adopts a phased approach to perform multi-stage parameter fine-tuning on the training Wensheng video model.

[0122] Specifically, in one embodiment, the sample pairs in the sample data set can be classified according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution; according to the model optimization requirements of each fine-tuning stage, the resolution data ratio requirement of each fine-tuning stage is determined; for any fine-tuning stage, according to the resolution data ratio requirement of the fine-tuning stage, a corresponding proportion of sample pairs is divided from the sample pair set corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

[0123] Specifically, Open-Sora's bucket mechanism can be used to classify target video data based on resolution, frame rate, and aspect ratio, forming different "buckets" to accommodate inputs of varying sizes, thereby supporting training requirements at multiple resolutions, aspect ratios, and frame rates. Before each training cycle, the sample dataset is reshuffled and assigned to corresponding buckets based on video characteristics, ensuring that the number of samples in each bucket is controlled. GPU load is balanced by searching for an optimal batch size for each bucket. The batch size is the number of samples that a single GPU can input at a time. Specifically, the corresponding batch size can be set for each bucket according to the resolution. For example, a larger batch size is set for buckets with lower resolutions to better balance computing resources and model learning efficiency during training at different resolutions.

[0124] Among them, Figure 4 As shown, it is a schematic diagram of the classification strategy of an exemplary sample data set provided in an embodiment of the present application. In the embodiment of the present application, the target video clips are assigned to 5 buckets (video classification) with different attributes. These 5 buckets represent video resolutions from low to high [256, 360p, 512, 480p, 720p]. Then all videos are cut to 51 frames, about 2 seconds, and all videos are scaled to a 1:1 aspect ratio. Then, the corresponding batch size is set according to the resolution size. By controlling the proportion of data falling into each bucket at different stages, the resolution of the staged training data can be controlled. Through this staged and bucketed training method, the model can gradually learn and adapt to video content of different resolutions and complexities, and ultimately achieve the generation of high-quality videos. This method not only improves the efficiency of training, but also ensures the diversity and quality of the generated videos. The "p" in the resolution indicates progressive scan. It indicates the number of pixels in the vertical direction of the video and also implies the number of pixels in the horizontal direction and the scanning method. For example, 720p means 720×1280 pixels, and 512 means 512×512 pixels.

[0125] Specifically, if Figure 5The figure shows a schematic diagram of the structure of sample sequences in each fine-tuning stage provided by an embodiment of the present application, taking the fine-tuning stages as the first fine-tuning stage, the second fine-tuning stage, and the third fine-tuning stage as an example. The model optimization requirement in the first fine-tuning stage is to achieve basic training, the model optimization requirement in the second fine-tuning stage is to achieve quality training, and the model optimization requirement in the third fine-tuning stage is to achieve detail training.

[0126] The first fine-tuning stage scales the video so that the target video clips are primarily distributed at lower resolutions of 256-360p, with the five resolutions accounting for 0.4, 0.3, 0.2, 0.1, and 0 data points, respectively. Using larger batch sizes, learning rates, and epochs in this stage helps the model quickly learn the fundamental concepts and structure of the video content without being overwhelmed by the complexity of the details. This allows the model to learn basic video generation capabilities, including understanding the fundamental connection between text descriptions and video content, and the ability to generate low-resolution videos, laying a solid foundation for subsequent high-resolution training.

[0127] The second fine-tuning stage scales the video so that the target video clips are primarily distributed at a medium resolution of 512, with the proportion of data at the five resolutions being 0.1, 0.2, 0.4, 0.2, and 0.1, respectively. In this stage, by using a medium batch size, medium learning rate, and medium number of epochs, the model improves the resolution and quality of the generated videos. The model is able to generate medium-resolution videos that are more consistent with the text description in content and style, while exhibiting better visual quality and motion coherence.

[0128] The third fine-tuning stage scales the videos so that the target video clips are primarily distributed in high-resolution 480p-720p, with the percentage of data at the five resolutions being 0, 0, 0.1, 0.4, and 0.5, respectively. In this stage, the model further improves the video resolution and detail by using a smaller batch size, a smaller learning rate, and fewer epochs. The model is able to generate high-resolution videos that not only closely match the text descriptions in content but also meet high standards of visual detail and motion fluidity.

[0129] Through these three stages of fine-tuning, the model gradually improves its video generation capabilities, from basic to advanced, ultimately achieving high-quality cross-lingual video generation. This phased training approach helps the model focus on different learning objectives at each stage, allowing it to more effectively master the skills required to generate complex video content.

[0130] On the basis of the above embodiment, as an implementable method, on the basis of the above embodiment, as Figure 6FIG. 1 is a schematic diagram of the structure of the embedding alignment layer to be trained provided in an embodiment of the present application. In one embodiment, based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector includes:

[0131] Step 2031: Input the cross-language feature vector into the first alignment block, and perform channel dimension compression on the cross-language feature vector based on the first alignment block to obtain a compressed cross-language feature vector;

[0132] Step 2032: Input the compressed cross-language feature vector to the second alignment block, so as to perform deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector.

[0133] In step 2033 , the deep semantic feature vector is input into the third alignment block, so as to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block to obtain an aligned embedding vector.

[0134] Among them, the embedded alignment layer to be trained includes a first alignment block, a second alignment block and a third alignment block. The embodiment of the present application mainly optimizes the parameters of the second alignment block in the embedded alignment layer to be trained.

[0135] Specifically, if Figure 6 As shown in the figure, the first alignment block and the second alignment block both include a fully connected layer, an activation layer, a regularization layer (Dropout layer) and a normalization layer, and the third alignment block includes a fully connected layer and a regularization layer.

[0136] Specifically, the cross-lingual feature vector is first input to the first alignment block. After entering the first alignment block, the cross-lingual feature vector undergoes a linear transformation through a fully connected layer, adjusting its dimensionality and compressing the channel dimension to half its original size. Next, the activation layer introduces nonlinearity, the dropout layer randomly drops neurons to prevent overfitting, and the normalization layer normalizes the data to accelerate model convergence. After these processes, a compressed cross-lingual feature vector is obtained. This compressed cross-lingual feature vector is then input to the second alignment block. In the second alignment block, a deep semantic transformation is performed on the compressed cross-lingual feature vector through a combination of fully connected layers, activation layers, dropout layers, and normalization layers. Although the number of channels in the output features remains unchanged after these layers, a deeper semantic transformation is achieved, resulting in a deep semantic feature vector. This deep semantic feature vector is finally input to the third alignment block. In the third alignment block, the deep semantic feature vector is mapped back to the original channel of the cross-lingual feature vector, and finally an aligned embedding vector is generated, realizing the conversion of the cross-lingual feature vector to the aligned embedding vector, ensuring the consistency of text and video features in the semantic space, and helping the subsequent model learn the mapping relationship from text to video.

[0137] On the basis of the above embodiment, as an implementable method, on the basis of the above embodiment, as Figure 7 FIG. 1 is a schematic diagram of the structure of a Vincent video model to be trained provided in an embodiment of the present application. The solid line represents the training process, and the dotted line represents the inference process. In one embodiment, based on the aligned embedding vector and the video feature vector, the parameters of the Vincent video model to be trained are optimized, including:

[0138] Step 2041: input the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned;

[0139] Step 2042 , with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned, optimize the parameters of the Wensheng video model to be trained.

[0140] The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

[0141] It should be noted that the Wensheng video model to be trained also includes a VAE encoder and a text encoder that supports multiple languages. The text encoder can use the pre-trained model mT5 to convert the input cross-language description into an intrinsic embedding representation (cross-language feature vector) that can be understood by the large model. The text encoder can capture the semantic information of the text and provide guidance for subsequent video generation. The text encoder keeps the weights frozen during training and inference. Then, through the embedding alignment layer, the text features generated by the text encoder are aligned with the video features to obtain an aligned embedding vector. This ensures the consistency of text and video features in the semantic space, which helps the model learn the mapping relationship from text to video. Taking the English description as a sample training example before the Wensheng video model to be trained, due to the distinction between cross-language semantics and English semantics, the embodiment of the present application can mainly train the parameters of the embedding alignment layer during the fine-tuning process, and realize semantic transfer without destroying the pre-training knowledge of the diffusion model DiT on image generation, so as to achieve good learning ability of the embedding alignment layer.

[0142] It should be further explained that the diffusion model DiT is a diffusion model combined with the Transformer architecture for image and video generation tasks. It gradually adds noise to the data by simulating the diffusion process, and then learns to reverse this process to construct the required data samples from the noise. The core idea of ​​DiT is to use Transformer as the backbone network of the diffusion model, rather than traditional convolutional neural networks (such as U-Net), to process the potential representation of the image. The DiT architecture is based on the Latent Diffusion Model (LDM) framework, uses Vision Transformer (ViT) as the backbone network, and constructs a scalable diffusion model by adjusting the normalization of ViT. The diffusion model to be fine-tuned used in the embodiment of the present application is an English language video pre-training model, which has been pre-trained on a large amount of English video data. The embodiment of the present application mainly transforms the text encoder and embedding alignment layer to enable it to adapt to the cross-language semantic environment, that is, by optimizing the structure of the English language video pre-training model, the migration ability of the English language video pre-training model to the cross-language language video model is improved.

[0143] Specifically, in one embodiment, after the aligned embedding vector and the video feature vector are input into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector; the diffusion model to be fine-tuned determines the predicted noise by learning the mapping relationship between the noise vector and the aligned embedding vector.

[0144] Specifically, during model training, the fine-tuned diffusion model simulates a forward diffusion process. When a video feature vector is input, Gaussian noise is gradually added to it according to the forward diffusion rules. In the first training iteration, only a small amount of noise can be added. As the number of training iterations increases, more and more noise is added, and the video feature vector gradually becomes a noise vector. This gradual noise addition process enables the model to learn how the video feature vector changes under varying levels of noise. Furthermore, during the inverse denoising process (the process by which the model generates video), it can accurately recover the video features corresponding to the cross-language description from the noise, ultimately generating a video that meets the requirements.

[0145] That is, the fine-tuned diffusion model learns how to find the correspondence between this noise vector and the aligned embedding vector during the inverse denoising process. The goal of the fine-tuned diffusion model is to generate the corresponding video (related to the video feature vector) based on the cross-lingual description (aligned embedding vector). By learning the mapping between the noise vector and the aligned embedding vector, the model can attempt to predict the noise during the noise addition process, thereby learning how to convert text features into video features.

[0146] Specifically, in one embodiment, the error of the predicted noise output by the diffusion model to be fine-tuned may be determined according to the error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

[0147] Specifically, the mean squared error (MSE) can be used to measure the error between the predicted noise and the noise added to the video feature vector by the fine-tuned diffusion model. During the training process, by minimizing the MSE loss, the fine-tuned diffusion model learns how to more accurately reconstruct the original latent feature representation from the noise representation.

[0148] Furthermore, in one embodiment, in the first fine-tuning stage, parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; in the second and third fine-tuning stages, parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned.

[0149] Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the Vincent video model to be trained in the first fine-tuning stage reaches the first round number threshold, it enters the second fine-tuning stage. When the number of sample learning rounds of the Vincent video model to be trained in the second fine-tuning stage reaches the second round number threshold, it enters the third fine-tuning stage. When the number of sample learning rounds of the Vincent video model to be trained in the third fine-tuning stage reaches the third round number threshold, the current Vincent video model to be trained is used as the target Vincent video model.

[0150] Specifically, during the first fine-tuning phase, the primary optimization goal is to optimize the parameters of the training embedding alignment layer. This is because, in the early stages of training, the focus is on achieving better alignment of text features (processed by the embedding alignment layer) and video features in the semantic space. During the second and third fine-tuning phases, optimization is performed not only on the training embedding alignment layer but also on the fine-tuned diffusion model. As training progresses, the diffusion model itself needs to better learn the mapping relationship between cross-lingual text and video, so that it can more accurately generate video data that matches the text description from noise.

[0151] The fine-tuned model (target language-based video model) was evaluated for performance using metrics such as the match between video content and text description, video realism, and action coherence. Experimental results show that after multi-stage fine-tuning and semantically enhanced annotation, the model significantly improves performance on cross-language video generation tasks. In particular, it outperforms English language-based video models that rely on translation or cue word optimizers in terms of video content-text match and video realism. Furthermore, the main training embedding alignment layer facilitates the transfer of knowledge from the pre-trained model, significantly accelerating model convergence while improving computational efficiency.

[0152] Based on the above embodiment, as an implementable manner, in one embodiment, a target cross-language description is input into a target language video model to generate corresponding video data based on the target language video model and the target cross-language description, including:

[0153] Step 2051 , inputting the target cross-language description into a text encoder, so as to convert the target cross-language description into a target cross-language feature vector based on the text encoder;

[0154] Step 2052: Input the target cross-lingual feature vector to the target embedding alignment layer to convert the target cross-lingual feature vector into a target aligned embedding vector based on the target embedding alignment layer;

[0155] Step 2053: input the target alignment embedding vector into the target diffusion model and input the target noise into the target diffusion model, so that the target diffusion model converts the target noise into target video feature code according to the target alignment embedding vector;

[0156] Step 2054 : Generate video data corresponding to the target cross-language description based on the target video feature code.

[0157] The target text-based video model includes a text encoder, a target embedding alignment layer, and a target diffusion model. Steps 2051 to 2052 are the inference process of the target text-based video model.

[0158] Specifically, the target cross-lingual description is first input into a text encoder, which converts it into a target cross-lingual feature vector based on the text encoder. The target cross-lingual feature vector includes the key features of the target cross-lingual description. The target cross-lingual feature vector is then input into a target embedding alignment layer. To ensure consistency between text and video features in the semantic space, the target cross-lingual feature vector is converted into a target aligned embedding vector. The target aligned embedding vector is then input into a target diffusion model, along with target noise (Gaussian noise). Based on the text-to-image mapping learned during training, the target diffusion model converts the target noise into a target video feature encoding based on the target aligned embedding vector. This means that a latent representation of video features (target video feature encoding) is generated from the noise and text feature information. The target diffusion model then inputs the target video feature encoding into a VAE decoder, which decodes the latent space representation (target video feature encoding) into video frames or images, generating the final video output: the video data corresponding to the target cross-lingual description.

[0159] In practical applications, based on the video data generation method provided in the embodiments of the present application, cross-language video data generation technology can be further developed to achieve seamless conversion and video content generation between different languages.

[0160] Specifically, in one embodiment, in order to further improve the reliability of the target text-based video model, a reinforcement learning mechanism can be introduced during the process of generating video data by the target text-based video model to optimize the generation strategy. Specifically, a reward function can be defined, which evaluates the generated video data based on multiple factors such as the degree of match between the video data and the target cross-language description, visual quality, and motion coherence. For example, a higher reward is given to videos with a high degree of match between the video and the text description, good visual effects, and smooth motion. Then, during the video generation process, the model continuously adjusts the generation strategy based on reward feedback, such as adjusting the parameters of the target embedding alignment layer or the target diffusion model, etc. The specific adjustment method is not limited in the embodiments of this application. Among them, reinforcement learning enables the model to continuously improve its behavior based on environmental feedback (rewards) during the generation process. In this way, the model can better adapt to different text description requirements, generate high-quality videos that better meet user expectations, and improve the model's ability to generate high-quality videos in complex scenarios.

[0161] The video data generation method provided in an embodiment of the present application obtains a sample dataset of a Vincent video model to be trained; wherein the sample dataset includes multiple sample pairs, each sample pair including a cross-language description and a target video clip; for any sample pair in the sample dataset, the cross-language description and the target video clip in the sample pair are converted into a cross-language feature vector and a video feature vector, respectively; based on the embedding alignment layer to be trained, the cross-language feature vector is converted into an aligned embedding vector; based on the aligned embedding vector and the video feature vector, the parameters of the Vincent video model to be trained are optimized to obtain a target Vincent video model; wherein the Vincent video model to be trained includes an embedding alignment layer to be trained, and the aligned embedding vector and the video feature vector are aligned in semantic space; the target cross-language description is input into the target Vincent video model to generate corresponding video data based on the target cross-language description. The method provided by the above scheme significantly improves the Vincent video model's ability to understand cross-language semantics by performing semantic space alignment processing on the cross-language feature vector and the video feature vector, thereby improving the accuracy of the video data generation results of the Vincent video model. In addition, by adopting multi-stage fine-tuning and semantic alignment technology, the computing resource consumption during model training and migration is optimized, the R&D cost is reduced, and it helps the popularization and application of the model.

[0162] An embodiment of the present application provides a video data generating device for executing the video data generating method provided in the above embodiment.

[0163] like Figure 8 FIG. 8 is a schematic diagram of the structure of a video data generating device according to an embodiment of the present invention. The video data generating device 80 includes: an acquisition module 801 , a conversion module 802 , an alignment module 803 , an optimization module 804 and a generation module 805 .

[0164] Among them, the acquisition module is used to obtain a sample data set of the cultural video model to be trained; wherein the sample data set includes multiple sample pairs, and the sample pairs include cross-language descriptions and target video clips; the conversion module is used to convert the cross-language descriptions and target video clips in the sample pairs into cross-language feature vectors and video feature vectors respectively for any sample pair in the sample data set; the alignment module is used to convert the cross-language feature vectors into aligned embedding vectors based on the embedding alignment layer to be trained; the optimization module is used to optimize the parameters of the cultural video model to be trained based on the aligned embedding vectors and video feature vectors to obtain the target cultural video model; wherein the cultural video model to be trained includes the embedding alignment layer to be trained, and the aligned embedding vectors and video feature vectors are aligned in the semantic space; the generation module is used to input the target cross-language description into the target cultural video model to generate corresponding video data based on the target cross-language description based on the target cultural video model.

[0165] Regarding the video data generating device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] The video data generating device provided in the embodiment of the present application is used to execute the video data generating method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0167] An embodiment of the present application provides an electronic device for executing the video data generation method provided in the above embodiment.

[0168] like Figure 9 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 90 includes: at least one processor 91 and a memory 92.

[0169] The memory stores computer-executable instructions; and at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video data generating method provided in the above embodiment.

[0170] The electronic device provided in the embodiment of the present application is used to execute the video data generation method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0171] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video data generation method provided in any of the above embodiments is implemented.

[0172] The storage medium containing computer-executable instructions provided in the embodiment of the present application can be used to store the computer-executable instructions of the video data generation method provided in the aforementioned embodiment. Its implementation method and principle are the same and will not be repeated here.

[0173] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the video data generation method provided in the aforementioned embodiment.

[0174] The embodiments of the present application provide computer program products that can be used to execute computer instructions of the video data generation method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0178] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0179] Part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes but is not limited to a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0180] Those skilled in the art will clearly understand that for the sake of convenience and brevity, the division of the above-mentioned functional modules is only used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating video data, characterized in that: include: Obtain a sample data set of a text-based video model to be trained; wherein the sample data set includes a plurality of sample pairs, and the sample pairs include a cross-language description and a target video clip; For any sample pair in the sample data set, converting the cross-language description and the target video clip in the sample pair into a cross-language feature vector and a video feature vector respectively; Based on the embedding alignment layer to be trained, converting the cross-language feature vector into an aligned embedding vector; Based on the aligned embedding vector and the video feature vector, the parameters of the to-be-trained Vincent video model are optimized to obtain a target Vincent video model; wherein the to-be-trained Vincent video model includes the to-be-trained embedding alignment layer, and the aligned embedding vector and the video feature vector are aligned in a semantic space; Inputting a target cross-language description into the target language video model to generate corresponding video data according to the target cross-language description based on the target language video model; The to-be-trained language video model is a language video model using English descriptions, and the cross-language description is a non-English description.

2. The method according to claim 1, characterized in that The step of obtaining a sample data set of a Wensheng video model to be trained includes: Get the original video data; Preprocessing the original video data to obtain multiple target video clips; Inputting each of the target video clips into a visual language model to obtain a cross-language description corresponding to each of the target video clips; The target video clips and the cross-language descriptions corresponding to the target video clips are summarized to obtain a sample data set of the Wensheng video model to be trained.

3. The method according to claim 2, characterized in that The preprocessing of the original video data to obtain a plurality of target video segments includes: In the original video data, target video data whose duration satisfies a preset constraint condition is selected; Performing scene detection on the target video data to obtain a scene detection result of the target video data; According to the scene detection result, segmenting the target video data to obtain a plurality of original video segments, so that each original video segment corresponds to a single scene; The multiple original video segments are filtered to obtain multiple target video segments.

4. The method according to claim 3, characterized in that The filtering of the multiple original video clips to obtain multiple target video clips includes: Scoring each of the original video clips; Filtering the multiple original video clips according to the scores of the original video clips to obtain multiple target video clips; The score includes an aesthetic score and an optical flow score, wherein the aesthetic score is used to characterize the quality of the original video clip, and the optical flow score is used to characterize whether the original video clip is a static video clip.

5. The method according to claim 1, characterized in that For any sample pair in the sample data set, converting the cross-language description and the target video segment in the sample pair into a cross-language feature vector and a video feature vector respectively includes: Dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage; For any sample pair in any sample sequence of the fine-tuning stage, the cross-language description and the target video segment in the sample pair are converted into a cross-language feature vector and a video feature vector respectively.

6. The method according to claim 5, characterized in that The step of dividing the sample data set into stages to obtain sample sequences of each fine-tuning stage includes: Classifying the sample pairs in the sample data set according to the resolution of each target video clip in the data set to obtain a set of sample pairs corresponding to each resolution; According to the model optimization requirements of each fine-tuning stage, determine the required resolution data ratio for each fine-tuning stage; For any of the fine-tuning stages, according to the resolution data ratio requirement of the fine-tuning stage, sample pairs of corresponding proportions are divided from the sample pair set corresponding to each resolution to obtain a sample sequence of the fine-tuning stage.

7. The method according to claim 1, characterized in that The converting the cross-language feature vector into an aligned embedding vector based on the embedding alignment layer to be trained comprises: Inputting the cross-language feature vector into a first alignment block, so as to perform channel dimension compression on the cross-language feature vector based on the first alignment block to obtain a compressed cross-language feature vector; Inputting the compressed cross-language feature vector into a second alignment block, so as to perform deep semantic transformation on the compressed cross-language feature vector based on the second alignment block to obtain a deep semantic feature vector; Inputting the deep semantic feature vector into a third alignment block, so as to map the deep semantic feature to the original channel of the cross-language feature vector based on the third alignment block, and obtain an aligned embedding vector; The embedded alignment layer to be trained includes the first alignment block, the second alignment block and the third alignment block.

8. The method according to claim 1, characterized in that: The step of optimizing parameters of the Wensheng video model to be trained based on the aligned embedding vector and the video feature vector comprises: Inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned; Optimizing the parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned; The Wensheng video model to be trained includes a diffusion model to be fine-tuned.

9. The method according to claim 8, characterized in that The step of inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned to obtain the predicted noise output by the diffusion model to be fine-tuned includes: After inputting the aligned embedding vector and the video feature vector into the diffusion model to be fine-tuned, the diffusion model to be fine-tuned adds noise to the video feature vector and performs noise diffusion to convert the video feature vector into a noise vector; The diffusion model to be fine-tuned determines the predicted noise by learning a mapping relationship between the noise vector and the aligned embedding vector.

10. The method according to claim 9, characterized in that The method further comprises: The error of the predicted noise output by the diffusion model to be fine-tuned is determined according to the error between the predicted noise and the noise added to the video feature vector by the diffusion model to be fine-tuned.

11. The method according to claim 8, characterized in that The step of optimizing the parameters of the Wensheng video model to be trained with the goal of reducing the error of the predicted noise output by the diffusion model to be fine-tuned includes: In the first fine-tuning stage, the parameters of the embedding alignment layer to be trained are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; In the second fine-tuning stage and the third fine-tuning stage, the parameters of the embedding alignment layer to be trained and the diffusion model to be fine-tuned are optimized with the goal of reducing the error of the prediction noise output by the diffusion model to be fine-tuned; Among them, the sample sequences used in the first fine-tuning stage, the second fine-tuning stage and the third fine-tuning stage are different. When the number of sample learning rounds of the to-be-trained Wensheng video model in the first fine-tuning stage reaches the first round number threshold, the second fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the second fine-tuning stage reaches the second round number threshold, the third fine-tuning stage is entered. When the number of sample learning rounds of the to-be-trained Wensheng video model in the third fine-tuning stage reaches the third round number threshold, the current Wensheng video model to be trained is used as the target Wensheng video model.

12. The method according to claim 1, characterized in that The step of inputting a target cross-language description into the target language video model to generate corresponding video data based on the target language video model and the target cross-language description includes: Inputting the target cross-lingual description into a text encoder to convert the target cross-lingual description into a target cross-lingual feature vector based on the text encoder; Inputting the target cross-language feature vector into a target embedding alignment layer to convert the target cross-language feature vector into a target aligned embedding vector based on the target embedding alignment layer; While inputting the target alignment embedding vector into the target diffusion model, target noise is input into the target diffusion model, so that the target diffusion model converts the target noise into target video feature encoding according to the target alignment embedding vector; Generate video data corresponding to the target cross-language description according to the target video feature encoding; The target text-generated video model includes the text encoder, the target embedding alignment layer and the target diffusion model.

13. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 12.

Images