Video generation method and system, and model training method and system

Abstract

The invention provides a video generation method and system and a model training method and system. According to the method, after video description information and N semantic extension instructions are obtained, semantic extraction is carried out on the video description information and the N semantic extension instructions based on a semantic feature generation network in a pre-trained video generation model to obtain target semantic features; and performing video generation based on the target semantic features through a video generation network in the video generation model to obtain and output a target video conforming to the semantics of the video description information. In the method, the semantic feature generation network and the video generation network are jointly trained in the training process of the video generation model, so that the semantic feature generation network can generate target semantic features which better meet the requirements of the video generation network; the video generation network can generate the high-quality target video more accurately according to the target semantic features, so that the overall performance of the video generation model and the generation effect of the target video are improved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a video generation, model training method and system. Background Art

[0002] Vincent video technology can convert text into video content and is widely used in advertising, content creation, education, and product display. Based on the video descriptive text input by the user, this technology can generate videos with a specific style and duration, greatly reducing the threshold and cost of video production and improving the efficiency and diversity of content creation. However, in this video generation method, the quality of the video mainly depends on the video description information input by the user, and there is a problem that the output video quality fluctuates too much and the video quality cannot be guaranteed.

[0003] Therefore, it is necessary to provide a method that can stably output high-quality video.

[0004] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0005] This specification provides a video generation and model training method and system, which can stably generate high-quality videos based on video description information.

[0006] In a first aspect, the present specification provides a video generation method, comprising: obtaining video description information and N semantic extension instructions, wherein N is an integer greater than or equal to 1; obtaining a pre-trained video generation model, wherein the video generation model comprises a semantic feature generation network and a video generation network, wherein, during the training process of the video generation model, the semantic feature generation network and the video generation network are jointly trained; performing semantic extraction on the video description information and the N semantic extension instructions through the semantic feature generation network to obtain target semantic features, and performing video generation based on the target semantic features through the video generation network to obtain a target video that is semantically consistent with the video description information, wherein the target semantic features represent the semantic features after the video description information is semantically extended through the N semantic extension instructions; and outputting the target video.

[0007] In a second aspect, the present specification also provides a method for training a video generation model, the method comprising: obtaining a sample set; the sample set comprising a plurality of sample pairs, each sample pair comprising a sample video and sample description information; obtaining a video generation model to be trained, the video generation model comprising a semantic feature generation network and a video generation network; and using the sample set to perform multiple iterative training on the video generation model to obtain a trained video generation model, wherein each iterative process comprises: performing semantic extraction on the sample description information and N semantic extension instructions through the semantic feature generation network to obtain sample semantic features, the sample semantic features representing the semantic features after the sample description information is semantically extended through the N semantic extension instructions; performing video generation based on the sample semantic features through the video generation network to obtain a predicted video, and updating the parameters of the semantic feature generation network and the video generation network with the training goal of minimizing the difference between the predicted video and the sample video.

[0008] In a third aspect, the present specification also provides a video generation system, comprising at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for performing a video generation method; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes the video generation method described in any one of the first aspects above according to the instructions of the at least one instruction set.

[0009] In a fourth aspect, the present specification also provides a training system for a video generation model, comprising at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for performing a training method for a video generation model; the at least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes according to the instructions of the at least one instruction set: the training method for a video generation model described in any one of the second aspect above.

[0010] In a fifth aspect, the present specification also provides a computer-readable non-volatile storage medium, wherein the computer-readable non-volatile storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, it implements the method described in any one of the first aspect or the second aspect above.

[0011] As can be seen from the above technical solutions, the video generation method provided in this specification obtains video description information and N semantic extension instructions, and obtains a pre-trained video generation model. The semantic feature generation network and the video generation network included in the video generation model are jointly trained during the training process of the video generation model, thereby achieving balanced learning of visual elements and language elements within a unified framework. The semantic feature generation network after joint training can generate target semantic features that are more in line with the needs of the video generation network, and the video generation network after joint training can more accurately generate high-quality target videos based on the target semantic features, thereby improving the performance of the video generation model and the video generation effect as a whole. In addition, the target semantic feature in this specification is comprehensively determined based on the video description information and N semantic extension instructions, and the video description information is supplemented and extended by introducing N semantic extension instructions, thereby achieving comprehensive and detailed feature acquisition at the semantic level. Thus, the feature information contained in the target semantic feature is not limited to the video description information input by the user, but can integrate more semantic elements contained in the semantic extension instructions, thereby improving the richness and accuracy of the target semantic feature expression. The video content of the target video generated according to such a target semantic feature is richer, and the quality of the video content is more stable.

[0012] The video generation, model training method and other functions of the system provided in this specification will be partially listed in the following description. The creative aspects of the video generation, model training method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of an application scenario provided according to an embodiment of this specification is shown;

[0015] Figure 2 A schematic diagram of the hardware structure of a computing device provided according to some embodiments of this specification is shown;

[0016] Figure 3 A schematic diagram of a process of training a video generation model provided according to an embodiment of this specification is shown;

[0017] Figure 4A schematic diagram of a flow chart of a method for training a video generation model provided according to another embodiment of the present specification is shown;

[0018] Figure 5 A schematic diagram showing a flow chart of a process for determining a semantic feature of a sample provided according to an embodiment of this specification;

[0019] Figure 6 A schematic diagram showing a flow chart of a video generation method provided according to an embodiment of the present specification; and

[0020] Figure 7 A schematic flow chart of a video generation method provided according to another embodiment of the present specification is shown. DETAILED DESCRIPTION

[0021] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0022] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0023] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0024] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0025] In this specification, "X includes at least one of A, B or C" means that X includes at least A, or X includes at least B, or X includes at least C. That is, X may include only any one of A, B, and C, or may include any combination of A, B, and C and other possible contents / elements at the same time. Any combination of A, B, and C may be A, B, C, AB, AC, BC, or ABC.

[0026] In this specification, unless explicitly stated otherwise, the association relationship between structures can be a direct association relationship or an indirect association relationship. For example, when describing "A is connected to B", unless it is explicitly stated that A is directly connected to B, it should be understood that A can be directly connected to B or indirectly connected to B; for another example, when describing "A is above B", unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements and A is above B). And so on.

[0027] It should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0028] For the convenience of description, the terms that will appear in the following text of this specification are first explained.

[0029] Large Language Models (LLMs): Usually used in the field of artificial intelligence for natural language processing (NLP), especially referring to large machine learning models with a large number of parameters and computing resources. Large models are named because of their huge number of parameters and complex network structures. They have powerful feature representation and feature understanding capabilities, and can better capture patterns and regularities in data when processing complex tasks. They are designed and trained to better understand and generate natural language.

[0030] Diffusion models: a type of generative network that performs well in image generation, audio synthesis, video generation and other fields. It aims to generate new data samples by learning the distribution of data. It is often used in the field of image generation and can generate extremely realistic and high-quality images and other data with rich details and natural textures. The working principle of the diffusion network includes the forward process and the reverse process. The forward process is also called the noise addition process, which gradually converts the input data into noise. The reverse process is also called the denoising process, which gradually recovers the input data from the noise.

[0031] In the scenario of video generation (generating a video corresponding to a description text based on a description text), the user generally inputs a video description information, and then the video generation model generates a video corresponding to the video description information based on the video description information input by the user. However, this video generation method has the problem that the quality of the generated video is highly correlated with the video description information input by the user. The more video description information the user inputs, the better the quality of the generated video. The less video description information the user inputs, the worse the quality of the generated video.

[0032] For example, if the video description information input by the user is "a woman with her family, driving a red car on an endless road, a river flowing around the road, and a plane flying in the sky", it is obvious that such description information is very rich, and the video generated based on such description information will also have rich content. However, if the video description information input by the user is "a car running on the road", it is obvious that such description information is too monotonous, and the video generation model cannot obtain more information based on the above description information, and the generated video content will also be monotonous.

[0033] That is to say, in the process of generating a video in the prior art, the quality of the generated video content mainly depends on the quality of the video description information input by the user. The more and richer the video description information input by the user, the better the video quality. The less video description information input by the user, the worse the video quality. The quality of the video content output by such a video generation method will fluctuate greatly with the quality of the video description information, and there is a problem of unstable quality of the generated video.

[0034] The video generation method provided in this specification obtains a pre-trained video generation model. The semantic feature generation network and the video generation network contained in the pre-trained video generation model are jointly trained in the process of training the video generation model. Therefore, the semantic feature generation network can generate target semantic features that are more suitable for the needs of the video generation network, and the video generation network can more accurately generate high-quality target videos based on the target semantic features, thereby improving the overall performance of the video generation model and the generation effect of the target video. In addition, after obtaining the video description information and N semantic extension instructions, the video description information and the N semantic extension instructions can be semantically extracted by the semantic feature generation network to obtain the target semantic features, and the target video can be generated based on the target semantic features by the video generation network. In this way, the target semantic features are supplemented by the N semantic extension instructions. Since the target semantic features are determined based on the video description information and the N semantic extension instructions, even for simple video description information, the semantic extension instructions can be used to semantically expand them, dig out more potential details and elements, and obtain more comprehensive and rich target semantic features. Then, the content of the target video generated according to the target semantic features will be richer. This setting method provides users with a more reliable video generation method, and users do not need to worry about whether the video description text is rich enough. For different quality video description texts input by different users, the video generation method provided in this manual can always obtain a target video with relatively guaranteed video content quality.

[0035] It should be noted that the above description of the application scenario is only one of the multiple usage scenarios provided in this specification. Those skilled in the art should understand that when the video generation, model training method and system provided in this specification are applied to other usage scenarios, their implementation methods and technical effects are similar.

[0036] Figure 1 A schematic diagram of an application scenario 100 provided according to an embodiment of this specification is shown.

[0037] like Figure 1 As shown, the application scenario 100 may include a video generation model training system 130 (hereinafter referred to as training system 130) and a video generation system 150. The application scenario 100 involves two execution stages.

[0038] In the first stage, the training system 130 can perform multiple iterations of training on the video generation model to be trained based on multiple sample pairs in the sample set to optimize the parameters in the semantic feature generation network and the video generation network included in the video generation model. Each sample pair includes a sample video and sample description information. The trained video generation model is deployed in the video generation system 150.

[0039] In the second stage, after obtaining the video description information and the N semantic extension instructions, the video generation system 150 can perform semantic extraction on the video description information and the N semantic extension instructions based on the semantic feature generation network in the video generation model to obtain the target semantic features. The video generation network in the video generation model generates the video based on the target semantic features to obtain and output the target video that is semantically consistent with the video description information.

[0040] The training system 130 may be a computing system with certain computing capabilities. The training system 130 may execute the training method of the video generation model provided in this specification, thereby training the video generation model to be trained based on the sample set to obtain the trained video generation model. The training system 130 may store data or instructions for executing the training method of the video generation model described in this specification, and may execute or be used to execute the data or instructions. The training system 130 may include a hardware device with data information processing functions and the necessary programs required to drive the hardware device to work.

[0041] The training system 130 may be a single computing device or a cluster system composed of multiple computing devices. The data or instructions stored in the training system 130 for executing the training method of the video generation model described in this specification may adopt any form of system architecture, such as a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture.

[0042] In some embodiments, the training system 130 may first obtain a sample set including a plurality of sample pairs. Each sample pair includes a sample video and sample description information. Further, the training system 130 performs multiple iterative training on the video generation model based on the sample set to obtain a trained video generation model. In each iterative process, the semantic feature generation network in the video generation model performs semantic extraction on the sample description information and N semantic extension instructions to obtain sample semantic features. The video generation network in the video generation model generates a video based on the sample semantic features to obtain a predicted video. Then, the parameters of the semantic feature generation network and the video generation network are updated with the training goal of minimizing the difference between the predicted video and the sample video.

[0043] The video generation model obtained by training the training system 130 can achieve that when the semantic feature generation network in the video generation model performs semantic extraction for different sample description information, the difference between the obtained sample semantic features fluctuates within a preset range. Thus, the video generation network in the video generation model can stably output a predicted video with the minimum difference from the sample video based on the sample semantic features.

[0044] The video generation system 150 may be a computing system with certain computing capabilities. The video generation system 150 may execute the video generation method provided in this specification, thereby generating and outputting a target video based on the obtained video description information and N semantic extension instructions. The video generation system 150 may store data or instructions for executing the video generation method described in this specification, and may execute or be used to execute the data or instructions. The video generation system 150 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0045] The video generation system 150 may be a single computing device or a cluster system composed of multiple computing devices. The data or instructions stored in the video generation system 150 for executing the video generation method described in this specification may adopt any form of system architecture, such as a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc.

[0046] In some embodiments, the video generation system 150 may first obtain a pre-trained video generation model, video description information, and N semantic extension instructions. Furthermore, the video generation system 150 may perform semantic extraction on the video description information and the N semantic extension instructions through the semantic feature generation network in the above video generation model to obtain target semantic features. The video generation system 150 then performs video generation on the target semantic features through the video generation network in the above video generation model to obtain a target video that is semantically consistent with the video description information. Finally, the video generation system 150 may output the target video.

[0047] The above method makes the target video finally output by the video generation system 150 to be a target video that is consistent with the semantics of the video description information. Compared with the method of directly determining and outputting the video based on the video description information, the above method makes the difference between the target semantic features output by the semantic feature generation network fluctuate within a preset range for different video description information. In other words, the difference between the target semantic features corresponding to the rich description information and the simple description information is within a preset range. Even for very simple video description information, target semantic features containing rich features can be output, which greatly reduces the impact of the video description information on the quality of the generated target video, thereby stabilizing the performance of the video generation model, and then stabilizing the video generation quality generated by the video generation model.

[0048] The training system 130 and the video generation system 150 may correspond to the same computing system or may correspond to different computing systems respectively, which is not limited in this specification.

[0049] Figure 2FIG. 1 is a schematic diagram of a hardware structure of a computing device 200 provided according to some embodiments of the present specification. The training system 130 and the video generation system 150 may have the following features: Figure 2 The structure of computing device 200 is shown.

[0050] like Figure 2 As shown, computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, computing device 200 may further include an internal communication bus 210. In some embodiments, computing device 200 may further include a communication port 250. In some embodiments, computing device 200 may further include an I / O component 260.

[0051] Internal communication bus 210 may connect various system components, including storage media 230 and processor 220. I / O components 260 support input / output between computing device 200 and other components.

[0052] The communication port 250 is used for data communication between the computing device 200 and the outside world. For example, the computing device 200 can be connected to a network through the communication port 250.

[0053] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the video generation method or the training method of the video generation model provided in this specification.

[0054] At least one processor 220 is connected to at least one storage medium 230 via an internal communication bus 210. At least one processor 220 is used to execute the at least one instruction set. When the system 130 is running, at least one processor 220 reads at least one instruction set and executes the video generation method or the video generation model training method provided in this specification according to the instructions of at least one instruction set.

[0055] The processor 220 may execute all steps included in the video generation method or the training method of the video generation model. The processor 220 may be in the form of one or more processors. The processor 220 may issue execution instructions. The processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0056] For illustrative purposes only, only one processor 220 is shown in the computing device 200 in the attached drawings in this specification. However, it should be noted that the computing device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as in this specification, or may be performed jointly by multiple processors. For example, if the processor 220 of the computing device 200 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0057] Figure 3 A schematic diagram of a process of training a video generation model provided according to an embodiment of this specification is shown; Figure 4 FIG. 1 is a flow chart of a method for training a video generation model according to another embodiment of the present specification; the method for training the video generation model P300 may be executed by the training system 130. Figure 3 and Figure 4 As shown, the method P300 provided in this specification may include S310-S350, wherein:

[0058] S310: Obtain a sample set; the sample set includes multiple sample pairs, and each sample pair includes a sample video and sample description information.

[0059] In order to improve the performance of the video generation model, it is necessary to ensure the diversity of samples during the training process. Therefore, the sample set may include a large number of sample videos of different types and scenes. For example, the sample set may include: landscape type videos, character type videos, animal type videos, videos of different styles (cartoon style, oil painting style, retro style, realistic style, etc.), architectural type videos, etc. The sample description information is used to describe the content in the sample video. The sample description information can be simple information describing the sample video, such as "autumn scenery"; "children flying kites"; "fishing", or it can be rich and complex information describing the sample video, such as: "Yellow maple leaves in the oil painting style slowly fall on the asphalt road, with some pedestrians and people riding bicycles next to them"; "A five-year-old boy and his parents are flying swallow kites in the park, and the family of three is happy"; "Two kittens are fishing, one kitten is very serious, and the other kitten is busy catching butterflies". It should be understood that the above embodiment is only a rational explanation, and the type of sample videos contained in the specific sample set and the type of sample description information can be flexibly adjusted according to user needs or actual applications, and are not limited to the above embodiment.

[0060] S330: Obtain a video generation model to be trained, where the video generation model includes a semantic feature generation network and a video generation network.

[0061] In the embodiment of this specification, the semantic feature generation network and the video generation network are used as a part of the video generation model, and the semantic feature generation network and the video generation network are jointly trained along with the iterative training of the video model, so that the balanced learning of visual elements and language elements in a unified framework is realized. The video generation model is optimized as a whole, and the compatibility problems and performance loss problems that may occur during use caused by training the semantic feature generation network and the video generation network separately and then combining them are avoided. In the process of joint training, the parameters of the video generation network and the parameters of the semantic feature generation network can be continuously adjusted following the iteration of the video generation model. The video generation network and the semantic feature generation network can be better adapted to each other, and then the performance of the video generation model in the face of various video descriptions and video generation tasks is improved. The semantic feature generation network and the video generation network after joint training can cooperate with each other and work in coordination.

[0062] S350: Perform multiple iterations of training on the video generation model using the sample set to obtain a trained video generation model.

[0063] In each iteration process, the training system 130 performs similar steps based on each sample pair. The following takes sample pair 1 as an example (sample pair 1 includes: sample video 1 and sample description information 1) to introduce the specific situation of the training system 130 executing S351-S353 based on sample pair 1. The steps executed by the training system 130 based on other sample pairs except sample pair 1 are similar and will not be repeated.

[0064] S351: semantically extracting the sample description information 1 and N semantic extension instructions through a semantic feature generation network to obtain a sample semantic feature, where the sample semantic feature represents the semantic feature after the sample description information 1 is semantically extended through the N semantic extension instructions.

[0065] In the embodiments of the present specification, N semantic extension instructions are used to provide a series of instruction prompts. The semantic extension instructions are some pre-set characteristic information with specific pointing or guiding functions, which, as a reference benchmark, provide an initial guiding direction for subsequent processing. For example, the semantic extension instructions may include any form such as "please describe the type of movement in the video", "please describe the relationship between the characters in the video", "please describe the theme color in the video", "please describe the environmental conditions in the video", "please describe the weather conditions in the video", etc., and are not limited to those given in the above embodiments.

[0066] Figure 5 A flow chart of a process for determining a sample semantic feature according to an embodiment of the present specification is shown. Figure 5 As shown, step S351 also includes:

[0067] S3511: Use the large model to perform semantic extraction on the sample description information 1 to obtain the first sample semantic feature.

[0068] In an embodiment of the present specification, the process of obtaining the first sample semantic feature can be: using a large model to perform semantic extraction on the sample description information 1 from multiple dimensions to obtain sample semantic sub-features corresponding to the multiple dimensions. Extracting sample semantic sub-features of multiple dimensions can make the extracted sample semantic sub-features cover multiple aspects of the sample description information. Compared with the feature extraction of a single dimension, the sample semantic sub-features extracted by the above-mentioned multiple-dimensional extraction method are richer and more comprehensive. Subsequently, the sample semantic sub-features corresponding to the multiple dimensions are spliced ​​to obtain a sample splicing feature. The sample splicing feature is then convolved to obtain the first sample semantic feature.

[0069] After convolution processing, the spliced ​​features can better extract, fuse and optimize the sample semantic sub-features corresponding to multiple dimensions. The feature interaction between the sample semantic sub-features corresponding to multiple dimensions is made more sufficient to obtain a more comprehensive, stable and representative first sample semantic feature. In addition, the spliced ​​features may have a higher dimension, and the convolution processing can map the high-dimensional features to a relatively low-dimensional space. In this way, a more compact and easier to understand first sample semantic feature is obtained, which helps to reduce the subsequent calculation amount and improve the operation efficiency of the model.

[0070] In an embodiment of the present specification, the video generation model also includes a convolution layer, through which the sample splicing features are convoluted to obtain a first sample semantic feature. In order to reduce the impact of the convolution layer on the backbone network (semantic feature generation network and video generation network) and improve the convergence speed of the model, the initial state of the convolution layer can be set to 0, and the parameters of the convolution layer gradually stabilize as the model is trained. During the training process of the video generation model, the convolution layer, the semantic feature generation network, and the video generation network are jointly trained to continuously adjust the parameters of the convolution layer, and the parameters in the trained convolution layer remain unchanged.

[0071] The sample semantic sub-features of each dimension may include question features and answer features. The question features are generated by the big model based on the sample description information. In other words, the big model directly extracts feature information that can reflect the basic situation and key elements of the video from the sample description information. This information constitutes the question features corresponding to the sample description information. The question features can be understood as the refinement of the inherent features of the sample description information itself, which can reflect the demand information in the sample description information.

[0072] The answer feature is determined by the large model through an autoregressive algorithm. When processing sample description information, the large model can continuously predict the next possible word based on the existing text sequence in the sample description information. By allowing the large model to continuously perform this operation of predicting the next word and continuously collecting relevant feature information during the prediction process, the answer feature corresponding to the sample description information is finally obtained. In other words, the answer feature is some of the features captured in the process of the large model dynamically generating subsequent text based on the sample description information.

[0073] For example, when the sample description information is "Kitten Fishing", the corresponding question features and answer features are as follows: The above sample description information may include core description dimension, content detail dimension and value dimension, and each dimension has its corresponding question features and answer features.

[0074] For example, the question features corresponding to the core description dimension may include: "What is the main story of the kitten fishing?" This question feature reflects the information demand for the story outline in the video, and clearly points to the core description of the story content of "The kitten fishing". The answer feature corresponding to the above question feature may be "The kitten went fishing with his mother. At first, the kitten was distracted, catching butterflies and dragonflies, and finally caught no fish. Later, the kitten listened to his mother, concentrated on fishing, and finally caught a fish." This answer feature provides a complete story content to meet the question feature's demand for an outline.

[0075] For the content detail dimension, the question feature may include: "Which animals did the kitten encounter while fishing?" This indicates an interest in the specific plot and characters in the story, and the question feature reflects the need to explore the details of the story. The answer feature corresponding to the above question feature may be "While fishing, the kitten encountered butterflies and dragonflies. It put down its fishing rod to chase butterflies and dragonflies, which delayed its fishing." Such an answer feature responds to the question feature's attention to details by listing specific animals and the kitten's interactions with them.

[0076] For the value dimension, the question feature may include: "What inspiration can the story of the kitten fishing bring to children?" It reflects the concern for the educational significance of the story, and is a question feature that rises from content understanding to value mining. The answer feature corresponding to the above question feature may be "This story tells children to concentrate on doing things and not to be distracted, just like the kitten can't catch fish if it is not focused at first, but can only catch fish after concentrating." This answer feature extracts educational value from the story content to meet the inspiration needs of the question feature. It should be understood that the above embodiments are only exemplary descriptions, and the specific circumstances of multiple dimensions, the question features and answer features that may be included in each dimension are determined according to the specific circumstances of the video description information and the knowledge storage and allocation of the large model, and are not limited to the above embodiments.

[0077] S3513: Use the large model to perform semantic extraction on the N semantic extension instructions to obtain second sample semantic features corresponding to the N semantic extension instructions.

[0078] Semantic extraction of semantic extension instructions is similar to analyzing text, which extracts key semantic information from semantic extension instructions. The second sample semantic features obtained focus on reflecting the semantic characteristics of the semantic extension instructions themselves, including the goals of the semantic extension instructions (such as extracting certain information), the objects involved (such as characters and scenes in the video, etc.), and the actions (such as analysis and recognition, etc.).

[0079] Taking the semantic expansion instruction "analyze the topic of conversation between characters in the video" as an example, the big model will identify the key elements "video", "character", "conversation topic", and "analysis". The big model may combine these key elements into a second semantic feature that can represent the corresponding semantic expansion instruction through word vector representation or other semantic representation methods.

[0080] S3515: Perform feature fusion on the first sample semantic feature and the second sample semantic features corresponding to the N semantic extension instructions to obtain a sample semantic feature.

[0081] The first sample semantic feature is extracted from the sample description information, which covers the basic content of the predicted video itself that the user expects to be generated later, such as the semantics of scenes, characters, events, etc. The second sample semantic feature comes from the semantic extension instruction, which contains the requirements for specific processing or mining of the predicted video, such as extracting emotional details, identifying specific objects, etc. By fusing the first sample semantic feature and the second sample semantic feature, the basic content of the predicted video and the requirements of specific analysis can be combined, thereby achieving a more comprehensive video semantic understanding.

[0082] In an embodiment of the present specification, the semantic feature generation network also includes a semantic stabilizer, which is used to fuse the first sample semantic feature and the second sample semantic feature. The fused sample semantic features contain more semantic dimensions, including both the content dimension of the predicted video itself that is expected to be generated subsequently in the sample description information, and the dimension of analyzing and mining the predicted video. This enables the sample semantic features to express the information of the video more richly. For example, in the sample description information, the semantic sub-feature about the scene may be "wedding scene". After fusing the above sample semantic sub-features with the second sample semantic sub-features about sentiment classification, a more detailed semantic feature "a wedding with a strong emotional atmosphere" can be obtained.

[0083] In the embodiments of the present specification, in order to further implement the constraint of the first sample semantic feature corresponding to the sample description text through the semantic stabilizer, in the process of determining the sample semantic feature, it is also necessary to obtain the third sample semantic features corresponding to N semantic extension instructions. For each semantic extension instruction, the second sample semantic feature and the third sample semantic feature corresponding to the semantic extension instruction are fused to obtain the sample comprehensive semantic feature corresponding to the semantic extension instruction. The first sample semantic feature and the sample comprehensive semantic features corresponding to each of the N semantic extension instructions are spliced ​​to obtain the sample semantic feature.

[0084] Among them, the third sample semantic features corresponding to each of the N semantic extension instructions are learned during the training process of the video generation model. The third sample semantic features can be weight features corresponding to each semantic extension instruction. In an embodiment of the present specification, a semantic stabilizer is used to first fuse the second sample semantic features and the third sample semantic features corresponding to each of the N semantic extension instructions to obtain a sample comprehensive semantic feature corresponding to the semantic extension instruction. The sample comprehensive semantic feature is then fused with the first sample semantic feature to obtain a stable sample semantic feature.

[0085] The above method of determining the comprehensive semantic features of the sample, since the third sample semantic feature is learned during the training process of the video generation model, represents a relatively stable and optimized feature, which has a certain degree of universality and representativeness. Fusion of the second sample semantic feature and the third sample semantic feature corresponding to the semantic extension instruction can make the second sample semantic feature and the third sample semantic feature complement each other, making the sample comprehensive semantic feature richer and more accurate. In addition, due to the stability of the third sample semantic feature, the sample comprehensive semantic feature is more stable.

[0086] The first sample semantic feature and the sample comprehensive semantic features corresponding to each of the N semantic extension instructions are spliced ​​together to obtain the sample semantic feature. The feature combination of two different sources (the first sample semantic feature comes from the sample description information; the sample comprehensive semantic feature comes from the N semantic extension instructions and the third sample semantic feature) is spliced ​​together to form a stable sample semantic feature.

[0087] Among them, the first sample semantic feature is obtained by processing the sample video description text, and contains rich detailed information extracted from the sample video description text. Such as the specific attributes of the object, the order of actions, the spatial relationship of the scene, etc., which can comprehensively and meticulously describe the semantic content of the video from the text level. The sample comprehensive semantic feature is formed by combining the third sample semantic feature under the guidance of the second sample semantic features corresponding to each of the N semantic extension instructions. The sample comprehensive semantic feature constrains and supplements the first sample semantic feature. Splicing the sample comprehensive semantic feature with the first sample semantic feature can realize the comprehensive integration of multi-dimensional and multi-level semantic information, so that the sample semantic feature can cover various semantic features required for predictive video generation. Including but not limited to macro-theme style requirements, general semantic patterns and specific text details. Provide a more sufficient and complete semantic basis for the subsequent generation of high-quality, rich and logical predictive videos.

[0088] S353: Generate a predicted video based on the sample semantic features through the video generation network, and take minimizing the difference between the predicted video and the sample video 1 as the training goal, and update the parameters of the semantic feature generation network and the video generation network.

[0089] Since the diffusion model has powerful semantic understanding ability, diverse generation ability and stability of progressive generation, in some embodiments of this specification, the video generation network may include a diffusion model. The diffusion model generates videos based on the acquired sample semantic features to obtain sample videos corresponding to the sample description information. Of course, in other possible embodiments, the video generation network may also include other models for video generation, such as adversarial generation networks, Transformer models, etc. The selection of specific models can be flexibly adjusted according to the needs of the application scenario and is not limited to the above embodiments.

[0090] In order to achieve a balance between visual elements and semantic elements and improve the performance of the video generation model, in the embodiments of this specification, visual features of sample video 1 in sample pair 1 will also be extracted. In some possible embodiments, the visual features of sample video 1 can be extracted by variational autoencoder (VAE). Of course, other feature extraction methods can also be selected, such as extracting visual features based on convolutional neural networks, or extracting visual features based on optical flow methods, etc. This specification does not impose any restrictions here. Subsequently, the extracted visual features corresponding to sample video 1 are fused with the sample semantic features corresponding to sample video description text 1, and the fused features are input into the video generation model, so that the video generation model outputs predicted video 1 based on the above fused features. Thereby achieving the fusion of visual features and semantic features during the training process.

[0091] After outputting the predicted video 1, the training system 130 determines the difference between the predicted video 1 and the sample video 1, and compares the difference with the preset threshold. When the difference is less than the preset threshold, it is considered that the error of the current video generation model is within the preset range, and the video generation model has been successfully trained, and the training of the video generation model is stopped. Otherwise, the video generation model is iteratively trained based on other sample pairs in the sample set except sample pair 1 until the difference between the predicted video and the sample video is minimized.

[0092] Since the visual features and semantic features are fused in the above training process, and video generation is performed based on the fused features, the visual features and semantic features are comprehensively considered in the training process, so that the visual elements and language elements can be balanced in learning within a unified framework. The semantic feature generation network after joint training can generate sample semantic features that are more in line with the needs of the video generation network. The video generation network after joint training can more accurately generate high-quality predicted videos based on the target semantic features, thereby improving the performance and generation effect of the video generation model as a whole.

[0093] It is understandable to those skilled in the art that, after the above-mentioned model training process, the video generation model is trained so that, for different sample description information, the differences between the sample semantic features output by the semantic feature generation network fluctuate within a preset range. In other words, whether it is simple sample description information or complex sample description information, the differences between the sample semantic features output by the semantic feature generation network are within a preset range. This reduces the degree to which the quality of the video generated by the video generation model depends on the video description information. Even in the case where the video description information is relatively simple, based on the semantic extension capabilities learned by the video generation model during the training process, it is possible to output target semantic features with rich and comprehensive semantics, thereby generating high-quality videos based on the target semantic features.

[0094] After the video generation model training is completed, the relevant parameters in the semantic feature generation network will remain unchanged, and the relevant parameters in the video generation network will remain unchanged. Until the next training of the video generation model, the relevant parameters in the semantic feature generation network and the relevant parameters in the video generation network will be further adjusted following the training of the video generation model.

[0095] For the embodiment in which the video generation model also includes a convolution layer, after the video generation model training is completed, it is also determined that the relevant parameters in the convolution layer have been trained. Until the next training of the video generation model, the relevant parameters in the convolution layer will be further adjusted following the training of the video generation model.

[0096] In summary, the training method and training system of the video generation model provided in this specification obtain a video generation model to be trained and a sample set including multiple sample pairs. The video generation model generates a predicted video based on the sample description information in the multiple sample pairs, and the predicted videos corresponding to the multiple sample pairs are differentially compared with the sample videos corresponding to the multiple sample pairs, with the goal of minimizing the difference between the predicted video and the sample video, and the video generation model is iteratively trained multiple times until the trained video generation model is obtained. Among them, the semantic feature generation network and the video generation network included in the video generation model will be jointly trained following the training process of the video generation model. When the trained video generation model generates a target video based on the video description information input by the user, the richness of the video content of the target video can be guaranteed, and the stability of the quality of the video content of the target video can be improved.

[0097] Figure 6 A schematic diagram of a process of a video generation method provided according to an embodiment of this specification is shown; Figure 7 FIG. 5 is a flow chart of a video generation method according to another embodiment of the present specification. The video generation method P500 may be executed by the video generation system 150. Figure 6 and Figure 7 As shown, the method P500 provided in this specification may include S510-S570, wherein:

[0098] S510: Obtain video description information and N semantic extension instructions, where N is an integer greater than or equal to 1.

[0099] In the embodiments of the present specification, the video description information is the video description information input by the user. The input method can be the video description text information directly input by the user, or it can be the video description information obtained by converting the voice information input by the user into text. The N semantic extension instructions can be N semantic extension instructions input by the user. Alternatively, they are N semantic extension instructions obtained from a preset instruction library. Alternatively, some semantic extension instructions are obtained from preset instructions, and some semantic extension instructions input by the user are obtained to form N semantic extension instructions. It should be understood that the above embodiments are only exemplary descriptions, and the specific methods for obtaining the video description information and semantic extension instructions can be flexibly adjusted according to the current application scenario and user needs, and are not limited to those given in the above embodiments.

[0100] S530: Obtain a pre-trained video generation model, where the video generation model includes a semantic feature generation network and a video generation network, wherein during the training process of the video generation model, the semantic feature generation network and the video generation network are jointly trained.

[0101] The semantic feature generation network and the video generation network are jointly trained in the training process of the video generation model. The trained semantic feature generation network and the video generation network achieve balanced learning of visual elements and language elements in a unified framework. This allows the semantic feature generation network and the video generation network to better adapt to each other, thereby improving the performance of the video generation model in the face of various video descriptions and video generation tasks. This allows the semantic feature generation network and the video generation network to cooperate and work together. In the subsequent video generation process, the video generation model trained in the above manner will neither overemphasize semantics and ignore visual presentation, nor only focus on video generation and weaken the integration of semantics. This ensures that the target video generated by the video generation model can accurately convey rich semantic content and have excellent visual performance, thereby improving the comprehensive quality of the generated target video in terms of content and form.

[0102] S550: semantically extract the video description information and N semantic extension instructions through a semantic feature generation network to obtain a target semantic feature, and generate a video based on the target semantic feature through a video generation network to obtain a target video that is semantically consistent with the video description information, wherein the target semantic feature represents the semantic feature after the video description information is semantically extended through N semantic extension instructions.

[0103] In the embodiment of the present specification, N semantic extension instructions are used to provide a series of instruction prompts. The semantic extension instructions are some pre-set characteristic information with specific pointing or guiding functions, which, as a reference benchmark, provide an initial guiding direction for subsequent processing. For example, the semantic extension instructions may include any form such as: "Please describe the type of movement in the video", "Please describe the relationship between the characters in the video", "Please describe the emotional color in the video", "Please describe the environmental atmosphere in the video", "Please describe the emotions of the characters in the video", etc.

[0104] In some possible embodiments, the semantic feature generation network may include a large model and a semantic stabilizer. In the process of determining the target semantic feature, the large model is used to perform semantic extraction on the video description information to obtain a first semantic feature. The large model is then used to perform semantic extraction on N semantic extension instructions to obtain second semantic features corresponding to the N semantic extension instructions. Subsequently, the semantic stabilizer performs feature fusion on the first semantic feature and the second semantic features corresponding to the N semantic extension instructions to obtain the target semantic feature.

[0105] That is, the method provided in this specification introduces N semantic extension instructions, and fuses the second semantic features corresponding to the N semantic extension instructions with the first semantic features corresponding to the video description information to obtain the target semantic features. The effect of supplementing the video description information through N semantic extension instructions is achieved. Whether it is simple video description information or complex video description information, it can be converted into relatively stable target semantic features, reducing the requirements for the video description information input by the user. The semantic generation network can better cope with video description information of various levels of detail. Even if the video description information input by the user is very simple, the semantic feature generation network can also dig out target semantic features with richer semantic information to a certain extent. The video quality of the target video generated based on the target semantic features containing rich semantic information will also be more stable. Thereby ensuring the stability of the quality of the target video generated by the video generation model based on the video description information.

[0106] S570: Output the target video.

[0107] The target video is output to the user, and the user can choose to download or save the target video. Alternatively, if the user is not satisfied with the output target video, the user can re-enter the video description information to regenerate and output the target video.

[0108] It can be understood that the technical details and beneficial effect descriptions involved in the video generation method P500 and the video generation model training method P300 can refer to each other and will not be repeated here.

[0109] In summary, in the video generation method and system provided by this specification, since the semantic feature generation network and the video generation network included in the video generation model are jointly trained during the training process of the video generation model, the semantic feature generation network can generate target semantic features that are more in line with the needs of the video generation network, and the video generation network can more accurately generate high-quality target videos based on the target semantic features, thereby improving the performance and generation effect of the video generation model as a whole. In addition, the target semantic features in this specification are comprehensively determined based on the video description information and N semantic extension instructions. By introducing N semantic extension instructions to supplement and extend the video description information, comprehensive and detailed feature acquisition at the semantic level is achieved. Therefore, the feature information contained in the target semantic features is not limited to the video description information input by the user, but can integrate more semantic elements contained in the semantic extension instructions, thereby improving the richness of the expression of the target semantic features.

[0110] In order to further improve the accuracy of the target semantic feature, this specification will first determine the first semantic feature of the video description information, and then determine the comprehensive semantic feature corresponding to the N semantic extension instructions. Then the target semantic feature is determined based on the first semantic feature and the comprehensive semantic feature. Wherein the semantic stabilizer first comprehensively determines the comprehensive semantic features of the N semantic extension instructions according to the second semantic features and the third semantic features corresponding to each of the N semantic extension instructions. Then the semantic stabilizer fuses the comprehensive semantic feature and the first semantic feature to obtain the target semantic feature. Since the third semantic feature is learned during the training process of the video generation model, the comprehensive semantic feature obtained in the above manner will also be more stable. The stability of the target semantic feature obtained based on such a stable comprehensive semantic feature and the first semantic feature has also been significantly improved. The target video generated according to such a target semantic feature not only ensures the richness of the video content of the target video, but also ensures the stability of the quality of the video content of the target video.

[0111] On the other hand, the present specification provides a computer-readable non-transitory storage medium, storing at least one instruction set, which is used to perform executable instructions for a video generation method or a training method for a video generation model. When at least one instruction set is executed by a processor, at least one instruction set instructs the processor to implement the steps of the video generation method P300 or the video generation model training method P500 of the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes program code. When the program product is run on the system 130, the program code is used to enable the system 130 to perform the steps of the video generation model training method P300 described in the present specification. The program product for implementing the above method can include the program code using a portable compact disk read-only memory (CD-ROM) and can be run on the system 130. When the program product is run on the system 150, the program code is used to enable the system 150 to perform the steps of the video generation method P500 described in the present specification. The program product for implementing the above method can include the program code using a portable compact disk read-only memory (CD-ROM) and can be run on the system 150. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.The program code for performing the operations of this specification can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0112] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0114] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0115] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0116] Each patent, patent application, publication of patent application, and other materials, such as articles, books, specifications, publications, documents, documents, etc., cited in this disclosure (excluding any historical review documents related thereto) are hereby incorporated by reference for all purposes related to this disclosure, such as in the specification and claims of this disclosure. However, if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology of the above materials and the descriptions, definitions, and / or terminology used in this disclosure, the descriptions, definitions, and / or terminology used in this disclosure shall prevail.

[0117] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A video generation method, comprising: Obtaining video description information and N semantic extension instructions, where N is an integer greater than or equal to 1; Obtaining a pre-trained video generation model, wherein the video generation model includes a semantic feature generation network and a video generation network, wherein during the training process of the video generation model, the semantic feature generation network and the video generation network are jointly trained; The video description information and the N semantic extension instructions are semantically extracted by the semantic feature generation network to obtain a target semantic feature, and a video is generated by the video generation network based on the target semantic feature to obtain a target video that is semantically consistent with the video description information, wherein the target semantic feature represents a semantic feature after the video description information is semantically extended by the N semantic extension instructions; and The target video is output.

2. The method according to claim 1, wherein: The step of performing semantic extraction on the video description information and the N semantic extension instructions to obtain target semantic features includes: Using the large model to perform semantic extraction on the video description information to obtain a first semantic feature; Using the large model to perform semantic extraction on the N semantic extension instructions to obtain second semantic features corresponding to the N semantic extension instructions; and Feature fusion is performed on the first semantic feature and the second semantic features corresponding to the N semantic extension instructions to obtain the target semantic feature.

3. The method of claim 2, wherein: The step of fusing the first semantic feature with the second semantic features corresponding to the N semantic extension instructions to obtain the target semantic feature includes: Obtaining third semantic features corresponding to the N semantic extension instructions; For each semantic extension instruction, fusing the second semantic feature and the third semantic feature corresponding to the semantic extension instruction to obtain a comprehensive semantic feature corresponding to the semantic extension instruction; and The first semantic feature and the comprehensive semantic features corresponding to each of the N semantic extension instructions are concatenated to obtain the target semantic feature.

4. The method of claim 3, wherein: The third semantic features corresponding to each of the N semantic extension instructions are learned during the training process of the video generation model.

5. The method of claim 2, wherein: The using of the large model to perform semantic extraction on the video description information to obtain a first semantic feature includes: Using the large model to perform semantic extraction on the video description information from multiple dimensions to obtain semantic sub-features corresponding to the multiple dimensions; splicing the semantic sub-features corresponding to the multiple dimensions to obtain a spliced ​​feature; and Perform convolution processing on the concatenated features to obtain the first semantic features.

6. The method of claim 5, wherein: The semantic sub-features corresponding to each dimension include question features and answer features. The question features are generated by the large model based on the video description information, and the answer features are determined by the large model through an autoregressive algorithm.

7. The method according to claim 1, wherein: The video generation model is trained such that, for different video description information, the differences between target semantic features output by the semantic feature generation network fluctuate within a preset range.

8. The method according to claim 1, wherein: Obtain video description information and N semantic extension instructions, including: Obtaining the video description information input by a user, and the N semantic expansion instructions input by the user; or The video description information input by the user is obtained, and the N semantic extension instructions are obtained from a preset instruction library.

9. A method for training a video generation model, the method comprising: Obtain a sample set; The sample set includes a plurality of sample pairs, each sample pair includes a sample video and sample description information; Obtaining a video generation model to be trained, wherein the video generation model includes a semantic feature generation network and a video generation network; as well as The video generation model is iteratively trained multiple times using the sample set to obtain a trained video generation model, wherein each iterative process includes: The sample description information and the N semantic extension instructions are semantically extracted by the semantic feature generation network to obtain a sample semantic feature, wherein the sample semantic feature represents the semantic feature after the sample description information is semantically extended by the N semantic extension instructions, and The video generation network generates a video based on the sample semantic features to obtain a predicted video, and the parameters of the semantic feature generation network and the video generation network are updated with the goal of minimizing the difference between the predicted video and the sample video.

10. The method according to claim 9, wherein: The performing semantic extraction on the sample description information and the N semantic extension instructions to obtain the sample semantic features includes: Using the large model to perform semantic extraction on the sample description information to obtain a first sample semantic feature; Using the large model to perform semantic extraction on the N semantic extension instructions to obtain second sample semantic features corresponding to the N semantic extension instructions; and Feature fusion is performed on the first sample semantic feature and the second sample semantic features corresponding to the N semantic extension instructions to obtain the sample semantic feature.

11. The method according to claim 10, wherein: The step of fusing the first sample semantic feature with the second sample semantic features corresponding to the N semantic extension instructions to obtain the sample semantic feature includes: Obtaining third sample semantic features corresponding to the N semantic extension instructions; For each semantic extension instruction, fusing the second sample semantic feature and the third sample semantic feature corresponding to the semantic extension instruction to obtain a sample comprehensive semantic feature corresponding to the semantic extension instruction; and The first sample semantic feature and the sample comprehensive semantic features corresponding to each of the N semantic extension instructions are concatenated to obtain the sample semantic feature.

12. The method according to claim 10, wherein: The using of the large model to perform semantic extraction on the sample description information to obtain a first sample semantic feature includes: Using the large model to perform semantic extraction on the sample description information from multiple dimensions to obtain sample semantic sub-features corresponding to the multiple dimensions; splicing the sample semantic sub-features corresponding to the multiple dimensions to obtain a sample splicing feature; and Convolution processing is performed on the sample splicing features to obtain the first sample semantic features.

13. The method according to claim 12, wherein: The sample semantic sub-features corresponding to each dimension include sample question features and sample answer features. The sample question features are generated by the large model based on the sample description information, and the sample answer features are determined by the large model through an autoregressive algorithm.

14. The method according to claim 9, wherein: The video generation model is trained such that, for different sample description information, the differences between the sample semantic features output by the semantic feature generation network fluctuate within a preset range.

15. A video generation system, comprising: At least one storage medium storing at least one instruction set for performing video generation; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes the method described in any one of claims 1-8 according to the instructions of the at least one instruction set.

16. A training system for a video generation model, comprising: At least one storage medium storing at least one instruction set for training a video generation model; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and executes the method described in any one of claims 9 to 14 according to the instructions of the at least one instruction set.

Images