Video generation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computer vision. According to the scheme, after a prompt text is obtained, static information is coded to obtain a first text feature, and motion information is coded to obtain a second text feature; the first text feature can be coded to generate a first potential representation and decoded to obtain an RGB image frame, and the second text feature can be coded to generate a second potential representation and decoded to obtain an optical flow field; and generating a target video by using the RGB image frame and the optical flow field. According to the scheme, the static information in the prompt text is used for executing RGB image reconstruction to provide rich visual appearance information, the motion information in the prompt text is used for executing optical flow field reconstruction to provide natural, smooth and real motion change information, mutual complementation can be achieved, the video generation requirements of large and fast motion change amplitude or complex actions are met, and the video generation efficiency is improved. The video generation quality is improved, the deviation is reduced, and the video content better conforms to the physical law.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, in particular to a video generation method, a video generation apparatus, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Video generation technology refers to a technology of generating video content on single modal or multi-modal data such as text, image and video by using trained AI (Artificial Intelligence).

[0003] In the field of text-to-video, only simple motion video generation can usually be achieved, and in the case of large and fast motion changes or complex actions described in the text, the generated video has the problems of low quality and large deviation, often does not conform to the physical law, and has poor practicability.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide a video generation method, a video generation apparatus, an electronic device and a computer readable storage medium, which can improve the video generation quality.

[0006] According to a first aspect of the present disclosure, a video generation method is provided, which can include: encoding static information of a prompt text to obtain a first text feature; encoding motion information of the prompt text to obtain a second text feature; performing encoding processing on the first text feature to generate a first latent representation, and decoding the first latent representation to obtain an RGB image frame; performing encoding processing on the second text feature to generate a second latent representation, and decoding the second latent representation to obtain an optical flow field; and generating a target video corresponding to the prompt text based on the RGB image frame and the optical flow field.

[0007] In an example embodiment of the present disclosure, the method is implemented based on a video generation model; the video generation model includes a first text encoder, a second text encoder, a first diffusion layer, a second diffusion layer, a first variational autoencoder and a second variational autoencoder.

[0008] In an example embodiment of the present disclosure, the encoding of the static information of the prompt text to obtain the first text feature includes: encoding the static information of the prompt text by the first text encoder to obtain the first text feature; the static information includes at least one of scene layout, object attribute, object color and object texture.

[0009] In an example embodiment of the present disclosure, the encoding of the prompt text with motion information obtains the second text feature, including: obtaining the second text feature by encoding the prompt text with motion information through a second text encoder; the motion information includes at least one of action type, motion trajectory, and speed change.

[0010] In an example embodiment of the present disclosure, the first diffusion layer encodes and processes the first text feature to generate the first latent representation, and decodes the first latent representation to obtain the RGB image frame, including: encoding and processing the first text feature through the first diffusion layer to obtain the first latent representation; decoding the first latent representation through a first variational autoencoder to obtain the RGB image frame.

[0011] In an example embodiment of the present disclosure, the second diffusion layer encodes and processes the second text feature to generate the second latent representation, and decodes the second latent representation to obtain the optical flow field, including: encoding and processing the second text feature through the second diffusion layer to generate the second latent representation; decoding the second latent representation through a second variational autoencoder to obtain the optical flow field.

[0012] In an example embodiment of the present disclosure, the training video data of the video generation model is filtered based on the complexity of the motion action.

[0013] In an example embodiment of the present disclosure, the filtering step of the training video data includes: performing target detection on the original video data to obtain preliminary filtered video data containing motion targets; calculating an action score of the preliminary filtered video data using an optical flow algorithm; filtering the preliminary filtered video data based on the action score to obtain training video data with the action score within the complexity range.

[0014] According to a second aspect of the present disclosure, a video generation device is provided, which can include: a text encoding module configured to encode a prompt text with static information to obtain a first text feature; the text encoding module is further configured to encode the prompt text with motion information to obtain a second text feature; an information reconstruction module configured to encode and process the first text feature to generate a first latent representation, and decode the first latent representation to obtain an RGB image frame; the information reconstruction module is further configured to encode and process the second text feature to generate a second latent representation, and decode the second latent representation to obtain an optical flow field; a video generation module configured to generate a target video corresponding to the prompt text based on the RGB image frame and the optical flow field.

[0015] In an example embodiment of the present disclosure, the device is implemented based on a video generation model; the video generation model includes a first text encoder, a second text encoder, a first diffusion layer, a second diffusion layer, a first variational autoencoder, and a second variational autoencoder.

[0016] In an example embodiment of the present disclosure, the text encoding module is specifically configured to obtain first text features by encoding the prompt text through a first text encoder based on static information; the static information includes at least one of scene layout, object attribute, object color, and object texture.

[0017] In an example embodiment of the present disclosure, the text encoding module is specifically configured to obtain second text features by encoding the prompt text through a second text encoder based on motion information; the motion information includes at least one of action type, motion trajectory, and speed change.

[0018] In an example embodiment of the present disclosure, the first diffusion layer information reconstruction module is specifically configured to obtain first latent representation by encoding and processing the first text features through a first diffusion layer; and obtain the RGB image frame by decoding the first latent representation through a first variational autoencoder.

[0019] In an example embodiment of the present disclosure, the information reconstruction module is specifically configured to generate second latent representation by encoding and processing the second text features through a second diffusion layer; and obtain the optical flow field by decoding the second latent representation through a second variational autoencoder.

[0020] In an example embodiment of the present disclosure, the training video data of the video generation model is obtained based on the complexity of the motion action.

[0021] In an example embodiment of the present disclosure, the device further includes a data acquisition module configured to perform target detection on the original video data to obtain preliminary screening video data containing motion targets; calculate the action score of the preliminary screening video data by using an optical flow algorithm; and perform screening on the preliminary screening video data based on the action score to obtain training video data with the action score within the complexity range.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to implement the above method by executing the executable instructions.

[0023] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0024] The embodiment of the present disclosure provides a video generation method, a video generation apparatus, an electronic device and a computer readable storage medium. The method comprises the following steps: obtaining prompt text; encoding static information of the prompt text to obtain first text features, and encoding motion information of the prompt text to obtain second text features; encoding the first text features to generate a first latent representation and then decoding to obtain an RGB image frame, and encoding the second text features to generate a second latent representation and then decoding to obtain an optical flow field; and generating a target video corresponding to the prompt text by using the RGB image frame and the optical flow field. In the process of generating the target video, the static information in the prompt text is used to perform RGB image reconstruction to provide rich visual appearance information, and the motion information in the prompt text is used to perform optical flow field reconstruction to provide natural, smooth and real motion change information. On this basis, the RGB image frame and the optical flow field can complement each other, can adapt to the video generation demand of large and fast motion change amplitude or complex action, improve the video generation quality, reduce the deviation, make the video content more in line with the physical law, and improve the practicability.

[0025] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0027] Figure 1 A step flowchart of a video generation method provided by an exemplary embodiment of the present disclosure is shown.

[0028] Figure 2 An architecture schematic diagram of a video generation model provided by an exemplary embodiment of the present disclosure is shown.

[0029] Figure 3 A structural block diagram of a video generation apparatus provided by an exemplary embodiment of the present disclosure is shown.

[0030] Figure 4 A composition schematic diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0032] In addition, the accompanying drawings are included to provide a thorough understanding of the present disclosure and are not intended to be exhaustive or to limit the present disclosure to the precise outlines described herein. The same or similar components have the same or similar reference numbers across the drawings and repeated description is omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] Figure 1 One of the step flowcharts of the video generation method provided according to an example embodiment of the present disclosure is shown, as shown in Figure 1 The method can include the following steps 101 to 105.

[0034] Step 101, encode static information in the prompt text to obtain first text features.

[0035] In an embodiment of the present disclosure, the prompt text can be a requirement description text provided when requesting to generate a video, which can include roles, scenes, actions and other description information, such as "a cat chasing a butterfly on the grass", "a female actress dancing ballet on the stage, the stage background is dark, and the light shines on the actress" and the like. Based on the obtained prompt text, the static information in the prompt text can be encoded, and the text description related to the static elements can be represented by features, so as to be converted into a representation form that can be understood, used and processed by a model. The first text features are obtained by encoding the static information, which can accurately guide the reproduction of the static elements in the text description in the subsequent video generation process.

[0036] In an embodiment of the present disclosure, the static information can be extracted by text recognition on the prompt text, and the static information can be encoded based thereon. The encoding can be performed by matching a static keyword dictionary, by recognizing a text classification model, or by representing a pre-trained language model, which is not specifically limited in the embodiment of the present disclosure.

[0037] Step 102, encode motion information in the prompt text to obtain second text features.

[0038] In the embodiments of the present disclosure, on the basis of obtaining the prompt text, the motion information in the prompt text can be encoded, and the text description related to the motion mode can be represented by features, so as to be converted into a representation form that can be understood, used and processed by a model. The second text feature is obtained by encoding the motion information, and can accurately guide the reproduction of the motion mode in the text description in the subsequent video generation process.

[0039] In the embodiments of the present disclosure, the motion information can be extracted by text recognition on the prompt text, and the motion information can be encoded on this basis. The encoding can be performed by matching a static keyword dictionary, by recognizing a text classification model, or by representing a pre-trained and fine-tuned language model. The embodiments of the present disclosure do not make specific limitations.

[0040] Through the steps 101 and 102, the prompt text can be divided into complementary information streams, one of which represents static information in the generated video, and the other of which represents motion information in the generated video. On this basis, the first text feature and the second text feature can be used to guide the video generation process in the subsequent process, thereby improving the quality of the generated video. In an optional method embodiment of the present disclosure, the decomposition of the double-branch information stream can also flexibly adjust the processing strategy of the static information and the motion information according to the distribution of the description information in the prompt text, such as adjusting the model structure, the weight distribution or other processing strategies, so as to adapt to different video generation requirements and be widely applicable in the fields of film production, game development, virtual reality and the like.

[0041] In step 103, the first text feature is encoded to generate a first latent representation, and the first latent representation is decoded to obtain an RGB image frame.

[0042] In the embodiments of the present disclosure, the first text feature provides static features of elements such as scenes and objects when generating a video. On this basis, the first text feature can be encoded to obtain a first latent representation of the static features. The first latent representation can encode the first text feature into an abstract latent space, simplify the expression of the first text feature to analyze the latent feature distribution rule, and then decode the first latent representation into an image representation to reconstruct an RGB (Red Green Blue) image frame corresponding to the static features, so as to reproduce the static information in the prompt text. The RGB image frame realizes the appearance reconstruction of the static information and provides rich visual information in the video generation.

[0043] In step 104, the second text feature is encoded to generate a second latent representation, and the second latent representation is decoded to obtain an optical flow field.

[0044] In the embodiments of the present disclosure, the second text feature provides dynamic features about elements such as actions and movements when generating a video. On this basis, the second text feature can be encoded to obtain a second latent representation of the aforementioned dynamic feature. The second latent representation can encode the second text feature into an abstract latent space, simplify the expression of the second text feature for analyzing the potential feature distribution rule; on this basis, the second latent representation is decoded into an image representation to reconstruct an optical flow field corresponding to the dynamic feature, so as to reproduce the dynamic information in the prompt text. The optical flow data represents the direction and speed of the pixel movement of the spatial moving object on the observation imaging plane. The optical flow field visually represents the change of the inter-frame pixels in the time domain based on the optical flow data, so as to realize the dynamic reconstruction of the motion information and provide natural, accurate and smooth action information in video generation.

[0045] Through the above steps 103 and 104, the RGB image frame and the optical flow field are reconstructed respectively under the guidance of the first text feature and the second text feature, so that not only the high-fidelity static information is visually reproduced in appearance, but also the real and coherent motion information is provided, making the video generation more practical. In an optional method embodiment of the present disclosure, the processing strategy for reconstructing the RGB image frame and the optical flow field can also be flexibly adjusted for different video content, such as adjusting the model structure, weight distribution or other processing strategies, so as to adapt to different video generation requirements and be widely applicable in the fields of film production, game development, virtual reality and the like.

[0046] Step 105, generating a target video corresponding to the prompt text based on the RGB image frame and the optical flow field.

[0047] In the embodiments of the present disclosure, the RGB image frame and the optical flow field can be combined to fuse the dynamic relationship of adjacent frame pixels reflected by the optical flow field and the pixel information of the RGB image frame, so as to generate a target video corresponding to the prompt text and accurately reproducing static information and motion information. By specializing the processing of static information and motion information respectively, the video generation quality and practicability can be significantly improved, the understanding and analysis of the prompt text and the task execution capability can be enhanced, and the video generation field can adapt to more extensive, flexible and strict business requirements.

[0048] In the embodiments of the present disclosure, the above method can be implemented by using artificial intelligence technology. For example, the decomposition and coding of the prompt text can be implemented by using a CLIP (Contrastive Language-Image Pre-training) text encoder, or can be implemented by using other text encoders. On the basis of obtaining the text features, the image reconstruction can be implemented by using a diffusion model, or can be implemented by using a variational auto-encoder (VAE), and the embodiments of the present disclosure do not make specific limitations in this regard.

[0049] In the embodiments of the present disclosure, the video generation method shown in Figure 1 may be implemented based on a video generation model. Figure 2 The architecture schematic diagram of the video generation model according to an exemplary embodiment of the present disclosure is shown in Figure 2 The video generation model 200 at least includes a first text encoder 210, a second text encoder 220, a first diffusion layer 230, a second diffusion layer 240, a first variational auto-encoder (hereinafter referred to as first VAE) 250, and a second variational auto-encoder (hereinafter referred to as second VAE) 260.

[0050] In an optional method embodiment of the present disclosure, Figure 2 The video generation model shown in may further include other conventional or additional model structures, such as a text recognition layer not shown, which is used to identify static information and motion information in the prompt text and send them into corresponding branch processes; a fusion generation layer not shown, which is used to fuse the RGB image frames and the optical flow field to generate a target video, and the like, and the embodiments of the present disclosure do not make specific limitations in this regard.

[0051] In the embodiments of the present disclosure, the foregoing video generation method can be implemented by using the video generation model 200 shown in Figure 2 The video generation model 200 can be obtained by training the model structure on the basis of initialization and for the video generation task. The video generation model 200 at least includes a first text encoder 210, a second text encoder 220, a first diffusion layer 230, a second diffusion layer 240, a first VAE 250, and a second VAE 260; wherein one text encoder, one diffusion layer, and one VAE form one processing branch, thereby constituting a double-branch network structure, and processing the branch processes of the static information, the first text features, the first latent representation, and the RGB image frames, and the branch processes of the motion information, the second text features, the second latent representation, and the optical flow field based on the prompt text, respectively. Through the above decoupling, the static and motion information are processed and reconstructed respectively, avoiding the omission and interference of key information in the video generation process, and a higher quality video content can be generated.

[0052] In the embodiments of the present disclosure, the first text encoder 210 and the second text encoder 220 can be independently deployed, trained and inferred, or can partially share the underlying feature representation, such as sharing the intermediate features transmitted by the first N layers in the M-layer network structure, and differentiating the processing in the last M-N layers. On this basis, the corresponding encoding tasks are differentiated and processed to realize the division of labor under the cooperation of the two, so that the video generation model 200 can effectively process static information and motion information. In the training process, the first text encoder 210 and the second text encoder 220 can be independently trained for different tasks first and then cooperatively optimized, or can directly start cooperative training. Through the dual-text encoder architecture of the first text encoder 210 and the second text encoder 220, the generalization ability and adaptability of the video generation model can be effectively improved.

[0053] In the embodiments of the present disclosure, the first VAE 250 and the second VAE 260 can be independently deployed, trained and inferred, or can partially share the underlying feature representation, such as sharing the intermediate features transmitted by the first N layers in the M-layer network structure, and differentiating the processing in the last M-N layers. On this basis, the corresponding reconstruction tasks are differentiated and processed to realize the division of labor under the cooperation of the two, so that the video generation model 200 can effectively reconstruct the RGB image frame and the optical flow field. In the training process, the first VAE 250 and the second VAE 260 can be independently trained for different tasks first and then cooperatively optimized, or can directly start cooperative training. Through the dual-VAE architecture of the first VAE 250 and the second VAE 260, the generalization ability and adaptability of the video generation model can be effectively improved.

[0054] In an optional method embodiment of the present disclosure, the foregoing step 101 can specifically include the following step A.

[0055] Step A, encoding the prompt text by the first text encoder to obtain first text features; the static information includes at least one of scene layout, object attribute, object color and object texture.

[0056] In the embodiments of the present disclosure, in the video generation model 200, the first text encoder 210 can be used to encode the static information of the prompt text. The static information can include scene layout, object attribute, object color, object texture, etc. The scene layout can include venue types such as grassland, stage and square, can include the spatial relationship of objects in the scene, and can include lighting information in the scene. The object attribute can be object type and object size, etc. The object texture can be smooth, wrinkled, etc., or can be materials such as leather, wood grain and marble. For details, please refer to the related description of the foregoing step 101, which will not be repeated here.

[0057] In an optional method embodiment of the present disclosure, the foregoing step 102 can specifically include the following step B.

[0058] Step B, obtaining a second text feature by encoding the prompt text with motion information by a second text encoder; the motion information includes at least one of action type, motion trajectory, and speed change.

[0059] In the video generation model 200 in the embodiments of the present disclosure, the prompt text can be encoded with motion information by the second text encoder 220. The motion information can include action type, motion trajectory, speed change, etc. The action type can include running, jumping, kicking a football, dribbling and shooting, balance beam action, complex diving action, etc. The motion trajectory can describe the motion direction and motion distance, etc. The speed change can include slow speed, fast speed, uniform speed, variable speed, etc. For details, please refer to the related description of the foregoing step 102, which will not be repeated here.

[0060] In an optional method embodiment of the present disclosure, the foregoing step 103 can include the following step C1 to step C2.

[0061] Step C1, encoding and processing the first text feature by a first diffusion layer to obtain a first latent representation.

[0062] In the embodiments of the present disclosure, the first diffusion layer can be implemented based on a diffusion model, such as DiT (Diffusion Transformer), which can encode the first text feature and generate a corresponding first latent representation at each time step to obtain a latent RGB image frame representation.

[0063] Step C2, decoding the first latent representation by a first variational autoencoder to obtain an RGB image frame.

[0064] In the embodiments of the present disclosure, in the video generation model 200, the first VAE 250 can be used to reconstruct the RGB image frame based on the first latent representation. After the design and adjustment of the model structure and the training task, the first VAE 250 can better preserve the static appearance information such as color, texture, shape, capture the light changes in the scene, object surface details, and complex background information, etc., so as to obtain the reconstructed RGB image frame by decoding the first latent representation.

[0065] In an optional method embodiment of the present disclosure, the foregoing step 104 includes the following step D1 to step D2.

[0066] Step D1, encoding and processing the second text feature by a second diffusion layer to generate a second latent representation.

[0067] In this embodiment of the disclosure, the second diffusion layer can be implemented based on a diffusion model, such as DiT, which can encode the second text features and generate the corresponding second latent representation at each time step to obtain the latent optical flow field representation.

[0068] Step D2: Decode the second latent representation using the second variational autoencoder to obtain the optical flow field.

[0069] In this embodiment of the disclosure, in the video generation model 200, the optical flow field can be reconstructed using a second VAE 260 based on a second latent representation. Through design and adjustment of the model structure and training task, the second VAE 260 can better preserve dynamic information such as motion, motion direction, distance, and speed, and capture the motion patterns of objects in the scene. Thus, by decoding the second latent representation, the reconstructed optical flow field can be obtained.

[0070] In an optional embodiment of the method disclosed herein, the training video data for the video generation model is obtained by filtering based on the complexity of the motion actions.

[0071] In this embodiment, the training video data used by the video generation model during training can be filtered based on the complexity of the motion movements. Unlike the general-purpose, multi-topic, large-scale video data used in existing related domain training, or video data collected for specific domains such as human-object interaction or facial expressions, this training data is filtered based on the complexity of the motion movements, thus adapting to the video generation needs of movements with varying degrees of complexity. The complexity of the motion movements can be evaluated using various indicators. For example, these may include the type of movement (e.g., walking is less complex than running); the number of movements (e.g., a single movement is less complex than a combination of movements); and the range of motion (e.g., a half-squat is less complex than a full squat), etc. These indicators are merely examples; those skilled in the art can select and expand upon them according to actual needs to ensure that the evaluation of the complexity of the motion movements meets business requirements.

[0072] In an optional embodiment of the method disclosed herein, the step of filtering training video data includes the following steps E1 to E3.

[0073] Step E1: Perform target detection on the raw video data to obtain preliminary video data containing moving targets.

[0074] In this embodiment, the original video data can be video data collected in the public domain. This video data can be obtained through shooting or by synthesizing and modifying existing videos. Target detection can be performed on the original video data to determine if there are moving targets. Moving targets can be human bodies, cats, dogs, cars, robots, etc. Based on this, the original video data containing moving targets is filtered to obtain preliminary video data containing moving targets.

[0075] Step E2: Calculate the action score of the initial screening video data using the optical flow algorithm.

[0076] In this embodiment of the disclosure, for moving targets in the initial screening video data, the motion score calculated by the optical flow algorithm can be used to evaluate the complexity of the motion. For example, the larger the pixel displacement vector corresponding to the moving target between frames, the larger the motion score calculated by the optical flow algorithm can be, thus indicating that the motion complexity of the moving target between adjacent frames is higher; conversely, the smaller the motion score, the lower the motion complexity of the moving target between adjacent frames. Furthermore, when the motion score is small enough, the moving target can be considered to be in a stationary state.

[0077] Step E3: Filter the initial video data based on action scores to obtain training video data whose action scores are within the complexity range.

[0078] In this embodiment, the initial video data can be further filtered based on action scores to retain training video data whose motion complexity meets the requirements of the video generation task. The complexity range can be limited by setting boundary thresholds for action scores. For example, a stationary threshold can be set; if the action score is less than or equal to the stationary threshold, the moving target in the initial video data is considered stationary and is thus filtered out. Similarly, simple action thresholds and complex action thresholds can be set from low to high. Furthermore, a maximum threshold can be set; if the action score is greater than the maximum threshold, the motion is considered too large or too fast, potentially affecting model training performance, and is also filtered out. For the obtained training video data, corresponding descriptive text can be constructed to form data pairs. During training, the initialized video generation model generates video data based on the descriptive text output, and the model parameters are updated based on the loss value of the objective function calculated using the generated video data and the training video data.

[0079] In this embodiment of the disclosure, based on the determined training video data, and for the dual-branch network structure of the video generation model, training optical flow data corresponding to the branching processes of motion information, second text features, second latent representation, and optical flow field can be constructed. Based on the training video data, optical flow data can be further extracted and visualized, such as by converting it to RGB format, mapping pixel motion direction to hue, and mapping intensity to brightness or transparency, making it available for model analysis and processing. The optical flow algorithm can employ FlowNet, FlowNet2, RAFT (Recurrent All-Pairs Field Transforms), etc., and this embodiment of the disclosure does not impose specific limitations on this.

[0080] In this embodiment of the disclosure, during the training process of the video generation model, the objective function can be extended. In addition to calculating the pixel reconstruction loss between the generated video data and the training video data, a potential representation loss corresponding to the static information branch process can be set to avoid the reconstructed RGB image frames deviating from the training video data. A potential representation loss corresponding to the dynamic information branch process can be set to avoid the reconstructed optical flow field deviating from the motion law of the real scene. Furthermore, the participation degree of the first potential representation and the second potential representation in video content generation can be evaluated through regularization loss to force the two branches to participate in cooperation when generating video content, thereby avoiding over-reliance on a single branch.

[0081] This disclosure provides a video generation method. Based on obtained prompt text, the method encodes static information of the prompt text to obtain first text features and encodes motion information to obtain second text features. Then, it encodes the first text features to generate a first latent representation, which is then decoded to obtain RGB image frames. Similarly, it encodes the second text features to generate a second latent representation, which is then decoded to obtain an optical flow field. The RGB image frames and optical flow field are used to generate a target video corresponding to the prompt text. In the process of generating the target video, this method performs RGB image reconstruction using static information from the prompt text, providing rich visual appearance information. It also performs optical flow field reconstruction using motion information from the prompt text, providing natural, smooth, and realistic motion change information. Furthermore, the RGB image frames and optical flow field complement each other, adapting to the needs of video generation with large or rapid motion changes or complex movements, improving video generation quality, reducing deviations, making video content more consistent with physical laws, and enhancing practicality.

[0082] Figure 3 A structural block diagram of a video generation apparatus 300 provided according to an exemplary embodiment of the present disclosure is shown, such as... Figure 3As shown, the device 300 may include: a text encoding module 301, used to encode static information of the prompt text to obtain a first text feature; the text encoding module 301 is also used to encode motion information of the prompt text to obtain a second text feature; an information reconstruction module 302, used to encode the first text feature to generate a first latent representation, and to decode the first latent representation to obtain an RGB image frame; the information reconstruction module 302 is also used to encode the second text feature to generate a second latent representation, and to decode the second latent representation to obtain an optical flow field; and a video generation module, used to generate a target video corresponding to the prompt text based on the RGB image frame and the optical flow field.

[0083] In an exemplary embodiment of this disclosure, the device is implemented based on a video generation model; the video generation model includes a first text encoder, a second text encoder, a first diffusion layer, a second diffusion layer, a first variational autoencoder, and a second variational autoencoder.

[0084] In an exemplary embodiment of this disclosure, the text encoding module 301 is specifically used to encode the prompt text with static information through a first text encoder to obtain a first text feature; the static information includes at least one of scene layout, object attributes, object color, and object texture.

[0085] In an exemplary embodiment of this disclosure, the text encoding module 301 is specifically used to encode motion information of the prompt text through a second text encoder to obtain second text features; the motion information includes at least one of action type, motion trajectory, and speed change.

[0086] In an exemplary embodiment of this disclosure, the first diffusion layer information reconstruction module 302 is specifically used to encode the first text features through the first diffusion layer to obtain a first latent representation; and to decode the first latent representation through the first variational autoencoder to obtain an RGB image frame.

[0087] In an exemplary embodiment of this disclosure, the information reconstruction module 302 is specifically used to encode the second text features through the second diffusion layer to generate a second latent representation; and to decode the second latent representation through the second variational autoencoder to obtain the optical flow field.

[0088] In an exemplary embodiment of this disclosure, the training video data for the video generation model is obtained by filtering based on the complexity of the motion actions.

[0089] In an exemplary embodiment of this disclosure, the apparatus further includes a data acquisition module, configured to perform target detection on the original video data to obtain preliminary screening video data containing moving targets; calculate the motion score of the preliminary screening video data using an optical flow algorithm; and filter the preliminary screening video data based on the motion score to obtain training video data with motion scores within the complexity range.

[0090] This disclosure provides a video generation apparatus. Based on obtained prompt text, the apparatus encodes static information of the prompt text to obtain first text features and encodes motion information to obtain second text features. Furthermore, it encodes the first text features to generate a first latent representation, which is then decoded to obtain RGB image frames. It also encodes the second text features to generate a second latent representation, which is then decoded to obtain an optical flow field. The RGB image frames and optical flow field are then used to generate a target video corresponding to the prompt text. In the process of generating the target video, this method uses the static information in the prompt text to perform RGB image reconstruction, providing rich visual appearance information, and uses the motion information in the prompt text to perform optical flow field reconstruction, providing natural, smooth, and realistic motion change information. Furthermore, the RGB image frames and optical flow field complement each other, adapting to the needs of video generation with large or rapid motion changes or complex movements, improving video generation quality, reducing deviations, making video content more consistent with physical laws, and improving practicality.

[0091] It should be noted that the specific details of each module in the above-mentioned video generation device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content in the method section. That is, the explanation and beneficial effects of the video generation method in the above-mentioned embodiments are also applicable to the video generation device of the present disclosure, and will not be elaborated here.

[0092] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0093] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0094] The following reference Figure 4To describe an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0095] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), and a display unit 440.

[0096] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure.

[0097] Storage unit 420 may include readable media in the form of volatile storage units, such as random access memory (RAM) 421 and / or cache memory 422, and may further include read-only memory (ROM) 423.

[0098] Storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0099] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0100] Electronic device 400 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0101] Furthermore, exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0102] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0103] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0105] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A video generation method, characterized in that, The method includes: The first text feature is obtained by encoding static information into the prompt text; The second text feature is obtained by encoding motion information into the prompt text; The first text features are encoded to generate a first latent representation, and the first latent representation is decoded to obtain an RGB image frame; The second text features are encoded to generate a second latent representation, and the second latent representation is decoded to obtain the optical flow field; Based on the RGB image frame and the optical flow field, a target video corresponding to the prompt text is generated.

2. The method according to claim 1, characterized in that, The method is based on a video generation model; the video generation model includes a first text encoder, a second text encoder, a first diffusion layer, a second diffusion layer, a first variational autoencoder, and a second variational autoencoder.

3. The method according to claim 2, characterized in that, The step of encoding static information into the prompt text to obtain the first text feature includes: The first text feature is obtained by encoding static information of the prompt text using a first text encoder; the static information includes at least one of scene layout, object attributes, object color, and object texture.

4. The method according to claim 2, characterized in that, The step of encoding motion information into the prompt text to obtain the second text feature includes: The second text feature is obtained by encoding motion information into the prompt text using a second text encoder; the motion information includes at least one of action type, motion trajectory, and speed change.

5. The method according to claim 2, characterized in that, The step of encoding the first text features to generate a first latent representation and decoding the first latent representation to obtain an RGB image frame includes: The first text features are encoded using the first diffusion layer to obtain a first latent representation. The RGB image frame is obtained by decoding the first latent representation using the first variational autoencoder.

6. The method according to claim 2, characterized in that, The steps of encoding the second text features to generate a second latent representation and decoding the second latent representation to obtain the optical flow field include: The second text features are encoded using the second diffusion layer to generate the second latent representation; The optical flow field is obtained by decoding the second latent representation using the second variational autoencoder.

7. The method according to any one of claims 2 to 6, characterized in that, The training video data for the video generation model is obtained by filtering based on the complexity of the motion actions.

8. The method according to claim 7, characterized in that, The steps for filtering the training video data include: Target detection is performed on the raw video data to obtain preliminary video data containing moving targets; The motion score of the initial screening video data is calculated using an optical flow algorithm. The video data is filtered based on the action scores to obtain training video data whose action scores fall within the complexity range.

9. A video generation apparatus, characterized in that, The device may include: The text encoding module is used to encode static information in the prompt text to obtain the first text feature; The text encoding module is also used to encode motion information into the prompt text to obtain a second text feature; The information reconstruction module is used to encode the first text features to generate a first latent representation, and to decode the first latent representation to obtain an RGB image frame; The information reconstruction module is further configured to encode the second text features to generate a second latent representation, and to decode the second latent representation to obtain an optical flow field; The video generation module is used to generate a target video corresponding to the prompt text based on the RGB image frame and the optical flow field.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images