Video generation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a video generation method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises: based on a mask, determining a key frame and a non-key frame from a first video, the mask being used for indicating the position of the key frame in the first video; generating a context feature based on the mask, the key frame and the non-key frame, the context feature being used for constraining content coherence between the video frames; generating a text feature based on a first text cue word, wherein the first text cue word is a content description text of the to-be-generated video; and generating a second video based on the context feature and the text feature, the second video comprising the key frame located at the key frame position and a generative video frame located at a non-key frame position outside the key frame position. By adopting the scheme provided by the invention, the video generation quality can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, device, and storage medium. Background Technology

[0002] The explosion of generative AI (Artificial Intelligence) technology has driven video generation to evolve from single text-driven to multi-condition controllable. The Diffusion Transformer (DiT) architecture, with its powerful scalability and generation quality, has become the mainstream basic framework in the field of video generation.

[0003] In related technologies, the method of generating the first frame is used. Given a starting image or text prompt, the model extrapolates and imagines the sequence based on this to achieve the video continuation task. This method is like toppling the first domino. The initial state determines the general direction, but the subsequent development is full of randomness. It is very easy to cause the subject's identity to drift, the scene structure to change drastically, or even the complete loss of narrative logic, resulting in a large deviation between the video generation result and the expectation. Summary of the Invention

[0004] This application provides a video generation method, apparatus, device, and storage medium, the technical solutions of which are as follows.

[0005] On one hand, embodiments of this application provide a video generation method, the method comprising: Based on a mask, keyframes and non-keyframes are determined from a first video, wherein the mask is used to indicate the location of keyframes in the first video; Based on the mask, the keyframes, and the non-keyframes, contextual features are generated, which are used to constrain the content coherence between video frames. Text features are generated based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated; Based on the context features and the text features, a second video is generated. The second video includes the keyframe located at the keyframe position and generative video frames located at non-keyframe positions outside the keyframe position.

[0006] On the other hand, embodiments of this application provide a video generation method, the method comprising: In response to a keyframe selection operation for a first video, the video frame indicated by the keyframe selection operation is determined as a keyframe of the first video. Based on the position of the keyframe in the first video, a mask is generated, and the mask is used to indicate the position of the keyframe in the first video. In response to a text input operation, a first text prompt word is obtained, wherein the first text prompt word is a content description text of the video to be generated; In response to a video generation operation, a second video is generated based on the first text prompt, the mask, keyframes and non-keyframes in the first video, wherein the keyframes and non-keyframes are determined from the first video based on the mask, and the second video includes the keyframes located at the keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0007] On the other hand, embodiments of this application provide a video generation apparatus, the apparatus comprising: A first determining module is used to determine keyframes and non-keyframes from a first video based on a mask, wherein the mask is used to indicate the positions of keyframes in the first video. The first feature generation module is used to generate context features based on the mask, the key frame, and the non-key frame, wherein the context features are used to constrain the content coherence between video frames. The second feature generation module is used to generate text features based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated. A video generation module is used to generate a second video based on the context features and the text features. The second video includes the key frame located at the key frame position and generative video frames located at non-key frame positions outside the key frame position.

[0008] On the other hand, embodiments of this application provide a video generation apparatus, the apparatus comprising: A keyframe determination module is used to determine the video frame indicated by the keyframe selection operation as a keyframe of the first video in response to a keyframe selection operation for the first video. A mask generation module is used to generate a mask based on the position of the keyframe in the first video, the mask being used to indicate the position of the keyframe in the first video; The acquisition module is used to acquire a first text prompt word in response to a text input operation. The first text prompt word is a content description text of the video to be generated. A video generation module is configured to generate a second video in response to a video generation operation, based on the first text prompt, the mask, keyframes and non-keyframes in the first video, wherein the keyframes and non-keyframes are determined from the first video based on the mask, and the second video includes the keyframes located at the keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0009] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the video generation method as described above.

[0010] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the video generation method as described above.

[0011] On the other hand, embodiments of this application provide a computer program product including at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the video generation method described above.

[0012] In this embodiment, by using a mask to indicate the keyframe positions in the first video, any number of keyframes can be defined at any position in the first video. This allows keyframes and non-keyframes to be determined from the first video based on the mask. Contextual features are then generated based on the mask, keyframes, and non-keyframes. These contextual features, along with text features generated based on the first text prompt, are then combined to generate the second video. This allows the generation process of video frames located at non-keyframe positions to be controlled using keyframes located at keyframe positions, resulting in a more coherent and natural second video and improving the generation quality of the second video. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This application shows a structural block diagram of a computer system provided in an exemplary embodiment; Figure 2 A flowchart illustrating a video generation method provided in an exemplary embodiment of this application is shown; Figure 3 This illustration shows a schematic diagram of the keyframe locations in a first video provided by an exemplary embodiment of this application; Figure 4 A flowchart of a video generation method provided by another exemplary embodiment of this application is shown; Figure 5 A schematic diagram of a masked hidden layer representation provided in an exemplary embodiment of this application is shown; Figure 6 A schematic diagram of a video generation method provided in an exemplary embodiment of this application is shown; Figure 7 A flowchart of a video generation method provided by another exemplary embodiment of this application is shown; Figure 8 This illustration shows a schematic diagram of the effect of a generative video frame provided in an exemplary embodiment of this application; Figure 9 A flowchart of a video generation method provided by another exemplary embodiment of this application is shown; Figure 10 A schematic diagram of a video generation method provided in another exemplary embodiment of this application is shown; Figure 11 A flowchart of a video generation method provided by another exemplary embodiment of this application is shown; Figure 12 This invention provides a structural block diagram of a video generation apparatus according to an exemplary embodiment of the present application. Figure 13 A structural block diagram of a video generation apparatus provided in another exemplary embodiment of this application is shown; Figure 14 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0018] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0019] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0020] DiT (Diffusion Transformer): This is a class of diffusion models built on the Transformer architecture. This type of model uses the core modules of Transformer (e.g., self-attention mechanism and feedforward neural network) to replace traditional networks such as U-Net, combining the progressive denoising process of diffusion models with the attention mechanism of Transformer, and taking advantage of Transformer's ability to process sequential data to model the generation process of images or videos.

[0021] VCU (Video Concept Units) are unified, fine-grained, and composable condition control units used for video generation and editing. Optionally, the control conditions for video generation can include multimodal inputs such as text, images, and masks.

[0022] VACE (All-in-One Video Creation and Editing): By integrating multimodal inputs such as text, images, and masks through the Unified Video Conditional Unit (VCU), it realizes diverse functions such as reference-to-video generation, video editing, and task combination.

[0023] Context Adapter: A pluggable module based on the DiT architecture. Its core function is to enable flexible injection of multiple tasks and decoupling of models, helping to build a unified video generation and editing framework.

[0024] Mask: A binary mask (single-channel tensor) with spatial dimensions identical to the video frame (h×w). Pixel values ​​contain only "0" and "1"—"1" represents the target area to be focused on (e.g., keyframe-forced retention area, area to be edited), and "0" represents the background area that does not require intervention (e.g., areas to retain the original image). The mask is the core input tool for achieving "precise region control." Essentially, it is a binary / feature tensor that matches the spatial dimensions of the video frame. By marking "valid areas" and "invalid areas," the model clearly defines "which areas should be subject to generation / editing / keyframe constraints."

[0025] Please refer to Figure 1 The diagram illustrates a computer system provided in one embodiment of this application. This computer system can implement a system architecture for a video generation method. The computer system may include a terminal 100 and a server 200.

[0026] Terminal 100 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, laptop computer, desktop computer, wearable device, in-vehicle terminal device, VR (Virtual Reality) device, AR (Augmented Reality) device, or MR (Mixed Reality) device. A client application for the target application can be installed and run on terminal 100. This target application has video generation capabilities. The target application can be a game application, a social application (such as an instant messaging application), a payment application, a video playback application, a streaming media application, or other applications that provide the video generation capabilities described in this application; this application does not limit its scope. Furthermore, this application does not limit the form of the target application, including but not limited to apps, mini-programs, etc., installed on terminal 100, and can also be in web page form.

[0027] Server 200 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 200 can be the backend server of the aforementioned target application, used to provide backend services to the client of the target application. Optionally, the video generation method provided in this application embodiment can be provided by a single server or by at least two servers, and this application does not limit this.

[0028] The video generation method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the video generation method can be executed by the terminal 100 (such as by the client of the target application installed and running in the terminal 100), or by the server 200, or by the interaction and cooperation between the terminal 100 and the server 200. This application does not limit this.

[0029] Please refer to Figure 2 The diagram illustrates a video generation method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps.

[0030] Step 210: Based on the mask, determine the keyframes and non-keyframes from the first video. The mask is used to indicate the location of the keyframes in the first video.

[0031] Optionally, the first video can be understood as the original video used for video generation. The first video is a composite frame sequence containing multiple video frames, including keyframes and non-keyframes. Non-keyframes are video frames that need to be edited and populated with video content during video generation, while keyframes are video frames that remain unchanged during video generation. Optionally, the video content contained in the keyframes can serve as cue information during the video generation process; for example, keyframes can be used to indicate the identity of an object, scene structure, etc.

[0032] The video generation method provided in this application supports setting any number and position of keyframes in the original video to control the video generation process. By setting binding keyframes at any point in the video timeline, it is possible to effectively control transitions in video content, changes in video composition, camera movement, or switching of object states in the video.

[0033] Indicative, such as Figure 3 As shown, the video generation method proposed in this application supports setting the first video frame as a keyframe to generate video from an image; it also supports setting the first and last video frames as keyframes to generate video from the first and last frames; and it also supports setting the first and last video frames and the middle video frames as keyframes to generate video from any keyframe.

[0034] Optional, multiple keyframes The constructed keyframe sequence can be represented as , The keyframes can be greater than or equal to 2. It can be an RGB image. Optionally, the keyframe sequence can also be associated with timestamp information. , .

[0035] Optionally, the first video includes a keyframe sequence and non-keyframe sequences, and the first video can be represented as follows: , This represents the total number of frames in the first video, which is also the total number of frames in the video to be generated.

[0036] In one possible implementation, to achieve flexibility in keyframe setting, the location of keyframes in the original video can be indicated by a mask, so that the computer device can determine keyframes and non-keyframes from the first video based on the mask.

[0037] Optionally, the mask corresponds one-to-one with a video frame, and the mask value is either 0 or 1. A mask value of 0 indicates that the video frame is a keyframe, and a mask value of 1 indicates that the video frame is a non-keyframe. Optionally, the mask sequence can be represented as follows: , This indicates the area that must be followed by the keyframe. Indicates the area that needs to be edited.

[0038] Optionally, keyframes can be determined based on the mask and the first video. Non-keyframes The process can be represented as: , .

[0039] Step 220: Generate contextual features based on the mask, keyframes, and non-keyframes. The contextual features are used to constrain the content coherence between video frames.

[0040] In some embodiments, after determining keyframes and non-keyframes, in order to combine the video content in the keyframes during the video generation process and generate video frames at non-keyframe locations, it is also necessary to first convert the existing video control information into feature vectors that the model can understand.

[0041] Optionally, the computer device obtains contextual features based on masks, keyframes, and non-keyframes through encoding and discretization, and then uses these contextual features to constrain the content coherence between video frames.

[0042] Optionally, by setting multiple keyframes as video generation control conditions, consistency in the generated video can be ensured across multiple dimensions, such as character consistency, scene consistency, inter-frame consistency, and motion rhythm consistency. This ensures the stability of the target object's identity and appearance on the timeline within the video frames, maintains the coherent evolution of scene layout and core elements, achieves smooth and natural transitions between keyframes, and precisely controls the speed and rhythm of dynamic changes.

[0043] Step 230: Generate text features based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated.

[0044] In some embodiments, in addition to using masks and keyframes as video generation control conditions, text prompts can also be added as video generation control conditions to constrain the macro framework of the video, such as the video's theme, scene, atmosphere, style, and basic narrative.

[0045] Optionally, the first text prompt is a descriptive text of the content to be generated in the video, such as "a girl is walking from a distance with flowers." In one possible implementation, the computer device inputs the first text prompt into a text encoder, and the text encoder outputs text features that are used to constrain the macroscopic framework of the video to be generated.

[0046] Step 240: Based on contextual features and textual features, generate a second video. The second video includes keyframes located at keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0047] In some embodiments, after obtaining the context features and text features, both can be used together as constraints in the video generation process. Optionally, the computer device injects the context features and text features into a diffusion model (such as a DiT model), performs denoising processing on the noisy video features through the diffusion model, and then decodes them to obtain the second video.

[0048] Optionally, the generated second video includes the keyframes that remain unchanged at the keyframe locations, as well as the generated video frames at non-keyframe locations outside the keyframe locations.

[0049] In summary, in this embodiment of the application, by using a mask to indicate the keyframe positions in the first video, it is possible to define any number of keyframes at any position in the first video. This allows keyframes and non-keyframes to be determined from the first video based on the mask. Contextual features are then generated based on the mask, keyframes, and non-keyframes. These contextual features, along with text features generated based on the first text prompt, are then combined to generate the second video. This approach allows the generation process of video frames located at non-keyframe positions to be controlled using keyframes located at keyframe positions, resulting in a more coherent and natural second video and improving the overall quality of the generated second video.

[0050] In some embodiments, the process of mapping the first video from pixel space to latent space involves downsampling the video frames in both spatial and temporal dimensions. In order to achieve precise spatiotemporal control of keyframes, it is necessary to extend the feature channel dimension corresponding to the mask to achieve downsampling adaptation between the video frames and the mask in both spatial and temporal dimensions.

[0051] Please refer to Figure 4The diagram illustrates a video generation method provided by another exemplary embodiment of this application. This embodiment uses the method for computer devices (including terminals and / or servers) as an example for illustration. The method includes the following steps.

[0052] Step 210: Based on the mask, determine the keyframes and non-keyframes from the first video. The mask is used to indicate the location of the keyframes in the first video.

[0053] In one possible implementation, to ensure smooth video transitions and avoid issues such as discontinuities, subject drift, or abrupt style changes, keyframes can be set at intermediate video frame positions in the first video. Specifically, keyframe positions include intermediate video frame positions excluding the first and last video frames. For example, the first video comprises 60 video frames, and the keyframes may include the 1st, 20th, 40th, and 60th video frames in the first video.

[0054] Step 221: Encode the keyframes and non-keyframes using an encoder to obtain the hidden layer representations of the keyframes and non-keyframes.

[0055] In one possible implementation, after determining the keyframes and non-keyframes, the computer device encodes the keyframes and non-keyframes respectively through an encoder, thereby mapping the keyframes and non-keyframes from the pixel space to the non-pixel space to obtain the hidden layer representation of the keyframes and the hidden layer representation of the non-keyframes.

[0056] Optionally, the encoder can be a variational autoencoder (VAE Encoder). Optionally, the keyframe hidden layer representation is... The hidden layer of non-keyframes is represented as .

[0057] Optionally, mapping keyframes and non-keyframes from pixel space to non-pixel space may include downsampling keyframes and non-keyframes by 8x in the spatial dimension and 4x in the temporal dimension.

[0058] Step 222: Perform reconstruction processing on the mask to obtain the hidden layer representation of the mask. The feature channel dimension of the hidden layer representation of the mask is matched with the feature channel dimension of the hidden layer representation of the keyframe.

[0059] In one possible implementation, to ensure the mask accurately marks the original pixel region corresponding to each hidden layer feature point, the feature channel dimension of the mask needs to be adapted to the downsampling of the video frame in both temporal and spatial dimensions. This avoids temporal misalignment between the mask's hidden layer representation and the video frame's hidden layer representation. Specifically, the computer device performs a reshape process on the mask to obtain the mask's hidden layer representation. The feature channel dimensions of the hidden layer representation of the key frame and the hidden layer representation of the non-key frame are matched.

[0060] Optionally, the feature channel dimension of the masked hidden layer representation is equal to the product of the spatial downsampling factor and the temporal downsampling factor used when encoding the keyframe, that is, the feature channel dimension is equal to the spatial downsampling factor × the temporal downsampling factor.

[0061] That is, the embodiments of this application are not limited to the setting of spatial dimensions. Instead of using 64 channels, the feature channels of the mask are further increased from 64 to 256, so that the feature channel dimension of the mask can cover the spatial downsampling factor and temporal downsampling factor of the video frame. That is, the number of feature channels of the mask hidden layer representation, 256, is equal to 8 (spatial height H) × 8 (spatial width W) × 4 (temporal dimension T), thereby ensuring that the mask hidden layer representation is fully aligned with the key frame hidden layer representation and the non-key frame hidden layer representation in three dimensions: spatial height, spatial width, and temporal dimension.

[0062] Optionally, to match the feature channel dimensions of the mask's hidden layer representation with those of the keyframe's hidden layer representation, the mask initialization method can be changed from interpolate to view. Interpolate requires calculating interpolation between pixels to change the size, while view only changes the shape description of the mask without actually moving the data. That is, if the mask is scaled by interpolation to match the 4x downsampling in the temporal dimension, the interpolated mask and keyframe may become misaligned or disappear. However, by using view to naturally match the 4x downsampling in the temporal dimension, the mask's hidden layer representation and the keyframe's hidden layer representation output by the encoder can achieve full alignment in both spatial and temporal dimensions, without information loss or distortion. This allows the model to accurately identify the spatiotemporal region corresponding to the keyframe during video generation, improving control accuracy.

[0063] Indicative, such as Figure 5 As shown, when the first and last video frames and the middle video frames are set as keyframes, the design of the 64-dimensional mask channel can only meet the feature expression requirements of basic video editing tasks (such as simple mask repair and global style conversion), and has obvious limitations in multi-keyframe control scenarios. The insufficient feature capacity and interpolation problems of the mask channel can easily lead to the loss of details, which in turn can cause problems such as blurred keyframe positioning and inaccurate control signal transmission.

[0064] The combination of Mask channel expansion (64-dimensional → 256-dimensional) and view initialization method enables the complete preservation of keyframe masks. The mask hidden layer representation and the video frame hidden layer representation can be aligned in all dimensions of space and time, forming a "high-capacity + high-precision" mask feature encoding system. The two work together to achieve precise keyframe control.

[0065] Step 223: Generate context features using a context embedder based on the keyframe hidden layer representation, the non-keyframe hidden layer representation, and the mask hidden layer representation.

[0066] In one possible implementation, after obtaining the keyframe hidden layer representation, the non-keyframe hidden layer representation, and the mask hidden layer representation, the computer device can input the three into a context embedder, which will then split the continuous hidden layer representation into discrete feature tokens to obtain context tokens.

[0067] Optionally, the context embedder is a module for generating context-aware embeddings that encodes input data into a vector representation that includes context information.

[0068] Step 231: Based on the first text prompt and the first video, generate a second text prompt using a visual language model. The content richness of the second text prompt is higher than that of the first text prompt.

[0069] Considering that language is abstract, it is usually difficult for users to describe specific visual details, textures, and precise color combinations through text. In other words, the first text prompt can only provide macro-level ideas and semantic context. Therefore, in order to further enrich the content description text of the video to be generated, the text prompt can be further optimized by combining the video content contained in the key frames of the first video.

[0070] In one possible implementation, the computer device can generate a second text prompt based on a first text prompt and a first video using a visual language model. The content richness of the second text prompt is higher than that of the first text prompt.

[0071] Optionally, a Visual Language Model (VLM) is an artificial intelligence model capable of simultaneously understanding and processing visual (images, videos) and linguistic (text, speech) information. Its core objective is to establish deep semantic alignment between visual and linguistic modalities, thereby enabling cross-modal understanding, reasoning, and generation tasks. In this embodiment, the input to the visual language model can be a first text prompt word and keyframes from a first video, and the output can be a second text prompt word.

[0072] Compared to the first text prompt, the second text prompt contains richer semantic information. For example, the first text prompt is "A girl is walking from a distance with flowers in her hand." By combining the video content contained in the keyframe and adding semantic information, the second text prompt can be "A girl in a red dress is walking from a distance with flowers in her hand and smiling."

[0073] Step 232: Based on the second text prompt word, generate text features through a text encoder.

[0074] Furthermore, after obtaining the second text prompt word, the computer device encodes the second text prompt word using a text encoder to obtain text tokens.

[0075] Step 241: Perform block-based processing on the noisy hidden layer representation to obtain noisy video features.

[0076] To convert the noise latent representation into a language that the DiT model can understand, the noise latent representation needs to be patched, flattening it into a one-dimensional feature vector, namely noise video tokens.

[0077] Step 242: Based on contextual features and text features, perform denoising processing on the noisy video features to obtain denoised video features.

[0078] In one possible implementation, the computer device inputs context features, text features, and noisy video features into the DiT model. The context features constrain the content coherence between video frames, the text features constrain the global framework of the video to be generated, and the noisy video features are denoised to obtain the denoised video tokens output by the DiT model.

[0079] Optionally, the DiT model is a diffusion model built on the Transformer architecture. Optionally, the DiT model can include n DiT Blocks, each DiT Block comprising the basic structure of a Transformer Block (residual connections, layer normalization, feedforward network), with key modifications for the diffusion task, primarily adding the fusion of conditional information. Optionally, the data flow within a DiT Block can be represented as: noisy features → adaptive layer normalization → multi-head self-attention → residual connections → adaptive layer normalization → feedforward network → residual connections → denoised features.

[0080] Step 243: Perform splicing processing on the denoised video features to obtain the hidden layer representation of the denoised video.

[0081] In one possible implementation, after obtaining the denoised video features, in order to convert them into an image format that the decoder can understand, it is necessary to perform unpatchify processing on the denoised video features to obtain the denoised video latent representation.

[0082] Step 244: Decode the hidden layer representation of the denoised video using a decoder to obtain the second video.

[0083] Finally, the computer device decodes the hidden layer representation of the denoised video through a decoder to obtain the second video, which includes both the unchanged keyframes located at keyframe positions and the generative video frames located at non-keyframe positions.

[0084] In the above embodiments, by performing reconstruction processing on the mask, the hidden layer representation of the mask can be matched with the hidden layer representations of keyframes and non-keyframes in the feature channel dimension. This ensures that the model can accurately identify keyframes based on the mask, enabling the use of keyframes to constrain the video generation process and improve video generation quality. Furthermore, by combining the keyframes in the first video with expanded text prompts, the semantic prompt information contained in the text prompts can be enriched, thereby optimizing the control effect based on text prompts and further improving the video generation quality.

[0085] Furthermore, the video generation method proposed in this application achieves a composite control mode of "keyframes defining structure + text supplementing details + masks for editing" by working collaboratively with multiple keyframes, text, masks, and other modalities. This optimizes the video generation control conditions, enables natural and smooth transitions between keyframes, and ensures spatiotemporal consistency.

[0086] Please refer to Figure 6 The diagram illustrates a video generation method provided in an exemplary embodiment of this application.

[0087] like Figure 6 As shown, the computer device performs concept decoupling on the first video 601 based on mask 602, and determines keyframe 603 from the first video 601. ) and non-keyframe 604 ( And through encoder 605, keyframes 603 and non-keyframes 604 are mapped from pixel space to hidden layer space to obtain keyframe hidden layer representation 606. ) and non-keyframe hidden layer representation 607 ( Furthermore, the computer device performs a reshape process on the mask 602 to obtain the hidden layer representation 608 of the mask. In this process, the mask hidden layer representation 608, the keyframe hidden layer representation 606, and the non-keyframe hidden layer representation 607 are fully aligned in both spatial and temporal dimensions. Furthermore, the computer device inputs the mask hidden layer representation 608, the keyframe hidden layer representation 606, and the non-keyframe hidden layer representation 607 into the context embedder 611, which discretizes the three representations to obtain the context feature 613.

[0088] Simultaneously, the computer device inputs the text prompt 610 into the text encoder 612, which encodes the text prompt 610 into text features 615. Furthermore, the computer device performs block-based processing on the noisy hidden layer representation 609 to obtain noisy video features 614.

[0089] Furthermore, based on contextual features 613 and text features 615, the computer device performs denoising processing on the noisy video features 614 through the DiT model 616 to obtain denoised video features 617, and performs splicing processing on the denoised video features 617 to obtain the denoised video hidden layer representation 618. Finally, the decoder 619 decodes the denoised video hidden layer representation 618 to obtain the second video 620.

[0090] In some embodiments, considering that in the case of generating long videos, for video frames that are far from the keyframes, the model may not be able to accurately recall and reproduce the target details from the video generation control conditions during the generation process, resulting in the problem of object consistency decay, a reference graph embedding mechanism can be introduced to optimize the video generation process, so that the model can dynamically refer to richer object visual features when generating each frame.

[0091] Please refer to Figure 7 The diagram illustrates a video generation method provided by another exemplary embodiment of this application. This embodiment uses the method for computer devices (including terminals and / or servers) as an example for illustration. The method includes the following steps.

[0092] Step 710: Based on the mask, determine the keyframes and non-keyframes from the first video. The mask is used to indicate the location of the keyframes in the first video.

[0093] The implementation method of this step can refer to step 210 in the above embodiment, and will not be repeated here.

[0094] Step 720: Perform semantic analysis and fusion on the keyframes in the first video to generate a reference feature image. The reference feature image is used to constrain the object consistency of the target object in the video to be generated.

[0095] In this embodiment of the application, a reference feature image is generated using keyframes in the first video, thereby using the reference feature image to constrain the object consistency of the target object in the video to be generated.

[0096] In one possible implementation, the computer device performs semantic analysis and fusion on keyframes in the first video, generating a reference feature image by combining the semantic information represented by multiple keyframes. Optionally, the reference feature image may be obtained by stitching together at least two keyframes, or it may be obtained by cropping and stitching together at least two keyframes.

[0097] Optionally, considering the differences in semantic information represented by different keyframes, in order to ensure that the reference feature image has comprehensive and multi-faceted high-value features, the computer device can first extract object visual features from the keyframes of the first video, and then perform feature alignment and fusion on at least two keyframes based on at least one of the keyframe number, resolution and object visual features to obtain the reference feature image.

[0098] Optionally, the computer device can adopt a content-aware adaptive grid stitching strategy, dynamically calculate the optimal grid layout based on the number of keyframes, resolution, and object visual features, and stitch together at least two keyframes into a reference feature image through feature-level alignment and fusion operations, so as to maximize the preservation and utilization of pixel-level information in the keyframes.

[0099] Optionally, the object visual features may include a first object visual feature and a second object visual feature. The first object visual feature is used to characterize the structured information of the target object from multiple perspectives, such as key viewpoint features of the target object (person) from the front, side, and back. The second object visual feature is used to characterize detailed information of key areas of the target object, such as fine-grained visual features including facial contours and local textures in the face region.

[0100] In one possible implementation, the computer device can generate visual features of a first object based on keyframes in the first video using a visual language model. Optionally, the keyframes can be understood and analyzed using a visual language model to automatically identify and classify the various perspectives of the target object in the keyframes, thereby constructing structured visual features of the first object with view semantics, which can provide a foundation for viewpoint perception generation in the video generation process.

[0101] In one possible implementation, the computer device can also generate visual features of a second object based on keyframes in the first video using a keypoint detection and localization model. Optionally, for key regions such as faces, the computer device can employ a high-precision keypoint detection and localization model to extract fine-grained visual features, including facial contours and local textures.

[0102] Optionally, the generation process of the visual features of the second object can be paralleled or cascaded with the generation process of the visual features of the first object, thereby realizing multi-level feature embedding from global structure to local details.

[0103] Step 730: Generate contextual features based on the mask, keyframes, non-keyframes, and reference feature image.

[0104] Furthermore, after obtaining the reference feature image, the computer device can combine the mask, keyframes, non-keyframes, and reference feature image to generate contextual features, thereby enabling the model to learn more comprehensive and multi-angle object visual features from the contextual features, and to enhance the appearance consistency of the target object on a global scale.

[0105] Indicative, such as Figure 8 As shown, compared to a no-reference feature image, setting a reference feature image can make the target object 801 in the generative video frame have richer and more consistent detailed features.

[0106] In one possible implementation, the computer device can place a reference feature image before the first video frame of the first video, and mark the reference feature image as a video frame that remains unchanged during video generation using a mask, meaning the mask value corresponding to the reference feature image is the same as that of the keyframe. Then, after generating a new video, the reference feature image is removed, resulting in a second video that does not contain the reference feature image.

[0107] Step 740: Generate text features based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated.

[0108] Step 750: Based on contextual features and textual features, generate a second video. The second video includes keyframes located at keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0109] The specific implementation of steps 740-750 can be referred to steps 230-240 in the above embodiments, and will not be repeated here.

[0110] In the above embodiments, a reference image embedding mechanism is introduced. By extracting more comprehensive and multi-angle visual features of the target object from keyframes, a reference feature image is generated. The reference feature image is then embedded as an additional visual control condition into the video generation process. This allows the model to dynamically reference richer object features during video generation, enabling the target object to maintain stable appearance details in long videos, thereby improving the quality of video generation.

[0111] In some embodiments, by simulating the creative logic of "sketching first, then refining," the video generation process can also be divided into two stages based on the MoE (Mixture of Experts) architecture: a high-noise stage and a low-noise stage. The high-noise stage is the sketching stage, used to achieve fast inference at low resolution; the low-noise stage is the refining stage, used to achieve high-resolution detail enhancement.

[0112] Please refer to Figure 9 The diagram illustrates a video generation method provided by another exemplary embodiment of this application. This embodiment uses the method for computer devices (including terminals and / or servers) as an example for illustration. The method includes the following steps.

[0113] Step 910: Based on the mask, determine the keyframes and non-keyframes from the first video. The mask is used to indicate the location of the keyframes in the first video.

[0114] The specific implementation of this step can be referred to step 210 in the above embodiment, and will not be repeated here.

[0115] Step 920: Generate the first contextual features based on the mask, keyframes, and non-keyframes.

[0116] Optionally, considering the different resolutions processed in the high-noise and low-noise stages, contextual features need to be adaptively constructed for each stage. Furthermore, considering that the ultimate goal of video generation is a high-resolution generative video, i.e., the first input video is also a high-resolution original video, the computer device generates the first contextual feature based on masks, keyframes, and non-keyframes, which is then used as the high-resolution contextual feature for the low-noise stage.

[0117] Step 930: Generate a second contextual feature based on the mask, the downsampled keyframes, and the non-keyframes.

[0118] To obtain low-resolution contextual features for the high-noise phase, the computer device needs to perform downsampling processing on the first video, first obtaining downsampled keyframes and non-keyframes, and then generating second contextual features based on the mask, the downsampled keyframes, and non-keyframes. Optionally, this downsampling process may only include spatial dimension downsampling.

[0119] For example, the resolution corresponding to the high noise stage can be 480P, and the resolution corresponding to the low noise stage can be 720P or 1080P. Thus, the computer device can first convert the 720P or 1080P first video into a 480P first video through downsampling processing, and then generate the second context features.

[0120] Optionally, the specific implementation process for generating the first context feature and the second context feature can be referred to the above embodiments, and will not be repeated here.

[0121] Step 940: Generate text features based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated.

[0122] The specific implementation method of this step can be referred to step 230 in the above embodiment, and will not be repeated here.

[0123] Step 950: Based on the second context features and text features, the noisy video features are denoised and decoded using the first diffusion model to obtain the third video.

[0124] In one possible implementation, during the high-noise stage, the computer device uses the second context features and text features as video generation control conditions to constrain the video generation process. The second context features and text features are injected into the first diffusion model. The first diffusion model is used to perform denoising processing on the noisy video features to obtain the first denoised video features. The first denoised video features are then spliced ​​to obtain the first denoised video hidden layer representation. Finally, the first denoised video hidden layer representation is decoded by the decoder to obtain the low-resolution third video.

[0125] Optionally, the third video has a lower resolution, but it is still able to represent a stable global structure and motion trajectory, meaning that "what happened" can be quickly determined based on the video content represented by the third video.

[0126] Optionally, the first diffusion model can be a lightweight 480P low-resolution base model, and due to the low resolution, it can effectively reduce computational overhead and time consumption, and achieve rapid convergence of model training.

[0127] Before applying the first diffusion model, low-resolution sample videos and ground truth videos can be used to train the first diffusion model. In one possible implementation, the computer device determines first sample keyframes and first sample non-keyframes from the first sample video based on a first sample mask. The first sample mask indicates the location of keyframes in the first sample video. Based on the first sample mask, the first sample keyframes, and the first sample non-keyframes, first sample context features are generated, and first sample text features are generated based on first sample text prompts. Then, based on the first sample context features and the first sample text features, the computer device performs denoising and decoding processing on the sample noisy video features using the first diffusion model to obtain a third sample video. Based on the third sample video and the first video ground truth corresponding to the first sample video, the first video generation loss is determined, thus enabling the training of the first diffusion model based on the first video generation loss. Finally, after the training completion conditions are met, the trained first diffusion model can be obtained.

[0128] Step 960: Based on the first context features and text features, the third video after upsampling and encoding is denoised and decoded using the second diffusion model to obtain the second video.

[0129] Optionally, in the low-noise stage, the focus is primarily on local texture refinement, detail enhancement, and image quality improvement based on the "sketch" generated in the high-noise stage. That is, in the low-noise stage, the second diffusion model no longer performs denoising processing on the features of the noisy video, but rather performs denoising processing on the third video.

[0130] Furthermore, considering the higher resolution corresponding to the low-noise stage, the computer equipment also needs to perform upsampling and encoding processing on the third video in order to adapt to the high-resolution second diffusion model.

[0131] In one possible implementation, during the low-noise stage, the computer device uses the first context features and text features as video generation control conditions to constrain the video generation process. The first context features and text features are injected into the second diffusion model. The second diffusion model is used to perform denoising processing on the third video after upsampling and encoding to obtain the second denoised video features. The second denoised video features are then spliced ​​to obtain the second denoised video hidden layer representation. Finally, the second denoised video hidden layer representation is decoded by the decoder to obtain the high-resolution second video.

[0132] Optionally, the second diffusion model can be a 720P or 1080P high-resolution refinement model. In this case, since the third video already has relatively stable video content, the pixel uncertainty that needs to be corrected is greatly reduced when using a high-resolution model in the low-noise stage. Therefore, it is possible to significantly improve the clarity and detail of the video while minimizing additional computational costs.

[0133] Before applying the second diffusion model, high-resolution sample videos and ground truth videos can be used to train the second diffusion model. In one possible implementation, the computer device determines second sample keyframes and second sample non-keyframes from the second sample video based on a second sample mask. The second sample mask is used to indicate the positions of keyframes in the second sample video. Based on the second sample mask, second sample keyframes, and second sample non-keyframes, second sample context features are generated, and second sample text features are generated based on second sample text prompts. Then, based on the second sample context features and second sample text features, the computer device performs denoising and decoding processing on the sample noisy video features using the second diffusion model to obtain a fourth sample video. Based on the fourth sample video and the corresponding second video ground truth videos, the second video generation loss is determined, thus enabling the training of the second diffusion model based on the second video generation loss. Finally, after the training completion conditions are met, the trained second diffusion model can be obtained.

[0134] Please refer to Figure 10 This illustrates a schematic diagram of a video generation method provided in another exemplary embodiment of this application. For example... Figure 10 As shown, the video generation process can be divided into a high-noise stage and a low-noise stage based on the MoE architecture. In the high-noise stage, the noisy video features are first analyzed using the first diffusion model 1001. Perform denoising to generate a low-resolution third-party video. In the low-noise stage, the third video, after upsampling and encoding, is further denoised using the second diffusion model 1002 to obtain a high-resolution second video. .

[0135] In the above embodiments, by combining the MoE architecture to divide the video generation process into a high-noise stage and a low-noise stage, a low-resolution third video is generated first and then a high-resolution second video is generated. This can save resource consumption in the high-noise stage and improve the image details in the low-noise stage, thereby shortening the model inference time while ensuring the quality of video generation.

[0136] Please refer to Figure 11The diagram illustrates a video generation method provided by another exemplary embodiment of this application. This embodiment uses the method for computer devices (including terminals and / or servers) as an example for illustration. The method includes the following steps.

[0137] Step 1110: In response to the keyframe selection operation for the first video, the video frame indicated by the keyframe selection operation is determined as the keyframe of the first video.

[0138] Optionally, the video generation method proposed in this application allows users to select any video frame in the first video as a keyframe and independently specify keyframe control conditions for each keyframe, realizing a paradigm shift from "passive generation" to "active directing." This means allowing users to place binding keyframe anchor points at any point on the video timeline of the first video. Each anchor point represents a clear creative intention, which could be a content transition, a compositional change, camera movement, or a switch in character status.

[0139] Based on this, in one possible implementation, in response to a keyframe selection operation for the first video, the computer device determines the video frame indicated by the keyframe selection operation as the keyframe of the first video. Optionally, the keyframe selection operation can be a point-and-click operation on the video frame or an input operation on the frame number corresponding to the video frame; this application embodiment does not limit this.

[0140] Step 1120: Generate a mask based on the position of the keyframe in the first video. The mask is used to indicate the position of the keyframe in the first video.

[0141] In one possible implementation, the computer device generates a mask based on the position of the keyframe in the first video, thereby enabling the use of the mask to indicate the position of the keyframe in the first video during the video generation process.

[0142] Optionally, the mask corresponds one-to-one with a video frame, and the mask value is either 0 or 1. A mask value of 0 indicates that the video frame is a keyframe, and a mask value of 1 indicates that the video frame is a non-keyframe. Optionally, the mask sequence can be represented as follows: , This indicates the area that must be followed by the keyframe. Indicates the area that needs to be edited.

[0143] Step 1130: In response to the text input operation, obtain the first text prompt word, which is the content description text of the video to be generated.

[0144] At the same time, the computer device can also receive the user's text input operation, thereby responding to the text input operation, obtaining the first text prompt word, and using the first text prompt word as a text control condition to control the video generation process.

[0145] Optionally, the first text cue word is a content description text for the video to be generated, such as "A girl is walking from a distance with flowers in her hand".

[0146] Step 1140: In response to the video generation operation, a second video is generated based on the first text prompt, a mask, keyframes and non-keyframes in the first video. The keyframes and non-keyframes are determined from the first video based on the mask. The second video includes keyframes located at keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0147] Optionally, the video generation operation can be a trigger operation of the video generation control or a voice control command for video generation; this application embodiment does not limit this.

[0148] In one possible implementation, in response to a video generation operation, a computer device determines keyframes and non-keyframes from a first video based on a mask, and generates contextual features based on the mask, keyframes, and non-keyframes. Simultaneously, it generates textual features based on a first text prompt, thereby using the contextual features and textual features as video generation control conditions. The noisy video features are then denoised using a diffusion model and decoded to obtain a second video.

[0149] Optionally, the generated second video includes the keyframes that remain unchanged at the keyframe locations, as well as the generated video frames at non-keyframe locations outside the keyframe locations.

[0150] In summary, in this embodiment of the application, by using a mask to indicate the keyframe positions in the first video, it is possible to define any number of keyframes at any position in the first video. This allows keyframes and non-keyframes to be determined from the first video based on the mask. Contextual features are then generated based on the mask, keyframes, and non-keyframes. These contextual features, along with text features generated based on the first text prompt, are then combined to generate the second video. This approach allows the generation process of video frames located at non-keyframe positions to be controlled using keyframes located at keyframe positions, resulting in a more coherent and natural second video and improving the overall quality of the generated second video.

[0151] Please refer to Figure 12 The diagram illustrates a structural block diagram of a video generation apparatus provided in an exemplary embodiment of this application, the apparatus comprising: The first determining module 1201 is used to determine key frames and non-key frames from the first video based on a mask, wherein the mask is used to indicate the position of key frames in the first video. The first feature generation module 1202 is used to generate context features based on the mask, the key frame and the non-key frame, wherein the context features are used to constrain the content coherence between video frames. The second feature generation module 1203 is used to generate text features based on the first text prompt word, wherein the first text prompt word is the content description text of the video to be generated. The video generation module 1204 is used to generate a second video based on the context features and the text features. The second video includes the key frame located at the key frame position and the generative video frames located at non-key frame positions outside the key frame position.

[0152] Optionally, the first feature generation module 1202 is used for: The keyframes and non-keyframes are encoded by an encoder to obtain the hidden layer representations of the keyframes and non-keyframes. The mask is reconstructed to obtain a hidden layer representation of the mask, and the feature channel dimension of the hidden layer representation of the mask matches the feature channel dimension of the hidden layer representation of the keyframe. The context features are generated by a context embedder based on the keyframe hidden layer representation, the non-keyframe hidden layer representation, and the mask hidden layer representation.

[0153] Optionally, the feature channel dimension represented by the mask hidden layer is equal to the product of the spatial downsampling factor and the temporal downsampling factor used when encoding the keyframe.

[0154] Optionally, the second feature generation module 1203 is used for: Based on the first text prompt and the first video, a second text prompt is generated using a visual language model. The content richness of the second text prompt is higher than that of the first text prompt. The text features are generated by a text encoder based on the second text prompt word.

[0155] Optionally, the device further includes: The image generation module is used to perform semantic analysis and fusion on the key frames in the first video to generate a reference feature image, which is used to constrain the object consistency of the target object in the video to be generated. The first feature generation module 1202 is used for: The context features are generated based on the mask, the keyframes, the non-keyframes, and the reference feature image.

[0156] Optionally, the image generation module is used for: Extract object visual features from the keyframes of the first video, wherein the object visual features are the appearance visual features corresponding to the target object in the keyframes; Based on at least one of the frame number, resolution, and visual features of the object, feature alignment and fusion are performed on at least two keyframes to obtain the reference feature image.

[0157] Optionally, the image generation module is used for: Based on the keyframes in the first video, a first object visual feature is generated through a visual language model. The first object visual feature is used to characterize the structured information of the target object from multiple perspectives. Based on the keyframes in the first video, a second object visual feature is generated through a key point detection and localization model. The second object visual feature is used to characterize the detailed information of the key areas of the target object.

[0158] Optionally, the video generation module 1204 is used for: The noisy hidden layer representation is segmented to obtain noisy video features; Based on the context features and the text features, denoising processing is performed on the noisy video features to obtain denoised video features; The denoised video features are spliced ​​together to obtain the hidden layer representation of the denoised video; The second video is obtained by decoding the hidden layer representation of the denoised video using a decoder.

[0159] Optionally, the keyframe position includes the position of an intermediate video frame in the first video, excluding the positions of the first and last video frames.

[0160] Optionally, the first feature generation module 1202 is used for: Based on the mask, the key frame, and the non-key frame, a first context feature is generated; Based on the mask, the downsampled keyframes, and the non-keyframes, a second contextual feature is generated. The video generation module 1204 is used for: Based on the second context features and the text features, the noisy video features are denoised and decoded using the first diffusion model to obtain the third video. Based on the first contextual features and the text features, the second video is obtained by performing denoising and decoding on the third video after upsampling and encoding through the second diffusion model.

[0161] Optionally, the device further includes: The second determining module is used to determine the first sample keyframe and the first sample non-keyframe from the first sample video based on the first sample mask. The third feature generation module is used to generate first sample context features based on the first sample mask, the first sample keyframe, and the first sample non-keyframe. The fourth feature generation module is used to generate features of the first sample text based on the prompt words of the first sample text; The first sample video generation module is used to perform denoising and decoding processing on the sample noisy video features based on the first sample context features and the first sample text features, and to obtain the third sample video by using a first diffusion model. The first training module is used to train the first diffusion model based on the third sample video and the first video ground truth corresponding to the first sample video.

[0162] Optionally, the device further includes: The third determining module is used to determine the second sample keyframes and the second sample non-keyframes from the second sample video based on the second sample mask. The fifth feature generation module is used to generate second sample context features based on the second sample mask, the second sample keyframe, and the second sample non-keyframe. The sixth feature generation module is used to generate second sample text features based on the prompt words in the second sample text; The second sample video generation module is used to perform denoising and decoding processing on the sample noisy video features based on the second sample context features and the second sample text features, and to obtain the fourth sample video by using the second diffusion model. The second training module is used to train the second diffusion model based on the fourth sample video and the ground truth value of the second video corresponding to the second sample video.

[0163] Please refer to Figure 13 The diagram illustrates a structural block diagram of a video generation apparatus provided in an exemplary embodiment of this application, the apparatus comprising: The keyframe determination module 1301 is configured to determine the video frame indicated by the keyframe selection operation as the keyframe of the first video in response to the keyframe selection operation for the first video. The mask generation module 1302 is used to generate a mask based on the position of the keyframe in the first video, the mask being used to indicate the position of the keyframe in the first video; The acquisition module 1303 is used to acquire a first text prompt word in response to a text input operation. The first text prompt word is a content description text of the video to be generated. The video generation module 1304 is configured to generate a second video in response to a video generation operation, based on the first text prompt, the mask, keyframes and non-keyframes in the first video, wherein the keyframes and non-keyframes are determined from the first video based on the mask, and the second video includes the keyframes located at the keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

[0164] In summary, in this embodiment of the application, by using a mask to indicate the keyframe positions in the first video, it is possible to define any number of keyframes at any position in the first video. This allows keyframes and non-keyframes to be determined from the first video based on the mask. Contextual features are then generated based on the mask, keyframes, and non-keyframes. These contextual features, along with text features generated based on the first text prompt, are then combined to generate the second video. This approach allows the generation process of video frames located at non-keyframe positions to be controlled using keyframes located at keyframe positions, resulting in a more coherent and natural second video and improving the overall quality of the generated second video.

[0165] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0166] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user-related data (e.g., operational data involved in this application). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their relevant data is being collected. This ensures that the application only begins the steps for collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without receiving confirmation from the user), the steps for collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0167] Please refer to Figure 14This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including a random access memory 1402 and a read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 may also include a basic input / output system (I / O system) 1406 to facilitate the transfer of information between various devices within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0168] In some embodiments, the basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to a system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0169] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer-readable media (not shown) such as a hard disk or drive.

[0170] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0171] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1401. The one or more programs contain instructions for implementing the methods described above. The central processing unit 1401 executes the one or more programs to implement the video generation methods provided in the various method embodiments described above.

[0172] According to various embodiments of this application, the computer device 1400 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer systems (not shown).

[0173] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the video generation method described in the above embodiments.

[0174] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0175] This application provides a computer program product including at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the video generation method described in the above embodiments.

[0176] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0177] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video generation method, characterized in that, The method includes: Based on a mask, keyframes and non-keyframes are determined from a first video, wherein the mask is used to indicate the location of keyframes in the first video; Based on the mask, the keyframes, and the non-keyframes, contextual features are generated, which are used to constrain the content coherence between video frames. Text features are generated based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated; Based on the context features and the text features, a second video is generated. The second video includes the keyframe located at the keyframe position and generative video frames located at non-keyframe positions outside the keyframe position.

2. The method according to claim 1, characterized in that, The generation of context features based on the mask, the keyframes, and the non-keyframes includes: The keyframes and non-keyframes are encoded by an encoder to obtain the hidden layer representations of the keyframes and non-keyframes. The mask is reconstructed to obtain a hidden layer representation of the mask, and the feature channel dimension of the hidden layer representation of the mask matches the feature channel dimension of the hidden layer representation of the keyframe. The context features are generated by a context embedder based on the keyframe hidden layer representation, the non-keyframe hidden layer representation, and the mask hidden layer representation.

3. The method according to claim 2, characterized in that, The feature channel dimension represented by the mask hidden layer is equal to the product of the spatial downsampling factor and the temporal downsampling factor used when encoding the keyframe.

4. The method according to any one of claims 1 to 3, characterized in that, The text features generated based on the first text prompt word include: Based on the first text prompt and the first video, a second text prompt is generated using a visual language model. The content richness of the second text prompt is higher than that of the first text prompt. The text features are generated by a text encoder based on the second text prompt word.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Semantic analysis and fusion are performed on the keyframes in the first video to generate a reference feature image, which is used to constrain the object consistency of the target object in the video to be generated. The generation of context features based on the mask, the keyframes, and the non-keyframes includes: The context features are generated based on the mask, the keyframes, the non-keyframes, and the reference feature image.

6. The method according to claim 5, characterized in that, The step of performing semantic analysis and fusion on the keyframes in the first video to generate a reference feature image includes: Extract object visual features from the keyframes of the first video, wherein the object visual features are the appearance visual features corresponding to the target object in the keyframes; Based on at least one of the frame number, resolution, and visual features of the object, feature alignment and fusion are performed on at least two keyframes to obtain the reference feature image.

7. The method according to claim 6, characterized in that, Extracting object visual features from the keyframes of the first video includes at least one of the following: Based on the keyframes in the first video, a first object visual feature is generated through a visual language model. The first object visual feature is used to characterize the structured information of the target object from multiple perspectives. Based on the keyframes in the first video, a second object visual feature is generated through a key point detection and localization model. The second object visual feature is used to characterize the detailed information of the key areas of the target object.

8. The method according to any one of claims 1 to 7, characterized in that, The process of generating a second video based on the context features and the text features includes: The noisy hidden layer representation is segmented to obtain noisy video features; Based on the context features and the text features, denoising processing is performed on the noisy video features to obtain denoised video features; The denoised video features are spliced ​​together to obtain the hidden layer representation of the denoised video; The second video is obtained by decoding the hidden layer representation of the denoised video using a decoder.

9. The method according to any one of claims 1 to 8, characterized in that, The keyframe positions include the positions of intermediate video frames in the first video, excluding the first and last video frames.

10. The method according to any one of claims 1 to 9, characterized in that, The generation of context features based on the mask, the keyframes, and the non-keyframes includes: Based on the mask, the key frame, and the non-key frame, a first context feature is generated; Based on the mask, the downsampled keyframes, and the non-keyframes, a second contextual feature is generated. The process of generating a second video based on the context features and the text features includes: Based on the second context features and the text features, the noisy video features are denoised and decoded using the first diffusion model to obtain the third video. Based on the first contextual features and the text features, the second video is obtained by performing denoising and decoding on the third video after upsampling and encoding through the second diffusion model.

11. The method according to claim 10, characterized in that, The method further includes: Based on the first sample mask, determine the first sample keyframes and the first sample non-keyframes from the first sample video; Based on the first sample mask, the first sample keyframe, and the first sample non-keyframe, generate the first sample context features; Generate first sample text features based on the prompt words in the first sample text; Based on the first sample context features and the first sample text features, the third sample video is obtained by performing denoising and decoding processing on the sample noisy video features through the first diffusion model. The first diffusion model is trained based on the third sample video and the ground truth value of the first video corresponding to the first sample video.

12. The method according to claim 10, characterized in that, The method further includes: Based on the second sample mask, determine the second sample keyframes and the second sample non-keyframes from the second sample video; Based on the second sample mask, the second sample keyframe, and the second sample non-keyframe, generate the second sample context features; Generate second sample text features based on the prompt words in the second sample text; Based on the second sample context features and the second sample text features, the second diffusion model is used to perform denoising and decoding on the sample noisy video features to obtain the fourth sample video. The second diffusion model is trained based on the fourth sample video and the ground truth value of the second video corresponding to the second sample video.

13. A video generation method, characterized in that, The method includes: In response to a keyframe selection operation for a first video, the video frame indicated by the keyframe selection operation is determined as a keyframe of the first video. Based on the position of the keyframe in the first video, a mask is generated, and the mask is used to indicate the position of the keyframe in the first video. In response to a text input operation, a first text prompt word is obtained, wherein the first text prompt word is a content description text of the video to be generated; In response to a video generation operation, a second video is generated based on the first text prompt, the mask, keyframes and non-keyframes in the first video, wherein the keyframes and non-keyframes are determined from the first video based on the mask, and the second video includes the keyframes located at the keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

14. A video generation apparatus, characterized in that, The device includes: A first determining module is used to determine keyframes and non-keyframes from a first video based on a mask, wherein the mask is used to indicate the positions of keyframes in the first video. The first feature generation module is used to generate context features based on the mask, the key frame, and the non-key frame, wherein the context features are used to constrain the content coherence between video frames. The second feature generation module is used to generate text features based on the first text prompt word, where the first text prompt word is the content description text of the video to be generated. A video generation module is used to generate a second video based on the context features and the text features. The second video includes the key frame located at the key frame position and generative video frames located at non-key frame positions outside the key frame position.

15. A video generation apparatus, characterized in that, The device includes: A keyframe determination module is used to determine the video frame indicated by the keyframe selection operation as a keyframe of the first video in response to a keyframe selection operation for the first video. A mask generation module is used to generate a mask based on the position of the keyframe in the first video, the mask being used to indicate the position of the keyframe in the first video; The acquisition module is used to acquire a first text prompt word in response to a text input operation. The first text prompt word is a content description text of the video to be generated. A video generation module is configured to generate a second video in response to a video generation operation, based on the first text prompt, the mask, keyframes and non-keyframes in the first video, wherein the keyframes and non-keyframes are determined from the first video based on the mask, and the second video includes the keyframes located at the keyframe positions and generative video frames located at non-keyframe positions outside the keyframe positions.

16. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one instruction, which is executed by the processor to implement the video generation method as described in any one of claims 1 to 12, or the video generation method as described in claim 13.

17. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is executed by a processor to implement the video generation method as described in any one of claims 1 to 12, or the video generation method as described in claim 13.

18. A computer program product, characterized in that, The computer program product includes at least one instruction stored in a computer-readable storage medium; a processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction to cause the computer device to implement the video generation method as described in any one of claims 1 to 12, or the video generation method as described in claim 13.

Images