A method, system and device for controlling scene changes in Wensheng video

Abstract

This application provides a method, system, and device for text-to-image generation with controllable scene changes, relating to the field of computer technology. The method includes: inputting descriptive text into a video generator based on a spatial layout fusion device to generate video segments. The video generator is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of a U-shaped network. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: a moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene. Through a cross-segment scheduling mechanism module, multiple video frames at the end of the previous video segment generated by the video generator are input into a frequency mixing control branch network. Through the frequency mixing control branch network, frequency information is mixed in an autoregressive manner based on the multiple video frames at the end of the previous video segment to generate a long video that conforms to the descriptive text.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system, and device for generating video with controllable scene changes. Background Technology

[0002] Due to the success of text-to-image generation models, generative video models (i.e., models that generate videos using text) have received increasing attention. However, current common text-to-video methods often fail to faithfully generate the trajectory of the subject described in the text prompt, and when it is necessary to change the scene according to the subject's movement trajectory, they often struggle to follow realistic logic and thus change the video background based on the prompt text.

[0003] Therefore, there is a need for a text-generated video method, system, and device with controllable scene changes to improve the video quality of videos where the scene changes with the subject's movement trajectory, thus achieving controllable scene changes. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a text-generated video method, system and device with controllable scene changes, which improves the video quality of videos where the scene changes with the subject's movement trajectory and achieves controllable scene changes.

[0005] A first aspect of this application provides a text-based video method with controllable scene changes, the method comprising:

[0006] The descriptive text describing the motion trajectory of the moving subject is input into a video generator based on a spatial layout fusion device to generate video clips. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of the U-net. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject from the first scene to the second scene.

[0007] The cross-segment scheduling mechanism module inputs multiple video frames from the end of the previous video segment generated by the video generator into the frequency mixing control branch network.

[0008] The frequency mixing control branch network uses an autoregressive approach to mix frequency information based on multiple video frames at the end of the previous video segment to generate a continuous long video that conforms to the description text.

[0009] In one possible implementation, the first training dataset used to train the video generator based on the spatial layout fusion processor is obtained according to the following steps:

[0010] Acquire training images and their layout information; the training images include: a first sample scene and a second sample scene, and a target subject;

[0011] The training image is cropped according to the movement trajectory of the target subject using a sliding window to obtain multiple sample video frames;

[0012] Based on the multiple sample video frames, a simulated video is generated;

[0013] The first training dataset is generated based on the training images, the layout information, and the simulated video.

[0014] In one possible implementation, the generation process of the first training dataset meets the following conditions:

[0015] The training data is labeled by adding a first suffix word to the prompt words;

[0016] Based on preset hyperparameters, noise is perturbed in each training data set;

[0017] Add manually labeled video segmentation datasets to the first training dataset.

[0018] In one possible implementation, the optimization objective of the video generator based on the spatial layout fusion unit is:

[0019] ;

[0020] in, It is a representation of the text prompt in the F-frame video. It is noise from the sampled F-frame video. It is the noisy latent variable of the F frames of video at time step t. It is noise after interference. It is the predicted noise output by the video generator. It is the representation of the text prompts in the encoded F-frame video. 't' represents the layout information, and 't' represents the time step.

[0021] In one possible implementation, the frequency mixing control branch network includes a frequency mixing module and a first encoding module; through the frequency mixing control branch network, using an autoregressive approach, frequency information is mixed based on multiple video frames at the end of the previous video segment to generate a continuous long video, including:

[0022] The sampled first noise is input into the frequency mixing module of the control branch network of the frequency mixing, and the first noise is used to generate the current video segment;

[0023] The frequency mixing module initializes the hidden variables of the current video segment according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency and low-frequency information of the reference information to obtain the processing result; the reference information consists of multiple video frames at the end of the previous video segment.

[0024] The processing result is encoded by the first encoding module to obtain the encoded result;

[0025] The encoding result is input into the decoding module of the video generator to generate the current video segment;

[0026] The long video is obtained by generating multiple video segments in succession.

[0027] In one possible implementation, the method further includes:

[0028] A second training dataset is obtained for training the control branch network of frequency mixing. The second training dataset consists of multiple sample videos. Each sample video corresponds to a sample text description and a sample layout information. The sample text description includes: the sample moving subject and the corresponding scene information. The sample layout information includes: the motion trajectory of the sample moving subject from the third sample scene to the fourth sample scene.

[0029] After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before training the frequency mixing control branch network. The frequency mixing control branch network is then trained on the second training dataset.

[0030] In one possible implementation, the method further includes:

[0031] The cross-segment scheduling mechanism module uses video information from multiple video frames at the end of the previous video segment generated by the video generator to replace video information from multiple video frames at the beginning of the next video segment.

[0032] In one possible implementation, the method further includes:

[0033] In the process of generating video segments based on the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the implicit variables of each video frame into the spatial layout fusion unit.

[0034] A second aspect of this application also provides a scene-change-controllable text-to-image system, applied to perform the scene-change-controllable text-to-image method described in the first aspect of this application, the system comprising:

[0035] A video generator based on a spatial layout fusion device is used to generate video clips based on descriptive text describing the motion trajectory of a moving subject. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of a U-net. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene.

[0036] The cross-segment scheduling mechanism module is used to input multiple video frames from the end of the previous video segment generated by the video generator into the frequency mixing control branch network;

[0037] The frequency mixing control branch network is used to mix frequency information based on multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, which is a video that conforms to the description text.

[0038] A third aspect of this application provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the scene-change-controllable text-to-video method described in the first aspect of this application.

[0039] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the scene-change-controllable text-to-video method described in the first aspect of this application.

[0040] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the scene-change-controllable text-to-video method as described in the first aspect of this application.

[0041] This application provides a method, system, and device for text-to-image generation with controllable scene changes. The method includes: inputting descriptive text describing the motion trajectory of a moving subject into a video generator based on a spatial layout fusion device to generate video segments. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of a U-net, wherein the spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene. Through a cross-segment scheduling mechanism module, multiple video frames at the end of the previous video segment generated by the video generator are input into a frequency mixing control branch network. Through the frequency mixing control branch network, frequency information is mixed in an autoregressive manner based on the multiple video frames at the end of the previous video segment to generate a continuous long video, wherein the long video is a video that conforms to the descriptive text.

[0042] The specific beneficial effects are as follows:

[0043] This application combines a spatial layout fusion processor with Unet to generate a video generator based on the spatial layout fusion processor, introducing layout knowledge into the text-based video task to improve the performance of the video generator. Furthermore, this application also uses a frequency mixing control branch network and an autoregressive approach to mix video information from multiple video frames at the end of the previous video segment (e.g., high-frequency and low-frequency information of each frame) to generate the next video segment consecutive to the previous one. This allows multiple video segments to be combined to form a continuous long video, thereby achieving controllable scene transitions. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the steps of a scene-change-controllable text-to-video generation method provided in an embodiment of this application.

[0046] Figure 2 This is a schematic diagram of the architecture of a video generator based on a spatial layout fusion device provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating the video clip generation process provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the execution flow of a cross-segment scheduling mechanism provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of a text-based video system provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0052] With advancements in diffusion model methods, text-to-image generation has achieved significant success in recent years. Driven by the success of image generation, researchers have made numerous efforts to incorporate temporal information into diffusion models for text-to-video generation. On one hand, some methods are trained directly on large-scale multimodal datasets. On the other hand, some methods utilize pre-trained text-to-image generation models and incorporate temporal information using various methods, such as cross-frame attention mechanisms or additional temporal modules, to obtain text-generated video models. Despite these progresses, general text-to-video methods often fail to faithfully generate the trajectory of the subject described in the text prompt and struggle to follow realistic logic to change the video background based on the prompt text.

[0053] However, current texturing videos still suffer from two main problems. First, they cannot control the transformation of the video background according to realistic generation logic; second, they cannot accurately control the movement trajectory of objects. For example, regarding the first problem, when generating an image of "a dog running from a meadow to a wheat field," the expected video logic is: initially, the dog stands on the meadow, with most of the video background being meadow. As time progresses, the dog begins to move in a certain direction, and wheat fields gradually appear at the edge of the video. By the end of the video, most of the background has completely transformed into wheat fields, and the dog has run into the wheat field. Current methods either use a background that is purely meadow, wheat field, or a mixture of both from the beginning, without any change, or a "transparent gradient" process, where wheat fields gradually emerge from the meadow, which does not conform to realistic logic. Regarding the second problem, when generating an image of "a dog running from left to right," existing methods cannot control the dog's movement rules, resulting in a random and disordered movement trajectory.

[0054] On the one hand, since the commonly used diffusion models are still probabilistic, in the absence of supervision, general text-to-video methods often fail to faithfully generate the trajectory of the subject described in the text prompt and struggle to follow real-world logic to change the video background according to the prompt text. On the other hand, current controllable generation methods, which incorporate control information, require each frame to have corresponding continuous conditional information constraints for video generation. This indicates that these methods are more suitable for video editing than generation; that is, to control the video in the desired way, a reference video must first be found in which the objects match the motion trajectory requirements of the text prompt, or the background changes match the desired requirements. However, given the reality, it is impossible to obtain an infinitely large video database and a sufficiently excellent or efficient retrieval module to find such a video for all text prompts. Therefore, previous controllable generation methods cannot solve this problem.

[0055] Current methods rely entirely on data-driven approaches, failing to meet the demands for controllable generation in response to changing scenes. They necessitate extensive trial and error to manually select videos that best meet expectations. Furthermore, most current controllable generation methods depend on continuous and complex control information. Therefore, before generating the desired video, a video with a similar transformation pattern must first be retrieved as a reference. This means they resemble video editing more than video generation, failing to fully satisfy users' imaginative generation needs and limiting the creative capabilities of text-based video models.

[0056] In view of the above problems, embodiments of this application propose a method, system, and device for generating text-based videos with controllable scene changes. By utilizing provided layout information, it adapts to the changing motion trajectories of different objects, and also introduces multiple methods to maintain the consistency of the generated video. The following, in conjunction with the accompanying drawings, provides a detailed description of the method, system, and device for generating text-based videos with controllable scene changes through some embodiments and application scenarios.

[0057] The first aspect of this application proposes a text-based video generation method with controllable scene changes, referring to... Figure 1 , Figure 1 A flowchart illustrating the steps of a text-based video generation method with controllable scene changes is shown, as follows: Figure 1 As shown, the method includes:

[0058] Step S101: Input the descriptive text describing the motion trajectory of the moving subject into the video generator based on the spatial layout fusion to generate a video clip. The video generator based on the spatial layout fusion is obtained by adding a spatial layout fusion at the beginning of the cross-attention module of each layer of the U-net. The spatial layout fusion is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject moving from the first scene to the second scene.

[0059] Specifically, the descriptive text used to describe the movement trajectory of the moving subject includes at least: the moving subject and the movement trajectory of the moving subject moving from the first scene to the second scene. For example, at the beginning, the puppy (the moving subject) is standing on the grass, and most of the background of the video is grass (the first scene). As time changes, the puppy begins to move in a certain direction (movement trajectory), and wheat fields gradually appear at the edge of the video. By the end of the video, most of the background has completely become a wheat field (the second scene), and the puppy has also run into the wheat field.

[0060] Reference Figure 2 , Figure 2 A schematic diagram of the architecture of a video generator based on a spatial layout fusion is shown, such as... Figure 2 As shown, the video generator consists of three parts: a spatial layout fusion unit, a U-net (text-to-image generation model), and an action extraction module. The spatial layout fusion unit is a text-to-image generation model, essentially utilizing the label-based mechanism and gated self-attention of GLIGEN (text-to-image generation model) as the spatial layout fusion unit. The action extraction module is used to extract dynamic information. The overall model architecture of this video generator is based on a U-Net with temporal attention capabilities for dynamic information extraction (e.g., the text-to-video generation model AnimateDiff). Figure 2As shown, a spatial layout fusion module is added before each cross-attention module of Unet, and an action extraction module is added at the end of each block of Unet. This action extraction module can use pre-trained weights from AnimateDiff (an open-source text-to-video generation model). In this embodiment, to control the trajectory of objects in each frame of the video, this embodiment utilizes GLIGEN's label-based mechanism and gated self-attention as the spatial layout fusion module, and extends it to the generated video.

[0061] Step S102: Through the cross-segment scheduling mechanism module, multiple video frames from the end of the previous video segment generated by the video generator are input into the frequency mixing control branch network.

[0062] The video generator generates individual video segments based on input text descriptions (layout information, etc.), and then combines multiple video segments into a continuous long video. To ensure content continuity and semantic coherence between adjacent video segments, this embodiment utilizes a cross-segment scheduling mechanism module to obtain multiple video frames from the end of the previous video segment and input them into a frequency-mixing control branch network (this frequency-mixing control branch network is described in detail in Section 1.2 below) as reference information to assist in the generation of the current video segment. That is, when generating the 6th video segment (i.e., the current video segment), the cross-segment scheduling mechanism module obtains the last multiple video frames of the 5th video segment (i.e., the previous video segment).

[0063] Step S103: Through the frequency mixing control branch network, using an autoregressive approach, frequency information is mixed based on multiple video frames at the end of the previous video segment to generate a continuous long video, which is a video that conforms to the description text.

[0064] The video generator, utilizing a trained spatial layout fusion unit, can already generate scene-controllable video clips based on the input text description. However, scene transformation is a dynamic process, making it difficult to effectively represent within a few frames (short video clips). Therefore, a sufficiently long video (long video) is needed to fully demonstrate the movement of a subject from a first scene to a second scene along a specific trajectory. Thus, this embodiment introduces a control branch network that uses an autoregressive approach to generate several consecutive video clips. Essentially, it generates the next video clip (the current video clip) based on the last few frames of the previous clip, and then splices them together to obtain the final longer video, i.e., a continuous long video (which can be longer than 10 seconds). This long video conforms to the described text and demonstrates the dynamic scene change process of a subject moving from a first scene to a second scene along a specific trajectory.

[0065] This application proposes a scene-change-controllable text-to-image method, which mainly includes three important parts: a video generator based on a spatial layout fusion unit (Section 1.1), a control branch network for frequency mixing (Section 1.2), and a cross-segment scheduling mechanism (Section 1.3). The scene-change-controllable text-to-image method will be described below through Sections 1.1-1.3.

[0066] 1.1 Video generator based on spatial layout fusion:

[0067] This embodiment proposes that if the GLIGEN module (i.e., spatial layout fusion unit) is directly applied to the video generator, the following problems are encountered: due to the lack of suitable training data (video data with dynamic scene changes), even if a satisfactory result is obtained in the first frame after adding descriptive text (layout information), the changes are not obvious enough in the subsequent frames and still cannot perfectly meet the requirements.

[0068] To address this issue, this application proposes a method for obtaining a first training dataset, used to obtain the first training dataset used to train the video generator based on the spatial layout fusion.

[0069] In one possible implementation, the first training dataset used to train the video generator based on the spatial layout fusion processor is obtained according to the following steps:

[0070] Step S201: Obtain training images and layout information of the training images; the training images include: a first sample scene and a second sample scene, and a target subject.

[0071] Specifically, a set of common scenes and moving subjects are collected and randomly paired to generate training images (which can be generated using the GLIGEN module, i.e., the spatial layout fusion unit) with different scenes on the left and right sides (first sample scene and second sample scene). Each training image also includes a target subject. For each training image, corresponding layout information can be obtained, which includes: textual description information, used to characterize each scene in the training image, the location of each scene, the target subject, and the location of the target subject, etc. For example, the layout information can be a pair of text and numbers. For instance, when generating a video of a person riding a horse, a total of 16 frames of images need to be obtained. The corresponding layout information would be 16 dictionaries, each like {'horse': [0.0,0.5, 0.35, 0.75], 'people': [0.15, 0.30, 0.25, 0.60]}. Then, [x1, y1, x2, y2] represents the position of the subject, where [x1, y1] is the coordinate of the top left corner and [x2, y2] is the coordinate of the bottom right corner.

[0072] Step S202: Use a sliding window to crop the training image according to the movement trajectory of the target subject to obtain multiple sample video frames.

[0073] In this embodiment, a bounding box (i.e., a sliding window) is used as the condition. The cue used is "a subject is doing something in a beautiful scene between scene A and scene B, masterpiece, 8K, 4K, high quality, best quality". Then, a direction is selected (as the direction of the target subject's motion trajectory), and the image is cropped using the sliding window to obtain video frames. That is, each time the sliding window moves over the training image, a sample video frame is obtained. Thus, by moving the sliding window, multiple sample video frames are obtained.

[0074] Step S203: Generate a simulated video based on the multiple sample video frames.

[0075] Step S204: Generate the first training dataset based on the training images, the layout information, and the simulated video.

[0076] In this first training dataset, each training data point contains: a training image, the layout information corresponding to the training image, and a simulated video generated using the training image.

[0077] It should be noted that the generation process of the first training dataset mentioned in steps S201-S204 is relatively simple. Therefore, the generated simulation data mainly contains translational motion, lacks dynamic elements, and has deficiencies in data quality. However, the goal of this application's embodiments is to capture knowledge of scene transformations, without needing to fully accept these data. Therefore, the impact of low-quality data can be mitigated by one or more of the following methods:

[0078] In one possible implementation, the generation process of the first training dataset meets the following conditions:

[0079] Condition 1: The training data is labeled by adding a first suffix to the prompt words; in the application phase, a second suffix is ​​added to the prompt words, wherein the first suffix includes the second suffix.

[0080] Specifically, instead of simply using "A subject is doing something, the scene is moving from scene A to scene B" as the prompt text, this embodiment of the application labels the simulated data (i.e., the training data in the first training dataset) by adding a first suffix "moving scenes, low quality, static subject" to the prompt text. The first suffix includes the second suffix "moving scenes". In the actual application stage after the video generator has been trained (i.e., in the inference stage), it is only necessary to describe the scene to be changed, and only the second suffix "moving scenes" is added to the prompt text.

[0081] Condition 2: Based on the preset hyperparameters, noise is interfered with for each training data.

[0082] Specifically, because the simulated videos in the generated first training dataset are of low quality and biased, the embodiments of this application do not strictly predict noise. Therefore, this application proposes a preset hyperparameter. The preset hyperparameters are parameter values ​​pre-set based on experience, used to introduce slight disturbances, that is, .in, This is additional noise sampled from a Gaussian distribution with the same shape; following a general optimization approach, the noise in the output is predicted by the video generator. The noise corresponding to the simulated video in the input training data (sampled out). The closer the better; however, considering the low quality of analog videos, the embodiments of this application do not strictly predict noise. Using preset hyperparameters To introduce interference, the video generator predicts the noise in the output. With the noise after interference With the same optimization objective, the video generator is trained.

[0083] Condition 3: Add manually labeled video segmentation datasets to the first training dataset.

[0084] Specifically, in order to achieve better performance and better training results, this application embodiment mixes the first training dataset with a small amount of manually labeled video segmentation dataset (including video clips obtained from actual shooting, which are also videos of moving subjects moving from scene A to scene B according to a certain motion trajectory, and each video clip carries manually labeled layout information, which is used to describe the video content, including the process of scene changes, etc.)

[0085] In one possible implementation, the optimization objective of the video generator based on the spatial layout fusion unit is:

[0086] ;

[0087] in, It is a representation of the text prompt in the F-frame video. It is noise from the sampled F-frame video. It is the noisy latent variable of the F frames of video at time step t. It is noise after interference. It is the predicted noise output by the video generator. It is the representation of the text prompts in the encoded F-frame video. 't' represents the layout information, and 't' represents the time step.

[0088] As shown in the loss function above, during the training of the video generator based on the first training dataset, in each iteration, the predicted noise output by the video generator is... , and noise after interference (That is, the noise obtained by slightly interfering with the noise obtained from the initial sampling of the input analog video) gets closer and closer to the optimization target, so the spatial layout fusion unit and motion extraction module in the video generator are updated.

[0089] In one possible implementation, the method further includes:

[0090] In the process of generating video segments based on the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the implicit variables of each video frame into the spatial layout fusion unit.

[0091] Specifically, in the process of the video generator generating video segments (i.e., step S101), before inputting the latent variables into the spatial layout fusion processor, this embodiment of the application changes the shape of the latent variables, combining the temporal dimension of the frame with the batch size. This change process can fuse dynamically changing layout information into the corresponding video frames, thereby allowing control over the positions of the subject and background in the video.

[0092] Section 1.2 Control Branch Network for Frequency Mixing:

[0093] Directly using control branch networks will cause discontinuities in video segments. Specifically, these discontinuities can be divided into two categories: 1. Inconsistencies within video segments, i.e., the inconsistency between two parts (the first half of the video segment) containing control information and the second half of the video segment not containing control information; 2. Inconsistencies between video segments, i.e., the inconsistency of overlapping parts between different video segments (the previous video segment and the adjacent next video segment).

[0094] In the first scenario, the inconsistency within video segments may be due to a significant information gap in the reference information provided to the control branch network. Since only the first half of the current video segment is known, the second half can only be completed using methods like zero padding. This results in the control branch network receiving reference information (i.e., control information) where only the first half possesses prior knowledge, while the second half is completely chaotic. This necessitates adjustments and planning to the control information received by the control network. This embodiment proposes a control branch network utilizing frequency mixing to blend the high and low frequency information of the previous video segment into the next video segment, thereby achieving content continuity in the video segments.

[0095] In one possible implementation, the frequency mixing control branch network includes: a frequency mixing module and a first encoding module; step S103, through the frequency mixing control branch network, using an autoregressive approach, frequency information is mixed based on multiple video frames at the end of the previous video segment to generate a continuous long video, including:

[0096] Step S1031: The sampled first noise is input into the frequency mixing module of the control branch network of the frequency mixing, and the first noise is used to generate the current video segment.

[0097] Specifically, refer to Figure 3 , Figure 3 A schematic diagram illustrating the generation process of a video clip is shown, such as... Figure 3As shown, the inputs to the control branch network include: first noise obtained by random sampling (noise sampled to generate the current video segment); reference information (multiple video frames at the end of the previous video segment, where each frame has high-frequency and low-frequency information); and mask information.

[0098] Step S1032: Through the frequency mixing module, according to the mask information, the initial hidden variables of the current video segment are initialized, the low-frequency information of the reference information is spliced, and the high-frequency information and low-frequency information of the reference information are mixed to obtain the processing result; the reference information is multiple video frames at the end of the previous video segment.

[0099] Specifically, for the diffusion model, the high-frequency information in the latent variables of each video frame usually represents dynamic, temporal, and motion information, which is more microscopic and specific; while the low-frequency information emphasizes static content, layout, and tone, which is more macroscopic. Since video is a continuous process, the low-frequency information between several adjacent frames is more similar, while the high-frequency information is different for each frame.

[0100] Based on this, the embodiments of this application use a frequency mixing module to mix the high-frequency and low-frequency information of the last multiple video frames of the previous video segment with the information of the current video segment. Correspondingly, the low-frequency information of the last frame of the previous video segment is selected for padding, and the high-frequency information of the noisy latent variables at each time step is used for splicing to estimate the knowledge content that may be contained in the latter half of the reference.

[0101] The mask information is used to indicate to the frequency mixing module which low-frequency information needs to be extracted from the last video frame of the previous video segment and spliced ​​into the current video segment; and to indicate which high-frequency information needs to be extracted from the multiple video frames (reference information) at the end of the previous video segment and mixed into the low-frequency information of the video frames of the current video segment.

[0102] The frequency mixing module, according to the mask information, initializes the latent variables of the current video segment and splices the low-frequency information of the reference information. Specifically, for video, each frame is relatively continuous, meaning that the overall layout between adjacent frames does not change significantly. In this embodiment, the low-frequency information of the last few frames of the previous video segment is extracted. This low-frequency information represents the overall layout (including the main body information, position information, etc.), and is spliced ​​into the first few frames of the next video segment to improve the continuity of the video segment. In this embodiment, when multiple video frames at the end of the previous video segment are selected as reference information, the more video frames selected, the better the final video continuity effect, but the lower the video generation efficiency. Considering the balance between efficiency and effect, in this embodiment, the last 2 to 4 frames of the previous video segment can be selected as reference information.

[0103] In addition, after the frequency information is mixed, a refiner module (Mixed FrequencyRefiner) can be used to transform the mixed result.

[0104] Step S1033: The processing result is encoded by the first encoding module to obtain the encoded result.

[0105] Step S1034: Input the encoding result into the decoding module of the video generator to generate the current video segment.

[0106] Specifically, the encoding module of the trained video generator is used as the first encoding module before training the frequency mixing control branch network. Figure 3 As shown, after encoding is completed using the first encoding module, the encoded result is inserted into the corresponding layer of the decoding module of the video generator for decoding, and finally the next video segment, i.e. the current video segment, is generated.

[0107] Step S1035: Obtain the long video based on the multiple continuously generated video segments.

[0108] In one possible implementation, the method further includes:

[0109] A second training dataset is obtained for training the control branch network of the frequency mixing. The second training dataset consists of multiple sample videos, each sample video corresponding to a sample text description and a sample layout information. The sample text description includes: the sample moving subject and corresponding scene information; the sample layout information includes: the motion trajectory of the sample moving subject moving from the third sample scene to the fourth sample scene. In this embodiment, the second training dataset can be the first training dataset in Section 1.1, or generated according to the method of steps S201-S204 (the method for generating the first training dataset).

[0110] After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before training the frequency mixing control branch network. The frequency mixing control branch network is then trained on the second training dataset.

[0111] Specifically, in each training iteration (similar to steps S102 and S1031-1035 above), the training data is first input into the video generator to obtain the first video segment. Then, the frequency mixing module of the control branch network, which mixes multiple video frames at the end of the first video segment, mask information, and sampled noise input frequencies, is used. Through the frequency mixing module, the initial latent variables of the currently generated video segment are initialized according to the mask information. The low-frequency information of the last few frames of the first video segment is concatenated, and the high-frequency and low-frequency information of the reference information (multiple video frames at the end of the first video segment) are mixed to obtain the sample processing result. The sample processing result is encoded by the first encoding module to obtain the sample encoding result. The sample encoding result is inserted into the corresponding layer of the decoding module of the video generator to generate the next video segment. By repeating the above steps, multiple video segments are continuously generated, reconstructing the video from the training data.

[0112] 1.3 Cross-segment scheduling mechanism.

[0113] To address the inconsistencies between video segments, this application embodiment considers planning the latent variables of different video segments, mainly including two aspects: First, when initializing the latent variables of a segment, the initialization latent variables of the previous video segment are used to mix high and low frequency information (implemented by the frequency mixing module in the frequency mixing control branch network in Section 1.2); Second, the latent variables of the previous video segment are partially reused (implemented by the cross-segment scheduling mechanism module in this section).

[0114] In one possible implementation, the method further includes:

[0115] The cross-segment scheduling mechanism module uses video information from multiple video frames at the end of the previous video segment generated by the video generator to replace video information from multiple video frames at the beginning of the next video segment.

[0116] Specifically, the cross-fragment scheduling mechanism module is used to execute the cross-fragment scheduling mechanism, as shown in the following figure. Figure 4 , Figure 4 A schematic diagram of the execution flow of a cross-segment scheduling mechanism is shown, such as... Figure 4 As shown, this mechanism refers to the following: when generating the next video segment, information from the previously generated video segment is retrieved and used to replace a portion of the currently generated video segment (i.e., the next video segment). This ensures that the first few frames of the generated video segment completely match the content of the last frame of the previous video segment. For example, if the previous video segment generated a total of 8 frames, then the last four frames of that segment are retrieved as reference information. Replacing these with the first four frames of the next video segment only requires generating the content of the last four frames.

[0117] This application combines a spatial layout fusion processor with Unet to generate a video generator based on the spatial layout fusion processor. This introduces layout knowledge into the text-based video generation task, improving the performance of the video generator and enabling more precise control over different objects. Furthermore, this application also utilizes a frequency mixing control branch network, employing an autoregressive approach, to mix video information from multiple video frames at the end of the previous video segment (e.g., high-frequency and low-frequency information from each frame) to generate the next video segment continuous with the previous one. This allows multiple video segments to be combined to form a continuous long video, better meeting the needs of dynamic scene changes while ensuring the continuity and quality of the generated video, thus achieving controllable scene transformations.

[0118] The second aspect of this application also provides a scene-change-controllable text-to-image system, applied to the text-to-image method described in the first aspect, with reference to... Figure 5 , Figure 5 A schematic diagram of the structure of a text-based video system is shown, such as... Figure 5 As shown, the system includes:

[0119] A video generator based on a spatial layout fusion device is used to generate video clips based on descriptive text describing the motion trajectory of a moving subject. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of a U-net. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene.

[0120] The cross-segment scheduling mechanism module is used to input multiple video frames from the end of the previous video segment generated by the video generator into the frequency mixing control branch network;

[0121] The frequency mixing control branch network is used to mix frequency information based on multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, which is a video that conforms to the description text.

[0122] In one possible implementation, the first training dataset used to train the video generator based on the spatial layout fusion processor is obtained according to the following steps:

[0123] Acquire training images and their layout information; the training images include: a first sample scene and a second sample scene, and a target subject;

[0124] The training image is cropped according to the movement trajectory of the target subject using a sliding window to obtain multiple sample video frames;

[0125] Based on the multiple sample video frames, a simulated video is generated;

[0126] The first training dataset is generated based on the training images, the layout information, and the simulated video.

[0127] In one possible implementation, the generation process of the first training dataset meets the following conditions:

[0128] The training data is labeled by adding a first suffix word to the prompt words;

[0129] Based on preset hyperparameters, noise is perturbed in each training data set;

[0130] Add manually labeled video segmentation datasets to the first training dataset.

[0131] In one possible implementation, the optimization objective of the video generator based on the spatial layout fusion unit is:

[0132] ;

[0133] in, It is a representation of the text prompt in the F-frame video. It is noise from the sampled F-frame video. It is the noisy latent variable of the F frames of video at time step t. It is noise after interference. It is the predicted noise output by the video generator. It is the representation of the text prompts in the encoded F-frame video. 't' represents the layout information, and 't' represents the time step.

[0134] In one possible implementation, the frequency mixing control branch network includes a frequency mixing module and a first encoding module; through the frequency mixing control branch network, using an autoregressive approach, frequency information is mixed based on multiple video frames at the end of the previous video segment to generate a continuous long video, including:

[0135] The sampled first noise is input into the frequency mixing module of the control branch network of the frequency mixing, and the first noise is used to generate the current video segment;

[0136] The frequency mixing module initializes the hidden variables of the current video segment according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency and low-frequency information of the reference information to obtain the processing result; the reference information consists of multiple video frames at the end of the previous video segment.

[0137] The processing result is encoded by the first encoding module to obtain the encoded result;

[0138] The encoding result is input into the decoding module of the video generator to generate the current video segment;

[0139] The long video is obtained by generating multiple video segments in succession.

[0140] In one possible implementation, the device further includes:

[0141] The training data generation module is used to acquire a second training dataset for training the frequency mixing control branch network. The second training dataset consists of multiple sample videos, each sample video corresponding to a sample text description and a sample layout information. The sample text description includes: the sample moving subject and corresponding scene information; the sample layout information includes: the motion trajectory of the sample moving subject from the third sample scene to the fourth sample scene.

[0142] The training module is used to train the frequency mixing control branch network on the second training dataset after the video generator has been trained, using the trained encoding module of the video generator as the first encoding module before training the frequency mixing control branch network.

[0143] In one possible implementation, the cross-segment scheduling mechanism module is further configured to replace the video information of multiple video frames at the beginning of the next video segment with the video information of multiple video frames at the end of the previous video segment generated by the video generator.

[0144] In one possible implementation, the video generator is further configured to, during the process of generating video segments based on the descriptive text, combine the temporal dimension of each video frame with the batch size before inputting the latent variables of each video frame into the spatial layout fusion unit.

[0145] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the structure of the electronic device proposed in the embodiments of this application. Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps of the scene change controllable text-generated video method described in the first aspect of the embodiments of this application.

[0146] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the scene-change-controllable text-to-video method described in the first aspect of this application.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the scene-change-controllable text-to-video method as described in the first aspect of this application.

[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0149] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0152] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0153] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0154] The above provides a detailed description of the text-based video method, system, and device with controllable scene changes provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A text-based video generation method with controllable scene changes, characterized in that, The method includes: The descriptive text describing the motion trajectory of the moving subject is input into a video generator based on a spatial layout fusion device to generate video clips. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of the U-net. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject from the first scene to the second scene. The cross-segment scheduling mechanism module inputs multiple video frames from the end of the previous video segment generated by the video generator into the frequency mixing control branch network. The frequency mixing control branch network uses an autoregressive approach to mix frequency information based on multiple video frames at the end of the previous video segment to generate a continuous long video, which is a video that conforms to the description text. Among them, GLIGEN's tag-based mechanism and gated self-attention are used as the spatial layout fusion machine; The frequency mixing control branch network includes a frequency mixing module and a first encoding module. Through the frequency mixing control branch network, using an autoregressive approach, frequency information is mixed based on multiple video frames at the end of the previous video segment to generate a continuous long video, including: The sampled first noise is input into the frequency mixing module of the control branch network of the frequency mixing, and the first noise is used to generate the current video segment; The frequency mixing module initializes the hidden variables of the current video segment according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency and low-frequency information of the reference information to obtain the processing result; the reference information consists of multiple video frames at the end of the previous video segment. The processing result is encoded by the first encoding module to obtain the encoded result; The encoding result is input into the decoding module of the video generator to generate the current video segment; The long video is obtained by generating multiple video segments in succession.

2. The text-based video generation method with controllable scene changes according to claim 1, characterized in that, The first training dataset used to train the video generator based on the spatial layout fusion processor was obtained according to the following steps: Acquire training images and their layout information; the training images include: a first sample scene and a second sample scene, and a target subject; The training image is cropped according to the movement trajectory of the target subject using a sliding window to obtain multiple sample video frames; Based on the multiple sample video frames, a simulated video is generated; The first training dataset is generated based on the training images, the layout information, and the simulated video.

3. The text-based video generation method with controllable scene changes according to claim 2, characterized in that, The generation process of the first training dataset meets the following conditions: The training data is labeled by adding a first suffix word to the prompt words; Based on preset hyperparameters, noise is perturbed in each training data set; Add manually labeled video segmentation datasets to the first training dataset.

4. The text-based video generation method with controllable scene changes according to claim 1, characterized in that, The optimization objective of the video generator based on spatial layout fusion is: ； in, It is a representation of the text prompt in the F-frame video. It is noise from the sampled F-frame video. It is the noisy latent variable of the F frames of video at time step t. It is noise after interference. It is the predicted noise output by the video generator. It is the representation of the text prompts in the encoded F-frame video. 't' represents the layout information, and 't' represents the time step.

5. The text-based video generation method with controllable scene changes according to claim 1, characterized in that, The method further includes: A second training dataset is obtained for training the control branch network of frequency mixing. The second training dataset consists of multiple sample videos. Each sample video corresponds to a sample text description and a sample layout information. The sample text description includes: the sample moving subject and the corresponding scene information. The sample layout information includes: the motion trajectory of the sample moving subject from the third sample scene to the fourth sample scene. After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before training the frequency mixing control branch network. The frequency mixing control branch network is then trained on the second training dataset.

6. The text-based video generation method with controllable scene changes according to claim 1, characterized in that, The method further includes: The cross-segment scheduling mechanism module uses video information from multiple video frames at the end of the previous video segment generated by the video generator to replace video information from multiple video frames at the beginning of the next video segment.

7. The text-based video generation method with controllable scene changes according to claim 1, characterized in that, The method further includes: In the process of generating video segments based on the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the implicit variables of each video frame into the spatial layout fusion unit.

8. A text-based video system with controllable scene changes, characterized in that, The system, applied to the text-based video method according to any one of claims 1-7, comprises: A video generator based on a spatial layout fusion device is used to generate video clips based on descriptive text describing the motion trajectory of a moving subject. The video generator based on the spatial layout fusion device is obtained by adding a spatial layout fusion device to the beginning of the cross-attention module of each layer of a U-net. The spatial layout fusion device is a text-to-image generation model. The descriptive text includes at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene. The cross-segment scheduling mechanism module is used to input multiple video frames from the end of the previous video segment generated by the video generator into the frequency mixing control branch network; The frequency mixing control branch network is used to mix frequency information based on multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, which is a video that conforms to the description text.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the scene-change-controllable text-to-video method as described in any one of claims 1-7.

Images