Scene change controllable text video method, system and equipment

Abstract

The invention provides a scene change controllable text video method, system and device, and relates to the technical field of computers, the method comprises the following steps: inputting a description text into a video generator based on a spatial layout fusion device to generate a video clip, the video generator is obtained by adding a spatial layout fusion device at the beginning of a cross attention module of each layer of the U-shaped network, and the spatial layout fusion device is a text-to-image generation model; the description text at least comprises a motion subject and a motion track of the motion subject moving from the first scene to the second scene; inputting a plurality of video frames at the tail of a previous video clip generated by a video generator into a frequency-mixed control branch network through a cross-clip scheduling mechanism module; and performing frequency information mixing according to the plurality of video frames at the tail of the previous video clip by adopting an autoregression mode through a frequency mixing control branch network so as to generate a long video which is a video conforming to the description text.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a scene-change controllable Wensheng video method, system and device. Background Art

[0002] Due to the success of text-to-image generative models, generative video models (i.e., models that generate videos using text) have received increasing attention. However, current common text-to-video methods are usually unable to faithfully generate the trajectory of the subject described in the text prompt, and when the scene needs to change with the subject's motion trajectory, it is often difficult to follow the logic of reality and change the background of the video according to the prompt text.

[0003] Therefore, a text-generated video method, system and device with controllable scene changes are needed to improve the video quality of text-generated videos in which the scene changes with the subject's motion trajectory and achieve controllable scene changes. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a text-generated video method, system and device with controllable scene changes, which improve the video quality of videos in which the scenes generated based on text change with the subject's motion trajectory and realize controllable scene changes.

[0005] In a first aspect of an embodiment of the present application, a scene change controllable Vincent video method is provided, the method comprising: Inputting a description text for describing the motion trajectory of the moving subject into a video generator based on a spatial layout fuser to generate a video clip, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text at least includes: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; Inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network through a cross-segment scheduling mechanism module; The frequency mixing control branch network adopts an autoregressive method to mix frequency information according to multiple video frames at the end of the previous video clip to generate a continuous long video, and the long video is a video that conforms to the description text.

[0006] In a possible implementation manner, the first training data set used for training the video generator based on the spatial layout fuser is obtained according to the following steps: Acquire a training image and layout information of the training image; the training image includes: a first sample scene and a second sample scene, and a target subject; Using a sliding window to crop the training image according to the moving trajectory of the target subject to obtain a plurality of sample video frames; generating a simulated video according to the plurality of sample video frames; The first training data set is generated according to the training image, the layout information and the simulation video.

[0007] In a possible implementation, the generation process of the first training data set meets the following conditions: The training data is labeled by adding the first suffix word to the prompt word; According to the preset hyperparameters, the noise of each training data is disturbed; A manually annotated video segmentation dataset is added to the first training dataset.

[0008] In a possible implementation, the optimization goal of the video generator based on the spatial layout fuser is: ; in, is the representation of the text prompt of the F frame video, is the noise of sampling F frames of video, is the noisy latent variable of F frames of video at time step t, is the post-interference noise, is the predicted noise output by the video generator, It is the representation of the text prompt of the encoded F frame video. is the layout information and t is the time step.

[0009] In a possible implementation, the frequency mixing control branch network includes: a frequency mixing module and a first encoding module; through the frequency mixing control branch network, frequency information is mixed according to multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, including: Inputting the sampled first noise into the frequency mixing module of the frequency mixing control branch network, wherein the first noise is used to generate the current video segment; The frequency mixing module initializes the hidden variables of the current video clip according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency information and the low-frequency information of the reference information to obtain a processing result; the reference information is a plurality of video frames at the end of the previous video clip; Encoding the processing result by the first encoding module to obtain an encoding result; Inputting the encoding result into a decoding module of the video generator to generate the current video segment; The long video is obtained according to a plurality of continuously generated video clips.

[0010] In a possible implementation, the method further includes: Acquire a second training data set for training the control branch network of the frequency mixing, wherein the second training data set is composed of a plurality of sample videos, each sample video corresponds to a sample text description and a sample layout information, wherein the sample text description includes: a sample moving subject and corresponding scene information; the sample layout information includes: a motion trajectory of the sample moving subject moving from a third sample scene to a fourth sample scene; After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before the frequency mixing control branch network is trained, and the frequency mixing control branch network is trained on the second training data set.

[0011] In a possible implementation, the method further includes: The cross-segment scheduling mechanism module uses the video information of multiple video frames at the end of a previous video segment generated by the video generator to replace the video information of multiple video frames at the beginning of a next video segment.

[0012] In a possible implementation, the method further includes: In the process of generating video clips according to the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the latent variables of each video frame into the spatial layout fuser.

[0013] The second aspect of the embodiment of the present application further provides a scene change controllable Vincent video system, which is applied to execute the scene change controllable Vincent video method described in the first aspect of the embodiment of the present application, and the system includes: A video generator based on a spatial layout fuser is used to generate a video clip according to a description text used to describe the motion trajectory of a moving subject, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text at least includes: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; A cross-segment scheduling mechanism module, used for inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network; The frequency mixing control branch network is used to mix frequency information in an autoregressive manner according to multiple video frames at the end of the previous video clip to generate a continuous long video, where the long video is a video that conforms to the description text.

[0014] A third aspect of an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the scene change controllable Vincent video method described in the first aspect of the embodiment of the present application are implemented.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the scene change controllable Vincent video method described in the first aspect of the embodiment of the present application are implemented.

[0016] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the scene change controllable Vincent video method as described in the first aspect of an embodiment of the present application.

[0017] The embodiment of the present application provides a scene-change controllable Wensheng video method, system and device, the method comprising: inputting a description text for describing the motion trajectory of a moving subject into a video generator based on a spatial layout fuser to generate a video segment, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross-attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text comprises at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; through a cross-segment scheduling mechanism module, multiple video frames at the end of a previous video segment generated by the video generator are input into a frequency mixing control branch network; through the frequency mixing control branch network, frequency information is mixed according to multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, wherein the long video is a video that conforms to the description text.

[0018] The specific beneficial effects are: The embodiment of the present application combines the spatial layout fuser with Unet to generate a video generator based on the spatial layout fuser, and introduces layout knowledge into the Wensheng video task to improve the performance of the video generator. In addition, the embodiment of the present application also uses a frequency mixed control branch network, adopts an autoregressive method, and uses the video information of multiple video frames at the end of the previous video clip (such as high-frequency information and low-frequency information of each frame) to mix frequency information to generate the next video clip that is continuous with the previous video clip, so that multiple video clips can be combined to obtain a continuous long video, thereby realizing controllable scene change. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 It is a flowchart of the steps of a scene change controllable Vincent video method provided in an embodiment of the present application; Figure 2 It is a schematic diagram of the architecture of a video generator based on a spatial layout amalgamator provided in an embodiment of the present application; Figure 3 is a schematic diagram of a process for generating a video clip provided in an embodiment of the present application; Figure 4 It is a schematic diagram of an execution flow of a cross-segment scheduling mechanism provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of a Vincent video system provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0022] With the advancement of diffusion model methods, text-to-image generation has achieved remarkable success in recent years. Driven by the success of image generation, researchers have made many efforts to incorporate temporal information into diffusion models to achieve text-to-video generation. On the one hand, some methods are directly trained on large-scale multimodal datasets. On the other hand, some methods utilize pre-trained text-to-image generation models and use different methods to introduce temporal information, such as cross-frame attention mechanisms or using additional temporal modules to obtain text-generated video models. Despite some progress, general text-to-video methods usually fail to faithfully generate the trajectory of the subject described in the text prompt and have difficulty following realistic logic to change the background of the video according to the prompt text.

[0023] However, there are still two problems with the current Wensheng videos. First, it is impossible to control the transformation of the video background according to the actual generation logic; second, it is impossible to accurately control the movement trajectory of the object. For example, for the first problem, when you want to generate a picture of "a dog running from the grass to the wheat field", the expected video logic is: at the beginning, the puppy is standing on the grass, and most of the background of the entire video is grass. As time changes, the puppy begins to move in a certain direction, and the edge of the video begins to gradually appear wheat fields. At the end of the video, most of the background completely turns into wheat fields, and the puppy also runs into the wheat field. The current method either starts with the background of the video being pure grass, wheat fields, or a mixture of the two, without any signs of change, or it is a "transparent gradient" process, that is, the wheat field gradually emerges from the grass, which does not conform to the real logic. For the second problem, when you want to generate "a puppy running from the left to the right", the existing method cannot control the movement rules of the puppy, making the puppy's movement trajectory in a random and disordered state.

[0024] On the one hand, since the commonly used diffusion model is still a probabilistic model, in the absence of supervision, general text-to-video methods are usually unable to faithfully generate the trajectory of the subject described in the text prompt, and it is difficult to follow the logic of reality, so as to change the background of the video according to the prompt text. On the other hand, the controllable generation method that has added control information requires that each frame has corresponding continuous conditional information restrictions for video generation. This shows that these methods are more suitable for video editing rather than generation, that is, in order to control the video in the desired way, you first need to find a reference video in which the objects in the video meet the requirements of the motion trajectory of your text prompt, or the background changes meet your requirements. However, based on the actual situation, it is impossible to obtain an infinitely large video database and a sufficiently good or efficient retrieval module so that such videos can be found for all text prompts. Therefore, the previous controllable generation cannot solve such problems.

[0025] The current methods are completely data-driven and cannot meet the controllable generation requirements for scene changes. They can only manually select videos that are more in line with expectations through a large number of attempts. In addition, most of the current controllable generation methods rely on continuous and complex control information. Therefore, before generating the required video, it is necessary to first retrieve a video with a similar transformation form as a reference video. This means that compared to video generation, it is more similar to video editing, which cannot fully meet the user's unrestrained generation needs and limits the creative ability of the Wensheng video model.

[0026] In view of the above problems, the embodiments of the present application propose a scene change controllable Vincent video method, system and device, which adapts to the change requirements of the motion trajectory of different objects by utilizing the provided layout information, and also introduces multiple methods to maintain the consistency of the generated video. In conjunction with the accompanying drawings, the scene change controllable Vincent video method, system and device provided by the embodiments of the present application are described in detail through some embodiments and their application scenarios.

[0027] The first aspect of the embodiment of the present application proposes a scene change controllable Vincent video method, referring to Figure 1 , Figure 1 A flowchart of the steps of a scene change controllable Vincent video method is shown, such as Figure 1 As shown, the method includes: Step S101, inputting a description text for describing the motion trajectory of a moving subject into a video generator based on a spatial layout fuser to generate a video clip, wherein the video generator based on a spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross-attention module of each layer of a U-net network U-net, wherein the spatial layout fuser is a text-to-image generation model; the description text includes at least: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene.

[0028] Specifically, the description text used to describe the motion trajectory of the moving subject at least includes: the moving subject and the motion trajectory of the moving subject moving from the first scene to the second scene. For example, at the beginning, the puppy (moving subject) stands on the grass, and most of the background of the entire video is grass (first scene). As time changes, the puppy begins to move in a certain direction (motion trajectory), and wheat fields gradually appear at the edge of the video. At the end of the video, most of the background completely turns into wheat fields (second scene), and the puppy also runs into the wheat fields.

[0029] Reference Figure 2 , Figure 2 A schematic diagram of the architecture of a video generator based on a spatial layout amalgamator is shown. Figure 2As shown in the figure, the video generator consists of three parts: a spatial layout fusion, a U-net network, and an action extraction module. The spatial layout fusion is a text-to-image generation model, which is equivalent to using the tag-based mechanism and gated self-attention of GLIGEN (text-to-image generation model) as a spatial layout fusion. The action extraction module is used to extract dynamic information. A U-Net with temporal attention and dynamic information extraction capability (such as the text-to-video generation model AnimateDiff) is used as the overall model architecture of the video generator, as shown in the figure. Figure 2 As shown, a spatial layout fuser is added before each cross-attention module of Unet, and an action extraction module is added at the end of each layer of Block of Unet. The action extraction module can use the pre-trained weights of AnimateDiff (an open source text-to-video generation model). In this embodiment, in order to control the trajectory of objects in each frame of the video, this embodiment uses GLIGEN's tag-based mechanism and gated self-attention as a spatial layout fuser and extends it to generate videos.

[0030] Step S102: inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network through a cross-segment scheduling mechanism module.

[0031] The video generator generates video clips one by one according to the input text description (layout information, etc.), and then combines multiple video clips into a continuous long video. In order to ensure the content continuity and semantic coherence between two adjacent video clips, the embodiment of the present application uses a cross-segment scheduling mechanism module to obtain multiple video frames at the end of the previous video clip, and inputs them into the frequency mixing control branch network (the frequency mixing control branch network is described in detail in Section 1.2 below) to serve as reference information to assist in the generation of the current video clip. That is, when generating the 6th video clip (i.e., the current video clip), the last multiple video frames of the 5th video clip (i.e., the previous video clip) are obtained through the cross-segment scheduling mechanism module.

[0032] Step S103, using the frequency mixing control branch network, using an autoregressive method, frequency information is mixed according to multiple video frames at the end of the previous video clip to generate a continuous long video, where the long video is a video that conforms to the description text.

[0033] The video generator using the trained spatial layout aggregator can generate scene-controllable video clips based on the input text description. However, the scene change is a dynamic process, so it is difficult to be well displayed in several frames of video (shorter video clips), that is, it is necessary to generate a sufficiently long video (long video) to fully show that the moving subject moves from the first scene to the second scene according to a certain motion trajectory. Therefore, the embodiment of the present application introduces a control branch network, which uses an autoregressive method to generate several consecutive video clips, which is equivalent to generating the next video clip (i.e., the current video clip) based on the last several frames of the previous video clip, and finally splicing them together to form a final longer video, i.e., a continuous long video (which can be a video with a length of more than 10s), and the long video is a video that conforms to the description text, showing a dynamic scene change process in which a moving subject moves from the first scene to the second scene according to a certain motion trajectory.

[0034] The embodiment of the present application proposes a scene change controllable Vincent video method, which mainly includes three important parts: Section 1.1 video generator based on spatial layout fusion, Section 1.2 control branch network of frequency mixing and Section 1.3 cross-segment scheduling mechanism. Below, the scene change controllable Vincent video method of the first aspect is introduced through Sections 1.1-1.3 respectively.

[0035] 1.1 Video generator based on spatial layout fuser: This embodiment proposes that if the GLIGEN module (i.e., spatial layout fuser) is directly applied to the video generator, the following problem will be encountered: due to the lack of suitable training data (video data of dynamic scene changes), after adding the descriptive text (layout information), even if a relatively satisfactory result is obtained in the first frame, the change is not obvious in the subsequent frames and still cannot perfectly meet the requirements.

[0036] In order to solve this problem, an embodiment of the present application proposes a method for obtaining a first training data set, which is used to obtain a first training data set used to train the video generator based on the spatial layout fuser.

[0037] In a possible implementation manner, the first training data set used for training the video generator based on the spatial layout fuser is obtained according to the following steps: Step S201, obtaining a training image and layout information of the training image; the training image includes: a first sample scene, a second sample scene, and a target subject.

[0038] Specifically, a set of common scenes and moving subjects are collected and randomly paired to generate some training images (which may be images generated using the GLIGEN module (i.e., spatial layout fusion)) with different scenes (first sample scene and second sample scene) on the left and right sides, and each training image also includes a target subject. For each training image, corresponding layout information can be obtained, and the layout information includes: text description information for characterizing each scene in the training image, the location of each scene, the target subject, and the location of the target subject. Exemplarily, the layout information can be a pair of text and numbers. For example, when you want to generate a video of a person riding a horse, you need to obtain a total of 16 frames of images. Then the corresponding layout information is 16 dicts, each dict is similar to {'horse': [0.0,0.5, 0.35, 0.75], 'people': [0.15, 0.30, 0.25, 0.60]}; then [x1, y1, x2, y2] represents the position of the subject, [x1, y1] is the coordinate of the upper left corner, and [x2, y2] is the coordinate of the lower right corner.

[0039] Step S202: cropping the training image according to the moving trajectory of the target subject using a sliding window to obtain a plurality of sample video frames.

[0040] In this embodiment, a bounding box (i.e., a sliding window) is used as a condition. The prompt used is "a subject is doing something in a beautiful scene between scene A and scene B, masterpiece, 8K, 4K, high quality, best quality". After that, a direction is selected (as the direction of the target subject's motion trajectory), and the image is cropped using the sliding window as a video frame, that is, each time the sliding window moves on the training image, a sample video frame is obtained. Thus, by moving the sliding window, multiple sample video frames are obtained.

[0041] Step S203: Generate a simulated video according to the plurality of sample video frames.

[0042] Step S204: generating the first training data set according to the training image, the layout information and the simulated video.

[0043] In the first training data set, each training data includes: a training image, layout information corresponding to the training image, and a simulated video generated using the training image.

[0044] It should be noted that the generation process of the first training data set mentioned in the above steps S201-S204 is relatively simple, so the generated simulation data mainly contains translational motion, lacks dynamic elements, and has deficiencies in data quality. However, the goal of the embodiment of the present application is to capture the knowledge of scene transitions without the need to fully accept these data, so the impact of low-quality data can be mitigated by one or more of the following methods: In a possible implementation, the generation process of the first training data set meets the following conditions: Condition 1: marking the training data by adding a first suffix word to the prompt word; in the application stage, adding a second suffix word to the prompt word, wherein the first suffix word includes the second suffix word.

[0045] Specifically, rather than simply using "A subject is doing something, the scene is moving from scene A to scene B" as the prompt text, the embodiment of the present application tags the simulation data (i.e., the training data in the first training data set) by adding a first suffix "moving scenes, low quality, static subject" to the prompt word, wherein the first suffix includes the second suffix "moving scenes". In the actual application stage after the video generator is trained (i.e., in the inference stage), it is only necessary to describe the scene that you want to change, and only add the second suffix "moving scenes" to the prompt word.

[0046] Condition 2: The noise of each training data is disturbed according to the preset hyperparameters.

[0047] Specifically, since the simulated video in the generated first training data set is of low quality and has deviations, the embodiment of the present application does not strictly predict the noise Therefore, the present embodiment proposes a preset hyperparameter , the preset hyperparameter is a parameter value pre-set based on experience to introduce slight interference, that is, .in, is the additional noise sampled from a Gaussian distribution with the same shape as ; according to the general optimization idea, the output noise is predicted by the video generator , which corresponds to the (sampled) noise in the simulated video in the input training data The closer the better, however, considering the low quality of analog video, the embodiment of the present application does not strictly predict the noise , using preset hyperparameters Interference is performed to predict the noise output of the video generator The noise after interference The video generator is trained with the same optimization goal.

[0048] Condition three: adding a manually annotated video segmentation dataset to the first training dataset.

[0049] Specifically, in order to achieve better performance and better training results, the embodiment of the present application mixes the first training data set with a small amount of manually annotated video segmentation data set (including video clips obtained by real shooting, which are also videos of the moving subject moving from scene A to scene B along a certain motion trajectory, and each video clip carries manually annotated layout information, which is used to describe the video content, including the process of scene change, etc.).

[0050] In a possible implementation, the optimization goal of the video generator based on the spatial layout fuser is: ; in, is the representation of the text prompt of the F frame video, is the noise of sampling F frames of video, is the noisy latent variable of F frames of video at time step t, is the post-interference noise, is the predicted noise output by the video generator, It is the representation of the text prompt of the encoded F frame video. is the layout information and t is the time step.

[0051] As shown in the above loss function, in the process of training the video generator based on the first training data set, in each iteration round, the predicted noise output by the video generator is , and the noise after interference (i.e., the noise obtained by slightly interfering with the noise obtained by the initial sampling corresponding to the input analog video) is getting closer and closer to the optimization goal, and the spatial layout fusion module and action extraction module in the video generator are updated.

[0052] In a possible implementation, the method further includes: In the process of generating video clips according to the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the latent variables of each video frame into the spatial layout fuser.

[0053] Specifically, in the process of generating a video clip by the video generator (i.e., step S101), before the latent variable is input into the spatial layout fuser, the embodiment of the present application changes the shape of the latent variable to combine the time dimension of the frame with the batch size. This change process can fuse the dynamically changing layout information into the corresponding video frame, thereby allowing the position of the subject and background in the video to be controlled.

[0054] Section 1.2 Frequency Mixing Control Branch Network: Directly using the control branch network will cause discontinuity in the video segments, which can be specifically divided into two types of discontinuities: 1. Inconsistency within the video segments, that is, the inconsistency between the two parts (the first half of the video segment) containing control information and the second half of the video segment) not containing control information; 2. Inconsistency between video segments, that is, the inconsistency of the overlapping parts of different video segments (the previous video segment and the adjacent next video segment).

[0055] For the first case, the inconsistency in the video clips may be due to the existence of too large an information gap in the reference information provided to the control branch network. Because only the content of the first half of the current video clip is known, the second half can only be completed by zero padding and other methods. This results in that among the reference information (i.e., control information) received by the control branch network, only the first half has prior knowledge, while the second half is completely chaotic, which means that the control information received by the control network needs to be adjusted and planned. This embodiment proposes to use a frequency-mixed control branch network to mix the high and low frequency information of the previous video clip into the next video clip, thereby achieving content continuity of the video clip.

[0056] In a possible implementation, the frequency mixing control branch network includes: a frequency mixing module and a first encoding module; step S103, using the frequency mixing control branch network, using an autoregressive method, mixing frequency information according to multiple video frames at the end of the previous video segment to generate a continuous long video, including: Step S1031, inputting the sampled first noise into the frequency mixing module of the frequency mixing control branch network, wherein the first noise is used to generate the current video segment; Specifically, refer to Figure 3 , Figure 3 A schematic diagram of the generation process of a video clip is shown, such as Figure 3 As shown, the input of the control branch network includes: a first noise obtained by random sampling (the noise sampled to generate the current video segment); reference information (a plurality of video frames at the end of the previous video segment, including high-frequency information and low-frequency information of each frame); and mask information mask.

[0057] Step S1032, through the frequency mixing module, according to the mask information, the initialization hidden variables of the current video clip are spliced ​​with the low-frequency information of the reference information, and the high-frequency information and low-frequency information of the reference information are mixed to obtain a processing result; the reference information is a plurality of video frames at the end of the previous video clip.

[0058] Specifically, for the diffusion model, the high-frequency information in the latent variables of each video frame usually represents dynamic, timing, and motion information, which is more microscopic and specific; the low-frequency information emphasizes static content, layout, and color tone, which is more macroscopic. Video is a continuous process. In comparison, the low-frequency information between adjacent frames is closer, while the high-frequency information is different.

[0059] Based on this, the embodiment of the present application uses a frequency mixing module to mix the high-frequency information and low-frequency information of the last multiple video frames of the previous video clip with the information of the current video clip. Correspondingly, the low-frequency information of the last frame of the previous video clip is selected for padding, and the high-frequency information of the noisy latent variables at each time step is used for splicing to estimate the knowledge content that may be contained in the second half of the reference.

[0060] Among them, the mask information is used to prompt the frequency mixing module which low-frequency information in the last video frame of the previous video clip needs to be extracted and spliced ​​into the current video clip; and to prompt which high-frequency information in multiple video frames (reference information) at the end of the previous video clip needs to be extracted and mixed into the low-frequency information of the video frames of the current video clip.

[0061] The frequency mixing module initializes the hidden variables of the current video clip according to the mask information and splices the low-frequency information of the reference information. Specifically, for the video, each frame is relatively continuous, that is, the change of the overall layout between adjacent frames will not be too large. In this embodiment, the low-frequency information of the last several frames of the previous video clip is taken out, and the low-frequency information represents the overall layout (including the subject information, position information, etc.), and is spliced ​​into the first few video frames of the next video clip to improve the continuity of the video clip. In this embodiment, when multiple video frames at the end of the previous video clip can be selected as reference information, the more video frames are selected, the better the final video continuity effect is, but the video generation efficiency is also lower. Considering the balance between efficiency and effect, in this embodiment, the last 2 to 4 frames of the previous video clip can be taken as reference information.

[0062] In addition, after the frequency information is mixed, a refiner module (Mixed FrequencyRefiner) can be used to transform the mixed processing results.

[0063] Step S1033: encode the processing result through the first encoding module to obtain an encoding result.

[0064] Step S1034: input the encoding result into the decoding module of the video generator to generate the current video segment.

[0065] The encoding module of the trained video generator is used as the first encoding module before the frequency mixing control branch network is trained. Figure 3 As shown, after the encoding is completed using the first encoding module, the encoding result is inserted into the corresponding layer of the decoding module of the video generator for decoding, and finally the next video segment, that is, the current video segment, is generated.

[0066] Step S1035, obtaining the long video according to the multiple video clips generated continuously.

[0067] In a possible implementation, the method further includes: A second training data set for training the control branch network of the frequency mixing is obtained, wherein the second training data set is composed of a plurality of sample videos, each sample video corresponds to a sample text description and a sample layout information, wherein the sample text description includes: a sample moving subject and corresponding scene information; and the sample layout information includes: a motion trajectory of the sample moving subject moving from a third sample scene to a fourth sample scene. In this embodiment, the second training data set can be the first training data set in Section 1.1, or can be generated according to the method of steps S201-S204 (the method of generating the first training data set).

[0068] After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before the frequency mixing control branch network is trained, and the frequency mixing control branch network is trained on the second training data set.

[0069] Specifically, in each training iteration process (similar to the above steps S102, S1031-1035), the training data is first input into the video generator to obtain the first video clip, and then the multiple video frames at the end of the first video clip, the mask information, and the noise obtained by sampling are input into the frequency mixing module of the frequency mixing control branch network. Through the frequency mixing module, according to the mask information, the initialization hidden variables of the video clip to be generated currently are spliced ​​with the low-frequency information of the last several frames of the first video clip, and the high-frequency information and low-frequency information of the reference information (multiple video frames at the end of the first video clip) are mixed to obtain the sample processing result. The sample processing result is encoded by the first encoding module to obtain the sample encoding result. The sample encoding result is inserted into the corresponding layer of the decoding module of the video generator to generate the next video clip. By repeating the above steps, multiple video clips are continuously generated to reconstruct the video in the training data.

[0070] 1.3 Cross-fragment scheduling mechanism.

[0071] Regarding the inconsistency between video clips, the embodiments of the present application consider planning the latent variables of different video clips, which mainly includes two aspects: on the one hand, when initializing the latent variables of the clip, the initialization latent variables of the previous video clip are used to mix the high and low frequency information (implemented by the frequency mixing module in the control branch network of frequency mixing in Section 1.2); on the other hand, the latent variables of the previous video clip are partially reused (implemented by the cross-segment scheduling mechanism module in this section).

[0072] In a possible implementation, the method further includes: The cross-segment scheduling mechanism module uses the video information of multiple video frames at the end of a previous video segment generated by the video generator to replace the video information of multiple video frames at the beginning of a next video segment.

[0073] Specifically, the cross-fragment scheduling mechanism module is used to execute the cross-fragment scheduling mechanism, refer to Figure 4 , Figure 4 A schematic diagram of the execution flow of a cross-fragment scheduling mechanism is shown, such as Figure 4 As shown in the figure, the mechanism means that when generating the next video segment, the information of the previous video segment generated is taken out and used to replace a part of the currently generated video segment (i.e., the next video segment), so that the content of the first few frames of the video segment generated in this way is completely consistent with the content of the last video frame of the previous video segment. For example, a total of 8 frames are generated for the previous video segment, and then the last four frames of the segment are taken out as reference information. To replace it with the first four frames of the next video segment, it is only necessary to continue to generate the content of the last four frames.

[0074] The embodiment of the present application combines the spatial layout fuser with Unet to generate a video generator based on the spatial layout fuser, introduces layout knowledge into the Wensheng video task to improve the performance of the video generator, and can more accurately control different objects. In addition, the embodiment of the present application also uses a frequency mixed control branch network, adopts an autoregressive method, and uses the video information of multiple video frames at the end of the previous video clip (such as high-frequency information and low-frequency information of each frame) to mix frequency information to generate the next video clip that is continuous with the previous video clip, so that multiple video clips can be combined to obtain a continuous long video, which can better meet the needs of dynamic scene changes while ensuring the continuity and quality of the generated video, thereby achieving controllable scene changes.

[0075] The second aspect of the embodiment of the present application further provides a scene change controllable Vincent video system, which is applied to the Vincent video method described in the first aspect, referring to Figure 5 , Figure 5 A schematic diagram of the structure of a Vincent video system is shown in FIG. Figure 5 As shown, the system comprises: A video generator based on a spatial layout fuser is used to generate a video clip according to a description text used to describe the motion trajectory of a moving subject, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text at least includes: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; A cross-segment scheduling mechanism module, used for inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network; The frequency mixing control branch network is used to mix frequency information in an autoregressive manner according to multiple video frames at the end of the previous video clip to generate a continuous long video, where the long video is a video that conforms to the description text.

[0076] In a possible implementation manner, the first training data set used for training the video generator based on the spatial layout fuser is obtained according to the following steps: Acquire a training image and layout information of the training image; the training image includes: a first sample scene and a second sample scene, and a target subject; Using a sliding window to crop the training image according to the moving trajectory of the target subject to obtain a plurality of sample video frames; generating a simulated video according to the plurality of sample video frames; The first training data set is generated according to the training image, the layout information and the simulation video.

[0077] In a possible implementation, the generation process of the first training data set meets the following conditions: The training data is labeled by adding the first suffix word to the prompt word; According to the preset hyperparameters, the noise of each training data is disturbed; A manually annotated video segmentation dataset is added to the first training dataset.

[0078] In a possible implementation, the optimization goal of the video generator based on the spatial layout fuser is: ; in, is the representation of the text prompt of the F frame video, is the noise of sampling F frames of video, is the noisy latent variable of F frames of video at time step t, is the post-interference noise, is the predicted noise output by the video generator, It is the representation of the text prompt of the encoded F frame video. is the layout information and t is the time step.

[0079] In a possible implementation, the frequency mixing control branch network includes: a frequency mixing module and a first encoding module; through the frequency mixing control branch network, frequency information is mixed according to multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, including: Inputting the sampled first noise into the frequency mixing module of the frequency mixing control branch network, wherein the first noise is used to generate the current video segment; The frequency mixing module initializes the hidden variables of the current video clip according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency information and the low-frequency information of the reference information to obtain a processing result; the reference information is a plurality of video frames at the end of the previous video clip; Encoding the processing result by the first encoding module to obtain an encoding result; Inputting the encoding result into a decoding module of the video generator to generate the current video segment; The long video is obtained according to a plurality of continuously generated video clips.

[0080] In a possible implementation, the device further includes: A training data generation module is used to obtain a second training data set for training the control branch network of the frequency mixing, wherein the second training data set is composed of a plurality of sample videos, each sample video corresponds to a sample text description and a sample layout information, wherein the sample text description includes: a sample motion subject and corresponding scene information; the sample layout information includes: a motion trajectory of the sample motion subject moving from a third sample scene to a fourth sample scene; A training module is used to, after the video generator is trained, use the encoding module of the trained video generator as the first encoding module before the training of the frequency mixing control branch network, and train the frequency mixing control branch network on the second training data set.

[0081] In a possible implementation, the cross-segment scheduling mechanism module is further used to replace video information of multiple video frames at the beginning of a next video segment with video information of multiple video frames at the end of a previous video segment generated by the video generator.

[0082] In a possible implementation, the video generator is further used to, in the process of generating video clips according to the description text, combine the time dimension of each video frame with the batch size before inputting the latent variables of each video frame into the spatial layout fuser.

[0083] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application. Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be executed on the processor 120 to implement the steps of the scene change controllable Vincent video method described in the first aspect of the embodiment of the present application.

[0084] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the scene change controllable Vincent video method described in the first aspect of the embodiment of the present application are implemented.

[0085] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the scene change controllable Vincent video method as described in the first aspect of the embodiment of the present application.

[0086] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0087] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0091] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0092] The above is a detailed introduction to the scene change controllable Vincent video method, system and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A scene change controllable Vincent video method, characterized in that: The method comprises: Inputting a description text for describing the motion trajectory of the moving subject into a video generator based on a spatial layout fuser to generate a video clip, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text at least includes: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; Inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network through a cross-segment scheduling mechanism module; The frequency mixing control branch network adopts an autoregressive method to mix frequency information according to multiple video frames at the end of the previous video clip to generate a continuous long video, and the long video is a video that conforms to the description text.

2. The scene change controllable Vincent video method according to claim 1, characterized in that: The first training data set used for training the video generator based on the spatial layout fuser is obtained according to the following steps: Acquire a training image and layout information of the training image; the training image includes: a first sample scene and a second sample scene, and a target subject; Using a sliding window to crop the training image according to the moving trajectory of the target subject to obtain a plurality of sample video frames; generating a simulated video according to the plurality of sample video frames; The first training data set is generated according to the training image, the layout information and the simulation video.

3. The scene change controllable Vincent video method according to claim 2, characterized in that: The generation process of the first training data set meets the following conditions: The training data is labeled by adding the first suffix word to the prompt word; According to the preset hyperparameters, the noise of each training data is disturbed; A manually annotated video segmentation dataset is added to the first training dataset.

4. The scene change controllable Vincent video method according to claim 1, characterized in that: The optimization goal of the video generator based on the spatial layout fuser is: ； in, is the representation of the text prompt of the F frame video, is the noise of sampling F frames of video, is the noisy latent variable of F frames of video at time step t, is the post-interference noise, is the predicted noise output by the video generator, It is the representation of the text prompt of the encoded F frame video. is the layout information and t is the time step.

5. The scene change controllable Vincent video method according to claim 1, characterized in that: The frequency mixing control branch network includes: a frequency mixing module and a first encoding module; through the frequency mixing control branch network, frequency information is mixed according to multiple video frames at the end of the previous video segment in an autoregressive manner to generate a continuous long video, including: Inputting the sampled first noise into the frequency mixing module of the frequency mixing control branch network, wherein the first noise is used to generate the current video segment; The frequency mixing module initializes the hidden variables of the current video segment according to the mask information, splices the low-frequency information of the reference information, and mixes the high-frequency information and the low-frequency information of the reference information to obtain a processing result; the reference information is a plurality of video frames at the end of the previous video segment; Encoding the processing result by the first encoding module to obtain an encoding result; Inputting the encoding result into a decoding module of the video generator to generate the current video segment; The long video is obtained according to a plurality of continuously generated video clips.

6. The scene change controllable Vincent video method according to claim 5, characterized in that: The method further comprises: Acquire a second training data set for training the control branch network of the frequency mixing, wherein the second training data set is composed of a plurality of sample videos, each sample video corresponds to a sample text description and a sample layout information, wherein the sample text description includes: a sample moving subject and corresponding scene information; the sample layout information includes: a motion trajectory of the sample moving subject moving from a third sample scene to a fourth sample scene; After the video generator is trained, the encoding module of the trained video generator is used as the first encoding module before the frequency mixing control branch network is trained, and the frequency mixing control branch network is trained on the second training data set.

7. The scene change controllable Vincent video method according to claim 1, characterized in that: The method further comprises: The cross-segment scheduling mechanism module uses the video information of multiple video frames at the end of a previous video segment generated by the video generator to replace the video information of multiple video frames at the beginning of a next video segment.

8. The scene change controllable Vincent video method according to claim 1, characterized in that: The method further comprises: In the process of generating video clips according to the description text, the video generator combines the time dimension of each video frame with the batch size before inputting the latent variables of each video frame into the spatial layout fuser.

9. A scene-change controllable Vincent video system, characterized in that: Applied to perform the Vincent video method according to any one of claims 1 to 8, the system comprising: A video generator based on a spatial layout fuser is used to generate a video clip according to a description text used to describe the motion trajectory of a moving subject, wherein the video generator based on the spatial layout fuser is obtained by adding a spatial layout fuser at the beginning of a cross attention module of each layer of a U-net network, wherein the spatial layout fuser is a text-to-image generation model; the description text at least includes: the moving subject and the motion trajectory of the moving subject moving from a first scene to a second scene; A cross-segment scheduling mechanism module, used for inputting a plurality of video frames at the end of a previous video segment generated by the video generator into a frequency mixing control branch network; The frequency mixing control branch network is used to mix frequency information in an autoregressive manner according to multiple video frames at the end of the previous video clip to generate a continuous long video, where the long video is a video that conforms to the description text.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the steps of the scene change controllable Vincent video method as described in any one of claims 1 to 8 are implemented.

Images