Long video generation method and system based on dynamic global-local memory mechanism

Abstract

The application relates to the technical field of computer vision, and discloses a long video generation method and system based on a dynamic global-local memory mechanism. The method comprises the following steps: acquiring the features of a reference image, a line drawing sequence and text; fusing visual features and text features through a mixed feature extractor to obtain mixed multi-modal features; extracting global memory representation of a generated historical video segment and local context features of a previous video segment, and performing cross-attention fusion on the mixed multi-modal features; inputting the obtained global visual features into a video generation model in a skip-layer injection mode, or inputting the mixed multi-modal features into the video generation model in a skip-layer injection mode to generate a video segment; and in a post-stage set in a denoising process, performing latent space fusion on an overlapping area of adjacent video segments. The application improves the control precision in the line drawing coloring process, and effectively guarantees the color attribute consistency of various elements in the video in the time dimension.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for generating long videos based on a dynamic global local memory mechanism. Background Technology

[0002] In recent years, video generation models have continued to evolve, demonstrating significant progress in cross-frame consistency modeling and semantic understanding, thereby driving the technological development of multiple downstream tasks. Controllable long video generation, as an important direction in this field, introduces explicit control conditions (such as actions, styles, or sketches) to achieve fine-grained control over long video content. While ensuring that the generated content does not deviate from the intended semantics, it transforms the creator's design intent into high-quality video effects, greatly expanding the flexibility of long video content creation, meeting the personalized needs of various application scenarios such as film, animation, and virtual reality, and promoting the intelligent production of digital media content.

[0003] Line art coloring is a crucial part of the animation and film production process, and it is highly labor-intensive. Traditional methods typically rely on a large number of people to complete it frame by frame, which is inefficient and costly. To improve production efficiency and reduce production costs, more and more research focuses on automating line art coloring using video generation models. This task usually uses the coloring of the first frame as a reference to guide the model to stably and consistently propagate color information to subsequent frames, ensuring the color continuity of each element in the video over time. As a typical application of controllable video generation, line art coloring not only places high demands on control precision and temporal consistency, but also fully demonstrates the practical value and industrial significance of controllable video generation in actual production processes.

[0004] However, existing methods primarily focus on maintaining color consistency across elements in a short timeframe (typically no more than 100 frames), and still face significant challenges when processing long-range line art videos (e.g., exceeding 300 frames). For long-range line art coloring tasks, mainstream solutions typically employ a segmented processing strategy: dividing the entire video into multiple shorter segments, using the last frame of the previous segment as the first frame of the next segment for supervision, and sequentially completing the coloring process to achieve cross-segment color information propagation. While this strategy achieves preliminary long-range color transfer to some extent, it easily leads to problems such as color drift and element color jumps in practical applications, resulting in decreased visual consistency and failing to meet the global color consistency requirements of long-range video production. Therefore, how to overcome the limitations of short-range modeling and achieve globally consistent coloring of long-range line art sequences remains one of the core problems that urgently need to be solved in this field. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a long video generation method and system based on a dynamic global-local memory mechanism for long-range line drawing coloring. This aims to improve control precision during the line drawing coloring process and effectively ensure the consistency of color attributes of various elements in the video over time. Specifically, during the coloring process of each short segment, a video understanding model extracts compressed information from all previously colorized segments as a global memory representation. Simultaneously, compressed information from the previous colorized segment is extracted as a local color context. The global and local information are dynamically and jointly injected into the video generation model of the current segment, thereby providing structured support for maintaining stable color attributes of various elements in the video over a long time span.

[0006] Furthermore, the global-local dynamic memory mechanism proposed in this invention is not only applicable to long-range line drawing coloring tasks but also possesses good versatility, extending to other controllable video generation tasks. For example, in style transfer tasks, this mechanism can effectively extract and fuse global style features with local inter-frame consistency information, achieving style-unified and temporally coherent video generation; in subject-action driven tasks, it can achieve more stable and natural motion generation by dynamically modeling historical action sequences and current action context. This mechanism provides an efficient structured information modeling paradigm for controllable video generation, helping to enhance the cross-frame consistency of elements in long-range videos, and has broad application potential and industrial value.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a long video generation method based on a dynamic global local memory mechanism, comprising:

[0009] Acquire features from the reference image, line drawing sequence, and text, and then concatenate the reference image features and line drawing sequence features into visual features;

[0010] The text features and the video latent features initialized as random noise are concatenated along the dimension and then input into the video generation model.

[0011] By fusing visual and textual features through a hybrid feature extractor, hybrid multimodal features are obtained. When the video generation model generates the first video segment, the hybrid multimodal features are input into the video generation model in a skip-layer injection manner to generate the video segment.

[0012] When the video generation model generates a video segment that is not the first one, it extracts the global memory representation of the previously generated historical video segments and the local context features of the previous video segment through a dynamic global local memory mechanism, and performs cross-attention fusion with the hybrid multimodal features. The resulting global visual features are then input into the video generation model in a skip-layer injection manner to generate video segments.

[0013] The video generation model generates video segments by denoising the initial latent features of the video. In the later stage of the denoising process, the overlapping video frames of adjacent video segments are latently fused to make the color of the output video segment sequence consistent.

[0014] In one embodiment, the step of acquiring features of the reference image, line drawing sequence, and text, and concatenating the reference image features and line drawing sequence features into visual features, specifically includes:

[0015] The line art sequence and reference image are encoded into line art sequence features and reference image features respectively using a 3D VAE encoder; the text is encoded into text features using a T5 text encoder.

[0016] The visual features are obtained by concatenating the features of the line drawing sequence with the features of the reference image along the feature channel dimension.

[0017] In one embodiment, the step of fusing visual and textual features through a hybrid feature extractor to obtain hybrid multimodal features specifically includes:

[0018] Visual features and text features are concatenated along the sequence dimension and input into a hybrid feature extractor to obtain hybrid multimodal features;

[0019] The hybrid feature extractor consists of L basic modules, each implemented through an attention mechanism and a fully connected layer.

[0020] In one embodiment, when the first video segment is generated, the hybrid multimodal features are input into the video generation model via a skip-layer injection method, specifically including:

[0021] The video generation model consists of N basic modules, each implemented through an attention mechanism and a fully connected layer. The hybrid multimodal features extracted by the hybrid feature extractor are added into the video generation model, as formally expressed below:

[0022] ;

[0023] in, These are the visual features obtained after passing through the attention mechanism of the nth basic module of the video generation model. This represents the visual features resulting from the fusion of the visual features output by the basic module and the hybrid multimodal features. It is a weighting factor. It represents a hybrid multimodal feature.

[0024] In one embodiment, when the video generation model generates a non-first video segment, the global memory representation of the previously generated historical video segments and the local context features of the previous video segment are extracted through a dynamic global-local memory mechanism, and then fused with the hybrid multimodal features across attention. The resulting global visual features are then input into the video generation model via a skip-layer injection method, specifically including:

[0025] The previously generated video segment is defined as a local video. All historically generated video clips are defined as global videos. Global video and partial video The data is input into a long video understanding model, where M-layer visual labels are extracted and stored in a key-value cache. The set of keys for the M-layer visual labels is denoted as... The set of values ​​is denoted as ,in, This indicates the number of visual label layers extracted from the long video understanding model; Represents the global video's first... The key of layer visual markers, The first part of the local video represents the... The key of layer visual markers, Represents the global video's first... The values ​​of the layer visual markers, The first part of the local video represents the... The values ​​of the layer visual markers, Recorded as the first video in the global video. Layered key-value cache features Recorded as the first partial video The key-value pairs are cached with features; then the keys and values ​​of the visual labels are fed into a mapper containing two layers of multilayer perceptrons to combine with the hybrid multimodal features. Align the dimensions; Key-value caching features of global and local videos at different layers. Input to In each cross-attention layer, global visual features relevant to the current generation are adaptively extracted. :

[0026] ;

[0027] Among them, cross-attention queries The key to cross attention Cross-attention value ; , and For weight parameters, Indicates feature splicing, This represents the number of dimensions in the attention mechanism. This represents the Softmax function.

[0028] In one embodiment, the step of inputting the obtained global visual features into the video generation model via a skip-layer injection method specifically includes:

[0029] ;

[0030] Represents global visual features. These are the visual features obtained after passing through the attention mechanism of the nth basic module of the video generation model. This represents the visual features resulting from the fusion of the visual features output by the basic module and the global visual features. It is a weighting factor. This represents the number of basic modules in the video generation model.

[0031] In one embodiment, the method further includes: improving the color consistency of the generated video clips through a non-gradient reward for color consistency, specifically including:

[0032] The video clip used to train the video generation model is used as the reference video clip;

[0033] After the generated video clips are fed into the long video understanding model, the obtained key-value cache features are aligned with the key-value cache features of the corresponding reference video clips through non-gradient rewards based on color consistency.

[0034] In one embodiment, the non-gradient reward for color consistency for:

[0035] ;

[0036] in, Indicates the first reference video clip The keys and values ​​of the visual markers for the layer. For generating the first video clip The keys and values ​​of the visual markers for the layer. It is the F2 norm.

[0037] In one embodiment, the video generation model generates video segments by denoising the initialized latent features of the video. In a later stage of the denoising process, latent space fusion is performed on overlapping video frames of adjacent video segments to ensure consistent color in the output video segment sequence. Specifically, this includes:

[0038] Given two adjacent video clips and Each video clip contains The number of video frames; the number of video frames overlapping between two video segments is . Set fusion factor To control the fusion intensity; during the denoising process, when When adjacent video clips are spliced ​​together, half of each overlapping portion is taken. Indicates the current denoising step number. This represents the total number of time steps for denoising. Indicates the number of denoising steps to begin latent space fusion; when At that time, latent spatial fusion is performed on the overlapping parts of adjacent video clips:

[0039] ;

[0040] in, Representing the One overlapping video frame; Indicates the first The noise reduction process, i-th video segment The Middle The video latent features of overlapping video frames; i is the i-th video segment. index, for The corresponding latent features of the video after latent space fusion.

[0041] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0042] Compared with the prior art, the beneficial technical effects of the present invention are:

[0043] This invention proposes a long-video colorization mechanism based on a dynamic global-local memory mechanism, comprising three parts: an efficient hybrid feature extractor, a dynamic global-local memory mechanism, and a color consistency fusion strategy. First, the efficient hybrid feature extractor can fuse multimodal features from the text and image domains, enabling the video generation model to accept more diverse control conditions while further optimizing and improving its generalization ability. Furthermore, the dynamic global-local memory mechanism, based on the long-video understanding model, dynamically compresses features from historically generated videos, avoiding information redundancy and computational explosion, and adaptively fuses global and local features, significantly improving feature consistency in long-video segments. Second, the color consistency fusion strategy further ensures the fusion features of overlapping video segments, ensuring smooth transitions in overlapping areas and avoiding interference with other frames. Attached Figure Description

[0044] Figure 1This is a flowchart of the method in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the overall architecture in an embodiment of the present invention. Detailed Implementation

[0046] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] like Figure 1 As shown, a long video generation method based on a dynamic global local memory mechanism according to the present invention includes the following steps:

[0048] S1, obtain the features of the reference image, line drawing sequence and text, and concatenate the features of the reference image and the line drawing sequence into visual features;

[0049] S2, after concatenating the text features and the video latent features initialized as random noise along the dimension, inputs them into the video generation model;

[0050] S3 uses a hybrid feature extractor to fuse visual and textual features to obtain hybrid multimodal features. When the video generation model generates the first video segment, the hybrid multimodal features are input into the video generation model in a skip-layer injection manner to generate the video segment.

[0051] S4. When the video generation model generates a video segment that is not the first video segment, the global memory representation of the previously generated historical video segments and the local context features of the previous video segment are extracted through the dynamic global local memory mechanism, and then fused with the hybrid multimodal features across attention. The resulting global visual features are then input into the video generation model in a skip-layer injection manner to generate video segments.

[0052] S5, the video generation model generates video segments by denoising the initial video latent features. In the later stage set during the denoising process, the overlapping video frames of adjacent video segments are latently fused to make the color of the output video segment sequence consistent.

[0053] Controllable long video generation with mixed conditions can be guided by various factors, such as line art, skeletons, optical flow, and trajectories. The task of controllable video generation with mixed conditions can typically be formalized as follows: given a reference image... Based on the corresponding guiding conditions and text, a long video that matches these conditions is ultimately generated. In this embodiment, the guiding condition is a line drawing sequence. ,in This represents the number of frames in the final generated video.

[0054] Currently, long video generation can typically be formally represented as: taking the long video to be ultimately generated... Divided into Video clip Each video clip has Each video frame ultimately passes the constraints. The final long video was synthesized:

[0055] .

[0056] The long video colorization algorithm based on dynamic global local memory mechanism proposed in this invention is as follows: Figure 2 As shown, it includes the following three parts: (1) Hybrid feature extractor, which efficiently fuses the three features of reference image, line drawing sequence and text to generate hybrid multimodal features; (2) Dynamic global local memory mechanism, which dynamically extracts color consistency features related to the currently generated video segment from the historical videos that have already been generated; (3) Color consistency fusion strategy, which performs latent space fusion operation only in the later stage of diffusion denoising to achieve a smooth transition.

[0057] (1) Highly efficient hybrid feature extractor.

[0058] Hybrid feature extractors are designed to efficiently extract a hybrid representation of reference images, line art, and text, facilitating the implementation of dynamic global-local memory. (Line art sequence) First, by the 3D VAE encoder Encoding as line drawing sequence features The reference image is encoded into reference image features using a 3DVAE encoder. via T5 text encoder Encode text into text features Then, the features of the line art sequence are compared with the features of the reference image along the feature channel dimension. The visual features are concatenated together and then combined with the text features along the sequence dimension. These are concatenated and used as input to a hybrid feature extractor. Textual conditions can further refine video generation. During training, reference images are randomly selected from historical key video frames, rather than being fixed to the first video frame of a video segment, to enhance robustness.

[0059] These multimodal features are fed into a hybrid feature extractor to obtain hybrid multimodal features. The hybrid feature extractor aims to enhance the guiding ability of input features while minimizing its impact on the video generation model, with minimal resource consumption. The hybrid feature extractor is fed into the video generation model via a skip-layer injection approach:

[0060] ;

[0061] in, These are the visual features obtained after passing through the attention mechanism of the nth basic module of the video generation model. This indicates the number of basic modules in the video generation model. It is a weighting factor. This is the visual feature resulting from the fusion of the visual features output by the basic module and the hybrid multimodal features. During training, this invention freezes the weights of the basic video generation model to preserve its original text guidance functionality.

[0062] The hybrid feature extractor includes L basic modules.

[0063] Each basic module is implemented through an attention mechanism and fully connected layers, specifically using 3D attention. The basic module structures in the hybrid feature extractor and the video generation model are identical, differing only in the number of layers and input features. The video generation model belongs to a diffusion model.

[0064] During the generation of the first video segment, hybrid multimodal features were used. The data is directly input into the video generation model. During subsequent video segment generation, multimodal features are mixed. After fusing with the global features output by the dynamic global-local memory mechanism, the data is input into the video generation model. At the selected layer, the original visual features are combined with the global visual features containing historical generation information. The layers that are not selected will retain their original computation paths and will not be injected.

[0065] (2) Dynamic global local memory mechanism.

[0066] To extract color consistency features from historically generated video clips that are relevant to the currently generated video clip, this invention proposes a Dynamic Global Local Memory (DGLM) mechanism.

[0067] This invention employs a long-video understanding model to extract historical features. After estimating visual feature changes between adjacent video frames using the CLIP model, video segments of different lengths (e.g., 2, 4, and 8 frames) are dynamically selected. These video segments are then fed into a multimodal large language model (MLLM), which autoregressively generates a visual tag for each video segment and unloads the original frame features. To improve computational efficiency, the keys and values ​​of the visual tags at each layer are stored in a key-value cache. This memorization method extracts relevant features from historical video segments as much as possible, reduces the impact of redundant information, and improves the model's performance on controllable video generation tasks.

[0068] Define the most recently generated video segment as a local video. All historically generated video clips are defined as global videos. These video segments are taken into account to achieve better temporal consistency. Specifically, the global video... and partial video It is fed into a long video understanding model to extract the key of visual tags. Sum It is stored in the KV cache, where This indicates the number of visual label layers extracted from the long video understanding model (less than the number of layers in the long video understanding model). Represents the global video's first... The key of layer visual markers, The first part of the local video represents the... The key of layer visual markers, Represents the global video's first... The values ​​of the layer visual markers, The first part of the local video represents the... The values ​​of the layer visual markers, Recorded as the first video in the global video. Layered key-value cache features Recorded as the first partial video The key-value pairs are cached with features; then the keys and values ​​of the visual labels are fed into a mapper containing two layers of multilayer perceptrons to align the mixed multimodal features. The dimensions. Then, the extracted... Layered key-value cache features and hybrid multimodal features Enter in sequence In each cross-attention layer, global visual features related to the currently generated video segment are adaptively extracted, as formalized as follows:

[0069] ;

[0070] Among them, cross-attention queries The key to cross attention Cross-attention value . , and For weight parameters, Indicates feature splicing, This represents the number of dimensions in the attention mechanism. This represents the Softmax function.

[0071] In a preferred embodiment, the long video understanding model can employ the Video-XL model.

[0072] To further address the issue of fine-grained local color inconsistencies, this invention proposes a non-gradient reward for color consistency to further improve the long-term color consistency of videos. Specifically, this invention feeds generated video segments into a long-video understanding model to obtain... The layer's key-value cache features are aligned with the corresponding key-value cache features of the reference video clips, making the colors of the generated video clips closer to those of the real video. The key-value cache features are extracted from the reference video clips during training, while these features are extracted from previously generated video clips during inference. Aligning the key-value cache features of the generated video clips with those of the reference video clips further reduces the gap between inference and training. The reward function is as follows:

[0073] ;

[0074] in, Indicates the first reference video clip The keys and values ​​of the visual markers for each layer, combined to form the first reference video segment. Key-value cache characteristics of the layer; For generating the first video clip The keys and values ​​of the visual tags in the layer are combined to generate the first video clip. Key-value cache characteristics of the layer; It is the F2 norm.

[0075] In summary, DGLM achieves long-term color consistency by dynamically compressing historical features and adaptively extracting current relevant features.

[0076] (3) Color consistency integration strategy.

[0077] To further smooth the transition of overlapping video segments, this invention proposes a color consistency fusion strategy. This invention chooses to fuse latent variable features in the later stages of denoising. Using a latent fusion strategy in all denoising steps would destroy visual details (e.g., brightness). On the one hand, the early stages focus on overall layout features, while the later stages focus on visual details (e.g., fine-grained color and brightness). The overall layout features obtained from early denoising fusion are redundant for animation coloring tasks. On the other hand, current video generation models typically decode latent video features jointly, rather than decoding each frame individually as in previous models. This means that unnecessary perturbations to some latent features will affect the decoding of all latent features. Since visual detail features (e.g., color and brightness) are mainly fused using fine-grained color consistency fusion in the later denoising stage, this invention ultimately chooses to fuse latent variable features in the later stages of denoising. Integrate latent variable features.

[0078] Given two adjacent video clips and Each video clip contains The number of video frames is [number]. The number of frames that overlap between the two video segments is [number]. Set the fusion factor To control the fusion strength, Ensure that video frames near the boundary have lower weights, while inner video frames have higher weights, thus mitigating the impact on non-overlapping video segments. During denoising, when... At that time, video clips are directly spliced ​​together, with half of each overlapping portion taken. In this case, latent spatial fusion will be applied to the overlapping portions of adjacent video clips:

[0079] ;

[0080] in, Representing the Several overlapping video frames. Indicates the first The noise reduction process, i-th video segment The Middle Potential video features of overlapping video frames.

[0081] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0082] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0083] In one embodiment, the present invention also provides a computer system, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor of the computer system provides computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer system stores data used in the methods described above. The network interface of the computer system is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the methods described above.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0086] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A long video generation method based on a dynamic global local memory mechanism, characterized in that, include: Acquire features from the reference image, line drawing sequence, and text, and then concatenate the reference image features and line drawing sequence features into visual features; The text features and the video latent features initialized as random noise are concatenated along the dimension and then input into the video generation model. By fusing visual and textual features through a hybrid feature extractor, hybrid multimodal features are obtained. When the video generation model generates the first video segment, the hybrid multimodal features are input into the video generation model in a skip-layer injection manner to generate the video segment. When the video generation model generates a video segment that is not the first one, it extracts the global memory representation of the previously generated historical video segments and the local context features of the previous video segment through a dynamic global-local memory mechanism. These are then fused with the hybrid multimodal features across attention. The resulting global visual features are input into the video generation model via a skip-layer injection method to generate video segments, including defining the previous video segment as a local video. All historically generated video clips are defined as global videos. ; and The data is input into a long video understanding model, where M-layer visual labels are extracted and stored in a key-value cache. The set of keys for the M-layer visual labels is denoted as... The set of values ​​is denoted as , This indicates the number of visual label layers extracted from the long video understanding model; express The The key of layer visual markers, express The The key of layer visual markers, express The The values ​​of the layer visual markers, express The The values ​​of the layer visual markers, Recorded as The Layered key-value cache features Recorded as The Layered KV cache features; inputting the keys and values ​​of visually labeled data into a mapper containing two layers of multilayer perceptrons to combine with hybrid multimodal features. Align the dimensions; layer and KV caching features and Input to In each cross-attention layer, global visual features relevant to the current generation are adaptively extracted. : ； Cross-attention queries ,key ,value ; , and For weight parameters, Indicates feature splicing, This represents the number of dimensions in the attention mechanism. This represents the Softmax function; The video generation model generates video segments by denoising the initial video latent features. In the later stage of the denoising process, the overlapping video frames of adjacent video segments are latently fused to make the output video segment sequence have consistent colors. This also includes improving the color consistency of generated video clips through non-gradient rewards for color consistency: using video clips used to train the video generation model as reference video clips; and feeding the generated video clips into the long video understanding model, then applying non-gradient rewards for color consistency. The obtained KV cache features are aligned with the KV cache features of the corresponding reference video segments; ； in, Indicates the first reference video clip The keys and values ​​of the visual markers for the layer. For generating the first video clip The keys and values ​​of the visual markers for the layer. It is the F2 norm.

2. The long video generation method based on a dynamic global local memory mechanism according to claim 1, characterized in that, The process of acquiring features from the reference image, line drawing sequence, and text, and concatenating the reference image features and line drawing sequence features into visual features, specifically includes: The line art sequence and reference image are encoded into line art sequence features and reference image features respectively using a 3D VAE encoder; the text is encoded into text features using a T5 text encoder. The visual features are obtained by concatenating the features of the line drawing sequence with the features of the reference image along the feature channel dimension.

3. The long video generation method based on a dynamic global local memory mechanism according to claim 1, characterized in that, The process of fusing visual and textual features using a hybrid feature extractor to obtain hybrid multimodal features specifically includes: Visual features and text features are concatenated along the sequence dimension and input into a hybrid feature extractor to obtain hybrid multimodal features; The hybrid feature extractor consists of L basic modules, each implemented through an attention mechanism and a fully connected layer.

4. The long video generation method based on a dynamic global local memory mechanism according to claim 1, characterized in that, When generating the first video segment, the hybrid multimodal features are input into the video generation model via a skip-layer injection method, specifically including: The video generation model consists of N basic modules, each implemented through an attention mechanism and a fully connected layer. The hybrid multimodal features extracted by the hybrid feature extractor are added into the video generation model, as formally expressed below: ； in, These are the visual features obtained after passing through the attention mechanism of the nth basic module of the video generation model. This represents the visual features resulting from the fusion of the visual features output by the basic module and the hybrid multimodal features. It is a weighting factor. It represents a hybrid multimodal feature.

5. The long video generation method based on a dynamic global local memory mechanism according to claim 1, characterized in that, The process of inputting the obtained global visual features into the video generation model via a skip-layer injection method specifically includes: ； Represents global visual features. These are the visual features obtained after passing through the attention mechanism of the nth basic module of the video generation model. This represents the visual features resulting from the fusion of the visual features output by the basic module and the global visual features. It is a weighting factor. This represents the number of basic modules in the video generation model.

6. The long video generation method based on a dynamic global local memory mechanism according to claim 1, characterized in that, The video generation model generates video segments by denoising the initialized latent features of the video. In a later stage of the denoising process, latent space fusion is performed on overlapping video frames of adjacent video segments to ensure consistent color in the output video segment sequence. Specifically, this includes: Given two adjacent video clips and Each video clip contains The number of video frames; the number of video frames overlapping between two video segments is . Set fusion factor To control the fusion intensity; during the denoising process, when When adjacent video clips are spliced ​​together, half of each overlapping portion is taken. Indicates the current denoising step number. This represents the total number of time steps for denoising. Indicates the number of denoising steps to begin latent space fusion; when At that time, latent spatial fusion is performed on the overlapping parts of adjacent video clips: ； in, Representing the One overlapping video frame; Indicates the first The noise reduction process, i-th video segment The Middle The video latent features of overlapping video frames; i is the i-th video segment. index, for The corresponding latent features of the video after latent space fusion.

7. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images