Multi-modal long story consistency generation method, computer equipment and program product
By using the VastStory generation model and the StoryBank data processing pipeline, the semantic alignment and long sequence consistency issues in multimodal long story generation were resolved, achieving high-quality multi-character scene generation, adapting to the variable frame length requirements of different series, and improving the generation effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-04-03
AI Technical Summary
Existing multimodal long story generation technologies have significant bottlenecks in semantic alignment and long sequence consistency, resulting in deviations from text prompts and confusion of character features in single-frame generation. Furthermore, the training data lacks specificity and is difficult to adapt to the variable frame length requirements of different series.
The VastStory generative model is adopted, which uses multi-cue cross-attention injection blocks and self-attention and cross-attention layers, combined with StoryBank data processing pipeline for role-centered annotation and frame filtering, to achieve independent semantic interaction with one cue per frame and hierarchical bucket frame training, enhance the model's adaptability to variable frame length, and constrain semantic independence through the joint loss function of rectified stream.
It improves the quality and applicability of multimodal long story generation, ensures accurate alignment between single frames and corresponding text, maintains the continuity of identity and action in multi-character scenes, adapts to the frame length requirements of different TV series, and solves the semantic leakage and character feature confusion problems existing in traditional methods.
Smart Images
Figure CN121788656A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and deep learning, specifically to a multimodal long story consistency generation method, computer device, and program product. Background Technology
[0002] Multimodal long story generation technology achieves the creation of coherent image sequences by fusing textual and visual information, and has significant application value in fields such as film and animation, and digital content production. Its core requirements are to achieve precise alignment of text-visual semantics and consistency between frames in long sequences. The former requires image generation to strictly match the semantics of text prompts, while the latter requires ensuring the cross-frame coherence of character features, action logic, and scene environment.
[0003] Current mainstream technologies such as StoryDiffusion, while improving sequence coherence through consistent self-attention mechanisms, still have significant bottlenecks. At the semantic alignment level, traditional models often employ global cross-attention interactions, which are prone to cross-frame semantic leakage, causing single-frame generation to deviate from the corresponding text prompts. At the long sequence consistency level, existing methods either rely on fixed frame length training, making it difficult to adapt to the variable frame length requirements of different series; or, due to a lack of targeted data processing, character feature confusion and identity recognition errors easily occur in multi-character scenarios.
[0004] At the data level, existing training data mostly uses raw video frames directly without character-centered selection and refined annotation, making it difficult for models to learn stable character feature representations. In terms of training mechanisms, traditional loss functions focus on image reconstruction accuracy and do not adequately constrain semantic independence, further exacerbating the problem of inter-frame semantic interference. Summary of the Invention
[0005] The purpose of this invention is to provide a multimodal long story consistency generation method, computer device, and program product to solve the problems mentioned in the background art.
[0006] According to one aspect of this application, a method for generating consistency in multimodal long stories is provided, comprising the following steps: The process involves acquiring input data, including a text cue sequence and a visual frame sequence; inputting the input data into a pre-trained VastStory generation model, and outputting a target image sequence; wherein the VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module, wherein the input module receives the input data; the encoding module independently encodes the text cue sequence into a text token sequence and independently encodes and segments the visual frame sequence into a visual token sequence; the core generation module includes a multi-cue cross-attention injection block and self-attention and cross-attention layers, including: inputting the text token sequence and visual frame sequence into a text token sequence and outputting a target image sequence; the input module receives the input data; the encoding module independently encodes the text cue sequence into a text token sequence and independently encodes and segments the visual frame sequence into a visual token sequence; the core generation module includes a multi-cue cross-attention injection block and a self-attention and cross-attention layer, including: inputting the text token sequence and visual frame sequence into a text token sequence and outputting a target image sequence. The frame dimension of the token sequence is merged into the batch dimension token sequence, and the merged token sequence is processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input to the self-attention and cross-attention layer, and a modulated token sequence is generated under the control of the temporal modulation parameter. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
[0007] Preferably, the VastStory generation model training includes: constructing a story library data processing pipeline, extracting shot-level frames from episode-level videos, performing character recognition and annotation on the shot-level frames to obtain character-centric story-level training data; in the static training phase, training the model with a continuous frame sequence of fixed frame length to learn basic consistency; in the dynamic training phase, grouping different episodes in the story-level training data into different buckets according to their frame count, and randomly selecting data from one bucket for training in each training optimization step, so that the model gradually adapts to the variable frame length sequence generation task.
[0008] Preferably, the story library data processing pipeline includes: performing character detection on the shot-level frames and extracting character embedding vectors; clustering the character embedding vectors to identify the optimal cluster center; matching each shot-level frame to the nearest cluster center to associate it with a specific character; excluding frames containing multiple characters and retaining only frames containing a single character; generating annotation content for the retained shot-level frames and assigning a unique identifier to each identified character.
[0009] Preferably, the training objective of the model is based on the definition of rectified flow, by constructing a time-related noise image and its velocity field, and minimizing the expected mean square error between the velocity field predicted by the model and the true velocity field determined by the starting and ending images, in order to optimize the model parameters.
[0010] Preferably, the training of the model further includes semantic independence constraints, which are achieved by calculating the feature differences of the generation results of adjacent frames at the partition boundary. These semantic independence constraints are used to constrain the generation result features of different frames at the sequence boundary during model training and together with the image reconstruction loss constitute a joint loss function.
[0011] Preferably, the cross-attention calculation includes normalizing the attention weights calculated from the query tensor and the key tensor based on the Sigmoid function, and multiplying the normalized weights by the value tensor to obtain the attention output corresponding to each index.
[0012] In another aspect, this application also provides a multimodal long story consistency generation system, comprising: A data acquisition module is used to acquire input data, which includes a text prompt sequence and a visual frame sequence. A target image sequence output module is used to input the input data into a pre-trained VastStory generation model and output a target image sequence. The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module. The input module receives the input data. The encoding module independently encodes the text prompt sequence into a text token sequence and independently encodes and segments the visual frame sequence into a visual token sequence. The core generation module includes a multi-prompt cross-attention injection block and self-attention and cross-attention layers, including: The frame dimensions of the text token sequence and the visual token sequence are merged into a batch-dimensional token sequence. The merged token sequence is then processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input to the self-attention and cross-attention layer to generate a modulated token sequence under the control of the temporal modulation parameters. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
[0013] This application also provides a computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multimodal long story consistency generation method as described above.
[0014] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the multimodal long story consistency generation method as described above.
[0015] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal long story consistency generation method as described above.
[0016] This application utilizes the StoryBank data processing pipeline to achieve refined character-centric annotation and frame selection, solving the problem of character feature confusion caused by the lack of targeted data processing in traditional methods, and providing high-quality training data for the model. Simultaneously, it employs a multi-cue cross-attention injection mechanism combined with the DiT module to achieve independent semantic interaction with one cue per frame, resolving cross-frame semantic leakage and ensuring accurate alignment between a single frame and its corresponding text. Furthermore, the hierarchical bucket training strategy balances fixed and variable frame length training, strengthening the basic consistency of short sequences while supporting flexible generation of long sequences up to 32 frames, adapting to the frame length requirements of different dramas. Finally, based on a joint loss function using rectified streams, semantic independence constraints are fused, improving image reconstruction accuracy while effectively ensuring the continuity of identity and action coherence in multi-character scenes, significantly enhancing the quality and applicability of multimodal long story generation. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of a multimodal long story consistency generation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the VastStory generation model provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the implementation process of the VastStory generation model provided in this embodiment of the invention; Figure 4 A schematic diagram of the workflow framework for generating a data processing pipeline for StoryBank. Figure 5 This is a schematic diagram of the role recognition and labeling operation process provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a multimodal long story consistency generation system provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this disclosure are information and data authorized by the user or fully authorized by all parties.
[0020] like Figure 1 As shown in the figure, an embodiment of the present invention discloses a multimodal long story consistency generation method 100, which includes the following method steps: S1, Obtain input data, the input data including a text prompt sequence and a visual frame sequence; S2, input the input data into the pre-trained VastStory generation model, and output the target image sequence; The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module. The input module is used to receive the input data. The encoding module is used to independently encode the text prompt sequence into a text token sequence and independently encode and segment the visual frame sequence into a visual token sequence. The core generation module includes a multi-cue cross-attention injection block and a self-attention and cross-attention layer, including: The frame dimension of the text token sequence and the visual token sequence is merged into the batch dimension token sequence, and the merged token sequence is processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input into the self-attention and cross-attention layer, and a modulated token sequence is generated under the control of the time modulation parameter. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
[0021] In some embodiments, for step S1, for example, the input data includes two types of key information: a text prompt sequence and a visual frame sequence. The text prompt sequence consists of several text descriptions, each corresponding to a specific scene or frame in a long story. Its content must clearly represent key information such as character characteristics, action state, background environment, lighting conditions, camera type, and atmosphere in the scene, ensuring accurate semantic guidance for subsequent image generation.
[0022] The visual frame sequence is derived from preprocessed shot-level frame data. Each frame in the visual frame sequence has undergone character selection, subtitle removal, clarity optimization, and resolution unification. Its resolution is preferably set to 720×1280 to balance generation efficiency and visual quality.
[0023] In some embodiments, step S2 is then performed, in which the input data is fed into the pre-trained VastStory generation model, and the model generates and outputs the target image sequence.
[0024] Specifically, please refer to Figure 2 , Figure 2 This is a schematic diagram of the VastStory generation model provided in an embodiment of the present invention. The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module.
[0025] For detailed implementation process, please refer to [link / reference]. Figure 3 , Figure 3 The flowchart of the VastStory generation model provided in this embodiment of the invention is shown. The input module receives text prompt sequences and visual frame sequences, the encoding module completes token sequence conversion, the core generation module performs semantic fusion and feature modulation, and finally the decoding and output modules synthesize and output the target image sequence.
[0026] Specifically, such as Figure 3 As shown, the visual input is a sequence of frames containing multiple characters, including labeled character IDs. The text input is a sequence of cues corresponding to the scene, such as a close-up shot of character 0. The two are concatenated before entering the VAE encoder. The time step t is processed by MLP to generate a temporal embedding. The VAE encoder encodes the visual frames into V-Tokens, and the umT5 encoder encodes the text cues into T-Tokens. The stacked DiT blocks in the core generation module receive the V-Tokens, T-Tokens, and temporal embeddings. After completing the independent interaction between the frame and the cues, they are passed to the VAE decoder to generate the output frame.
[0027] Among them, V-Tokens are first normalized by layers and self-attention modeled for inter-frame association, and then multi-cue cross-attention calculation is performed with T-Tokens. That is, independent frame-cue interaction is achieved through Q / K / V linear mapping. Finally, after layer normalization and FFN optimization of features, the modulated features can be used for subsequent decoding.
[0028] This model achieves precise text-image alignment and maintains consistency across long sequences of frames through a modular architecture, effectively solving problems such as semantic leakage, dynamic target blurring, and multi-role scene distortion in traditional generation methods. The following details the composition of the VastStory generation model and the implementation of each module.
[0029] The input module serves as the data flow entry point for the VastStory generation model. Its core function is to receive and temporarily store input data, providing a standardized data input format for subsequent encoding module processing.
[0030] Specifically, the input module receives a sequence of text prompts that is an ordered set of natural language descriptions. The sequence length is consistent with the number of frames in the visual frame sequence, meaning that each visual frame corresponds to a unique text prompt. After receiving the data, the input module performs format standardization processing on the text prompt sequence, removing redundant characters and standardizing punctuation usage to ensure the integrity and consistency of the text information. Simultaneously, the visual frame sequence undergoes format validation to confirm that the resolution and number of channels (RGB three channels) of each frame meet preset requirements. If any frame data does not meet the requirements, it is marked as invalid data and filtered to avoid affecting the processing effect of subsequent modules. Understandably, the input module's processing is concise and efficient, its core purpose being to provide high-quality, standardized input data for the encoding module.
[0031] The core task of the encoding module is to independently encode the input text cue sequence and visual frame sequence, converting them into token sequences that the model can process, namely, text token sequences and visual token sequences. This module is designed following the principles of independent encoding and dimensional uniformity, ensuring that the semantic features of the two types of data do not interfere with each other, while also providing a suitable tensor format for subsequent attention calculations.
[0032] Regarding the encoding process of the text prompt sequence, specifically, the encoding module uses the umT5 model as the text encoder to independently encode each text description in the text prompt sequence. For the i-th text description in the text prompt sequence, the umT5 model converts it into a fixed-dimensional semantic embedding vector. This vector accurately represents the semantic information in the text description, including key content such as character features, action commands, and environmental descriptions. The semantic embedding vectors corresponding to all text descriptions are arranged in sequence to form a text token sequence.
[0033] During the encoding process, the tensor dimension of the text token sequence is set to [B, f, ... Where B is the batch dimension and f is the length of the text prompt sequence, i.e., the number of frames. The number of tokens corresponding to each text description. Let B be the dimension of the text embedding vector. Preferably, B is set to 1 to avoid semantic interference between multiple batches of text data and to ensure the semantic independence of each text prompt.
[0034] For the encoding process of the visual frame sequence, the encoding module exemplarily uses the Wan VAE model as the visual encoder to independently encode each frame image in the visual frame sequence. Specifically, the i-th frame image in the visual frame sequence is first input into the Wan VAE encoder. Through the network structure of the encoder, such as convolutional layers and pooling layers, the visual features of the image are extracted to obtain a high-dimensional visual embedding vector. Subsequently, the visual embedding vector is subjected to patch segmentation processing, that is, the high-dimensional visual embedding vector is divided into several visual patches according to a preset patch size. Each visual patch corresponds to a local region feature in the image. The visual patches corresponding to all visual frames are arranged in sequence to form a visual token sequence.
[0035] The tensor dimension of the visual token sequence is set to [B, f, ... Where B is the batch dimension, which is consistent with the B in the text token sequence, i.e., B=1, and f is the number of frames in the visual frame sequence, which is the same as the length f of the text cue sequence. The number of visual patches corresponding to each frame of image. The dimension for each visual patch. It should be noted that the visual token sequence... Dimensions and text token sequences The dimensions are set to the same value to ensure compatibility of tensor dimensions during subsequent attention calculations.
[0036] In the implementation of the encoding module, the encoding processes of the text prompt sequence and the visual frame sequence are independent of each other, and there is no feature interaction or fusion operation. This design can preserve the integrity of text semantic features and visual image features to the greatest extent and avoid semantic confusion between the two types of features during the encoding stage.
[0037] The core generation module consists of a multi-cue cross-attention injection block and self-attention and cross-attention layers. The core function of this module is to achieve accurate interaction between text token sequences and visual token sequences, temporal consistency modeling, and feature modulation.
[0038] Specifically, before performing attention calculations, the text token sequence and the visual token sequence are first adjusted in terms of dimensions, that is, the frame dimensions of the two types of token sequences are merged into the batch dimension.
[0039] Specifically, for a sequence of text tokens, its original tensor dimension is [B, f, ... Through dimension reshaping operations, it is transformed into [B×f, The tensor format; similarly, the original tensor dimension of the visual token sequence is [B×f, After dimensional reshaping, it is converted to [B×f, The tensor format. Since B was set to 1 during the encoding stage, after dimension reshaping, the tensor dimensions of the text token sequence and the visual token sequence are simplified to [f, and [f, .
[0040] This dimensionality adjustment operation treats the token corresponding to each frame in the sequence as an independent batch sample. This allows the subsequent multi-cue cross-attention injection block to perform independent one-to-one interaction between the text token and the visual token in each frame, avoiding semantic interference between different frames and providing structural support for accurate generation of one cue per frame. It is understandable that the dimensionality adjustment process only changes the organization of the tensor, without altering the semantic features and data information of the token sequence, ensuring the accuracy of subsequent processing.
[0041] The multi-cue cross-attention injection block is used to achieve precise text-image alignment. Its core design idea is to provide independent cross-attention calculations for each index (i.e., the text token and visual token corresponding to each frame) of the dimension-adjusted token sequence. This ensures that the generation of each frame's image strictly adheres to the semantic requirements of the corresponding text cue, while avoiding cross-frame semantic leakage. The specific implementation process of this module is as follows: For the dimension-adjusted text token sequence, dimension [f, and visual token sequence dimension [f, For each index i, where i ranges from 1 to f, a multi-hint cross-attention injection block performs cross-attention calculation.
[0042] Specifically, for the i-th index, the visual token sequence corresponding to that index is used as the query tensor. Its dimensions are Use the text token sequence corresponding to the same index as the key tensor. Sum tensor Their dimensions are respectively and Because it has been ensured during the coding phase. and Dimensions are consistent, therefore the key tensor Sum tensor It can be directly used in attention calculations without the need for additional dimensional transformations.
[0043] In the cross-attention calculation process, the query tensor is calculated first. With key tensor The inner product of the transposes of the given tensors yields the original attention weight matrix. To avoid excessively large inner product values due to the high tensor dimension, the original attention weight matrix is divided by a scaling factor. Where d is the dimension of the query tensor. This operation can effectively mitigate the numerical fluctuations of the weight matrix and improve computational stability.
[0044] Subsequently, the scaled attention weight matrix is normalized using the Sigmoid function, which maps the attention weight values to the [0,1] interval, ensuring the rationality and interpretability of the weights. Finally, the normalized attention weight matrix and the value tensor are... Perform matrix multiplication to obtain the attention output corresponding to the i-th index, whose dimension is... .
[0045] For all indices i, including 1 to f, repeat the above cross-attention calculation process to obtain f attention outputs. Arrange these results in index order to form a sequence with dimension [f, The attention-processed token sequence is then processed. After obtaining the attention output, a residual connection mechanism is used to add the attention output to the visual tokens at the corresponding indices in the original visual token sequence, resulting in the final multi-cue cross-attention injection block output sequence. The introduction of the residual connection mechanism can effectively alleviate the gradient vanishing problem during deep network training, while preserving the spatial feature information of the original visual tokens, thus improving the model's feature representation ability and training stability.
[0046] Preferably, the application of the Sigmoid function in the cross-attention calculation process is key to achieving precise modulation of attention weights. Compared to the traditional Softmax function, it can more flexibly control the activation range of attention, avoiding semantic deviations caused by excessive concentration of some weights, and further improving the alignment accuracy of text and image. Furthermore, since the cross-attention calculations corresponding to each index are independent and there is no information interaction, cross-frame semantic leakage can be effectively prevented, ensuring that the generation of each frame image is only guided by the corresponding text prompt, thus guaranteeing the consistency of long sequence generation.
[0047] The self-attention and cross-attention layer is used to model the temporal consistency and spatial correlation of token sequences, ensuring that the generated target image sequence maintains the coherence of character features, background environment, and action logic between frames, while further optimizing the semantic fusion effect of text-image. This layer's processing takes the output sequence of the multi-cue cross-attention injection block as input and completes feature modulation under the control of temporal modulation parameters. The self-attention and cross-attention layer contains multiple stacked multi-cue DiT blocks, i.e., diffusion transformers. The basic structure of a DiT block includes layer normalization, self-attention and cross-attention calculation, FFN, and residual connections.
[0048] Specifically, first, time modulation parameters are generated. These parameters are generated based on time step t, where t is sampled from a logit-normal distribution and its value ranges from [0,1]. The time step t is then input into a multilayer perceptron (MLP) for mapping processing to obtain a time embedding vector. Its dimension is the same as the feature dimension of the output sequence of the multi-cue cross-attention injection block. Maintain consistency. Subsequently, the temporal embedding vector... Layer normalization is performed to eliminate numerical fluctuations, ensuring that the value matches the feature distribution of the subsequent input sequence and improving the stability of the modulation process.
[0049] Inject multi-cue cross-attention into the token sequence of the block output, dimension [f, The input sequence is fed into a self-attention and cross-attention layer, which first performs layer normalization to unify the feature distribution. Subsequently, self-attention and cross-attention calculations are performed respectively.
[0050] Specifically, in the self-attention calculation process, the token sequence after layer normalization is used as the query tensor, key tensor, and value tensor to calculate the attention weights within the sequence.
[0051] Specifically, self-attention computation includes two parts: intra-frame self-attention and inter-frame self-attention. Intra-frame self-attention is used to model the spatial relationship between different visual patches within a single frame, ensuring the coherence of local features in a single frame image, such as the spatial positional relationship of different parts of the character's body and the reasonable layout of background elements. Inter-frame self-attention is used to model the temporal relationship between corresponding visual patches between different frames, ensuring the consistency of character features such as clothing, hairstyle, facial features, and background environment across frames in a long sequence, and avoiding abrupt changes in features between frames.
[0052] In the cross-attention calculation process, the token sequence after layer normalization is used as the query tensor, and the temporal embedding vector is used as the query tensor. The vectors after linear transformation serve as key and value tensors, enabling semantic fusion of time modulation parameters and token sequences. This cross-attention computation allows the features of the token sequence to be dynamically adjusted with time step t, adapting to the requirements of rectified flow training mechanisms, improving the model's ability to generate dynamic scenes such as characters' violent movements, and avoiding problems such as blurring and ghosting of dynamic targets.
[0053] The results of self-attention and cross-attention computations are fused using a weighted summation to obtain a pre-modulated token sequence. This pre-modulated token sequence is then input into a feedforward network (FFN) for nonlinear transformation. The FFN consists of two fully connected layers and one activation function such as GELU, which enhances the feature representation of the token sequence and captures more complex semantic associations and spatial-temporal features. Finally, a residual connection mechanism is used to add the output of the FFN to the fusion result of self-attention and cross-attention computations, resulting in the final modulated token sequence, whose dimension remains [f, .
[0054] It should be noted that the processing of the self-attention and cross-attention layers is carried out under the unified control of the temporal modulation parameters. The introduction of the temporal modulation parameters enables the feature modulation of the token sequence to adapt to the generation requirements of different time steps, providing support for subsequent training optimization based on rectified streams. At the same time, this layer effectively balances the generation quality of single-frame images and the inter-frame consistency of long sequences through the combination of intra-frame and inter-frame self-attention, which is an important guarantee for achieving consistent generation of multimodal long stories.
[0055] The decoding and output module decodes the modulated token sequence output by the core generation module into the target image and outputs the target image sequence in sequence.
[0056] The decoding and output module uses a Wan VAE decoder as the core decoding unit. This decoder is paired with the Wan VAE encoder in the encoding module to ensure the reversibility of the encoding and decoding process. First, the modulated token sequence output by the core generation module, i.e., the sequence of dimensions [f, ...], is processed. Perform dimensional reshaping to convert to an input format compatible with the Wan VAE decoder, i.e., [B, ,in and These are the height and width of the feature map, respectively, derived from the number of patches and patch size of the visual token.
[0057] Subsequently, the reshaped feature tensor is input into the Wan VAE decoder. The decoder performs an inverse transformation on the feature tensor through network structures such as transposed convolutional layers and upsampling layers, gradually restoring the spatial resolution of the image and finally generating a pixel-level image tensor with dimensions [B, 3, H, W], where 3 is the number of RGB channels, and H and W are the height and width of the image, respectively, which are 720 and 1280. During the decoding process, the network parameters of each layer of the decoder have been co-optimized with the encoder during the model training phase to ensure that the decoded image can accurately restore the semantic features and visual information in the modulated token sequence. The training of the model is described in detail below and will not be repeated here.
[0058] Finally, the generated pixel-level image tensors are normalized to map pixel values from the [-1,1] interval to the [0,255] interval, conforming to the format requirements of standard RGB images. The normalized image tensors are then converted into image files such as PNG format in sequence to form the target image sequence, which is then output through the model's output interface.
[0059] Understandably, the implementation of the decoding and output module mainly relies on the efficient decoding capability of the Wan VAE decoder. Its core is to ensure that the semantic information and temporal consistency features in the modulated token sequence can be accurately converted into visual images, and the final output target image sequence meets the core requirements such as text-image alignment, inter-frame consistency, and clear dynamic targets.
[0060] In some embodiments, the training process of the VastStory generation model adopts a three-stage training strategy of data preprocessing, hierarchical training, and loss optimization. It combines story library data processing pipeline to build high-quality training data, and gradually improves the model's consistent generation capability through static and dynamic training. At the same time, it introduces multiple loss constraints to ensure the accuracy of model parameter optimization.
[0061] Specifically, please refer to Figure 4 , Figure 4 This diagram illustrates the framework for the StoryBank data processing pipeline. The core purpose of building the StoryBank data processing pipeline is to extract high-quality, human-centered story-level training data from TV series videos, providing rich and standardized sample support for model training.
[0062] The implementation process of this pipeline specifically includes the following steps: First, extracting shot-level frames from the episode-level video. For example, MoviePy is used to perform shot-level segmentation on the episode-level video. MoviePy can automatically identify shot boundaries based on the video's shot transition features (such as sudden brightness changes and scene jumps), dividing the episode-level video into several independent shot-level video segments. For each shot-level video segment, a representative frame is selected as a visual summary. The selection principle for the representative frame is to most comprehensively reflect the core content of the shot, preferably selecting a frame in the middle of the shot to avoid incompleteness of frames at the beginning or end of the shot.
[0063] Subsequently, preprocessing operations were performed on the extracted shot-level frames. Specifically, the EasyOCR tool was used to detect the presence of visible text such as subtitles and watermarks in the shot-level frames. If visible text was detected, the frame was discarded. For shot-level video clips where all frames contain subtitles, the middle frame was selected, and Flux-Inpainting technology was applied to remove the subtitles while maintaining the visual continuity of the frames and avoiding image content distortion caused by subtitle removal. Preprocessing operations also included resolution unification and blur frame filtering: all shot-level frames that had undergone subtitle processing were adjusted to a uniform resolution of 720×1280 to ensure the consistency of the training data format; the Laplacian variance of each frame was calculated, and if the variance was less than 50, the frame was determined to be a blur frame and removed to ensure the clarity of the training data.
[0064] Next, please refer to Figure 5 , Figure 5 This is a schematic diagram of the character recognition and annotation operation process provided in an embodiment of the present invention. The character recognition and annotation operation is performed on the preprocessed shot-level frames. This operation is to construct human-centered training data.
[0065] Specifically, in S501, character detection and embedding vector extraction are performed. Specifically, target detection algorithms, such as the YOLO series algorithms, are used to detect characters in each shot-level frame, identify the character regions in the frame, and extract the embedding vectors of the character regions through feature extraction networks, such as the ResNet series networks. These embedding vectors can represent the appearance features of the character, such as facial features and clothing features. In S502, character clustering and frame matching are performed using the DBSCAN clustering algorithm to cluster the character embedding vectors of all shot-level frames. DBSCAN automatically divides the frames into clusters based on the similarity of the embedding vectors, with each cluster corresponding to a specific character. The optimal cluster center is identified through clustering, selected based on the mean vector of the embedding vectors within the cluster, which most accurately represents the appearance features of the corresponding character. Subsequently, the distance between the character embedding vector of each shot-level frame and each optimal cluster center is calculated, and the frame is matched to the nearest cluster center, thus associating the frame with a specific character. In S503, to ensure the single-character focus of the training data, frames containing multiple characters are excluded, and only frames containing a single, clear character are retained. Specifically, a face detection algorithm is used to detect whether multiple facial regions exist in a frame. If multiple facial regions are detected, the frame is determined to contain multiple characters and is discarded; only frames with a single detected facial region and clear facial features are retained to ensure the uniqueness and completeness of character features in the training data. In S504, fine-grained annotation and character ID assignment are exemplarily achieved by using the Qwen2.5-vl model to generate fine-grained annotations for the retained shot-level frames. The annotations must comprehensively cover the key information of the frames, including the character's facial appearance (e.g., skin tone, facial features), clothing (e.g., color, style, material), hairstyle (e.g., length, color, shape), actions (e.g., standing, running, gestures), background (e.g., indoor, outdoor), scene elements, camera type (e.g., close-up, panoramic, distant view), lighting (e.g., soft, bright, dim), and environmental atmosphere (e.g., tense, relaxed, sad). Simultaneously, a unique identifier ID is assigned to each character identified through clustering, and this ID is associated with the annotation content of the corresponding frame, achieving accurate character annotation across frames and ensuring consistency of the same character's annotation across different frames. It is understood that other models capable of achieving similar annotation functions can also be used, and this invention does not impose limitations.
[0066] Through the processing of the story library data pipeline described above, the final result is human-centered story-level training data. This training data contains several episode samples, each consisting of several shot-level frames, corresponding fine-grained labeled text, and character IDs, which can provide rich semantic and visual feature information for model training.
[0067] The static training phase is the initial stage of model training. Its core objective is to enable the model to learn the basic consistency between short sequence frames, establishing preliminary temporal modeling capabilities and text-image alignment capabilities. The specific implementation process of this phase is as follows: First, a fixed frame length of 8 frames is set for the static training phase. For each episode sample in the story-level training data, it is determined whether its frame count is greater than or equal to 8 frames: if the episode sample has more than 8 frames, a consecutive sequence of 8 frames is randomly sampled as a training sample; if the episode sample has less than 8 frames, the sample is discarded and does not participate in the static training phase. The purpose of this setting is to ensure that the training samples in the static training phase have a uniform length, which facilitates the model learning the inter-frame consistency rules of fixed-length sequences.
[0068] Subsequently, the 8 sampled training frames are input into the VastStory generation model for training. During training, the model's input consists of 8 visual frame sequences and their corresponding 8 fine-grained text annotations (text prompt sequences). The model outputs 8 target image sequences according to the aforementioned encoding, core generation, and decoding process. In the static training phase, the model's training parameters, such as the learning rate and optimizer configuration, remain fixed. The learning rate is preferably set to 1e-5, and the AdamW optimizer is used. This optimizer effectively alleviates the generalization problem caused by weight decay and improves the model's training stability.
[0069] The number of training iterations in the static training phase is determined by the convergence of the model's loss function. The static training phase ends when the model's loss function value stabilizes below a preset threshold and the generated 8-frame target image sequence meets the basic requirements for text-image alignment and inter-frame consistency. It can be understood that the core of the static training phase is to enable the model to master basic temporal correlation and semantic alignment capabilities, laying the foundation for the generation of long sequences in the subsequent dynamic training phase. Therefore, the training sample length is fixed, the training objective is clear, and the training process is relatively simple.
[0070] The core objective of the dynamic training phase is to allow the model to gradually adapt to the task of generating sequences with variable frame lengths, improving the model's consistent generation capability for long sequences of up to 32 frames, while maintaining the model's flexibility and generalization ability. The specific implementation process of this phase is as follows: First, all episode samples in the story-level training data are grouped by their frame count, forming different "buckets". Each bucket corresponds to a specific frame count, such as a 2-frame bucket, a 4-frame bucket, a 6-frame bucket, ..., a 32-frame bucket. Each bucket contains all episode samples with the same frame count as that bucket. For example, episode samples with 2 frames are assigned to the 2-frame bucket, and episode samples with 32 frames are assigned to the 32-frame bucket, ensuring that each episode sample is assigned to its corresponding bucket.
[0071] Subsequently, the iterative training process of the dynamic training phase begins. In each training optimization step, a bucket is randomly selected as the current training bucket, and a preset number of episode samples, such as 16 samples, are randomly extracted from this bucket as the training data for the current training batch. The training data is input into the VastStory generation model, and the model adaptively adjusts its internal processing flow according to the number of frames in the input samples, outputting the target image sequence with the corresponding number of frames.
[0072] During the dynamic training phase, the model's training parameters remain consistent with those in the static training phase; the learning rate remains 1e-5, and the optimizer remains the AdamW optimizer. As training iterates, the model gradually learns the inter-frame consistency patterns of sequences with different frame lengths, from short sequences like 2 or 4 frames to long sequences like 32 frames. The model's temporal modeling and adaptive generation capabilities continuously improve. The number of training iterations in the dynamic training phase is typically greater than in the static training phase. The dynamic training phase ends when the model can generate target image sequences that meet the requirements of text-image alignment and inter-frame consistency in all frame length buckets, and the loss function value stabilizes at a low level.
[0073] Preferably, in the dynamic training phase, randomly selecting buckets for each training optimization step ensures that the model learns evenly from training samples of different frame lengths, avoiding excessive bias towards sequences of a particular frame length and improving the model's generalization ability. Simultaneously, the dynamic training phase supports the generation of sequences up to 32 frames, meeting the needs of most multimodal long story generation application scenarios, thus balancing the model's flexibility and practicality.
[0074] The training objective of the VastStory generative model is based on the definition of rectified flow. It optimizes model parameters by constructing time-correlated noisy images and their velocity fields, and minimizing the expected mean square error between the model's predicted velocity field and the true velocity field. Simultaneously, semantic independence constraints are introduced, forming a joint loss function with the image reconstruction loss, further improving the model's generation quality and consistency.
[0075] Specifically, first, define the intermediate state tensor in the rectified flow. Its expression is: ,in, Let N be the initial noise tensor, which follows a standard normal distribution N(0,I), where I is the identity matrix; t is the target feature tensor, which is the feature tensor of the visual token sequence output by the encoding module after processing; t is the time step, which is sampled from the logit-normal distribution, t∈[0,1].
[0076] Then, define the velocity tensor. The velocity tensor represents the intermediate state tensor. The rate of change of time step t is expressed as: This velocity tensor is the true velocity field, which can reflect the change law of the feature from the initial noise state to the target feature state.
[0077] The model's predicted velocity field is determined by the model's prediction function. It means that, among them For intermediate state tensors, Let t be the umT5 embedding vector of the text token sequence, and t be the time step. These are all the trainable parameters of the model. Prediction function. The output is the velocity tensor predicted by the model.
[0078] Based on the above definition, the loss function related to rectified flow, i.e., the velocity prediction loss, is the expected mean square error between the model's predicted velocity tensor and the actual velocity tensor, and its expression is:
[0079] in, Expressing expectations, This represents the L2 norm. This loss function can constrain the model to accurately learn the variation of features from noise to the target, improving the model's generation accuracy and stability.
[0080] To further prevent cross-frame semantic leakage and improve the inter-frame consistency of long sequence generation, semantic independence constraints are introduced during model training. Specifically, semantic independence constraints are achieved by calculating the feature differences between the generation results of adjacent frames at the partition boundaries: For long sequence generation tasks, the sequence is divided into several sub-intervals according to preset rules, such as every 8 frames as a sub-interval. The feature differences between the generation results of the previous and next frames at the boundary of each sub-interval are calculated, using metrics such as cosine similarity or Euclidean distance. These feature differences are then used as semantic independence constraint terms. Semantic independence constraint terms Its function is to constrain the generated result features of different frames at the sequence boundary to remain continuous or similar, and to avoid semantic abrupt changes across frames.
[0081] At the same time, image reconstruction loss is introduced. This loss function is achieved by calculating pixel-level errors such as mean square error and perceptual loss between the target image sequence generated by the model and the real image sequence. It can constrain the image generated by the model to be consistent with the real image at the pixel level, thereby improving the image fidelity.
[0082] Predicting loss based on speed Semantic independence constraint terms and image reconstruction loss The joint loss function is formed by combining the preset weights, and its expression is as follows: , in, , , These are weighting coefficients, and their values range from [0,1]. Preferably, The value is 0.5. The value is 0.3. The value of 0.2 is used to balance the training objectives of velocity prediction, semantic constraints and image reconstruction, ensuring the overall generative performance of the model.
[0083] During model training, the joint loss function is calculated using the backpropagation algorithm. For each trainable parameter of the model The gradient is calculated, and the AdamW optimizer is used to update the model parameters based on the gradient information to minimize the joint loss function. This process is iterated until the model converges, meaning the joint loss function value stabilizes below a preset threshold, and the target image sequence generated by the model meets preset requirements in terms of text-image alignment, inter-frame consistency, and image fidelity.
[0084] It should be noted that during model training, the parameters of the Wan VAE encoder and the umT5 encoder were frozen, and only the parameters of modules such as the multi-cue cross-attention injection block, self-attention and cross-attention layers, and feedforward network were fine-tuned. The purpose of this setting is to fully utilize the feature extraction capabilities of the pre-trained encoder, avoid the decrease in semantic feature extraction accuracy caused by encoder parameter updates, and at the same time reduce the number of training parameters of the model, thereby improving training efficiency.
[0085] Therefore, this invention solves the problems of semantic leakage, dynamic target ambiguity, multi-role scene distortion, and poor adaptability of long and short sequences in traditional multimodal long story generation methods. This invention enables the method to generate high-quality, highly consistent multimodal long story image sequences, which are suitable for various application scenarios such as film, TV series, and animation production.
[0086] Figure 6 A multimodal long story consistency generation system 600 is illustrated. Implementations of this system are similar to... Figure 1 Corresponding to the illustrated method embodiments, this system can be specifically applied to various computer devices. The system specifically includes: Data acquisition module 601 is used to acquire input data, which includes a text prompt sequence and a visual frame sequence; The target image sequence output module 602 is used to input the input data into the pre-trained VastStory generation model and output the target image sequence. The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module. The input module is used to receive the input data. The encoding module is used to independently encode the text prompt sequence into a text token sequence and independently encode and segment the visual frame sequence into a visual token sequence. The core generation module includes a multi-cue cross-attention injection block and a self-attention and cross-attention layer, including: The frame dimension of the text token sequence and the visual token sequence is merged into the batch dimension token sequence, and the merged token sequence is processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input into the self-attention and cross-attention layer, and a modulated token sequence is generated under the control of the time modulation parameter. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
[0087] Based on the same inventive concept, this application also provides a computer device, the method corresponding to which can be the method in the foregoing embodiments, and the principle of solving the problem is similar to that method. The computer device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.
[0088] The computer device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and smart bands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.
[0089] Figure 7The diagram illustrates the structure of an apparatus suitable for implementing the methods and / or technical solutions in the embodiments of this application. The apparatus 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0090] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 708 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 709 including a network interface card such as a LAN (local area network) card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet.
[0091] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 701, it performs the functions defined in the methods of this application.
[0092] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.
[0093] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0094] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0095] Furthermore, the inclusion of a single word does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any particular order.
Claims
1. A method for generating consistency in multimodal long stories, characterized in that, Includes the following steps: Acquire input data, which includes a text prompt sequence and a visual frame sequence; The input data is fed into the pre-trained VastStory generation model, which outputs a target image sequence. The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module. The input module is used to receive the input data. The encoding module is used to independently encode the text prompt sequence into a text token sequence and independently encode and segment the visual frame sequence into a visual token sequence. The core generation module includes a multi-cue cross-attention injection block and a self-attention and cross-attention layer, including: The frame dimension of the text token sequence and the visual token sequence is merged into the batch dimension token sequence, and the merged token sequence is processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input into the self-attention and cross-attention layer, and a modulated token sequence is generated under the control of the time modulation parameter. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
2. The method for generating consistency in multimodal long stories according to claim 1, characterized in that, The VastStory generative model training includes: A story library data processing pipeline is constructed to extract shot-level frames from TV series videos, and to perform character recognition and annotation on the shot-level frames to obtain story-level training data centered on the characters. During the static training phase, the model is trained with a series of consecutive frames of fixed frame length to learn basic consistency. During the dynamic training phase, different episodes in the story-level training data are grouped into different buckets according to their frame count. In each training optimization step, data from one bucket is randomly selected for training, so that the model gradually adapts to the variable frame length sequence generation task.
3. The method for generating consistency in multimodal long stories according to claim 2, characterized in that, The pipeline for building the story library data processing includes: Perform character detection and extract character embedding vectors for the aforementioned shot-level frames; Cluster the character embedding vectors to identify the optimal cluster centers; Each of the aforementioned shot-level frames is matched to the nearest cluster center and associated with a specific character; Frames containing multiple characters are excluded; only frames containing a single character are retained. Annotations are generated for the retained shot-level frames, and a unique identifier is assigned to each identified character.
4. The method for generating consistency in multimodal long stories according to claim 2, characterized in that, The training objective of the model is based on the definition of rectified flow. It optimizes the model parameters by constructing a time-correlated noise image and its velocity field, and minimizing the expected mean square error between the velocity field predicted by the model and the true velocity field determined by the starting and ending images.
5. The method for generating consistency in multimodal long stories according to claim 1, characterized in that, The training of the model also includes semantic independence constraints, which are achieved by calculating the feature differences of the generation results of adjacent frames at the partition boundary. These constraints are used to constrain the generation result features of different frames at the sequence boundary during model training and together with the image reconstruction loss constitute a joint loss function.
6. The method for generating consistency in multimodal long stories according to claim 1, characterized in that, The cross-attention calculation includes normalizing the attention weights calculated from the query tensor and key tensor based on the Sigmoid function, and multiplying the normalized weights by the value tensor to obtain the attention output corresponding to each index.
7. A multimodal long story consistency generation system, characterized in that, include: The data acquisition module is used to acquire input data, which includes a text prompt sequence and a visual frame sequence; The target image sequence output module is used to input the input data into the pre-trained VastStory generation model and output the target image sequence; The VastStory generation model includes an input module, an encoding module, a core generation module, and a decoding and output module. The input module is used to receive the input data. The encoding module is used to independently encode the text prompt sequence into a text token sequence and independently encode and segment the visual frame sequence into a visual token sequence. The core generation module includes a multi-cue cross-attention injection block and a self-attention and cross-attention layer, including: The frame dimension of the text token sequence and the visual token sequence is merged into the batch dimension token sequence, and the merged token sequence is processed by the multi-cue cross-attention injection block. For each index i in the batch, the visual token sequence corresponding to the current index is used as the query tensor, and the text token sequence corresponding to the same index is used as the key tensor and value tensor to perform cross-attention calculation. The token sequence processed by the multi-cue cross-attention injection block is input into the self-attention and cross-attention layer, and a modulated token sequence is generated under the control of the time modulation parameter. The decoding and output module is used to receive the modulated token sequence, decode and synthesize it to generate the target image sequence, and output the target image sequence.
8. A computer device, wherein the computer device is characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
9. A computer-readable medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.