An AIGC multi-modal short play generation method and system

Abstract

This invention discloses an AIGC multimodal short drama generation method, belonging to the field of short drama technology. The method involves: constructing a preprocessed multimodal dataset; obtaining visually dense one-dimensional discrete tag sequences and visually lightweight one-dimensional discrete tag sequences; obtaining audio one-dimensional discrete tag sequences and text one-dimensional discrete tag sequences; generating a total multimodal one-dimensional discrete tag sequence; constructing a two-layer scalable xLSTM model composed of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module; completing the pre-training of the two-layer scalable xLSTM model; generating a complete total multimodal one-dimensional discrete tag sequence; obtaining preliminary video frame sequences, preliminary audio waveforms, and preliminary text content; and outputting the target multimodal short drama. This invention achieves consistent modeling of local details and global plot while maintaining controllable computational complexity, enabling the generated short drama to maintain semantic coherence and plot stability over long periods.

Description

Technical Field

[0001] This invention relates to the field of short drama technology, and in particular to an AIGC multimodal short drama generation method and system. Background Technology

[0002] With the rapid development of multimodal generation technology, deep learning-based methods for the joint generation of video, audio, and text have gradually become an important technological path for the automated production of short drama content. Existing multimodal generation methods typically rely on the Transformer architecture to uniformly model data from different modalities, generating data end-to-end by encoding visual, audio, and text data into a unified sequence.

[0003] In existing technologies, multimodal data representations generally employ high-dimensional continuous features or simple discrete label concatenation, lacking modeling of the differences in temporal scale and spatial structure between different modalities. This leads to information redundancy and semantic misalignment issues during unified sequence modeling. Furthermore, traditional methods often fail to distinguish between high-frequency detail information and low-frequency structural information when encoding visual information, making it difficult to achieve a balance between detail richness and overall structural consistency in the generated results.

[0004] Existing methods based on Transformer or RNN models suffer from significant computational complexity and memory bottlenecks when processing long sequences. On the one hand, Transformers incur high computational costs in long sequence modeling, making it difficult to efficiently handle multimodal short drama generation tasks spanning long periods. On the other hand, traditional recurrent neural networks are prone to information decay during long-distance dependency modeling, making it difficult to simultaneously maintain local detail information and global semantic consistency, resulting in deficiencies in the generated content's narrative coherence and detail representation.

[0005] In terms of multimodal fusion, existing technologies typically employ simple splicing or attention mechanisms to fuse different modalities, lacking fine-grained modeling of cross-modal temporal alignment relationships. This leads to asynchrony or semantic inconsistencies between different modalities during the generation process. Furthermore, during the generation phase, most methods fail to structurally optimize the autoregressive inference process of the unified multimodal sequence, making it difficult to dynamically coordinate information interaction between different modalities during generation. Summary of the Invention

[0006] One objective of this invention is to propose an AIGC multimodal short drama generation method. This invention achieves consistent modeling of local details and global plot while keeping computational complexity under control, so that the generated short dramas maintain semantic coherence and plot stability over a long period of time.

[0007] A method for generating AIGC multimodal short dramas according to an embodiment of the present invention includes:

[0008] Receive and preprocess user input of script text prompts, storyboard descriptions, and interactive control instructions, and construct a preprocessed multimodal data set;

[0009] Each frame of video sequence data in the preprocessed multimodal dataset is input into a one-dimensional discretization coding module to obtain a visually dense one-dimensional discrete label sequence and a visually lightweight one-dimensional discrete label sequence.

[0010] The audio sequence data and text data in the preprocessed multimodal dataset are input into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, to obtain the audio one-dimensional discrete tag sequence and the text one-dimensional discrete tag sequence.

[0011] The visually dense one-dimensional discrete label sequence, the audio one-dimensional discrete label sequence, the text one-dimensional discrete label sequence, and the visually lightweight one-dimensional discrete label sequence are alternately arranged on the same time axis according to a unified timestamp to generate a multimodal one-dimensional discrete label sequence.

[0012] Construct a two-layer scalable xLSTM model consisting of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module;

[0013] The total sequence of multimodal one-dimensional discrete labels is used as input, and the pre-training of the two-layer scalable xLSTM model is completed using cross-entropy loss and memory alignment loss.

[0014] In the short drama generation stage, a trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence to generate a complete multimodal one-dimensional discrete tag sequence.

[0015] Modal separation is performed on the generated multimodal one-dimensional discrete marker sequence to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content.

[0016] Perform frame-level clock synchronization on the initial video frame sequence and the initial audio waveform, and perform subtitle alignment on the initial text content and the initial video frame sequence based on the text timestamp to generate a target multimodal short drama with synchronized audio and video and synchronized subtitles, and output the target multimodal short drama.

[0017] Optionally, the step of receiving and preprocessing the script text prompts, storyboard descriptions, and interactive control instructions input by the user includes:

[0018] The system collects user-input script text prompts, storyboard descriptions, and interactive control commands, and aggregates multimodal training data including video sequence data, audio sequence data, and text data to generate a multimodal raw dataset. Frame-level sampling, color space normalization, and noise suppression are performed on the video sequence data in the multimodal raw dataset. Temporal framing and silent segment removal are performed on the audio sequence data. Sentence segmentation and punctuation normalization are performed on the text data to form a preprocessed multimodal dataset.

[0019] Optionally, inputting each frame of video sequence data from the preprocessed multimodal dataset into a one-dimensional discretization coding module includes:

[0020] Video sequence data is read frame by frame in chronological order from the preprocessed multimodal dataset, and the video sequence data at the current moment is defined as the current frame video data;

[0021] In a visually dense one-dimensional discretized encoder, the current frame video data is divided into multiple dense visual blocks of the same size;

[0022] Perform feature mapping operation on each dense visual block to obtain the corresponding dense continuous feature vector;

[0023] The distance between each dense continuous feature vector and all codebook vectors in the visual dense codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the dense continuous feature vector, thus obtaining a visual dense one-dimensional discrete label sequence. The discrepancies are then connected in series according to the increasing order of the time index t to form the overall visual dense one-dimensional discrete label sequence.

[0024] In the visual lightweight one-dimensional discretization encoder, the current frame video data is divided into large-scale spatial partitions and low-frequency structure aggregation is performed to divide the current frame video data into multiple lightweight visual blocks with a size larger than dense visual blocks.

[0025] Perform feature mapping operations on each lightweight visual block to obtain the corresponding lightweight continuous feature vector;

[0026] The distance between each lightweight continuous feature vector and all codebook vectors in the visual lightweight codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the lightweight continuous feature vector, resulting in a visual lightweight one-dimensional discrete label sequence. These labels are then concatenated in ascending order of time index t to form the overall visual lightweight one-dimensional discrete label sequence.

[0027] Optionally, the step of inputting the audio sequence data and text data from the preprocessed multimodal dataset into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, includes:

[0028] Audio sequence data is read from the preprocessed multimodal dataset in chronological order, and the audio sequence data corresponding to the t-th time index is defined as the audio data at the current time.

[0029] The audio data at the current moment is input into the audio one-dimensional discretization encoder, and the audio data at the current moment is processed into equal-duration frames to obtain multiple audio encoded frames of the same length.

[0030] Each audio encoded frame is input into the feature mapping function in the audio one-dimensional discretization encoder to obtain the corresponding audio continuous feature vector. The distance between each audio continuous feature vector and all codebook vectors in the audio codebook is calculated one by one, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the audio continuous feature vector to obtain the audio one-dimensional discrete label sequence corresponding to the audio data at the current time.

[0031] The audio one-dimensional discrete tag sequences corresponding to each time index t are sequentially connected in ascending order of time index t to form the overall audio one-dimensional discrete tag sequence.

[0032] Read text data sequentially from the preprocessed multimodal dataset, and define the text data corresponding to the r-th text sequential index as the current text segment;

[0033] Input the current text segment into the sub-word encoder, perform character-level splitting and sub-word matching on the current text segment, and obtain the sub-word unit sequence corresponding to the current text segment;

[0034] Map each sub-word unit sequence corresponding to the current text segment to a word index in the text sub-word vocabulary, and convert each sub-word unit sequence into a corresponding discrete tag to obtain the text one-dimensional discrete tag sequence corresponding to the current text segment;

[0035] The one-dimensional discrete text marker sequences corresponding to each text sequence index r are concatenated in ascending order of text sequence index r to form the overall one-dimensional discrete text marker sequence.

[0036] Optionally, the step of alternating the visually dense one-dimensional discrete label sequence, the audio one-dimensional discrete label sequence, the text one-dimensional discrete label sequence, and the visually lightweight one-dimensional discrete label sequence on the same time axis according to a unified timestamp includes:

[0037] Using the time index t of the video sequence data as a unified timestamp reference, the overall visually dense one-dimensional discrete label sequence, the overall audio one-dimensional discrete label sequence, and the overall visually lightweight one-dimensional discrete label sequence are aligned by time index. The overall text one-dimensional discrete label sequence is then mapped to the corresponding time index interval according to the text order index r, thus obtaining the visually dense one-dimensional discrete label subsequence, the audio one-dimensional discrete label subsequence, the text one-dimensional discrete label subsequence, and the visually lightweight one-dimensional discrete label subsequence.

[0038] For the visually dense one-dimensional discrete label subsequence, audio one-dimensional discrete label subsequence, text one-dimensional discrete label subsequence and visually lightweight one-dimensional discrete label subsequence corresponding to the t-th time index, sequential concatenation is performed according to a preset alternating order, and all labels in each subsequence are connected sequentially to obtain the local multimodal one-dimensional discrete label sequence corresponding to the t-th time index.

[0039] Following the ascending order of time index t, the local multimodal one-dimensional discrete label sequences corresponding to each time index are sequentially spliced ​​together to generate a single time series form of multimodal one-dimensional discrete label sequence.

[0040] Optionally, the construction of a two-layer scalable xLSTM model consisting of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module includes:

[0041] The input label sequence is constructed by arranging the total sequence of multimodal one-dimensional discrete labels in ascending order of sequence position indices.

[0042] A micro-layer scalable xLSTM network is constructed. Each multimodal one-dimensional discrete label in the input label sequence is converted into an input embedding vector through a label embedding mapping function. The relative delay position embedding vector is superimposed on the input embedding vector to obtain the input embedding sequence.

[0043] The input embedding sequence is divided into multiple grouped embedding sub-vectors according to the embedding dimension. A groupable gating operation is performed on each grouped embedding sub-vector to obtain dynamically shrunken grouped embedding sub-vectors. All dynamically shrunken grouped embedding sub-vectors are concatenated to form the gated input embedding sequence.

[0044] The gated input embedding sequence is input into multiple stacked xLSTM blocks in the micro-layer scalable xLSTM network in ascending order of sequence position index. Token-level state transfer is performed in ascending order of sequence position index to obtain the local context feature sequence output by the micro-layer scalable xLSTM network.

[0045] A macro-layer scalable xLSTM network is constructed, and the local context feature sequence is divided into multiple continuous segments according to a preset dynamic truncation boundary. The local context feature sequence corresponding to each segment is averaged and aggregated to obtain the segment feature vector. The block covariance matrix is ​​constructed based on the segment feature vector to remember the state.

[0046] In the macro-layer scalable xLSTM network, the block covariance matrix memory state of the previous segment is exponentially decayed, and the segment feature vector of the current segment is written on the exponentially decayed block covariance matrix memory state to obtain the block covariance matrix memory state corresponding to the current segment.

[0047] The fragment feature vector and block covariance matrix memory state corresponding to the current fragment are input into the state update unit of the macro-layer scalable xLSTM network to recursively update the macro-layer state and obtain the macro-layer context feature vector.

[0048] To construct a context bridging module, in order to follow the causal temporal logic of autoregressive generation, the macro-level context feature vector of the previous segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a macro-to-micro bridging vector. At the same time, the segment feature vector of the current segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a micro-to-macro bridging vector. The macro-to-micro bridging vector is added to each local context feature sequence corresponding to the current segment to obtain an enhanced local context feature sequence. At the same time, the micro-to-macro bridging vector is added to the segment feature vector to obtain an enhanced segment feature vector.

[0049] The micro-layer scalable xLSTM network and the macro-layer scalable xLSTM network interact in real time through the context bridging module, forming a two-layer loop of detail layer and global layer, and constructing a joint probability prediction head at the output end to build an end-to-end two-layer scalable xLSTM model.

[0050] Optionally, the pre-training of the two-layer scalable xLSTM model using cross-entropy loss and memory alignment loss includes:

[0051] The total sequence of multimodal one-dimensional discrete labels is input into a two-layer scalable xLSTM model in the order of sequence positions. Based on the historical multimodal one-dimensional discrete labels before the current sequence position, the target multimodal one-dimensional discrete label corresponding to the current sequence position is predicted, and the label prediction results corresponding to each sequence position are obtained. The label prediction results corresponding to each sequence position are compared with the corresponding real multimodal one-dimensional discrete labels position by position to construct the cross-entropy loss function.

[0052] Extract the macro-layer context feature vectors and block covariance matrix memory states of the two-layer scalable xLSTM model corresponding to adjacent segments, and construct a memory alignment loss function to constrain the smooth transition of the global plot between adjacent segments;

[0053] The cross-entropy loss value and the memory alignment loss value are weighted and summed to obtain the total training loss value. The parameters in the two-layer scalable xLSTM model are then updated by backpropagation based on the total training loss value, and the pre-training of the two-layer scalable xLSTM model is completed iteratively.

[0054] After pre-training is completed, the two-layer scalable xLSTM model with updated parameters and a multimodal one-dimensional discrete label total sequence input that is longer than the input length of the pre-training stage is trained again in the same way as the pre-training stage, including label prediction, loss calculation and parameter update, to complete the long sequence fine-tuning of the two-layer scalable xLSTM model.

[0055] Optionally, in the short drama generation stage, the trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence, including:

[0056] In the short drama generation stage, the currently generated multimodal one-dimensional discrete tag sequence is arranged in ascending order of sequence position index, and the two-layer scalable xLSTM model is constructed after the current generated input tag sequence is input into the trained model.

[0057] The forward inference process of the two-layer scalable xLSTM model is invoked to perform a sequence-level feature calculation on the current generated input label sequence, and the predicted feature vector corresponding to the end sequence position index of the current generated input label sequence is obtained.

[0058] By performing similarity calculation and normalization on the predicted feature vector and the parameter vector corresponding to each word index in the multimodal one-dimensional discrete tag vocabulary, the predicted probability distribution of the tag corresponding to the next sequence position index is obtained.

[0059] Select the multimodal one-dimensional discrete label corresponding to the vocabulary index with the highest prediction probability from the label prediction probability distribution as the target one-dimensional discrete label for the next time step.

[0060] The target one-dimensional discrete label for the next time step is appended to the end of the currently generated input label sequence, and the total order of the currently generated multimodal one-dimensional discrete labels is updated in the AIGC multimodal short drama generation method list;

[0061] Determine whether the currently generated multimodal one-dimensional discrete label sequence meets the preset termination condition. If the preset termination condition is not met, return to the forward inference process and continue to generate the next time step. If the preset termination condition is met, stop the generation process and output the generated multimodal one-dimensional discrete label sequence.

[0062] Optionally, the modality separation of the generated multimodal one-dimensional discrete marker sequence includes:

[0063] The generated multimodal one-dimensional discrete tag sequence is processed by modality separation according to modality type. Based on the tag source identifier, the multimodal one-dimensional discrete tag sequence is divided into visually dense one-dimensional discrete tag sequence, visually lightweight one-dimensional discrete tag sequence, audio one-dimensional discrete tag sequence and text one-dimensional discrete tag sequence.

[0064] Each lightweight one-dimensional discrete visual label is mapped to a corresponding lightweight continuous visual feature vector sequence. The temporal resolution of the feature domain is aligned with the high frame rate benchmark through temporal interpolation to obtain the global structure prior feature sequence.

[0065] The visually dense one-dimensional discrete marker mapping is decoded, spatial affine parameters are calculated with global structure priors, spatial adaptive modulation is performed on the intermediate feature map, and the fused video frame sequence is output.

[0066] After mapping each one-dimensional discrete audio marker to a corresponding continuous audio feature vector, waveform reconstruction processing is performed to restore the continuous audio waveform.

[0067] Each one-dimensional discrete text marker is mapped to a corresponding sub-word unit, and all sub-word units are concatenated in ascending order of text sequential index to restore the corresponding text content;

[0068] The fused video frame sequence, continuous audio waveform, and text content are output as the initial video frame sequence, initial audio waveform, and initial text content, respectively.

[0069] An AIGC multimodal short drama generation system includes:

[0070] The data receiving and processing module is used to receive and preprocess user-input script text prompts, storyboard descriptions, and interactive control instructions, and to construct a preprocessed multimodal data set.

[0071] The video processing module is used to input each frame of video sequence data in the preprocessed multimodal dataset into the one-dimensional discretization encoding module to obtain visually dense one-dimensional discrete label sequences and visually lightweight one-dimensional discrete label sequences.

[0072] The audio-text processing module is used to input the audio sequence data and text data from the preprocessed multimodal dataset into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, to obtain the audio one-dimensional discrete tag sequence and the text one-dimensional discrete tag sequence.

[0073] The orchestration module is used to alternately orchestrate the visually dense one-dimensional discrete tag sequence, the audio one-dimensional discrete tag sequence, the text one-dimensional discrete tag sequence, and the visually lightweight one-dimensional discrete tag sequence on the same time axis according to a unified timestamp, to generate a multimodal one-dimensional discrete tag sequence.

[0074] A two-layer scalable xLSTM model building module for constructing two-layer scalable xLSTM models;

[0075] The model training module is used to pre-train a two-layer scalable xLSTM model using cross-entropy loss and memory alignment loss.

[0076] The short drama generation module is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence using a trained two-layer scalable xLSTM model to generate a complete multimodal one-dimensional discrete tag sequence.

[0077] The modality separation module is used to perform modality separation on the generated multimodal one-dimensional discrete marker sequence to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content.

[0078] The output module is used to perform frame-level clock synchronization on the initial video frame sequence and the initial audio waveform, and to perform subtitle alignment on the initial text content and the initial video frame sequence based on the text timestamp, generating a target multimodal short drama with synchronized audio and video and synchronized subtitles, and outputting the target multimodal short drama.

[0079] The beneficial effects of this invention are:

[0080] (1) This invention constructs a dual-branch 1DAIGC multimodal short drama generation method Tokenizer mechanism based on visually dense one-dimensional discrete marker sequences and visually lightweight one-dimensional discrete marker sequences. This divides video sequence data into two types of discrete representations: high-frequency detail representation and low-frequency structural representation. During the encoding stage, the dense visual blocks and lightweight visual blocks are discretized and mapped respectively, so that visual information retains both detail texture information and overall structural contour information in the discrete space. This effectively reduces visual information redundancy and improves the structural separability of visual representation during the unified sequence modeling process. During the decoding stage, a global structural prior feature sequence is generated through the visually lightweight one-dimensional discrete marker sequence. A spatial adaptive modulation mechanism is introduced during the hierarchical decoding process of the visually dense one-dimensional discrete marker sequence, so that the high-frequency detail reconstruction process is constrained by the low-frequency structure. This improves the ability of the generated video to balance spatial structural consistency and detail clarity, and avoids structural distortion or detail blurring.

[0081] (2) This invention constructs a two-layer scalable xLSTM model, introducing a hierarchical memory modeling mechanism between the micro-layer scalable xLSTM network Micro-xLSTM and the macro-layer scalable xLSTM network Macro-xLSTM. This enables collaborative modeling of token-level local dependencies and episode-level global dependencies. The micro-layer scalable xLSTM network performs dynamic shrinkage processing on the input embedding sequence in the embedding dimension direction through a groupable gating mechanism, effectively suppressing noise dimensions and enhancing the ability to express effective features. The macro-layer scalable xLSTM network models the segment-level semantic distribution through a block covariance matrix memory structure and achieves smooth transition across segment memories through an exponential handover gating mechanism. While maintaining controllable computational complexity, this invention achieves consistent modeling of local details and global plot, enabling the generated short drama to maintain semantic coherence and plot stability over long periods of time.

[0082] (3) This invention constructs a multimodal one-dimensional discrete marker sequence based on a unified timestamp, and uses autoregressive inference to call templates during the generation stage. Combined with the context bridging module, it realizes bidirectional information interaction between the micro-layer and the macro-layer, enabling the multimodal information to achieve dynamic alignment and co-evolution during the generation process. At the same time, during the decoding stage, through modal separation and structural constraint reconstruction mechanisms, different modal markers in the unified sequence are mapped back to video frame sequences, audio waveforms and text content respectively. Furthermore, cross-modal consistency enhancement is achieved through visual structure prior and detail interpolation fusion. This invention can maintain the consistency of multimodal information in the time axis and semantic space throughout the generation process, enabling the generated short drama to have a higher degree of matching and overall coordination between the visuals, sound and text expression. Attached Figure Description

[0083] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0084] Figure 1 This is a flowchart of an AIGC multimodal short drama generation method proposed in this invention;

[0085] Figure 2 This is a structural block diagram of a two-layer scalable xLSTM model in an AIGC multimodal short drama generation method proposed in this invention. Detailed Implementation

[0086] Example 1: Reference Figures 1-2 A method for generating AIGC multimodal short dramas, comprising:

[0087] Receive and preprocess user input of script text prompts, storyboard descriptions, and interactive control instructions, and construct a preprocessed multimodal data set;

[0088] In this embodiment, the script text prompts, scene descriptions, and interactive control commands input by the user are received and preprocessed, including:

[0089] The system collects user-input script text prompts, storyboard descriptions, and interactive control commands, and aggregates multimodal training data including video sequence data, audio sequence data, and text data to generate a multimodal raw dataset. Frame-level sampling, color space normalization, and noise suppression are performed on the video sequence data in the multimodal raw dataset. Temporal framing and silent segment removal are performed on the audio sequence data. Sentence segmentation and punctuation normalization are performed on the text data to form a preprocessed multimodal dataset.

[0090] Each frame of video sequence data in the preprocessed multimodal dataset is input into a one-dimensional discretization coding module to obtain a visually dense one-dimensional discrete label sequence and a visually lightweight one-dimensional discrete label sequence.

[0091] In this embodiment, each frame of video sequence data in the preprocessed multimodal dataset is input into a one-dimensional discretization coding module, including:

[0092] Video sequence data is read frame by frame in chronological order from the preprocessed multimodal dataset, and the video sequence data at the current moment is defined as the current frame video data;

[0093] In a visually dense one-dimensional discretized encoder, the current frame video data is divided into multiple dense visual blocks of the same size;

[0094] Perform feature mapping operation on each dense visual block to obtain the corresponding dense continuous feature vector;

[0095] The distance between each dense continuous feature vector and all codebook vectors in the visual dense codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the dense continuous feature vector, thus obtaining a visual dense one-dimensional discrete label sequence. The discrepancies are then connected in series according to the increasing order of the time index t to form the overall visual dense one-dimensional discrete label sequence.

[0096] In Example 1, the one-dimensional discretization coding module adopts a multi-resolution visual discrete representation system based on the 1DTokenizer coding mechanism to compress video sequence data from a high-dimensional pixel space into a unified one-dimensional discrete tokenized representation. The one-dimensional discretization coding module includes a visually dense one-dimensional discretization encoder and a visually lightweight one-dimensional discretization encoder. The visually dense one-dimensional discretization encoder and the visually lightweight one-dimensional discretization encoder perform information extraction and discrete mapping at different scales for the same frame of video data to form a complementary visual representation.

[0097] For the input video sequence data, it is read frame by frame in chronological order. The video frame at the current moment is taken as the current frame video data. Each frame of video data is fed into a visual dense one-dimensional discretization encoder and a visual lightweight one-dimensional discretization encoder for parallel processing.

[0098] In the visually dense one-dimensional discretized encoder, fine-grained spatial partitioning is performed on the current frame video data. The image is segmented by a sliding window according to a preset small-scale block size (set to 8×8 pixels, and a non-overlapping sliding window is used in this embodiment), resulting in multiple dense visual blocks that maintain the original pixel arrangement order. Each dense visual block corresponds to local region information in the video frame, including high-frequency visual features such as facial expressions, lip movements, edge details, and local texture structures. The dense visual blocks are mapped to corresponding dense continuous feature vectors. The mapping process is implemented by an encoding network based on a visual autoencoder training model. The encoding network is trained by minimizing the video reconstruction error, so that the output continuous features represent the local spatial structure information of the original video frame. Each dense continuous feature vector is matched with all codebook vectors in the pre-trained visually dense codebook. The index corresponding to the codebook vector with the smallest distance is selected as the discrete representation of the feature vector. By performing discretization mapping operations on all dense visual blocks, the visually dense one-dimensional discrete label sequence corresponding to the current frame is obtained.

[0099] The visually dense codebook is a set of discrete representations obtained by constructing a visual autoencoder training model based on video sequence data in a preprocessed multimodal dataset and training it with video reconstruction error as the optimization objective. The visually dense one-dimensional discrete label sequence is obtained by performing minimum distance matching on each dense continuous feature vector.

[0100] In the visual lightweight one-dimensional discretization encoder, the current frame video data is divided into large-scale spatial partitions and low-frequency structure aggregation is performed to divide the current frame video data into multiple lightweight visual blocks with a size larger than dense visual blocks.

[0101] Perform feature mapping operations on each lightweight visual block to obtain the corresponding lightweight continuous feature vector;

[0102] The distance between each lightweight continuous feature vector and all codebook vectors in the visual lightweight codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the lightweight continuous feature vector, resulting in a visual lightweight one-dimensional discrete label sequence. These labels are then concatenated in ascending order of time index t to form the overall visual lightweight one-dimensional discrete label sequence.

[0103] The visual lightweight one-dimensional discrete label sequence is obtained by performing minimum distance matching on each lightweight continuous feature vector.

[0104] In Example 1, in the lightweight one-dimensional discretized visual encoder, large-scale spatial partitioning and low-frequency structure extraction are performed on the same frame of video data. The current frame of video data is divided according to the block size (set to 32×32 pixels, covering an area 16 times that of dense visual blocks). Through downsampling and structure aggregation operations, only the main contour information and scene structure information in the image are retained, while local detail changes are ignored, resulting in multiple lightweight visual blocks. Each lightweight visual block corresponds to a large-scale region in the video frame and is used to describe the overall composition, background layout, and scene semantic structure. The lightweight visual blocks are mapped to corresponding lightweight continuous feature vectors. The mapping process is implemented by a visual autoencoder training model that introduces low-frequency structure preservation constraints, so that the encoding result can maintain the overall contour consistency and structural stability of the video frame while reducing spatial resolution.

[0105] Downsampling and structural aggregation operations can be implemented by using a cascaded 3D convolutional kernel with a stride of 4 and an average pooling layer with Gaussian smoothing to filter out high-frequency noise. Each lightweight continuous feature vector is matched with all codebook vectors in the lightweight visual codebook, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete representation of the lightweight continuous feature vector. By performing a discretization mapping operation on all lightweight visual blocks, the lightweight one-dimensional discrete label sequence corresponding to the current frame is obtained.

[0106] The audio sequence data and text data in the preprocessed multimodal dataset are input into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, to obtain the audio one-dimensional discrete tag sequence and the text one-dimensional discrete tag sequence.

[0107] In this embodiment, the audio sequence data and text data in the preprocessed multimodal dataset are input into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, including:

[0108] Audio sequence data is read from the preprocessed multimodal dataset in chronological order, and the audio sequence data corresponding to the t-th time index is defined as the audio data at the current time.

[0109] The audio data at the current moment is input into the audio one-dimensional discretization encoder, and the audio data at the current moment is processed into equal-duration frames to obtain multiple audio encoded frames of the same length.

[0110] In Example 1, the audio one-dimensional discretization encoder is an encoding model obtained by constructing an audio autoencoder training model and optimizing the parameters based on the audio sequence data in the preprocessed multimodal dataset. The audio autoencoder training model uses the reconstruction error of the audio sequence data as the optimization target, so that the one-dimensional discrete label of the encoded output can characterize the temporal structural features and acoustic variation features of the audio sequence data.

[0111] Each audio encoded frame is input into the feature mapping function in the audio one-dimensional discretization encoder to obtain the corresponding audio continuous feature vector. The distance between each audio continuous feature vector and all codebook vectors in the audio codebook is calculated one by one, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the audio continuous feature vector to obtain the audio one-dimensional discrete label sequence corresponding to the audio data at the current time.

[0112] In Example 1, the audio codebook is a discrete representation set obtained by constructing an audio autoencoder training model and training it with audio reconstruction error as the optimization objective, based on the audio sequence data in the preprocessed multimodal dataset. The one-dimensional discrete audio label sequence is obtained by performing minimum distance matching on each audio continuous feature vector.

[0113] The audio one-dimensional discrete tag sequences corresponding to each time index t are sequentially connected in ascending order of time index t to form the overall audio one-dimensional discrete tag sequence.

[0114] The overall audio one-dimensional discrete token sequence maintains the same time index alignment relationship as the video sequence data. The alignment mapping rule is as follows: to address the issue of inconsistent audio and video sampling rates, an audio token packaging method is used to group multiple audio discrete tokens into the same video frame time window t to ensure strict alignment of modalities on the same time axis.

[0115] Read text data sequentially from the preprocessed multimodal dataset, and define the text data corresponding to the r-th text sequential index as the current text segment;

[0116] Input the current text segment into the sub-word encoder, perform character-level splitting and sub-word matching on the current text segment, and obtain the sub-word unit sequence corresponding to the current text segment;

[0117] In Example 1, the sub-word segmentation encoder is an encoding model obtained by constructing a sub-word vocabulary learning model and performing sub-word frequency statistics and merging rule optimization based on the text data in the preprocessed multimodal dataset. The sub-word unit sequence output by the sub-word segmentation encoder is used to represent the local semantic structure in the text data.

[0118] Map each sub-word unit sequence corresponding to the current text segment to a word index in the text sub-word vocabulary, and convert each sub-word unit sequence into a corresponding discrete tag to obtain the text one-dimensional discrete tag sequence corresponding to the current text segment;

[0119] The one-dimensional discrete text marker sequences corresponding to each text sequence index r are concatenated in ascending order of text sequence index r to form the overall one-dimensional discrete text marker sequence.

[0120] The timeline mapping of text data is not continuously aligned frame by frame. Instead, it is based on the start and end timestamps of the script dialogue, anchoring the entire text's one-dimensional discrete marker sequence within a specific time window of the multimodal timeline. The remaining intervals without text time are filled with placeholder markers.

[0121] The visually dense one-dimensional discrete label sequence, the audio one-dimensional discrete label sequence, the text one-dimensional discrete label sequence, and the visually lightweight one-dimensional discrete label sequence are alternately arranged on the same time axis according to a unified timestamp to generate a multimodal one-dimensional discrete label sequence.

[0122] In this embodiment, the visually dense one-dimensional discrete label sequence, the audio one-dimensional discrete label sequence, the text one-dimensional discrete label sequence, and the visually lightweight one-dimensional discrete label sequence are alternately arranged on the same time axis according to a unified timestamp, including:

[0123] Using the time index t of the video sequence data as a unified timestamp reference, the overall visually dense one-dimensional discrete label sequence, the overall audio one-dimensional discrete label sequence, and the overall visually lightweight one-dimensional discrete label sequence are aligned by time index. The overall text one-dimensional discrete label sequence is then mapped to the corresponding time index interval according to the text order index r, thus obtaining the visually dense one-dimensional discrete label subsequence, the audio one-dimensional discrete label subsequence, the text one-dimensional discrete label subsequence, and the visually lightweight one-dimensional discrete label subsequence.

[0124] In Example 1, based on the fact that the overall audio one-dimensional discrete label sequence maintains the same time index alignment relationship with the video sequence data, and using the time index t of the video sequence data as a unified timestamp reference, time-index partitioning processing is performed on the overall visual dense one-dimensional discrete label sequence, the overall audio one-dimensional discrete label sequence, and the overall visual lightweight one-dimensional discrete label sequence.

[0125] The time-index partitioning process is as follows: Based on the value range of time index t, extract all visually dense one-dimensional discrete markers with time index equal to t from the overall visually dense one-dimensional discrete marker sequence and form corresponding visually dense one-dimensional discrete marker subsequences; extract all audio one-dimensional discrete markers with time index equal to t from the overall audio one-dimensional discrete marker sequence and form corresponding audio one-dimensional discrete marker subsequences; and extract all visually lightweight one-dimensional discrete markers with time index equal to t from the overall visually lightweight one-dimensional discrete marker sequence and form corresponding visually lightweight one-dimensional discrete marker subsequences.

[0126] Simultaneously, time interval mapping processing is performed on the overall one-dimensional discrete text tag sequence according to the text order index r. The time interval mapping processing is as follows: based on the start time index and end time index of the text segment corresponding to each text order index r in the video sequence data, all one-dimensional discrete text tags corresponding to the corresponding text segment are assigned to the corresponding time index interval, so that each time index t located in the corresponding time index interval corresponds to at least one one-dimensional discrete text tag. Under the unified timestamp constraint, visually dense one-dimensional discrete tag subsequence, audio one-dimensional discrete tag subsequence, text one-dimensional discrete tag subsequence, and visually lightweight one-dimensional discrete tag subsequence are obtained.

[0127] For the visually dense one-dimensional discrete label subsequence, audio one-dimensional discrete label subsequence, text one-dimensional discrete label subsequence and visually lightweight one-dimensional discrete label subsequence corresponding to the t-th time index, sequential concatenation is performed according to a preset alternating order, and all labels in each subsequence are connected sequentially to obtain the local multimodal one-dimensional discrete label sequence corresponding to the t-th time index.

[0128] Following the ascending order of time index t, the local multimodal one-dimensional discrete label sequences corresponding to each time index are sequentially spliced ​​together to generate a single time series form of multimodal one-dimensional discrete label sequence.

[0129] In Example 1, the total sequence of multimodal one-dimensional discrete labels represents the ordered connection result of all local multimodal one-dimensional discrete label sequences arranged in time index order under a unified timestamp constraint.

[0130] The generated multimodal one-dimensional discrete label sequence is subjected to a sequence consistency check. The sequence consistency check is performed position by position to check whether the adjacent labels before and after any label in the multimodal one-dimensional discrete label sequence satisfy the preset alternation order relationship under the unified timestamp constraint. When all labels satisfy the preset alternation order relationship under the unified timestamp constraint, the multimodal one-dimensional discrete label sequence is output to the input sequence of the two-layer scalable xLSTM model.

[0131] Construct a two-layer scalable xLSTM model consisting of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module;

[0132] In this embodiment, the total sequence of multimodal one-dimensional discrete markers is arranged in ascending order of sequence position index to construct the input marker sequence;

[0133] In Example 1, the input label sequence is composed of multimodal one-dimensional discrete labels corresponding to the sequence position index. The input label sequence is used as the input sequence of the two-layer scalable xLSTM model, and the sequence position index is used to indicate the position order of the multimodal one-dimensional discrete labels in the input label sequence.

[0134] A micro-layer scalable xLSTM network is constructed. Each multimodal one-dimensional discrete label in the input label sequence is converted into an input embedding vector through a label embedding mapping function. The relative delay position embedding vector is superimposed on the input embedding vector to obtain the input embedding sequence.

[0135] In Example 1, the delay position difference is calculated based on the difference between the current sequence position index and the sequence position index corresponding to the starting position of the local time window. The delay position difference is input into the relative delay position embedding mapping function to obtain the relative delay position embedding vector. The input embedding vector and the corresponding relative delay position embedding vector are added element by element and connected in ascending order of the sequence position index to form the input embedding sequence.

[0136] The relative delay position embedding mapping function is a trainable embedding mapping function constructed based on a preset range of delay position difference values. The delay position difference is obtained by subtracting the sequence position index corresponding to the starting position of the local time window from the current sequence position index. The relative delay position embedding mapping function is embodied in a pre-initialized learnable lookup table matrix.

[0137] The input embedding sequence is divided into multiple grouped embedding sub-vectors according to the embedding dimension. A groupable gating operation is performed on each grouped embedding sub-vector to obtain dynamically shrunken grouped embedding sub-vectors. All dynamically shrunken grouped embedding sub-vectors are concatenated to form the gated input embedding sequence.

[0138] In Example 1, each grouped embedding subvector is linearly mapped to the corresponding gating weight matrix and gating bias vector and then input into the Sigmoid activation function to obtain gating coefficients with values ​​ranging from 0 to 1. The gating coefficients are then multiplied element-wise with the corresponding grouped embedding subvectors to obtain dynamically shrunken grouped embedding subvectors. All dynamically shrunken grouped embedding subvectors are then concatenated according to the original grouping order to form the gating input embedding sequence.

[0139] The grouped embedding sub-vectors are obtained by dividing the input embedding vector into equal-length parts along the embedding dimension. Each grouped embedding sub-vector corresponds to an independent gating weight matrix and gating bias vector. The dynamically shrunk grouped embedding sub-vectors are used to adaptively suppress or retain information in different dimensions of the input embedding vector.

[0140] The gated input embedding sequence is input into multiple stacked xLSTM blocks in the micro-layer scalable xLSTM network in ascending order of sequence position index. Token-level state transfer is performed in ascending order of sequence position index to obtain the local context feature sequence output by the micro-layer scalable xLSTM network.

[0141] In Example 1, at each sequence position index n, the current input vector, along with the hidden state vector and memory state vector corresponding to the previous sequence position index n-1, are input into the state update function corresponding to the xLSTM block. The hidden state vector and memory state vector corresponding to the current sequence position index n are calculated. The hidden state vector output by the current xLSTM block is used as the input vector of the next xLSTM block at the same sequence position index n. This process is passed layer by layer according to the stacking order of the xLSTM blocks to complete the serial state update processing of multiple xLSTM blocks. The hidden state vectors output by the last xLSTM block at each sequence position index n are then concatenated in ascending order of sequence position index n to form the local context feature sequence output by the micro-layer scalable xLSTM network.

[0142] The state update function obtains the hidden state vector and memory state vector corresponding to the current sequence position index n by performing gating calculation and state fusion operation on the current input vector, the hidden state vector corresponding to the previous sequence position index n-1, and the memory state vector. These vectors are used to represent the context dependency of the current multimodal one-dimensional discrete label in the local time range.

[0143] A macro-layer scalable xLSTM network is constructed, and the local context feature sequence is divided into multiple continuous segments according to a preset dynamic truncation boundary. The local context feature sequence corresponding to each segment is averaged and aggregated to obtain the segment feature vector. The block covariance matrix is ​​constructed based on the segment feature vector to remember the state.

[0144] In Example 1, the dynamic truncation boundary is a sequence boundary point generated based on a preset fixed marker step-size window or a discrete marker in a visual lightweight one-dimensional discrete marker sequence that represents scene switching, thereby realizing the division of continuous segments that conforms to the storyboard logic of a physical short drama.

[0145] For each segment, sum all local context feature vectors in the local context feature sequence and divide by the number of sequence position indices contained in the segment to obtain the segment feature vector. Based on the segment feature vector, construct the block covariance matrix to remember the state.

[0146] The fragment feature vector corresponding to the q-th fragment is obtained by summing all local context feature vectors vector by vector and dividing by the number of sequence position indices contained in the fragment. The fragment feature vector is used to represent the overall semantic distribution of multimodal one-dimensional discrete markers in the local context within the fragment.

[0147] The segment feature vector corresponding to the q-th segment is transposed and multiplied with itself to obtain the corresponding covariance contribution matrix. The covariance contribution matrix is ​​then divided into multiple sub-blocks according to a preset embedding dimension. Only the diagonal sub-block elements in each sub-block are retained as block diagonal information. A portion of the off-diagonal sub-blocks are selected within a preset remote location index range as sparse remote slots. The retained block diagonal information and the sparse remote slots are combined to obtain the block covariance matrix memory state corresponding to the q-th segment.

[0148] In the macro-layer scalable xLSTM network, the block covariance matrix memory state of the previous segment is exponentially decayed, and the segment feature vector of the current segment is written on the exponentially decayed block covariance matrix memory state to obtain the block covariance matrix memory state corresponding to the current segment.

[0149] In Example 1, the exponential handover gating coefficient is obtained by performing an exponential function operation on the gating decay parameter corresponding to the current segment. The exponential handover gating coefficient is then multiplied element-wise with the memory state of the block covariance matrix corresponding to the previous segment to obtain the memory state of the block covariance matrix after exponential decay. The gating decay parameter is a non-negative real number calculated by parametric mapping based on the segment feature vector corresponding to the current segment. It is used to control the retention ratio of the memory state of the block covariance matrix corresponding to the previous segment when the current segment is updated.

[0150] Based on the memory state of the exponentially decaying block covariance matrix, the segment eigenvector corresponding to the current segment is multiplied by its transpose to obtain the covariance contribution matrix corresponding to the current segment.

[0151] The covariance contribution matrix is ​​superimposed onto the exponentially decayed block covariance matrix memory state to obtain the joint update matrix. Block retention processing and sparse retention processing are performed simultaneously on the joint update matrix. Block retention processing retains only the diagonal sub-block elements of each sub-block in the joint update matrix. Sparse retention processing selects some off-diagonal sub-blocks as sparse remote slots within a preset remote position index range. The results of block retention processing and sparse retention processing are combined to obtain the block covariance matrix memory state corresponding to the current segment.

[0152] ;

[0153] in, This represents the block retention operator that retains only the diagonal terms of the block covariance matrix memory state. This indicates a sparsity retention operator that only retains remote sparse slots. Represents the feature vector of a segment The transpose of , This represents the memory state of the block covariance matrix after exponential decay. This represents the memory state of the block covariance matrix corresponding to the current segment.

[0154] The fragment feature vector and block covariance matrix memory state corresponding to the current fragment are input into the state update unit of the macro-layer scalable xLSTM network to recursively update the macro-layer state and obtain the macro-layer context feature vector.

[0155] In Example 1, the segment feature vector corresponding to the current segment is concatenated with the vector representation of the block covariance matrix memory state after expansion to obtain the fused input vector of the current segment. The fused input vector and the macro-layer context feature vector corresponding to the previous segment are input together into the state update unit of the macro-layer scalable xLSTM network. The state update function in the state update unit obtains the macro-layer context feature vector corresponding to the current segment through gating calculation and state update operation.

[0156] The state update function performs gating control on the state fusion process of the fused input vector and the macro-layer context feature vector corresponding to the previous segment, thereby achieving adaptive fusion of historical macro-layer context information and current segment information.

[0157] To construct a context bridging module, in order to follow the causal temporal logic of autoregressive generation, the macro-level context feature vector of the previous segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a macro-to-micro bridging vector. At the same time, the segment feature vector of the current segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a micro-to-macro bridging vector. The macro-to-micro bridging vector is added to each local context feature sequence corresponding to the current segment to obtain an enhanced local context feature sequence. At the same time, the micro-to-macro bridging vector is added to the segment feature vector to obtain an enhanced segment feature vector.

[0158] In Example 1, the macro-layer context feature vector corresponding to the previous segment is linearly mapped through a low-rank projection matrix to obtain the low-dimensional projection vector corresponding to the macro-layer context feature vector. Then, a rotation position encoding transformation is performed on the low-dimensional projection vector according to the segment position index corresponding to the previous segment to obtain the macro-layer to micro-layer bridging vector.

[0159] The low-rank projection matrix is ​​a mapping matrix with a dimension smaller than the original dimension of the macro-layer context feature vector. The low-dimensional projection vector is obtained by performing matrix multiplication on the macro-layer context feature vector. The rotation position encoding is obtained by performing pairwise rotation transformations on each dimension component of the low-dimensional projection vector according to a preset angular frequency. It is used to inject fragment-level position information into the projection space. The fragment feature vector corresponding to the current fragment is linearly mapped through another low-rank projection matrix to obtain the low-dimensional projection vector corresponding to the fragment feature vector. Then, the rotation position encoding transformation is performed on the low-dimensional projection vector according to the fragment position index corresponding to the current fragment to obtain the micro-layer to macro-layer bridging vector.

[0160] The macro-to-micro bridging vector is aligned according to the fragment index q, and vector-wise addition is performed on all local context feature vectors corresponding to the current fragment. The macro-to-micro bridging vector is added to each local context feature vector in the current fragment to obtain an enhanced local context feature sequence, in order to avoid causal leakage of future information. At the same time, the micro-to-macro bridging vector and the fragment feature vector corresponding to the current fragment are added element-wise to obtain an enhanced fragment feature vector.

[0161] The macro-to-micro bridging vector is used to inject global context information into the micro-layer scalable xLSTM network, while the micro-to-macro bridging vector is used to feed back local detail information to the macro-layer scalable xLSTM network, thus forming a two-way information exchange.

[0162] The micro-layer scalable xLSTM network and the macro-layer scalable xLSTM network interact in real time through the context bridging module, forming a two-layer loop of detail layer and global layer, and constructing a joint probability prediction head at the output end to build an end-to-end two-layer scalable xLSTM model.

[0163] In Example 1, the joint probability prediction head includes a linear projection mapping layer and a Softmax normalization layer. The joint probability prediction head projects the enhanced local context feature sequence into a high-dimensional space, maps it to the cross-modal unified tag vocabulary space, and outputs the joint probability distribution of various multimodal one-dimensional discrete tags corresponding to the next time step.

[0164] The dual-layer scalable xLSTM algorithm constructed in this implementation decouples the architecture in multimodal short drama generation by finely controlling the local audio-visual coherence at the micro-layer and coordinating the overall plot direction at the macro-layer. Combined with a block covariance matrix memory structure with significantly reduced dimensionality, it breaks through the computational power bottleneck in the generation of ultra-long videos, making it possible to stream ultra-long short dramas of several minutes with limited computing power. At the same time, the introduced exponential transition gating and the temporal bridging mechanism of fused rotation position encoding not only eliminates screen tearing and state abrupt changes when switching scenes in short dramas, but also achieves precise forward guidance of the global plot on the consistency of character costumes, and the alignment of lines and lip movements, under the premise of strictly following autoregressive causal logic. It can generate high-quality multimodal short dramas with high fidelity that span ultra-long contexts, with absolutely consistent characters and perfect audio-visual synchronization.

[0165] The total sequence of multimodal one-dimensional discrete labels is used as input, and the pre-training of the two-layer scalable xLSTM model is completed using cross-entropy loss and memory alignment loss.

[0166] In this embodiment, the pre-training of the two-layer scalable xLSTM model is completed using cross-entropy loss and memory alignment loss, including:

[0167] The total sequence of multimodal one-dimensional discrete labels is input into a two-layer scalable xLSTM model in the order of sequence positions. Based on the historical multimodal one-dimensional discrete labels before the current sequence position, the target multimodal one-dimensional discrete label corresponding to the current sequence position is predicted, and the label prediction results corresponding to each sequence position are obtained. The label prediction results corresponding to each sequence position are compared with the corresponding real multimodal one-dimensional discrete labels position by position to construct the cross-entropy loss function.

[0168] Extract the macro-layer context feature vectors and block covariance matrix memory states of the two-layer scalable xLSTM model corresponding to adjacent segments, and construct a memory alignment loss function to constrain the smooth transition of the global plot between adjacent segments;

[0169] ;

[0170] in, This represents the memory alignment loss value. Indicates the total number of segments. and These represent adjacent macro-layer context feature vectors, Denotes the Frobenius norm of a matrix. and These are the preset loss weight hyperparameters.

[0171] The cross-entropy loss value and the memory alignment loss value are weighted and summed to obtain the total training loss value. The parameters in the two-layer scalable xLSTM model are then updated by backpropagation based on the total training loss value, and the pre-training of the two-layer scalable xLSTM model is completed iteratively.

[0172] After pre-training is completed, the two-layer scalable xLSTM model with updated parameters and a multimodal one-dimensional discrete label total sequence input that is longer than the input length of the pre-training stage is trained again in the same way as the pre-training stage, including label prediction, loss calculation and parameter update, to complete the long sequence fine-tuning of the two-layer scalable xLSTM model.

[0173] In the short drama generation stage, a trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence to generate a complete multimodal one-dimensional discrete tag sequence.

[0174] In this embodiment, during the short drama generation stage, a trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence, including:

[0175] In the short drama generation stage, the currently generated multimodal one-dimensional discrete tag sequence is arranged in ascending order of sequence position index, and the two-layer scalable xLSTM model is constructed after the current generated input tag sequence is input into the trained model.

[0176] The forward inference process of the two-layer scalable xLSTM model is invoked to perform a sequence-level feature calculation on the current generated input label sequence, and the predicted feature vector corresponding to the end sequence position index of the current generated input label sequence is obtained.

[0177] In Example 1, the forward inference process is as follows: the currently generated input tag sequence is sequentially input into the micro-layer scalable xLSTM network, the macro-layer scalable xLSTM network, and the context bridge module ContextBridge. Token-level state transfer, fragment-level state update, and cross-layer information interaction processing are performed in ascending order of sequence position index, and the predicted feature vector corresponding to the end sequence position index of the currently generated input tag sequence is output.

[0178] By performing similarity calculation and normalization on the predicted feature vector and the parameter vector corresponding to each word index in the multimodal one-dimensional discrete tag vocabulary, the predicted probability distribution of the tag corresponding to the next sequence position index is obtained.

[0179] In Example 1, the label prediction probability distribution is obtained by performing a one-to-one matching calculation between the predicted feature vector and the parameter vector corresponding to each vocabulary index, and is converted into a probability form through exponential normalization. The label prediction probability distribution is used to represent the generation probability of each candidate multimodal one-dimensional discrete label under the current generation conditions.

[0180] Select the multimodal one-dimensional discrete label corresponding to the vocabulary index with the highest prediction probability from the label prediction probability distribution as the target one-dimensional discrete label for the next time step.

[0181] The target one-dimensional discrete label for the next time step is appended to the end of the currently generated input label sequence, and the total sequence of multimodal one-dimensional discrete labels that has been generated is updated.

[0182] Determine whether the currently generated multimodal one-dimensional discrete label sequence meets the preset termination condition. If the preset termination condition is not met, return to the forward inference process and continue to generate the next time step. If the preset termination condition is met, stop the generation process and output the generated multimodal one-dimensional discrete label sequence.

[0183] Modal separation is performed on the generated multimodal one-dimensional discrete marker sequence to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content.

[0184] In this embodiment, modality separation is performed on the generated multimodal one-dimensional discrete marker sequence, including:

[0185] The generated multimodal one-dimensional discrete tag sequence is processed by modality separation according to modality type. Based on the tag source identifier, the multimodal one-dimensional discrete tag sequence is divided into visually dense one-dimensional discrete tag sequence, visually lightweight one-dimensional discrete tag sequence, audio one-dimensional discrete tag sequence and text one-dimensional discrete tag sequence.

[0186] The modality separation process is as follows: each multimodal one-dimensional discrete label corresponding to each sequence position index in the total multimodal one-dimensional discrete label sequence is identified one by one. Based on the encoding source corresponding to it in the construction stage, it is assigned to the corresponding modal one-dimensional discrete label sequence and recombined in ascending order of sequence position index to form the one-dimensional discrete label sequence of each modality.

[0187] Each lightweight one-dimensional discrete visual label is mapped to a corresponding lightweight continuous visual feature vector sequence. The temporal resolution of the feature domain is aligned with the high frame rate benchmark through temporal interpolation to obtain the global structure prior feature sequence.

[0188] The visually dense one-dimensional discrete marker mapping is decoded, spatial affine parameters are calculated with global structure priors, spatial adaptive modulation is performed on the intermediate feature map, and the fused video frame sequence is output.

[0189] In Example 1, each visually dense one-dimensional discrete marker is mapped to a corresponding visually dense continuous feature vector sequence, and input into a multi-layer hierarchical decoding network for upsampling reconstruction. When the multi-layer hierarchical decoding network performs feature decoding, the global structural prior feature sequence is used as a structural constraint condition input to the spatial adaptive modulation unit. The spatial scaling parameter matrix and spatial translation parameter matrix corresponding to each network layer are calculated through affine transformation. The spatial scaling parameter matrix and spatial translation parameter matrix are used to perform channel-wise spatial adaptive feature modulation on the intermediate feature map of the visually dense continuous feature vector sequence at the corresponding layer, and the fused video frame sequence is output.

[0190] After mapping each one-dimensional discrete audio marker to a corresponding continuous audio feature vector, waveform reconstruction processing is performed to restore the continuous audio waveform.

[0191] Each one-dimensional discrete text marker is mapped to a corresponding sub-word unit, and all sub-word units are concatenated in ascending order of text sequential index to restore the corresponding text content;

[0192] The fused video frame sequence, continuous audio waveform, and text content are output as the initial video frame sequence, initial audio waveform, and initial text content, respectively.

[0193] Perform frame-level clock synchronization on the initial video frame sequence and the initial audio waveform, and perform subtitle alignment on the initial text content and the initial video frame sequence based on the text timestamp to generate a target multimodal short drama with synchronized audio and video and synchronized subtitles, and output the target multimodal short drama.

[0194] In Example 1, a global media presentation time reference is established. The system extracts the unified timestamp index assigned during the generation stage of the multimodal one-dimensional discrete marker sequence and maps it to the presentation timestamp and decoding timestamp in the multimedia playback system.

[0195] To achieve frame-level clock synchronization between audio and video, since the reconstructed initial video frame sequence has a fixed spatial frame rate while the initial audio waveform has a continuous sampling rate, the system uses the audio master clock or the system master clock as the synchronization anchor. By comparing the PTS of the video frame with the playback timestamp of the current audio waveform, microsecond-level clock drift compensation is performed. If a slight clock phase shift is detected, the time difference is eliminated through audio resampling or fine-tuning of the video frame, ensuring a strict correspondence between the video image (lip movements) and the audio waveform (speech pronunciation) on the physical timeline.

[0196] The system performs subtitle alignment and rendering. It extracts the initial text content and transforms it into a subtitle data structure with time boundary attributes based on the start and end timestamps anchored to the text token during autoregression generation. In practice, a hard subtitle rendering strategy is adopted, which uses the graphics rendering engine to rasterize and draw the text content directly onto the specified pixel area of ​​the initial video frame according to the corresponding timestamp, or a soft subtitle strategy is adopted to preserve the text content as an independent text track.

[0197] The process involves multiplexing and encapsulating multiple media streams, inputting the clock-synchronized video stream, audio stream, and subtitle track into the multimedia multiplexer, packaging them according to the time interleaving mechanism, outputting the target multimodal short drama file that conforms to the standard streaming media container format, and storing it in a computer-readable storage medium or a specified file path. This completes the end-to-end generation process of the multimodal short drama.

[0198] An AIGC multimodal short drama generation system includes:

[0199] The data receiving and processing module is used to receive and preprocess user-input script text prompts, storyboard descriptions, and interactive control instructions, and to construct a preprocessed multimodal data set.

[0200] The video processing module is used to input each frame of video sequence data in the preprocessed multimodal dataset into the one-dimensional discretization encoding module to obtain visually dense one-dimensional discrete label sequences and visually lightweight one-dimensional discrete label sequences.

[0201] The audio-text processing module is used to input the audio sequence data and text data from the preprocessed multimodal dataset into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, to obtain the audio one-dimensional discrete tag sequence and the text one-dimensional discrete tag sequence.

[0202] The orchestration module is used to alternately orchestrate the visually dense one-dimensional discrete tag sequence, the audio one-dimensional discrete tag sequence, the text one-dimensional discrete tag sequence, and the visually lightweight one-dimensional discrete tag sequence on the same time axis according to a unified timestamp, to generate a multimodal one-dimensional discrete tag sequence.

[0203] A two-layer scalable xLSTM model building module for constructing two-layer scalable xLSTM models;

[0204] The model training module is used to pre-train a two-layer scalable xLSTM model using cross-entropy loss and memory alignment loss.

[0205] The short drama generation module is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence using a trained two-layer scalable xLSTM model to generate a complete multimodal one-dimensional discrete tag sequence.

[0206] The modality separation module is used to perform modality separation on the generated multimodal one-dimensional discrete marker sequence to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content.

[0207] The output module is used to perform frame-level clock synchronization on the initial video frame sequence and the initial audio waveform, and to perform subtitle alignment on the initial text content and the initial video frame sequence based on the text timestamp, generating a target multimodal short drama with synchronized audio and video and synchronized subtitles, and outputting the target multimodal short drama.

[0208] Example 2: During the operation of a multimodal content generation system, the system receives a short drama generation task request. The input content includes script text prompting characters to enter the room and have a dialogue, storyboard description switching between long shot → medium shot → close-up, and interactive control commands with background music gradually increasing.

[0209] The system also retrieves a multimodal training dataset from the training data warehouse, consisting of 10,240 samples. The video sequence data is 720×1280 resolution, 30 frames per second, with a total of approximately 3.07×10⁻⁶ frames. 6 The audio sequence data has a sampling rate of 16000Hz and a total duration of approximately 85200 seconds. The text data contains approximately 186,000 sentences.

[0210] In the data preprocessing stage, the system performs frame-level sampling processing on the video sequence data, extracts 30 frames of image data from a continuous 1-second video segment, performs color space normalization processing on each frame of image, normalizes the RGB pixel values ​​to the [0,1] interval, and removes high-frequency noise through median filtering, reducing the average noise variance of each frame of image from 0.021 to 0.008.

[0211] The audio sequence data is processed by temporal framing, dividing the audio per second into 40 audio frames, each frame being 25 milliseconds long. Silent frames with energy below the threshold of 0.01 are detected and removed, reducing the proportion of silent frames from the original 19.4% to 3.2%.

[0212] Sentence segmentation and punctuation normalization are performed on the text data, and the original sentences with an average length of 23 characters are uniformly split into text fragments with an average length of 12 characters, resulting in a preprocessed multimodal data set.

[0213] In the visual discretization coding stage, the system processes the video frame with the current time index t=15, dividing the video frame with t=15 into 16×16 dense visual blocks, resulting in a total of 3600 dense visual blocks; and performs feature mapping operation on each dense visual block to obtain a dense continuous feature vector with a length of 128 dimensions.

[0214] The system calculates the distance between each dense continuous feature vector and a set of codebook vectors of size 8192 in the visually dense codebook. The average minimum distance in this frame is 0.0132 and the maximum minimum distance is 0.0285, generating 3600 visually dense one-dimensional discrete labels.

[0215] Within the same frame, the system performs visual lightweight discretization coding, dividing the frame into 64×64 lightweight visual blocks, resulting in 225 lightweight visual blocks. This generates 225 lightweight one-dimensional discrete visual tags to represent the overall structural information.

[0216] In the audio discretization coding stage, the system extracts 40 audio coding frames from the audio segment corresponding to time index t=15, performs feature mapping operation on each audio coding frame to obtain a 64-dimensional continuous audio feature vector, and matches it with an audio codebook of size 4096 to obtain 40 one-dimensional discrete audio tags.

[0217] In the text discretization encoding stage, the system reads the text segment corresponding to the text sequence index r=8 into the room, splits it into sub-word units into the room, and maps them to the corresponding word index [125, 873] to generate a one-dimensional discrete tag sequence of text.

[0218] During the multimodal sequence construction phase, the system uses time index t as a unified timestamp and alternately arranges 3600 visual dense markers, 40 audio markers, 225 visual lightweight markers, and corresponding text markers to generate a local multimodal one-dimensional discrete marker sequence with a length of 3867 at time index t=15.

[0219] The system concatenates time indices t from 1 to 900 sequentially to generate a total sequence of multimodal one-dimensional discrete markers, with a total length of approximately 3.48 × 10⁻⁶. 6 A tag.

[0220] During the model building phase, the system constructs a two-layer scalable xLSTM model and inputs the total sequence of multimodal one-dimensional discrete labels into the model.

[0221] At sequence position index n=1200, the system maps the corresponding label to an input embedding vector with an embedding dimension of 256, and divides it into 8 grouped embedding sub-vectors, each with a dimension of 32.

[0222] The gating coefficient was calculated for the third group embedding subvector, and the gating value was 0.72. The amplitude of the corresponding group embedding subvector decreased by about 28% after scaling.

[0223] In the micro-layer xLSTM, the system obtains a hidden state vector norm of 3.84 at the sequence position index n=1200 and a hidden state vector norm of 3.79 at the previous position n-1, with a state change amplitude of 0.05, indicating that the local information is continuous and stable.

[0224] During the macro-level modeling stage, the system divides the sequence into segments, each segment having a length of 60 time steps. At the segment index q=10, the system calculates the segment feature vector, which has a mean of 0.013 and a variance of 0.087.

[0225] The system multiplies the fragment feature vector with itself to generate a covariance contribution matrix, and performs block retention and sparsity retention processing to obtain the memory state of the block covariance matrix, in which the proportion of non-zero elements is approximately 12.6%.

[0226] The memory state of the block covariance matrix of the previous segment is subjected to exponential decay with a decay coefficient of 0.83, so that 83% of the historical information is retained.

[0227] During the generation phase, the system performs prediction at the end index Ng=2450 of the current generated sequence to obtain the predicted feature vector.

[0228] The system performs matching calculations between the predicted feature vector and the multimodal one-dimensional discrete tag vocabulary with a size of 16384, and obtains a maximum probability value of 0.217, corresponding to a vocabulary index of 5482. The system selects the tag corresponding to this index as the target one-dimensional discrete tag for the next time step.

[0229] The system continued the generation process, accumulating approximately 3.5 × 10⁻⁶ tags. 6 One, with a total generation time of 30 seconds.

[0230] During the decoding phase, the system performs mode separation on the total sequence of multimodal one-dimensional discrete tags, and statistically obtains the following from all tags:

[0231] The number of visually dense one-dimensional discrete labels is 3.2 × 10⁻⁶. 6 indivual;

[0232] The number of lightweight one-dimensional discrete labels for visual imaging is 2.0 × 10⁻⁶. 5 indivual;

[0233] The number of one-dimensional discrete audio labels is 3.6 × 10⁻⁶. 4 indivual;

[0234] The number of one-dimensional discrete markers in the text is 1.2 × 10⁻⁶. 4 indivual;

[0235] The system maps the visual lightweight one-dimensional discrete label sequence into a visual lightweight continuous feature vector sequence, and increases the frame rate from 5 frames per second to 30 frames per second through feature domain temporal interpolation to generate a global structural prior feature sequence.

[0236] The visually dense one-dimensional discrete label sequence is mapped to a visually dense continuous feature vector sequence. During the decoding process, spatial affine parameters are calculated, with the spatial scaling parameter averaging 1.12 and the spatial translation parameter averaging 2.3 pixels. Spatial modulation is then performed on the intermediate feature map to generate the fused video frame sequence.

[0237] The one-dimensional discrete audio marker sequence was decoded into a continuous audio waveform, and the signal-to-noise ratio of the recovered audio signal reached 28.6 dB.

[0238] The one-dimensional discrete token sequence of the text was mapped into sub-word units and concatenated into complete text content, with a text recovery accuracy of 94.1%.

[0239] In the comparative experiment, the system used the same input data and was compared with the traditional Transformer method:

[0240] Video structure stability error: 1.9 pixels for this method, 4.7 pixels for the traditional method;

[0241] Audio-visual synchronization error: 21 milliseconds for this method, 68 milliseconds for the traditional method;

[0242] Text matching accuracy: 94.1% for this method, compared to 82.6% for traditional methods;

[0243] Generation time: 14.2 seconds for this method, 27.8 seconds for the traditional method.

[0244] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating AIGC multimodal short dramas, characterized in that, include: Receive and preprocess user input of script text prompts, storyboard descriptions, and interactive control instructions, and construct a preprocessed multimodal data set; Each frame of video sequence data in the preprocessed multimodal dataset is input into a one-dimensional discretization coding module to obtain a visually dense one-dimensional discrete label sequence and a visually lightweight one-dimensional discrete label sequence. The audio sequence data and text data in the preprocessed multimodal dataset are input into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, to obtain the audio one-dimensional discrete tag sequence and the text one-dimensional discrete tag sequence. The visually dense one-dimensional discrete tag sequence, the audio one-dimensional discrete tag sequence, the text one-dimensional discrete tag sequence, and the visually lightweight one-dimensional discrete tag sequence are alternately arranged on the same time axis according to a unified timestamp to generate a multimodal one-dimensional discrete tag sequence. Construct a two-layer scalable xLSTM model consisting of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module; The total sequence of multimodal one-dimensional discrete labels is used as input, and the pre-training of the two-layer scalable xLSTM model is completed using cross-entropy loss and memory alignment loss. In the short drama generation stage, a trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence to generate a complete multimodal one-dimensional discrete tag sequence. Modal separation is performed on the generated multimodal one-dimensional discrete marker sequence to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content. Perform frame-level clock synchronization on the initial video frame sequence and the initial audio waveform, and perform subtitle alignment on the initial text content and the initial video frame sequence based on the text timestamp to generate a target multimodal short drama with synchronized audio and video and synchronized subtitles, and output the target multimodal short drama.

2. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The process of receiving and preprocessing user-input script text prompts, storyboard descriptions, and interactive control commands includes: The system collects user-input script text prompts, storyboard descriptions, and interactive control commands, and aggregates multimodal training data including video sequence data, audio sequence data, and text data to generate a multimodal raw dataset. Frame-level sampling, color space normalization, and noise suppression are performed on the video sequence data in the multimodal raw dataset. Temporal framing and silent segment removal are performed on the audio sequence data. Sentence segmentation and punctuation normalization are performed on the text data to form a preprocessed multimodal dataset.

3. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The step of inputting each frame of video sequence data from the preprocessed multimodal dataset into a one-dimensional discretization encoding module includes: Video sequence data is read frame by frame in chronological order from the preprocessed multimodal dataset, and the video sequence data at the current moment is defined as the current frame video data; In a visually dense one-dimensional discretized encoder, the current frame video data is divided into multiple dense visual blocks of the same size; Perform feature mapping operation on each dense visual block to obtain the corresponding dense continuous feature vector; The distance between each dense continuous feature vector and all codebook vectors in the visual dense codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the dense continuous feature vector, thus obtaining a visual dense one-dimensional discrete label sequence. The discrepancies are then connected in ascending order of time index to form the overall visual dense one-dimensional discrete label sequence. In the visual lightweight one-dimensional discretization encoder, the current frame video data is divided into large-scale spatial partitions and low-frequency structure aggregation is performed to divide the current frame video data into multiple lightweight visual blocks with a size larger than dense visual blocks. Perform feature mapping operations on each lightweight visual block to obtain the corresponding lightweight continuous feature vector; The distance between each lightweight continuous feature vector and all codebook vectors in the lightweight visual codebook is calculated, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the lightweight continuous feature vector, resulting in a lightweight one-dimensional discrete label sequence. These labels are then concatenated in ascending order of time index to form the overall lightweight one-dimensional discrete label sequence.

4. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The step of inputting the preprocessed multimodal dataset's audio sequence data and text data into the audio one-dimensional discretization encoder and the word segmentation encoder, respectively, includes: Audio sequence data is read from the preprocessed multimodal dataset in chronological order, and the audio sequence data corresponding to the time index is defined as the audio data at the current moment. The audio data at the current moment is input into the audio one-dimensional discretization encoder, and the audio data at the current moment is processed into equal-duration frames to obtain multiple audio encoded frames of the same length. Each audio encoded frame is input into the feature mapping function in the audio one-dimensional discretization encoder to obtain the corresponding audio continuous feature vector. The distance between each audio continuous feature vector and all codebook vectors in the audio codebook is calculated one by one, and the index corresponding to the codebook vector with the smallest distance is selected as the discrete label of the audio continuous feature vector to obtain the audio one-dimensional discrete label sequence corresponding to the audio data at the current time. The audio one-dimensional discrete tag sequences corresponding to each time index are sequentially connected in ascending order of time index to form the overall audio one-dimensional discrete tag sequence. Read text data sequentially from the preprocessed multimodal dataset, and define the text data corresponding to the text sequence index as the current text segment; Input the current text segment into the sub-word encoder, perform character-level splitting and sub-word matching on the current text segment, and obtain the sub-word unit sequence corresponding to the current text segment; Map each sub-word unit sequence corresponding to the current text segment to a word index in the text sub-word vocabulary, and convert each sub-word unit sequence into a corresponding discrete tag to obtain the text one-dimensional discrete tag sequence corresponding to the current text segment; The one-dimensional discrete text marker sequences corresponding to each text sequential index are concatenated in ascending order of the text sequential index to form the overall one-dimensional discrete text marker sequence.

5. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The method of alternating the visually dense one-dimensional discrete label sequence, the audio one-dimensional discrete label sequence, the text one-dimensional discrete label sequence, and the visually lightweight one-dimensional discrete label sequence on the same time axis according to a unified timestamp includes: Using the time index of video sequence data as a unified timestamp benchmark, the overall visually dense one-dimensional discrete label sequence, the overall audio one-dimensional discrete label sequence, and the overall visually lightweight one-dimensional discrete label sequence are aligned by time index. The overall text one-dimensional discrete label sequence is then mapped to the corresponding time index interval according to the text order index, resulting in visually dense one-dimensional discrete label subsequences, audio one-dimensional discrete label subsequences, text one-dimensional discrete label subsequences, and visually lightweight one-dimensional discrete label subsequences. The visually dense one-dimensional discrete label subsequence, audio one-dimensional discrete label subsequence, text one-dimensional discrete label subsequence and visual lightweight one-dimensional discrete label subsequence are sequentially spliced ​​according to a preset alternating order, and all labels in each subsequence are connected in sequence to obtain the local multimodal one-dimensional discrete label sequence corresponding to the time index. Following the ascending order of the time indices, the local multimodal one-dimensional discrete label sequences corresponding to each time index are sequentially spliced ​​together to generate a single time series form of the total multimodal one-dimensional discrete label sequence.

6. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The construction of the two-layer scalable xLSTM model, consisting of a micro-layer scalable xLSTM network, a macro-layer scalable xLSTM network, and a context bridging module, includes: The input label sequence is constructed by arranging the total sequence of multimodal one-dimensional discrete labels in ascending order of sequence position indices. A micro-layer scalable xLSTM network is constructed. Each multimodal one-dimensional discrete label in the input label sequence is converted into an input embedding vector through a label embedding mapping function. The relative delay position embedding vector is superimposed on the input embedding vector to obtain the input embedding sequence. The input embedding sequence is divided into multiple grouped embedding sub-vectors according to the embedding dimension. A groupable gating operation is performed on each grouped embedding sub-vector to obtain dynamically shrunken grouped embedding sub-vectors. All dynamically shrunken grouped embedding sub-vectors are concatenated to form the gated input embedding sequence. The gated input embedding sequence is input into multiple stacked xLSTM blocks in the micro-layer scalable xLSTM network in ascending order of sequence position index. Token-level state transfer is performed in ascending order of sequence position index to obtain the local context feature sequence output by the micro-layer scalable xLSTM network. A macro-layer scalable xLSTM network is constructed, and the local context feature sequence is divided into multiple continuous segments according to a preset dynamic truncation boundary. The local context feature sequence corresponding to each segment is averaged and aggregated to obtain the segment feature vector. The block covariance matrix is ​​constructed based on the segment feature vector to remember the state. In the macro-layer scalable xLSTM network, the block covariance matrix memory state of the previous segment is exponentially decayed, and the segment feature vector of the current segment is written on the exponentially decayed block covariance matrix memory state to obtain the block covariance matrix memory state corresponding to the current segment. The fragment feature vector and block covariance matrix memory state corresponding to the current fragment are input into the state update unit of the macro-layer scalable xLSTM network to recursively update the macro-layer state and obtain the macro-layer context feature vector. A context bridging module is constructed. The macro-level context feature vector of the previous segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a macro-to-micro bridging vector. At the same time, the segment feature vector of the current segment is mapped through a low-rank projection matrix and superimposed with rotational position encoding to obtain a micro-to-macro bridging vector. The macro-to-micro bridging vector is added to each local context feature sequence corresponding to the current segment to obtain an enhanced local context feature sequence. At the same time, the micro-to-macro bridging vector is added to the segment feature vector to obtain an enhanced segment feature vector. The micro-layer scalable xLSTM network and the macro-layer scalable xLSTM network interact in real time through the context bridging module, forming a two-layer loop of detail layer and global layer, and constructing a joint probability prediction head at the output end to build an end-to-end two-layer scalable xLSTM model.

7. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The pre-training of the two-layer scalable xLSTM model using cross-entropy loss and memory alignment loss includes: The total sequence of multimodal one-dimensional discrete labels is input into a two-layer scalable xLSTM model in the order of sequence positions. Based on the historical multimodal one-dimensional discrete labels before the current sequence position, the target multimodal one-dimensional discrete label corresponding to the current sequence position is predicted, and the label prediction results corresponding to each sequence position are obtained. The label prediction results corresponding to each sequence position are compared with the corresponding real multimodal one-dimensional discrete labels position by position to construct the cross-entropy loss function. Extract the macro-layer context feature vectors and block covariance matrix memory states of the two-layer scalable xLSTM model corresponding to adjacent segments, and construct a memory alignment loss function to constrain the smooth transition of the global plot between adjacent segments; The cross-entropy loss value and the memory alignment loss value are weighted and summed to obtain the total training loss value. The parameters in the two-layer scalable xLSTM model are then updated by backpropagation based on the total training loss value, and the pre-training of the two-layer scalable xLSTM model is completed iteratively. After pre-training is completed, the two-layer scalable xLSTM model with updated parameters and a multimodal one-dimensional discrete label total sequence input that is longer than the input length of the pre-training stage is trained again in the same way as the pre-training stage, including label prediction, loss calculation and parameter update, to complete the long sequence fine-tuning of the two-layer scalable xLSTM model.

8. The AIGC multimodal short drama generation method according to claim 1, characterized in that, In the short drama generation stage, a trained two-layer scalable xLSTM model is used to perform autoregressive inference on the currently generated multimodal one-dimensional discrete tag sequence, including: In the short drama generation stage, the currently generated multimodal one-dimensional discrete tag sequence is arranged in ascending order of sequence position index, and the two-layer scalable xLSTM model is constructed after the current generated input tag sequence is input into the trained model. The forward inference process of the two-layer scalable xLSTM model is invoked to perform a sequence-level feature calculation on the current generated input label sequence, and the predicted feature vector corresponding to the end sequence position index of the current generated input label sequence is obtained. By performing similarity calculation and normalization on the predicted feature vector and the parameter vector corresponding to each word index in the multimodal one-dimensional discrete tag vocabulary, the predicted probability distribution of the tag corresponding to the next sequence position index is obtained. Select the multimodal one-dimensional discrete label corresponding to the vocabulary index with the highest prediction probability from the label prediction probability distribution as the target one-dimensional discrete label for the next time step. The target one-dimensional discrete label for the next time step is appended to the end of the currently generated input label sequence, and the total sequence of multimodal one-dimensional discrete labels that has been generated is updated. Determine whether the currently generated multimodal one-dimensional discrete label sequence meets the preset termination condition. If the preset termination condition is not met, return to the forward inference process and continue to generate the next time step. If the preset termination condition is met, stop the generation process and output the generated multimodal one-dimensional discrete label sequence.

9. The AIGC multimodal short drama generation method according to claim 1, characterized in that, The modality separation of the generated multimodal one-dimensional discrete marker sequence includes: The generated multimodal one-dimensional discrete tag sequence is processed by modality separation according to modality type. Based on the tag source identifier, the multimodal one-dimensional discrete tag sequence is divided into visually dense one-dimensional discrete tag sequence, visually lightweight one-dimensional discrete tag sequence, audio one-dimensional discrete tag sequence and text one-dimensional discrete tag sequence. Each lightweight one-dimensional discrete visual label is mapped to a corresponding lightweight continuous visual feature vector sequence. The temporal resolution of the feature domain is aligned with the high frame rate benchmark through temporal interpolation to obtain the global structure prior feature sequence. The visually dense one-dimensional discrete marker mapping is decoded, spatial affine parameters are calculated with global structure priors, spatial adaptive modulation is performed on the intermediate feature map, and the fused video frame sequence is output. After mapping each one-dimensional discrete audio marker to a corresponding continuous audio feature vector, waveform reconstruction processing is performed to restore the continuous audio waveform. Each one-dimensional discrete text marker is mapped to a corresponding sub-word unit, and all sub-word units are concatenated in ascending order of text sequential index to restore the corresponding text content; The fused video frame sequence, continuous audio waveform, and text content are output as the initial video frame sequence, initial audio waveform, and initial text content, respectively.

10. An AIGC multimodal short drama generation system, used to execute the AIGC multimodal short drama generation method according to any one of claims 1-9, characterized in that, include: The data receiving and processing module is used to construct a preprocessed multimodal data set; The video processing module is used to obtain visually dense one-dimensional discrete label sequences and visually lightweight one-dimensional discrete label sequences; The audio-text processing module is used to obtain a one-dimensional discrete audio tag sequence and a one-dimensional discrete text tag sequence. The orchestration module is used to generate a total sequence of multimodal one-dimensional discrete markers; A two-layer scalable xLSTM model building module for constructing two-layer scalable xLSTM models; The model training module is used to pre-train a two-layer scalable xLSTM model using cross-entropy loss and memory alignment loss. The short drama generation module is used to generate a complete multimodal one-dimensional discrete marker sequence; The modality separation module is used to obtain preliminary video frame sequences, preliminary audio waveforms, and preliminary text content; The output module is used to output the target multimodal short drama.

Images