Video sub-mirror layout generation method and system based on autoregression model

Abstract

The invention relates to a video sub-mirror layout generation method and system based on an autoregression model. According to the method, structured layout data is extracted from an original video to construct a multi-task training set containing sub-mirror generation, interactive editing, middle frame filling and sub-mirror verification, and a single autoregression language model is trained to achieve the unified multi-task processing capacity. The model can directly and automatically convert natural language originality into a continuous shot split sequence, and supports that a user interactively edits any frame and then only incrementally generates subsequent content without complete sequence recalculation; and meanwhile, the consistency of layout and text description is automatically checked, and an intermediate transition picture is intelligently completed according to head and tail frames. And finally, the complete sub-mirror sequence is integrated and generated, so that end-to-end automatic generation of the long video sub-mirror from creativity to manuscript formation is realized, and the core problems of lack of long video sequence planning ability, low editing efficiency, high labor cost and incoherent generation rhythm in the prior art are fundamentally solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and in particular relates to a method and system for generating video storyboard layouts based on an autoregressive model. Background Technology

[0002] Currently, video content creation has become a core component of the digital media field, especially in film and television production, animation, advertising, and short video industries. High-quality video storyboard design is a key factor in determining the smoothness and expressiveness of a work. However, under current technological conditions, the planning and generation of storyboards for long videos still faces a series of significant bottlenecks. Most mainstream video generation tools are based on diffusion model technology, and their capabilities are usually limited to generating short video clips of about ten seconds. Users must provide extremely detailed text descriptions or hand-drawn sketches for each single shot to specify the composition. These tools are essentially "snapshot generators" and lack the ability to plan and automatically lay out complete video sequences with long durations and multiple shots.

[0003] Therefore, when producing videos that are several minutes long or even longer, the production process still heavily relies on professional directors and editors manually designing the storyboards. Directors and editors need to plan character positions, object layouts, camera movements, and transitions frame by frame. This process is not only time-consuming—averaging nearly half the entire production cycle—but also demands a high level of professional experience from the staff. A more significant problem is that even with the assistance of existing AI generation tools, the results often have a noticeable "AI feel," manifested in distorted character movements, disjointed physical motion, and illogical camera angles, resulting in mechanically rigid content that is difficult to directly apply to high-quality applications.

[0004] From a technical implementation perspective, existing solutions generally lack the ability to perform structured and lightweight representation of video content. They typically manipulate pixel-level video data directly, resulting in a massive number of model parameters and extremely high computational resource consumption, making it difficult to support efficient inference and real-time interactive editing of long sequences. In addition, some existing auxiliary tools (such as stick figure sketching tools) have limited functionality, only able to handle simple static or single-shot compositions, and cannot support dynamic interactions of multiple characters, complex shot language conversions, and time rhythm control that conforms to animation principles.

[0005] In conclusion, the industry has long lacked a technological solution capable of automatically transforming users' high-level creative intentions (such as a natural language story description) into an editable, adjustable sequence of shots with precise spatiotemporal layout. This has resulted in video creation, especially long-form video creation, being characterized by high barriers to entry, low efficiency, and persistently high costs. Summary of the Invention

[0006] Therefore, it is necessary to provide a video storyboard layout generation method and system based on an autoregressive model to address the aforementioned technical problems.

[0007] Firstly, this application provides a video storyboard layout generation method based on an autoregressive model, including:

[0008] S1. Extract information and track targets from the original video to generate video annotation data including shot text descriptions, camera parameters, character pose sequences, and object bounding box sequences; construct a multi-task training dataset based on the video annotation data, including storyboard generation tasks, interactive editing tasks, in-between frame filling tasks, and storyboard verification tasks.

[0009] S2. Using a multi-task training dataset, train the parameters of a Transformer-based autoregressive language model by minimizing the multi-task joint loss function, so that the autoregressive language model can obtain a unified layout sequence processing capability.

[0010] S3. The user-input creative description text is concatenated with the storyboard generation task identifier and then input into the autoregressive language model. The corresponding initial continuous shot storyboard layout sequence is generated through autoregressive decoding.

[0011] S4. Responding to the user's editing command, the historical shot layout sequence before the frame to be edited, the editing command, and the interactive editing task identifier are concatenated and input into the autoregressive language model, and the edited subsequent shot layout sequence is output through conditional generation.

[0012] S5. Responding to the user's verification command, the sequence of shots to be verified and its content description text are concatenated with the shot verification task identifier and input into the autoregressive language model, and the consistency judgment result is output; where, when the consistency judgment result is qualified, it means that the sequence of shots to be verified conforms to the content description text.

[0013] S6. Respond to the user's fill command, concatenate the specified first and last frame shot layout, the number of fill frames, and the middle frame fill task identifier, and input them into the autoregressive language model. The model then outputs the middle frame shot layout sequence.

[0014] S7. Integrate the initial continuous shot layout sequence, the subsequent shot layout sequence with a consistency judgment result of qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

[0015] Secondly, this application also provides a video storyboard layout generation system based on an autoregressive model, used to implement the method described in the first aspect, the system comprising:

[0016] The video semantic annotation construction module is used to extract information and track targets from the original video, and generate video annotation data including shot text descriptions, camera parameters, human pose sequences and object bounding box sequences. Based on the video annotation data, a multi-task training dataset is constructed, including storyboard generation tasks, interactive editing tasks, intermediate frame filling tasks and storyboard verification tasks.

[0017] The multi-task model training module is used to train the parameters of a Transformer-based autoregressive language model by minimizing the joint loss function of the multi-task training dataset, thereby enabling the autoregressive language model to obtain a unified layout sequence processing capability.

[0018] The creative-driven storyboard generation module is used to concatenate the creative description text input by the user with the storyboard generation task identifier and input it into the autoregressive language model. The corresponding initial continuous shot storyboard layout sequence is generated through autoregressive decoding.

[0019] The interactive storyboard editing module is used to respond to user editing commands. It concatenates the historical shot storyboard layout sequence before the frame to be edited, the editing command, and the interactive editing task identifier, and inputs them into the autoregressive language model. It then outputs the edited subsequent shot storyboard layout sequence through conditional generation.

[0020] The storyboard consistency verification module is used to respond to user verification commands. It concatenates the storyboard layout sequence of the shot to be verified and its content description text with the storyboard verification task identifier and inputs it into the autoregressive language model, and outputs the consistency judgment result. When the consistency judgment result is qualified, it means that the storyboard layout sequence of the shot to be verified conforms to the content description text.

[0021] The intermediate frame intelligent filling module is used to respond to user filling instructions. It concatenates the specified first and last frame shot layout, the number of filling frames, and the intermediate frame filling task identifier and inputs them into the autoregressive language model. The model outputs the intermediate frame shot layout sequence.

[0022] The shot sequence fusion module is used to integrate the initial continuous shot layout sequence, the subsequent shot layout sequence with a consistency judgment result of qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

[0023] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a video storyboard layout generation method based on an autoregressive model as described in the first aspect.

[0024] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a video storyboard layout generation method based on an autoregressive model as described in the first aspect.

[0025] The aforementioned video storyboard layout generation method and system based on an autoregressive model automatically extracts information and tracks targets from the original video, transforming the video content into structured labeled data containing shot text descriptions, camera parameters, character pose sequences, and object bounding box sequences. This data is then used to construct a multi-task training dataset covering four task types: storyboard generation, interactive editing, in-between frame filling, and storyboard verification. Next, an autoregressive language model based on a Transformer architecture is trained using this dataset. The model parameters are optimized through a multi-task joint loss function, enabling the model to uniformly understand and generate structured layout sequences. In practical applications, users only need to input a natural language creative description, and the model, combined with the storyboard generation task identifier, automatically outputs an initial, semantically consistent continuous shot layout sequence. If the user modifies the layout of a frame in the sequence... The system is adjusted to input historical layouts, editing instructions, and interactive editing task identifiers into the same model, generating subsequent layouts incrementally based only on existing context, avoiding full sequence recalculation. Simultaneously, a storyboard verification task verifies the consistency between the generated layouts and text descriptions, ensuring the output content aligns with creative intent. Furthermore, an intermediate frame filling task automatically completes smooth transitions between intermediate frames based on user-specified first and last frames. Finally, all generated and edited layout sequences are integrated to form a complete and coherent long video storyboard plan, achieving end-to-end automation from creative input to storyboard output. This significantly reduces reliance on manual professional storyboard design in long video production, improving creative efficiency and layout consistency. Additionally, lightweight sequence modeling and incremental generation mechanisms reduce computational resource consumption, enhancing practicality in real-time interaction and long sequence generation scenarios. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a video scene layout generation method based on an autoregressive model provided by the present invention;

[0028] Figure 2 This is a schematic diagram of the process of training an autoregressive language model in one optional embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of a video storyboard layout generation system based on an autoregressive model provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] refer to Figure 1 The document presents a flowchart illustrating a video storyboard layout generation method based on an autoregressive model, which includes the following steps:

[0032] S1. Extract information and track targets from the original video to generate video annotation data including shot text descriptions, camera parameters, character pose sequences, and object bounding box sequences; construct a multi-task training dataset based on the video annotation data, including storyboard generation tasks, interactive editing tasks, in-between frame filling tasks, and storyboard verification tasks.

[0033] Specifically, when extracting information from the original video, the original video is first sampled at a preset frame rate. The sampling frame rate can be flexibly adjusted according to the video type to ensure that the extracted information can fully cover the details of human actions and object motion in the video. For human pose extraction, the OpenPose model is used to extract two-dimensional joint coordinates. This model uses a convolutional neural network architecture to extract features from video frames. First, the input frame is downsampled using convolutional layers to generate multi-scale feature maps. Then, a heatmap prediction branch is used to generate heatmaps for each joint point, and a connection probability map between joint points is generated using a limb affinity domain prediction branch. Finally, peak detection is performed on the heatmaps to obtain the two-dimensional coordinates of each joint point. At the same time, the SAM Humanpose model is used in conjunction with monocular vision depth estimation technology to recover the three-dimensional joint coordinates based on the two-dimensional joint coordinates. Specifically, this model introduces the camera intrinsic parameter matrix, combines the pixel coordinates and depth information of the joint points, and uses the triangulation principle to calculate the three-dimensional joint coordinates, thereby achieving accurate representation of human pose.

[0034] For object bounding box extraction, a YOLOv8 model is used for 2D bounding box extraction. This model employs an end-to-end detection architecture, dividing the input frame into several grid cells. Each grid cell is responsible for predicting the bounding box parameters and class probability of objects falling within that cell. Simultaneously, a 6DoF detector is used for 3D bounding box extraction. This detector estimates the object's rotation matrix and translation vector, combining this with the object's 3D dimensions to calculate the object's 3D bounding box parameters, thus achieving accurate representation of the object's spatial location. For generating shot text descriptions, a visual language model is used. This model achieves the fusion of visual and linguistic features through a cross-modal attention mechanism. First, a convolutional neural network is used to extract visual features from video frames. Then, a pre-trained language model is used to generate corresponding text descriptions. Furthermore, by combining the temporal information of the video frames, a sequence of text descriptions for consecutive shots is generated. For target tracking, pedestrian re-identification technology is adopted. This technology extracts the appearance features of people, including color features, texture features and shape features, and uses the cosine similarity algorithm to calculate the matching degree of people's features in adjacent frames. A unique and fixed identifier is assigned to each person. At the same time, a multi-object tracking algorithm is used to track objects to ensure that the object identifiers remain consistent throughout the video sequence and avoid the problem of identifier confusion.

[0035] After completing information extraction and target tracking, the video annotation data is integrated. This data is structured as a combination of shot text descriptions and camera parameters, character icons and corresponding poses, and object icons and corresponding bounding boxes. When constructing a multi-task training dataset based on this video annotation data, the annotation data is split and reorganized according to the needs of different tasks. For the storyboard generation task, shot text descriptions and camera parameters are selected from the video annotation data as model inputs, and character pose sequences and object bounding box sequences are selected as the model's target outputs. For the interactive editing task, editing instructions and historical layout sequences are added to the model inputs, and the edited character pose sequences and object bounding box sequences are used as the target outputs. For the in-between frame filling task, the layout sequences of the first and last frames, camera parameters, and the number of frames to be filled are selected as model inputs, and the layout sequences of the in-between frames are used as the target outputs. For the storyboard verification task, positive and negative sample datasets are constructed. Positive samples are data matching shot text descriptions and layout sequences, while negative samples are data obtained by shuffling layout sequences or replacing text descriptions. Shot text descriptions and layout sequences are used as model inputs, and the consistency judgment results are used as the target outputs. During the dataset construction process, data augmentation was also performed, including randomly adjusting camera parameters, adding Gaussian noise to pose coordinates and bounding box parameters, and randomly shuffling the labeling order of people and objects to improve the model's generalization ability. At the same time, the dataset was divided into training, validation and test sets according to the usual proportions to ensure the effectiveness and reliability of the training process.

[0036] S2. Using a multi-task training dataset, train the parameters of a Transformer-based autoregressive language model by minimizing the multi-task joint loss function, so that the autoregressive language model can obtain a unified layout sequence processing capability.

[0037] Specifically, the Transformer-based autoregressive language model used in this step comprises an encoder and a decoder. The encoder consists of multiple stacked Transformer encoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network. The decoder consists of multiple stacked Transformer decoder layers, each containing a multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network. During the model input phase, different types of input data undergo tokenization. For textual data, a byte-by-byte encoding algorithm is used for word segmentation, converting the text into corresponding token sequences. For numerical data, a numerical tokenization algorithm is used to map numerical values ​​to corresponding token sequences. Simultaneously, unique task identifiers are added for different tasks to guide the model in recognizing different task types.

[0038] Before model training, the model parameters are initialized. The Xavier initialization method is used to initialize the weight matrices of the attention mechanism and the feedforward neural network, ensuring the stability of the model training process. The expression for the multi-task joint loss function can be: . In the formula, α represents the joint loss value of multiple tasks, which is the objective function that needs to be minimized during model training; α, β, γ, and δ are all weight coefficients of the loss function, used to balance the contribution of different task losses to the overall loss, and can be determined by optimization through the validation set. The ID number error loss is used to measure the difference between the model's predicted person and object identifiers and the actual identifiers. L2 loss represents the layout similarity and is used to measure the difference between the model's predicted pose coordinates and bounding box parameters and the true parameters. Represents motion smoothing loss, used to measure the motion coherence of the intermediate frame layout sequence generated by the model; Represents the instruction correctness loss, used to measure the degree of matching between the layout sequence generated by the model and the text description.

[0039] The cross-entropy loss function is used to calculate and quantify the difference between the model's predicted ID and the true ID. Its specific expression is as follows: In the formula, N represents the number of samples in a single training run (i.e., the number of shot sequences in a single batch); K represents the total number of ID categories for people and objects; This represents the true label of the k-th ID in the i-th sample, using one-hot encoding. If the true ID of the i-th sample is k, then... ,otherwise ; This represents the probability that the model predicts the i-th sample belongs to the k-th ID; log represents the natural logarithm, and the negative sign is used to convert the loss value to a non-negative value. The summation and average are used to obtain the ID error loss for a single batch. The purpose of this loss function is to ensure that the model can accurately maintain ID consistency when dealing with scenes with multiple people and multiple objects, and to avoid ID confusion.

[0040] The L2 loss function (Euclidean distance loss) is used for calculation, and its specific expression is as follows: In the formula, M represents the total number of parameters that need to be matched in a single frame layout (including all character pose coordinates and object bounding box parameters); D represents the dimension of each parameter (pose coordinates are 2 or 3-dimensional, and bounding box parameters are 4 or 6-dimensional). This represents the true value of the m-th parameter in the d-th dimension; The value of the m-th parameter in the d-th dimension predicted by the model is represented by the square root of the Euclidean distance of a single parameter. The average value of the summation results in the layout similarity loss. This loss function ensures that the layout sequence generated by the model is consistent with the actual layout sequence in spatial location.

[0041] The mean squared error loss function is used to constrain the motion coherence of the intermediate frame layout sequence. The specific expression is as follows: In the formula, T represents the total number of frames in the shot sequence; T-1 represents the number of pairs of adjacent frames; and M represents the total number of parameters in a single frame layout. This represents the value of the m-th parameter in the t-th frame; This represents the value of the m-th parameter in the (t+1)-th frame. First, the squared difference of the same parameter in adjacent frames is calculated. Then, the average of all parameters in a single frame is calculated, and finally, the average of all adjacent frames is calculated to obtain the motion smoothing loss. This loss function can avoid the problem of sudden changes in action.

[0042] The contrastive loss function is used to measure the matching degree between the layout sequence and the text description. The specific expression is as follows: In the formula, P represents the number of positive samples in a single batch; The interval threshold representing the similarity between positive and negative samples (usually set to 0.5 to distinguish the boundary between positive and negative samples). Represents the feature similarity (calculated using cosine similarity) of the i-th positive sample pair (the matched layout sequence and the text description). Let represent the feature similarity of the i-th negative sample pair (the mismatched layout sequence and text description); by maximizing the positive sample similarity and minimizing the negative sample similarity, we ensure that the layout sequence generated by the model matches the text description.

[0043] During model training, the AdamW optimizer is used to update the model parameters. The parameter update formula is as follows: ,in This represents the model parameters in the t-th training round. This indicates the updated parameters. This represents the current learning rate. This represents the gradient of the loss function with respect to the parameters. This represents the weight decay coefficient (used to prevent overfitting). The learning rate is adjusted using a cosine annealing learning rate scheduling strategy, with the following formula: ,in This represents the learning rate in round t. This represents the initial maximum learning rate. This represents the minimum learning rate. This indicates the maximum number of training epochs. An early stopping strategy is also employed: training stops when the validation set loss no longer decreases after several consecutive epochs, thus preventing overfitting.

[0044] Through the above training process, the autoregressive language model based on Transformer can learn the processing logic of different tasks, obtain a unified layout sequence processing capability, and realize end-to-end processing of tasks such as storyboard generation, interactive editing, in-between frame filling and storyboard verification.

[0045] S3. The user-input creative description text is concatenated with the storyboard generation task identifier and then input into the autoregressive language model. The corresponding initial continuous shot storyboard layout sequence is generated through autoregressive decoding.

[0046] Specifically, the user-input creative description text is natural language text, which can intuitively express the user's creative intent. Before being input into the model, it undergoes preprocessing. First, text cleaning is performed to remove special characters, redundant spaces, and typos. Then, word segmentation is performed, and a byte-pair encoding algorithm is used to convert the text into corresponding tag sequences. The storyboard generation task identifier is a unique tag that guides the model to recognize the current task type as a storyboard generation task. The preprocessed creative description text tag sequence is concatenated with the storyboard generation task identifier to obtain the model's input sequence. The structure of the input sequence is a combination of the storyboard generation task identifier and the creative description text tag sequence.

[0047] After inputting the input sequence into the model, the encoder first extracts features from the input sequence. The multi-head self-attention mechanism in the encoder can capture long-distance dependencies in the text sequence, and its calculation formula is as follows:

[0048]

[0049]

[0050]

[0051] In the formula, Q, K, and V represent the query vector, key vector, and value vector, respectively. , , Let represent the query, key, and value weight matrices for the i-th attention head, respectively; h represents the number of attention heads. Represents the dimensions of the Q and K vectors; The scaling factor is used to avoid the inner product result from being too large; the Softmax function is used to normalize the attention weights. This represents the concatenated weight matrix of the multi-head attention output. The feedforward neural network can perform non-linear transformations on the extracted features, as shown in the formula:

[0052]

[0053] in, , This represents the weight matrix of the feedforward network. , Indicates the bias term. This represents the ReLU activation function, used to introduce non-linear features. The text feature representation output by the encoder is fed into the decoder, which generates an initial sequence of consecutive shot layouts using autoregressive decoding. The probability formula for autoregressive decoding is:

[0054]

[0055] In the formula, This represents the historical marker sequence generated at a known time t-1. Given the encoder input sequence X, the layout marker is generated at time t. The probability of this is the core quantitative indicator of autoregressive decoding, which determines the selection of the optimal label at the current moment. This indicates the layout markers generated at the current moment (such as the markers corresponding to the pose parameters and bounding box parameters); This represents the historical marker sequence generated at time t-1. This represents the encoder input sequence (a sequence of creative description text tags). This represents the text feature sequence output by the encoder; the first MultiHeadAttn represents the decoder's masked self-attention layer (focusing only on historical markers and masking future markers), and the second MultiHeadAttn represents the encoder-decoder attention layer (focusing on the association between text features and historical markers). This represents the weight matrix of the decoder output layer; the Softmax function is used to transform the output into a probability distribution, and the model selects the label with the highest probability as the output at the current time step.

[0056] The core principle of autoregressive decoding is that when the model generates each marker, it relies on the previously generated markers. Specifically, the decoder first generates the first marker (corresponding to the layout parameter start marker of the first frame), then generates the second marker based on the first marker (corresponding to the character pose coordinate parameters of the first frame), and so on, gradually generating complete layout sequence markers.

[0057] During the decoding process, a beam search strategy is used to filter the generated tags in order to balance generation quality and decoding efficiency. The formula for beam search is: ,in This represents the set (bundle) of candidate label sequences retained at time t. Represents the set of all possible labels at the current moment; Indicates that the label is generated at time i. The logarithmic probability is used to avoid numerical underflow; the summation represents the cumulative logarithmic probability of the candidate sequences, and the model selects the top-K candidate sequences with the highest cumulative logarithmic probabilities as the bundle at the current time step (K is the bundle width, set according to conventional decoding). This strategy can improve decoding efficiency while ensuring generation quality and avoid the problems of duplicate or meaningless marking.

[0058] The initial continuous shot layout sequence generated by the model is structured as one line per frame. Each line contains information such as camera parameters, character identifiers, character 2D pose, character 3D pose, object identifiers, object 2D bounding boxes, and object 3D bounding boxes. During the layout sequence generation process, the model incorporates key information from the user's creative description text. For example, when the description includes descriptions related to camera movement, the model automatically adjusts the position parameters in the camera parameters to achieve the effect of camera movement. When the description includes descriptions related to character movement, the model generates corresponding character movement poses, manifested as periodic changes in the character's leg joint coordinates. The periodicity of the character movement pose is determined based on the temporal variation of the joint coordinates, calculated using the following formula: ,in Indicates the period of motion; Indicates the candidate period (frame interval); Indicates the total number of frames in the motion sequence; This represents the value of the d-th dimension of the joint coordinates in frame t; by minimizing the mean square error of the joint coordinates in adjacent periods, the period of the character's movement is determined, ensuring that the generated movement posture conforms to natural laws.

[0059] To ensure that the generated initial sequence of consecutive shots conforms to the principles of video creation, the model also incorporates the camera language knowledge learned during training to automatically add appropriate transition effects, improving the smoothness of the sequence. After generating the initial sequence of consecutive shots, post-processing is required. First, the labeled sequences in the layout sequence are converted into numerical parameters. Then, the numerical parameters are normalized to ensure that their values ​​are within a reasonable range, facilitating subsequent editing and integration. Finally, the post-processed layout sequence is stored in a structured data format for easy viewing and use by users.

[0060] S4. Responding to the user's editing instructions, the historical shot layout sequence before the frame to be edited, the editing instructions, and the interactive editing task identifier are concatenated and input into the autoregressive language model. The edited subsequent shot layout sequence is output through conditional generation.

[0061] Specifically, user editing instructions are either natural language text or parameter adjustment instructions, accurately expressing the user's editing needs. Before being input into the model, these instructions undergo preprocessing. For natural language editing instructions, the same text cleaning and word segmentation methods as in step S3 are used to convert them into a tokenized sequence. For parameter adjustment editing instructions, the parameter values ​​are converted into corresponding tokenized sequences to ensure that the editing instructions can be accurately recognized by the model. The interactive editing task identifier is a unique tag whose purpose is to guide the model to recognize the current task type as an interactive editing task.

[0062] The historical shot layout sequence preceding the edited frame is the layout sequence of the edited frame and all frames preceding it in the initial continuous shot layout sequence generated in step S3. This sequence provides contextual information to the model, ensuring that the edited subsequent shot layout sequence remains coherent with the historical sequence. The historical shot layout sequence marker sequence, the editing instruction marker sequence, and the interactive editing task identifier are concatenated to obtain the model's input sequence. The input sequence is a combination of the interactive editing task identifier, the historical shot layout sequence marker sequence, and the editing instruction marker sequence. After inputting the input sequence into the model, the encoder extracts features from the input sequence, capturing not only key information from the editing instructions but also spatiotemporal features from the historical layout sequence, such as character movement trajectories and camera movement patterns. The decoder then generates the edited subsequent shot layout sequence through conditional generation. The core principle of conditional generation is that when generating the subsequent layout sequence, the model uses the historical layout sequence and editing instructions as constraints to ensure that the generated layout sequence meets the user's editing needs while maintaining coherence with the historical sequence.

[0063] During the generation process, the model first parses the type of editing instruction. If it's a position adjustment instruction, the model directly modifies the pose coordinates or bounding box parameters of the corresponding character or object, and generates the layout sequence of subsequent frames based on the modified parameters, ensuring a smooth transition of motion trajectories. If it's an instruction to add a character or object, the model selects an appropriate pattern from the character interaction patterns or object layout patterns learned from the training data to generate the corresponding layout sequence. Unlike traditional full-sequence regeneration methods, this step uses incremental generation, which only generates the layout sequence of subsequent shots after the edited frame, without needing to regenerate the entire sequence. This effectively saves computational resources and improves editing efficiency. Specifically, the model generates the layout parameters of each subsequent frame step by step based on the layout parameters of the edited frame and the requirements of the editing instruction, ensuring that the motion trajectories between adjacent frames conform to physical laws and avoiding abrupt changes in action.

[0064] After generating the edited sequence of subsequent shot layouts, its coherence is verified by calculating the difference between the pose coordinates and bounding box parameters between the edited frame and the first subsequent frame. If the difference exceeds a preset threshold, the subsequent sequence is smoothed by adjusting the parameters to bring the difference within the threshold range. The preset threshold can be adjusted according to the video type. Through this process, the model can quickly respond to the user's editing commands and generate a sequence of subsequent shot layouts that meets the requirements, enabling interactive editing of the shot layout.

[0065] S5. Responding to the user's verification command, the sequence of shots to be verified and its content description text are concatenated with the shot verification task identifier and input into the autoregressive language model, and the consistency judgment result is output; where, when the consistency judgment result is qualified, it means that the sequence of shots to be verified conforms to the content description text.

[0066] Specifically, the user verification instruction is triggered when the user completes the generation or editing of the storyboard layout and needs to verify whether the layout sequence meets the requirements of the creative description text. The storyboard layout sequence to be verified is the initial layout sequence generated in step S3 or the edited layout sequence generated in step S4, and the content description text is the creative description text entered by the user or the edited description text.

[0067] Before inputting the data into the model, preprocessing operations are performed on the storyboard layout sequence and content description text of the shots to be verified. For the storyboard layout sequence, the numerical parameters are converted into corresponding labeled sequences. For the content description text, the same text cleaning and word segmentation methods as in step S3 are used to convert it into a labeled sequence. The storyboard verification task identifier is a unique identifier, whose purpose is to guide the model to recognize the current task type as a storyboard verification task. The labeled sequence of the storyboard layout sequence, the labeled sequence of the content description text, and the storyboard verification task identifier are concatenated to obtain the input sequence of the model. The structure of the input sequence is a combination of the storyboard verification task identifier, the labeled sequence of the storyboard layout sequence, and the labeled sequence of the content description text.

[0068] After the input sequence is fed into the model, the encoder performs cross-modal feature extraction, extracting the visual spatial features of the layout sequence and the linguistic semantic features of the descriptive text, respectively. Then, a cross-modal attention mechanism is used to fuse these two types of features. The formula for calculating cross-modal attention is as follows:

[0069]

[0070] in, The visual feature matrix representing the layout sequence. A linguistic feature matrix representing a text sequence; , These represent the projection weight matrices for visual features and linguistic features, respectively; Dimensions representing cross-modal features; is the scaling factor; the Softmax function is used to normalize the attention weights, and finally outputs the fused cross-modal features.

[0071] The decoder then calculates a matching score between the layout sequence and the text description based on the fused features. The formula for calculating the matching score is: ,in The sigmoid activation function is represented by the formula: (z is the input variable, and the output range is [0,1]); This represents the weight matrix of the fully connected layer; This represents the fused cross-modal feature vector; represents the bias term of the fully connected layer; score represents the matching score, ranging from [0,1]. A higher score indicates a higher degree of matching between the layout sequence and the text description. Specifically, the model maps the fused features into a one-dimensional vector through a fully connected layer, and then compresses the vector's value range to [0,1] using a sigmoid activation function to obtain the matching score. Among these, the fused features... The formula for generating it is:

[0072]

[0073] in, This represents the global average pooling operation, which sums and averages the feature matrix over the time dimension to obtain the global features; Indicates feature concatenation operation; This represents the concatenated feature fusion weight matrix, used to fuse global visual and linguistic features into a feature vector of a unified dimension.

[0074] The model presets a matching threshold, which is determined through optimization on a validation set. When the matching score is higher than the threshold, the output consistency judgment result is "qualified," indicating that the shot layout sequence to be verified meets the requirements of the content description text. When the matching score is lower than the threshold, the output consistency judgment result is "unqualified," and the reason for the mismatch is also output, providing the user with directions for modification. During the shot verification process, the model can not only verify the overall layout sequence but also perform fine-grained verification of the single-frame layout, achieving accurate verification of the shot layout.

[0075] To improve the reliability of the verification results, the model can also incorporate the camera language knowledge learned during training to verify the rationality of the layout sequence. This includes verifying the smoothness of transitions, the conformity of character movements to motion principles, and the spatial logic of object positions. This ensures that the storyboard layout sequence not only meets the requirements of the text description but also possesses feasibility for actual video creation. When the consistency judgment result is unsatisfactory, the user can re-enter editing instructions to adjust the layout sequence based on the model's output reason for the mismatch. After adjustment, the verification is repeated until the consistency judgment result is satisfactory. This verification process ensures that the generated storyboard layout sequence accurately matches the user's creative intent, improving the quality of the storyboard layout.

[0076] S6. Respond to the user's fill command, concatenate the specified first and last frame shot layout, the number of fill frames, and the middle frame fill task identifier, and input them into the autoregressive language model. The model then outputs the middle frame shot layout sequence.

[0077] Specifically, the trigger condition for the user fill instruction is that the user needs to generate a sequence of intermediate frames between the specified first and last frames in order to achieve a smooth transition of the action. The specified first and last frame shot layout can be manually edited and generated by the user, or it can be selected from the initial layout sequence generated in step S3 or the edited layout sequence generated in step S4. The number of fill frames is the number of intermediate frames that the user specifies to be generated.

[0078] Before inputting the data into the model, preprocessing is performed on the specified first and last frame shot layouts and fill frame counts. For the first and last frame shot layouts, the numerical parameters are converted into corresponding label sequences; for the fill frame counts, they are also converted into corresponding label sequences. The middle frame fill task identifier is a unique identifier used to guide the model in recognizing the current task type as a middle frame fill task. The first and last frame shot layout label sequences, the fill frame count label sequences, and the middle frame fill task identifier are concatenated to obtain the model's input sequence. The input sequence is a combination of the middle frame fill task identifier, the first frame shot layout label sequence, the last frame shot layout label sequence, and the fill frame count label sequence.

[0079] After the input sequence is fed into the model, the encoder extracts features from the input sequence, capturing the character pose features, object bounding box features, and camera parameter features in the layout sequence of the first and last frames. It also captures the numerical features of the fill frame count. The decoder, based on the extracted features, generates the intermediate frame shot layout sequence through autoregressive decoding. The core principle is that the model predicts the motion trajectories between the first and last frames based on the character and object motion patterns learned during training, thus generating a smoothly transitioning intermediate frame layout sequence. During the generation process, the model first interpolates the layout parameters of the first and last frames to obtain the initial parameters of the intermediate frames. The specific formula for linear interpolation is:

[0080]

[0081] in, This represents the initial interpolation result of the m-th parameter in the t-th frame; Indicates the frame number of the first frame; The frame number indicating the last frame; This represents the value of the m-th parameter in the first frame; This represents the value of the m-th parameter in the last frame; t represents the frame number of the intermediate frame. ).

[0082] Subsequently, the initial parameters are optimized and adjusted based on the constraints of the motion smoothing loss function. The optimization objective function can be:

[0083]

[0084] in, This represents the optimized intermediate frame parameters; Indicates the initial interpolation parameters; Indicates the interpolation constraint weights (to ensure that the optimized parameters closely approximate the interpolation results); The first term represents the motion smoothing constraint weight (ensuring smooth parameter changes between adjacent frames); the second term is the interpolation constraint term, and the third term is the motion smoothing constraint term. The optimized intermediate frame parameters are obtained by minimizing the objective function, ensuring that the parameter change rate between adjacent frames remains consistent.

[0085] For camera parameter generation, the model predicts the camera's motion trajectory based on the camera position parameters of the first and last frames, achieving a smooth transition in the shot motion effect. For object bounding box parameter generation, the model predicts the object's position change based on the object's motion state, conforming to the object's motion trajectory. After generating the intermediate frame shot layout sequence, its motion smoothness is verified by calculating the rate of change of layout parameters between adjacent frames. If the rate of change exceeds a preset threshold, the parameters are adjusted. The preset threshold can be adjusted according to the video type. The verified intermediate frame shot layout sequence ensures the continuity and smoothness of the action.

[0086] Through the above process, the model can quickly generate a sequence of intermediate frame shot layouts between specified first and last frames, achieving interpolation filling of the shot layout and improving the integrity of the shot sequence.

[0087] S7. Integrate the initial continuous shot layout sequence, the subsequent shot layout sequence with a consistency judgment result of qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

[0088] Specifically, the core of this step is to integrate the layout sequences generated at different stages according to chronological order and logical relationships, ensuring the coherence, completeness, and rationality of the final shot storyboard layout sequence. Before the integration operation, each layout sequence undergoes preprocessing. First, the initial continuous shot storyboard layout sequence, subsequent shot storyboard layout sequences, and intermediate frame shot storyboard layout sequences are formatted uniformly, converting all parameters in the sequences to the same numerical format and unit. Second, each sequence is marked with a frame number. The frame number of the initial continuous shot storyboard layout sequence increments sequentially from the starting frame. The frame number of the subsequent shot storyboard layout sequences increments sequentially from the frame number of the edited frame. The frame number of the intermediate frame shot storyboard layout sequence is filled with the frame numbers of the first and last frames.

[0089] After preprocessing, the integration process begins. First, using the initial continuous shot layout sequence as the base sequence, subsequent shot layout sequences that pass the consistency check replace the frames after the edited frames in the base sequence, ensuring the replaced sequence meets the user's editing requirements. Second, intermediate frame shot layout sequences are filled between the corresponding first and last frames in the base sequence, replacing the original blank frame sequences to achieve smooth transitions. During integration, the continuity of the frame sequence is verified, including the rate of change of character poses, object bounding boxes, and camera parameters between adjacent frames. If the rate of change exceeds a preset threshold, the corresponding frame parameters are smoothly adjusted to ensure the motion trajectory of the entire sequence conforms to physical laws. Simultaneously, the transition effects are verified to ensure accurate transition markers between different shots, conforming to the principles of video creation.

[0090] After the integration process is complete, a global verification of the final shot layout sequence is performed. The final sequence is compared with the user's creative description text in step S5, and a consistency judgment result is output. If the result is satisfactory, the generation of the final shot layout sequence is complete; if the result is unsatisfactory, the final sequence is adjusted according to the reasons for the mismatch in the verification result until the consistency judgment result is satisfactory. The structure of the final shot layout sequence is a frame sequence arranged in frame number order. Each frame contains information such as camera parameters, character identifiers, character 2D pose, character 3D pose, object identifiers, object 2D bounding boxes, object 3D bounding boxes, and shot transition markers. This sequence can be stored in various structured data formats, which can be easily imported into video generation models or third-party interfaces to achieve automated generation from the shot layout sequence to the final video.

[0091] Through the above integration process, the layout sequences generated at different stages can be merged into a complete, coherent final shot layout sequence that conforms to the user's creative intent, providing precise layout guidance for subsequent video generation and significantly improving the efficiency and quality of video creation.

[0092] The aforementioned video storyboard layout generation method based on an autoregressive model automatically extracts information and tracks targets from the original video, transforming the video content into structured labeled data containing shot descriptions, camera parameters, character pose sequences, and object bounding box sequences. This data is then used to construct a multi-task training dataset covering four task types: storyboard generation, interactive editing, in-between frame filling, and storyboard verification. Next, an autoregressive language model based on a Transformer architecture is trained using this dataset. The model parameters are optimized through a multi-task joint loss function, enabling the model to uniformly understand and generate structured layout sequences. In practical applications, users only need to input a natural language creative description, and the model, combined with the storyboard generation task identifier, automatically outputs an initial, semantically consistent sequence of consecutive shot layouts. If the user modifies the layout of a frame in the sequence... The process involves adjusting the layout, inputting historical layouts, editing instructions, and interactive editing task identifiers into the same model, and incrementally generating subsequent layouts based solely on the existing context, avoiding full sequence recalculation. Simultaneously, a storyboard verification task verifies the consistency between the generated layouts and text descriptions, ensuring the output content aligns with the creative intent. Furthermore, an intermediate frame filling task automatically completes smooth transitions between intermediate frames based on user-specified first and last frames. Finally, all generated and edited layout sequences are integrated to form a complete and coherent long video storyboard plan, achieving an end-to-end automated workflow from creative input to storyboard output. This significantly reduces the reliance on manual professional storyboard design in long video production, improving creative efficiency and layout consistency. Additionally, lightweight sequence modeling and incremental generation mechanisms reduce computational resource consumption, enhancing practicality in real-time interaction and long sequence generation scenarios.

[0093] refer to Figure 2 In one alternative embodiment, using a multi-task training dataset, the parameters of a Transformer-based autoregressive language model are trained by minimizing the joint multi-task loss function, including the following steps:

[0094] S11. Convert the input information of each training sample in the multi-task training dataset into a discrete token sequence, and add a task identifier token corresponding to the task type at the beginning of the discrete token sequence.

[0095] Specifically, differentiated discretization strategies are employed for different types of input information in the multi-task training dataset to ensure that all types of information can be transformed into token sequences recognizable by the model. Text-based input information includes shot descriptions, user editing instructions, and content descriptions. This type of information is segmented using the Byte-Pair Encoding (BPE) algorithm. This algorithm iteratively merges the most frequent character pairs in the corpus to generate a fixed-size vocabulary, effectively covering core semantic units and converting text of arbitrary length into a fixed-length discrete token sequence. The specific processing steps are as follows: First, the text information is cleaned, removing special characters, redundant spaces, and invalid characters to avoid interfering with model training. Then, based on the pre-trained vocabulary, the cleaned text is split into several basic tokens, each mapped to a unique index value, facilitating numerical processing by the model. For numerical input information, including camera parameters (viewpoint, focal length, position coordinates), character pose coordinates, object bounding box parameters, and fill frame count, a numerical tokenization algorithm is used for discretization. This algorithm maps numerical values ​​to preset numerical intervals, with each interval corresponding to a unique discrete token, ensuring accurate representation of numerical information.

[0096] Specifically, all numerical information is first normalized to map the value range to a fixed interval. This interval is then evenly divided into several sub-intervals, each assigned a unique token, thus converting numerical information into discrete tokens. After discretization of the input information, a task identifier token corresponding to the task type is added to the beginning of the discrete token sequence. This identifier token is a model-preset token used to explicitly inform the model of the current task type, ensuring the model can quickly call the corresponding task processing logic and avoid confusion between different task processing logics. Different task types correspond to unique task identifier tokens; for example, storyboard generation, interactive editing, in-between frame filling, and storyboard verification tasks all have their own unique identifiers. The discrete token sequence after adding the task identifier token has a clear structure and a clear task orientation, effectively improving the targeting of model training. Simultaneously, to ensure the uniformity of token sequence length and avoid affecting model training performance due to inconsistent sequence lengths, all discrete token sequences are padded to a fixed length. The padding token uses a model-preset padding token.

[0097] S12. Input the discrete token sequence with the task identifier token added into the autoregressive language model, and predict the probability distribution of the next token through forward propagation calculation of the autoregressive language model.

[0098] Specifically, the model's forward propagation process encompasses four core stages: the embedding layer, encoder, decoder, and output layer. These stages work together to transform input tokens into predicted probabilities. First, the embedding layer converts the input discrete token sequence (including task identifier tokens, text tokens, numerical tokens, and padding tokens) into a high-dimensional dense embedding vector. Each token is mapped to a fixed-dimensional embedding vector, and the transformation formula is as follows: .in The embedding vector matrix is ​​used to store the high-dimensional dense feature representation of each token in the input token sequence, which can accurately capture the semantic or numerical attributes of the tokens. This represents the embedding weight matrix, which consists of trainable parameters of the model. Through training, these parameters are continuously optimized to improve the accuracy of token representation. This represents the input token index matrix, used to store the unique index value corresponding to each input token.

[0099] After the embedding vector matrix is ​​generated, it is input into the encoder of the model for feature extraction. The encoder consists of multiple stacked Transformer encoder layers, each containing two core modules: a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism effectively captures long-distance dependencies in the token sequence, including the association between the task identifier and the input information, and the association between text tokens and numerical tokens, ensuring the comprehensiveness of feature extraction. The feedforward neural network is used to perform non-linear transformations on the extracted features, improving the model's feature representation ability. The text feature representation output by the encoder is input into the decoder, which consists of multiple stacked Transformer decoder layers. Each decoder layer contains three core modules: a mask self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network.

[0100] The masked self-attention mechanism is used to mask token information in future time steps, ensuring that the model makes predictions based only on historical tokens (the first t-1 tokens), which is consistent with the core characteristics of autoregressive models. The encoder-decoder attention mechanism is used to capture the correlation between the encoder feature matrix and the decoder's historical token features, ensuring that the model makes predictions based on the core features of the input information, and avoiding the prediction results from being decoupled from the input information.

[0101] After being processed through multiple layers, the decoder's output features are input to the output layer for probability distribution prediction. The output layer first maps the decoder's output features into a vector of vocabulary size through linear projection, and then converts the vector into a probability distribution through the Softmax activation function, finally obtaining the probability distribution of the next token. This distribution can quantify the model's prediction confidence for each possible token, providing a direct basis for subsequent loss calculation.

[0102] S13. Based on the probability distribution of the next token and the corresponding real target token in the training samples, calculate the corresponding task sub-loss value according to the task type of the current training samples; where the task sub-loss value includes the identity matching loss for storyboard generation tasks and interactive editing tasks. Layout similarity loss Motion smoothing loss for intermediate frame filling task And the loss of instruction correctness in the storyboard verification task. .

[0103] Specifically, this step calculates the corresponding task sub-loss value based on the task type of the current training sample. The loss value quantifies the difference between the model prediction result and the real target, providing a clear direction for model parameter optimization and ensuring that the training needs of different tasks can be effectively met.

[0104] Different task types correspond to different sub-loss combinations to avoid the problem that a single loss function cannot adapt to multi-task training: both the storyboard generation task and the interactive editing task involve maintaining the consistency of character and object IDs and accurately representing layout parameters, so the identity matching loss is calculated simultaneously. Similarity loss with layout The core requirement of the intermediate frame padding task is to ensure the motion continuity of the intermediate frame layout sequence; therefore, it is necessary to calculate the motion smoothing loss. The core requirement of the storyboard verification task is to ensure the matching between the layout sequence and the text description; therefore, the error in calculating the instruction correctness loss is crucial. .

[0105] Identity matching loss To quantify the difference between the predicted person and object ID tokens and the actual target ID tokens, thus avoiding ID confusion, this loss is calculated using the cross-entropy loss function, and the formula is as follows: .in The loss value represents the identity identifier matching. This represents the total number of people and objects in the current sample. This represents the total number of ID categories. Let the one-hot encoded real label be the ID of the k-th class corresponding to the i-th person / object. The smaller the loss value, the higher the match between the model's predicted ID and the real ID.

[0106] Layout similarity loss The spatial difference between the layout parameter tokens predicted by the quantification model and the actual target layout parameter tokens is used to ensure that the layout sequence generated by the model is consistent with the actual layout in spatial location. This loss is calculated using the L2 loss function, and the calculation formula is as follows: ,in This represents the layout similarity loss value. This represents the total number of layout parameters that need to be matched in a single frame. For each parameter's dimension, This is the normalized value corresponding to the actual parameter token. The normalized value corresponding to the predicted parameter token is used. The smaller the loss value, the higher the layout similarity.

[0107] Motion smoothing loss To quantify the differences in motion coherence of intermediate frame layout sequences, avoid abrupt changes in motion, conform to the laws of motion physics, and use the mean square error loss function for calculation, the smaller the loss value, the better the motion coherence.

[0108] Loss of instruction correctness To quantify the difference in matching degree between layout sequence tokens and text description tokens, ensuring that the layout sequence generated by the model can accurately match the core intent of the text description, a contrastive loss function is used for calculation. The smaller the loss value, the higher the matching degree between the layout sequence and the text description.

[0109] S14. The weighted sum of the sub-loss values ​​of each task is used to obtain the joint loss function for multiple tasks. The value of ; where the multi-task joint loss function The calculation formula is ;in, , , , These are the preset weighting coefficients corresponding to the sub-losses of each task.

[0110] Specifically, in the multi-task joint loss function, This represents the joint loss value across multiple tasks, which is the core objective that needs to be minimized during model training. It comprehensively reflects the prediction error of the model on various tasks. The smaller the value, the better the overall performance of the model. , , , Loss for matching identity identifiers Layout similarity loss Motion smoothing loss Loss of instruction correctness The corresponding preset weight coefficients are used to adjust the contribution ratio of each sub-loss in the total loss. The value of the weight coefficient directly affects the optimization priority of the model for different tasks.

[0111] The values ​​of each preset weight coefficient are determined through optimization on the validation set. The core principle is to ensure that the loss of each type of task accounts for a reasonable proportion of the total loss, so that the model can learn the processing logic of each type of task in a balanced way. The specific optimization process is as follows: First, preset a set of initial weight coefficients to ensure that the initial loss contributes evenly; then, substitute this set of weight coefficients into the model training process. After training, calculate the performance indicators of each type of task in the validation set, including ID matching accuracy, layout parameter prediction error, motion coherence score, and text matching accuracy; adjust each weight coefficient according to the performance indicators. If the performance indicator of a certain type of task does not reach the preset threshold, increase the weight coefficient of the corresponding loss; if the performance indicator of a certain type of task has reached the threshold and is much higher than that of other tasks, appropriately decrease the weight coefficient of the corresponding loss; repeat the above process until the performance indicators of all tasks in the validation set reach the preset threshold. The weight coefficients at this point are the final preset weight coefficients.

[0112] S15. Based on the value of the multi-task joint loss function, calculate the parameter gradient of the autoregressive language model using the backpropagation algorithm, and update the parameters of the autoregressive language model according to the parameter gradient.

[0113] Specifically, the backpropagation algorithm is based on the chain rule, starting from the joint loss function of multiple tasks. Starting from this point, we iterate backward through all layers of the model, including the output layer, decoder, encoder, and embedding layer, calculating the gradient of each trainable parameter. The physical meaning of the gradient is the rate of change of the loss function with respect to that parameter; the sign of the gradient indicates the direction of parameter adjustment, and the absolute value of the gradient indicates the magnitude of parameter adjustment.

[0114] The gradient calculation of trainable parameters at each layer needs to be combined with the characteristics of the output and activation function of the corresponding layer. For example, the trainable parameters of the output layer include the weight matrix and bias term, and its gradient calculation needs to be based on the difference between the predicted probability distribution and the real target label. The trainable parameters inside the encoder and decoder include the projected weight matrix of the attention mechanism, the weight matrix of the multi-head attention output concatenation, and the weight matrix and bias term of the feedforward neural network. The gradient calculation of these parameters needs to be combined with the output gradient of the attention mechanism and the feedforward neural network, and derived layer by layer in reverse using the chain rule. The gradient calculation of the weight matrix of the embedding layer needs to be based on the output gradient of the embedding vector, and associated with the mapping relationship between the input token index and the embedding vector.

[0115] After calculating the gradients for all trainable parameters, a suitable optimizer is used for parameter updates. Here, the AdamW optimizer is chosen. This optimizer combines the advantages of momentum optimization and adaptive learning rate adjustment, and introduces a weight decay mechanism, which can effectively avoid model overfitting and improve training stability and convergence speed. The formula for parameter updates is as follows: ,in Let represent a trainable parameter in the t-th training round. This indicates the updated parameters. This represents the learning rate in the current training round. The first-order momentum of the parameter is represented by... The second momentum of the parameter is represented by . Represents the numerical stability coefficient. This represents the weight decay coefficient, and all parameters are determined through validation set tuning. The learning rate is dynamically adjusted using a cosine annealing learning rate scheduling strategy. This strategy allows the learning rate to gradually decay with each training epoch. A larger learning rate is used in the early stages of training to accelerate convergence, while a smaller learning rate is used in the later stages to precisely optimize the parameters, ensuring the stability and convergence of the training process. During parameter updates, all trainable parameters are updated synchronously in batches. That is, after processing each batch of training samples, the average gradient of that batch is calculated, and a parameter update is performed based on the average gradient. This avoids interference from single samples in parameter updates and improves training stability.

[0116] Meanwhile, an early stopping strategy is employed to ensure the effectiveness of model training and avoid overfitting. Specifically, during training, the multi-task joint loss value of the validation set is monitored in real time. If the validation set loss value no longer decreases for a preset number of rounds, and the decrease is less than a preset threshold, model training is stopped, and the model parameters for the current round are saved as the final training parameters. If the validation set loss value continues to decrease, training continues until the maximum number of training rounds is reached. Through this backpropagation and parameter update process, the model's trainable parameters are continuously optimized, the multi-task joint loss function value gradually decreases, and the model's task processing capabilities (including storyboard generation, interactive editing, intermediate frame filling, and storyboard verification) are continuously improved, ultimately resulting in an autoregressive language model with unified layout sequence processing capabilities.

[0117] In one optional embodiment, the calculation steps for layout similarity loss include:

[0118] S21. Based on the complete token sequence predicted by the autoregressive language model for the current training sample, decode to obtain the predicted layout element position information for the current frame; based on the real complete target token sequence in the current training sample, decode to obtain the real labeled layout element position information for the current frame.

[0119] The layout element position information includes joint coordinates in the character pose sequence and corner coordinates in the object bounding box sequence; the predicted layout element position information set is represented as follows. The set of actual marked layout element position information is represented as This indicates the number of elements that need to be laid out in the current frame. This represents the first element in the set of predicted layout element position information. A vector of coordinates of elements. The first element in the set of actual labeled layout element position information A vector of coordinates of elements. .

[0120] Specifically, this step converts the discrete token sequence output by the model into layout element position information that can be used for spatial difference calculation, laying the foundation for subsequent distance quantization and matching. First, for the current training sample, the predicted complete token sequence is extracted from the output of the autoregressive language model. This sequence is then decoded using a preset decoding rule to reconstruct the predicted layout element position information corresponding to the current frame. The preset decoding rule refers to a fixed mapping and transformation that restores the discrete token sequence to layout element position information. This can be achieved by converting token index values ​​to coordinate values ​​using a preset token-coordinate mapping table (for numerical tokens, inverse normalization is required to restore the original coordinate range), and then organizing them into standard format coordinate vectors according to element type. Simultaneously, the actual complete target token sequence is extracted from the current training sample and decoded using the same decoding rule to obtain the actual labeled layout element position information for the current frame.

[0121] The layout element position information specifically encompasses two types of core coordinate data: one is the joint coordinates in the character pose sequence, used to represent the spatial position of each joint; the other is the corner coordinates in the object bounding box sequence, used to represent the spatial extent of the object's bounding box. To facilitate subsequent calculations and matching, the predicted layout element position information is organized into a set format. Organize the actual position information of the layout elements into a set. .in, This indicates the total number of elements that need to be matched in the layout of the current frame. This number is determined by the scene content of the current frame and covers all elements corresponding to the character joints and object bounding boxes. This represents the first element in the set of predicted layout element position information. The coordinate vector of each element, with the vector dimension set according to the element type (joint coordinates are usually two-dimensional or three-dimensional vectors, and corner coordinates are usually two-dimensional vectors), stores the spatial coordinate values ​​of the element; The first element in the set of actual labeled layout element position information The coordinate vector of each element, and The vector dimensions are consistent, storing the actual spatial coordinate values ​​of the corresponding elements; and All are element indices, with values ​​ranging from 1 to... This ensures that all elements in the set are covered.

[0122] S22. Calculate the coordinate distance between all element pairs in the predicted layout element location information set and the actual labeled layout element location information set, and construct a distance matrix based on the coordinate distance; where the distance matrix is... matrix Distance matrix The Middle Line number Column elements Represents the predicted element With real elements The Euclidean distance between them.

[0123] Specifically, this step quantifies the spatial distance between predicted layout elements and actual layout elements, and organizes this distance information in matrix form to provide a quantitative basis for subsequent optimal matching. In detail, it requires traversing the set of predicted layout element position information. Collection of position information of actual labeled layout elements For all element pairs, calculate the coordinate distance between each pair. Considering the need for accurate quantification of spatial coordinate differences, Euclidean distance is used as the calculation method for coordinate distance. This distance can effectively reflect the straight-line distance between two points in space, adapting to scenarios where the positional differences of layout elements are quantified.

[0124] Based on the Euclidean distance calculation results for all element pairs, construct the distance matrix. The distance matrix is A square matrix of dimension 1, where the number of rows and columns is equal to the number of layout elements in the current frame. Consistency is ensured by storing the distance between each predicted element and each true element in a matrix. Distance matrix The Middle Line number The elements of the column are used This indicates that its physical meaning is the prediction of the first element in the set of layout elements. element The first in the set of actual layout elements element The Euclidean distance between them. By constructing this matrix, the distance relationship between all predicted elements and the true elements can be clearly presented. Subsequent steps will be based on this matrix to perform element matching, ensuring the orderliness and accuracy of the matching process.

[0125] S23. Based on the distance values ​​in the distance matrix as the matching cost, and with the goal of minimizing the total matching cost, the Hungarian algorithm is used to solve for the optimal bijective matching result between the predicted layout element position information set and the actual labeled layout element position information set.

[0126] Specifically, this step establishes a unique correspondence between predicted layout elements and actual layout elements, avoiding the confusion of many-to-one or one-to-many matching and ensuring the targeted nature of subsequent loss calculations. Here, the distance values ​​in the distance matrix are used as the matching cost. The smaller the distance value, the closer the corresponding predicted element is to the actual element in spatial location, and the lower the matching cost. The core objective of matching is to minimize the total matching cost, that is, to minimize the sum of the distance values ​​of all matched element pairs. This objective ensures the optimality of the overall layout matching and best reflects the real layout scenario.

[0127] To achieve the above objectives, the Hungarian algorithm is used for optimal matching. This algorithm is specifically designed for solving optimal matching problems in bipartite graphs and can find the bijective matching result that minimizes the total matching cost within a finite number of steps. In the specific implementation, the predicted layout element set... With the collection of real layout elements As two sets of vertices in a bipartite graph, the distance matrix elements in As the corresponding vertex in the two vertex sets (the first The predicted element and the first The edge weights (matching costs) between the real elements are calculated using the Hungarian algorithm on the bipartite graph, ultimately yielding an optimal bijective matching result. This result ensures that each predicted element matches only one real element, and each real element also matches only one predicted element, while minimizing the sum of the total distances of all matching pairs.

[0128] S24. Based on the optimal bijective matching result, calculate the layout similarity sub-loss of the current frame; where, the layout similarity loss of the current frame... The calculation formula is:

[0129]

[0130] in, Indicates the result of the optimal bijective matching with the th The index of the real element that matches each predicted element. This represents the L2 norm of a vector.

[0131] Specifically, this step, based on the optimal bijective matching result, quantifies the overall spatial difference between the predicted and true layouts in the current frame, obtaining the single-frame layout similarity sub-loss. After obtaining the optimal bijective matching result, the unique true element corresponding to each predicted element can be determined. This indicates that the matching result is related to the first... Each prediction element The index of the matching real element, i.e., the first... The true element corresponding to each predicted element is: By using this index, the true matching object of each predicted element can be accurately located, ensuring the targeted nature of the difference calculation.

[0132] The layout similarity sub-loss of the current frame is calculated using the formula... The calculation yielded the following result. This represents the layout similarity sub-loss value of the current frame. The smaller this value, the higher the spatial similarity between the predicted layout and the actual layout of the current frame. The squared L2 norm of a vector is used to quantify the spatial difference between two coordinate vectors. It is calculated as the sum of the squares of the differences in the corresponding dimensions of the two vectors. Compared to the L1 norm, the squared L2 norm is more sensitive to larger differences, can more accurately amplify spatial deviations, and improve the discriminative power of loss calculations; the summation symbol... Indicates all The differences between each pair of matched elements are accumulated to comprehensively reflect the overall differences among all layout elements in the current frame. The specific calculation process is as follows: first, for each pair of matched elements... and Calculate the difference between coordinate vectors, then take the L2 norm square of the difference, and finally sum the calculation results of all matching pairs to obtain the layout similarity sub-loss of the current frame.

[0133] S25. Sum the layout similarity sub-losses of all frames in the sequence to obtain the layout similarity loss of the current training sample. .

[0134] Specifically, this step integrates the layout differences of all frames in the current training sample to obtain the layout similarity loss of the entire sample, providing basic data for the calculation of the multi-task joint loss. Since the layout sequence is usually composed of multiple consecutive layouts, the single-frame loss can only reflect the layout difference at a certain moment and cannot fully characterize the layout similarity of the entire sequence. Therefore, it is necessary to accumulate the layout similarity sub-losses of all frames in the sequence.

[0135] In the specific implementation process, each frame of the current training sample layout sequence is traversed, and the layout similarity sub-loss corresponding to each frame is extracted. The sub-loss values ​​of all frames are summed, and the final sum is the layout similarity loss of the current training sample. This loss value comprehensively covers the spatial differences of the entire layout sequence, and can fully reflect the accuracy of the model's prediction of the current sample layout. Subsequently, this loss value is substituted into the calculation of the multi-task joint loss to participate in the optimization process of model parameters, ensuring that the model can optimize for the similarity of the entire layout sequence, thereby improving the stability and accuracy of continuous layout prediction.

[0136] In one optional embodiment, the calculation step of motion smoothing loss includes:

[0137] S31. Based on the complete token sequence predicted by the autoregressive language model for the intermediate frame filling task, decode to obtain the generated intermediate frame layout sequence.

[0138] The intermediate frame layout sequence includes the joint coordinates of the character's pose sequence and the corner coordinates of the object's bounding box sequence in each frame; the intermediate frame layout sequence is represented as follows: , The number of intermediate frames. Indicates the first All in the intermediate frame layout sequence of the frame A set of coordinate vectors of elements. .

[0139] Specifically, this step converts the discrete token sequence output by the autoregressive language model for the intermediate frame filling task into an intermediate frame layout sequence that can be used for motion coherence analysis, laying the data foundation for subsequent error calculation. Specifically, for the current training sample corresponding to the intermediate frame filling task, the predicted complete token sequence is extracted from the model output and decoded using a preset decoding rule consistent with the layout similarity loss calculation process. This decoding rule converts token indices into coordinate values ​​through a token-coordinate mapping table (inverse normalizing numerical tokens to restore the original coordinate range), and organizes them into standard coordinate vectors according to element type.

[0140] The decoded intermediate frame layout sequence contains the joint coordinates of the character's pose sequence and the corner coordinates of the object's bounding box sequence in each frame. This coordinate data directly represents the spatial position of the layout elements in each intermediate frame and is the core basis for judging motion continuity. This intermediate frame layout sequence is represented as... ,in This represents the number of intermediate frames, i.e., the total number of consecutive frames generated by padding; Representing the The set of coordinate vectors of all elements in the intermediate frame layout sequence of a frame. For frame index, the value range is 1 to ; This represents the total number of elements in each frame's layout, encompassing all elements corresponding to the character's joints and object bounding boxes. Includes A coordinate vector of elements.

[0141] S32, for the first frame in the intermediate frame layout sequence The frame calculates the motion smoothing error for each element of the intermediate frame layout sequence based on the set of coordinate vectors; where, , No. Motion smoothing error of each element The calculation formula is:

[0142]

[0143] in, Indicates the first The set of coordinate vectors of a frame The Middle A coordinate vector of elements.

[0144] Specifically, this step involves quantifying the smoothness of the transition between the position of a single layout element in the current frame and the positions of elements in the preceding and following frames, except for the first and last frames in the intermediate frame layout sequence. This means determining whether the position of the element in the current frame conforms to the inertial movement patterns of the preceding and following frames, avoiding abrupt displacements. Because the first frame ( ) and tail frame ( The reference frame for the intermediate frame filling task (usually determined by the input information) does not need to participate in motion smoothness verification, therefore it is only used for... Error calculation is performed on the intermediate frames.

[0145] For the The first frame Each element has a motion smoothing error expressed by the formula. The calculation yielded the following. Representing the Frame number The motion smoothing error of each element is such that the smaller the error value, the smoother the transition between the position of the element in the current frame and the previous and next frames. Representing the Frame coordinate vector set The Middle A coordinate vector of elements, storing the spatial coordinates of the element in the current frame; and Representing the first Frame and the The first frame The coordinate vector of each element, that is, the position coordinates of the current element in the previous frame and the next frame; This represents the theoretically smoothed coordinates of the current frame calculated based on the positions of the preceding and following frames. These coordinates are the average of the positions of the preceding and following frames, conforming to the inertial laws of element movement. The L2 norm squared of a vector is used to quantify the spatial difference between the actual coordinates and the theoretical smooth coordinates of the current frame. It can accurately amplify large displacement deviations and improve error identification.

[0146] S33, regarding the first The motion smoothing errors of all elements in the intermediate frame layout sequence are summed to obtain the frame-level smoothing loss for the corresponding frame; where, the frame-level smoothing loss... The calculation formula is .

[0147] Specifically, this step integrates the first... The motion smoothing error of all layout elements in a frame is used to obtain the motion smoothing loss at the single-frame level, quantifying the motion coherence of the entire frame layout. The motion smoothing error of each element within a single frame can only reflect the connection between individual elements, while the frame-level smoothing loss can comprehensively characterize the overall motion smoothing effect of all elements in the current frame, providing core data at the single-frame level for subsequent overall loss calculation.

[0148] No. The frame-level smoothing loss is expressed by the formula The calculation yielded the following. Representing the The frame-level smoothing loss value; the smaller the value, the better the overall motion continuity of the current frame. Indicates the first All in the frame The motion smoothing error of each layout element is accumulated; For the first Frame number The motion smoothing error of each element is the result calculated in S32. The specific calculation process is as follows: traverse the... All frames For each element, extract the motion smoothing error. Sum all the error values ​​to get the frame-level smoothing loss for the current frame.

[0149] S34, Layout sequence of intermediate frames Frame-level smoothing loss for all intermediate frames Summing is performed to obtain the overall motion smoothing loss for the current training samples. The formula for calculating the overall motion smoothing loss is as follows: .

[0150] Specifically, this step integrates the frame-level smoothing loss of all valid intermediate frames in the intermediate frame layout sequence to obtain the overall motion smoothing loss of the current training sample, providing core sub-loss data for the calculation of the multi-task joint loss. A single frame-level smoothing loss can only reflect the motion coherence of a certain frame, while the overall motion smoothing loss can comprehensively cover the motion connection effect of the entire intermediate frame sequence, ensuring that the model optimizes the motion smoothness of the entire sequence.

[0151] The overall motion smoothing loss of the current training samples is expressed by the formula The calculation yielded the following. This represents the overall motion smoothing loss value. The smaller the value, the better the motion coherence of the entire intermediate frame layout sequence. This represents all valid intermediate frames in the intermediate frame layout sequence (i.e., The frame-level smoothing loss of the frames (excluding the first and last frames) is accumulated. For the first The frame-level smoothing loss is the result calculated in S33. The specific calculation process is as follows: traverse all valid intermediate frames in the intermediate frame sequence, extract the frame-level smoothing loss of each frame, sum all the loss values, and the sum is the overall motion smoothing loss of the current training sample. This loss value will be substituted into the multi-task joint loss formula to participate in the optimization process of model parameters and improve the motion coherence accuracy of the intermediate frame filling task.

[0152] In one optional embodiment, the user-input creative description text is concatenated with the storyboard generation task identifier and then input into an autoregressive language model. The corresponding initial sequence of consecutive shot layouts is generated through autoregressive decoding, including the following steps:

[0153] S41. Tokenize the creative description text to obtain a text token sequence; concatenate the predefined storyboard generation task identifier token with the text token sequence to form the initial decoding input sequence.

[0154] Specifically, the user-input creative description text is first tokenized. The text is segmented using a byte-pair encoding (BPE) algorithm, and redundant characters and invalid information are removed. The text is then divided into several core semantic tokens, each mapped to a unique index value, ultimately forming an ordered sequence of text tokens. Next, a predefined storyboard generation task identifier token is extracted. This token is a model-specific token used to explicitly inform the model that the current task is storyboard generation, prompting it to invoke the corresponding storyboard layout generation logic. Finally, the storyboard generation task identifier token is concatenated at the beginning of the text token sequence to form the initial decoding input sequence. This sequence contains both the core semantic information of the creative text and clearly defines the task type, ensuring the model can accurately locate the processing scenario.

[0155] S42. Input the initial decoded input sequence into the autoregressive language model, use the forward propagation of the autoregressive language model to calculate the next token based on all tokens in the initial decoded input sequence, and output the probability distribution of the first predicted token.

[0156] Specifically, after the constructed initial decoded input sequence is input into the autoregressive language model, the model first converts each token in the sequence into a high-dimensional dense embedding vector through an embedding layer, capturing the semantic features and task attributes of the token. The embedding vector is then processed by the encoder to extract global association features of the sequence. The decoder, based on all tokens in the initial input sequence (including task identifiers and text tokens), combines the features output by the encoder to make predictions. The decoder features are mapped to a vector of vocabulary size through linear projection of the output layer, and then transformed by the Softmax activation function, finally outputting the probability distribution of the first predicted token. In this probability distribution, each element corresponds to the prediction confidence of a token in the vocabulary, providing a quantitative basis for subsequent sampling.

[0157] S43. Sample according to the probability distribution of the first predicted token to obtain the first generated token, and append the first generated token to the end of the initial decoding input sequence to form a new decoding input sequence.

[0158] Specifically, for the probability distribution of the first predicted token in the output, a sampling strategy based on probability confidence is used to select the token. Tokens with higher confidence are prioritized as the first generated token to ensure the rationality and validity of the sampling results. After sampling, this token is appended to the end of the initial decoding input sequence to form a new decoding input sequence. The updated sequence retains both the original creative text information and task identifier, and adds the information of the generated tokens, providing a complete input basis for the model to predict the next token based on historical tokens, which aligns with the predictive characteristics of the autoregressive model.

[0159] S44. Input the new decoded input sequence into the autoregressive language model, predict the next token through the autoregressive language model, output the probability distribution of the next predicted token and sample it, and continue to append the newly sampled token to the input sequence.

[0160] Specifically, the newly decoded input sequence formed in S43 is input again into the autoregressive language model. Based on all historical tokens in the sequence (including the initial input and generated tokens), the model predicts the probability distribution of the next token through a forward propagation process. The prediction logic is consistent with S42, implemented through embedding, encoding, decoding, and probability transformation. Subsequently, this probability distribution is sampled, and tokens with reasonable confidence are selected as newly generated tokens and appended to the end of the current decoded input sequence, completing another update of the input sequence. The entire process achieves a closed loop of "prediction-sampling-update," ensuring the continuity of sequence generation.

[0161] S45. Repeat step S44 to iteratively generate subsequent tokens until the generated token is a predefined sequence end token or the sequence length reaches the preset maximum value, thus obtaining a complete output token sequence.

[0162] Specifically, the S44 iterative process is continuously repeated. After each iteration, the currently generated sequence is validated based on two termination conditions: first, whether the newly generated token is a predefined sequence end token, which is a model-preset exclusive token used to indicate the completion of the storyboard layout sequence generation; and second, whether the total length of the current sequence has reached a preset maximum value, which is set according to the actual needs of storyboard generation to ensure that the sequence length conforms to the application scenario (avoiding both excessively short sequences leading to incomplete information and excessively long sequences causing redundancy). When either termination condition is met, the iteration process stops, and the resulting sequence containing all generated tokens is the complete output token sequence.

[0163] S46. Detoxify the output token sequence and decode it into a structured initial continuous shot layout sequence; wherein, the initial continuous shot layout sequence includes camera parameters, character identification and pose coordinate sequence, object identification and bounding box coordinate sequence.

[0164] Specifically, the complete output token sequence is first detoxified. This process is the reverse of the tokenization process in S41. Using a pre-defined token-information mapping table, the index value corresponding to each token is restored to its original information (textual semantic information or numerical information). Task identifier tokens and any possible padding tokens are removed from the sequence, retaining the core generated information. Subsequently, the detoxified information is decoded and, according to the structured requirements of the storyboard layout, an initial sequence of consecutive shots is obtained. This sequence contains four types of core information: camera parameters (viewpoint, focal length, position, etc.), character identification and corresponding pose coordinate sequences (character joint coordinates in each frame), and object identification and corresponding bounding box coordinate sequences (object bounding box corner coordinates in each frame). This information together constitutes the complete storyboard layout, providing a structured foundation for subsequent storyboard optimization or editing.

[0165] The aforementioned video storyboard layout generation method based on an autoregressive model automatically extracts information and tracks targets from the original video, transforming the video content into structured labeled data containing shot descriptions, camera parameters, character pose sequences, and object bounding box sequences. This data is then used to construct a multi-task training dataset covering four task types: storyboard generation, interactive editing, in-between frame filling, and storyboard verification. Next, an autoregressive language model based on a Transformer architecture is trained using this dataset. The model parameters are optimized through a multi-task joint loss function, enabling the model to uniformly understand and generate structured layout sequences. In practical applications, users only need to input a natural language creative description, and the model, combined with the storyboard generation task identifier, automatically outputs an initial, semantically consistent sequence of consecutive shot layouts. If the user modifies the layout of a frame in the sequence... The process involves adjusting the layout, inputting historical layouts, editing instructions, and interactive editing task identifiers into the same model, and incrementally generating subsequent layouts based solely on the existing context, avoiding full sequence recalculation. Simultaneously, a storyboard verification task verifies the consistency between the generated layouts and text descriptions, ensuring the output content aligns with the creative intent. Furthermore, an intermediate frame filling task automatically completes smooth transitions between intermediate frames based on user-specified first and last frames. Finally, all generated and edited layout sequences are integrated to form a complete and coherent long video storyboard plan, achieving an end-to-end automated workflow from creative input to storyboard output. This significantly reduces the reliance on manual professional storyboard design in long video production, improving creative efficiency and layout consistency. Additionally, lightweight sequence modeling and incremental generation mechanisms reduce computational resource consumption, enhancing practicality in real-time interaction and long sequence generation scenarios.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] Based on the same inventive concept, this application also provides a system for implementing the video storyboard layout generation method based on the autoregressive model described above. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the video storyboard layout generation system based on the autoregressive model provided below can be found in the limitations of the video storyboard layout generation method based on the autoregressive model described above, and will not be repeated here.

[0168] In one exemplary embodiment, such as Figure 3 As shown, a video storyboard layout generation system 30 based on an autoregressive model is provided to implement the methods in the above-described embodiments. The system includes:

[0169] The video semantic annotation construction module 31 is used to extract information and track targets from the original video, and generate video annotation data including shot text descriptions, camera parameters, human pose sequences and object bounding box sequences; based on the video annotation data, a multi-task training dataset is constructed, including storyboard generation tasks, interactive editing tasks, intermediate frame filling tasks and storyboard verification tasks.

[0170] The multi-task model training module 32 is used to train the parameters of a Transformer-based autoregressive language model by minimizing the multi-task joint loss function using a multi-task training dataset, thereby enabling the autoregressive language model to obtain a unified layout sequence processing capability.

[0171] The creative-driven storyboard generation module 33 is used to concatenate the creative description text input by the user with the storyboard generation task identifier and input it into the autoregressive language model, and generate the corresponding initial continuous shot storyboard layout sequence through autoregressive decoding.

[0172] The interactive storyboard editing module 34 is used to respond to user editing commands. It concatenates the historical shot storyboard layout sequence before the frame to be edited, the editing command, and the interactive editing task identifier, and inputs them into the autoregressive language model. It then outputs the edited subsequent shot storyboard layout sequence through a conditional generation method.

[0173] The storyboard consistency verification module 35 is used to respond to user verification commands, concatenate the storyboard layout sequence of the shot to be verified and its content description text with the storyboard verification task identifier, input it into the autoregressive language model, and output the consistency judgment result; wherein, when the consistency judgment result is qualified, it means that the storyboard layout sequence of the shot to be verified conforms to the content description text.

[0174] The intermediate frame intelligent filling module 36 is used to respond to user filling instructions. It concatenates the specified first and last frame shot layout, the number of filling frames, and the intermediate frame filling task identifier and inputs them into the autoregressive language model. The model outputs the intermediate frame shot layout sequence.

[0175] The shot sequence fusion module 37 is used to integrate the initial continuous shot layout sequence, the subsequent shot layout sequence with a consistency judgment result of qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

[0176] Embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the aforementioned method embodiments.

[0177] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0178] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0179] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A video storyboard layout generation method based on an autoregressive model, characterized in that, The method includes: S1. Extract information and track targets from the original video to generate video annotation data including shot text descriptions, camera parameters, character pose sequences, and object bounding box sequences; construct a multi-task training dataset based on the video annotation data, including storyboard generation tasks, interactive editing tasks, intermediate frame filling tasks, and storyboard verification tasks. S2. Using the multi-task training dataset, train the parameters of a Transformer-based autoregressive language model by minimizing the multi-task joint loss function, so that the autoregressive language model can obtain a unified layout sequence processing capability. S3. The user-input creative description text and the storyboard generation task identifier are concatenated and input into the autoregressive language model. The corresponding initial continuous shot storyboard layout sequence is generated through autoregressive decoding. S4. In response to the user's editing command, the historical shot layout sequence before the frame to be edited, the editing command, and the interactive editing task identifier are concatenated and input into the autoregressive language model, and the edited subsequent shot layout sequence is output through a conditional generation method. S5. In response to the user's verification command, the sequence of shots to be verified and its content description text are concatenated with the shot verification task identifier and input into the autoregressive language model, and the consistency judgment result is output; wherein, when the consistency judgment result indicates that the sequence of shots to be verified conforms to the content description text. S6. In response to the user's fill command, the specified first and last frame shot layout, fill frame number and middle frame fill task identifier are concatenated and input into the autoregressive language model, and the middle frame shot layout sequence is output through the model. S7. Integrate the initial continuous shot layout sequence, the subsequent shot layout sequence whose consistency judgment result is qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

2. The method according to claim 1, characterized in that, The step of training the parameters of a Transformer-based autoregressive language model using the multi-task training dataset by minimizing the joint multi-task loss function includes: S11. Convert the input information of each training sample in the multi-task training dataset into a discrete token sequence, and add a task identifier token corresponding to the task type at the beginning of the discrete token sequence. S12. Input the discrete token sequence with the added task identifier token into the autoregressive language model, and predict the probability distribution of the next token through the forward propagation calculation of the autoregressive language model. S13. Based on the probability distribution of the next token and the corresponding real target token in the training sample, calculate the corresponding task sub-loss value according to the task type of the current training sample; wherein, the task sub-loss value includes the identity matching loss for the storyboard generation task and the interactive editing task. Layout similarity loss Motion smoothing loss for intermediate frame filling task And the loss of instruction correctness in the storyboard verification task. ; S14. The weighted sum of the sub-loss values ​​of each task is used to obtain the multi-task joint loss function. The value of ; where the multi-task joint loss function The calculation formula is ;in, , , , These are the preset weighting coefficients corresponding to the sub-losses of each task; S15. Based on the value of the multi-task joint loss function, calculate the parameter gradient of the autoregressive language model using the backpropagation algorithm, and update the parameters of the autoregressive language model based on the parameter gradient.

3. The method according to claim 2, characterized in that, The calculation steps for the layout similarity loss include: S21. Based on the complete token sequence predicted by the autoregressive language model for the current training sample, decode to obtain the predicted layout element position information for the current frame; based on the actual complete target token sequence in the current training sample, decode to obtain the actual labeled layout element position information for the current frame. The layout element position information includes joint coordinates in the character pose sequence and corner coordinates in the object bounding box sequence; the predicted layout element position information set is represented as follows. The set of actual marked layout element position information is represented as This indicates the number of elements that need to be laid out in the current frame. Indicating the first element in the set of predicted layout element position information A vector of coordinates of elements. This indicates the first element in the set of actual labeled layout element position information. A vector of coordinates of elements. ; S22. Calculate the coordinate distance between all element pairs in the predicted layout element position information set and the actual labeled layout element position information set, and construct a distance matrix based on the coordinate distance; wherein, the distance matrix is... matrix The distance matrix The Middle Line number Column elements Represents the predicted element With real elements The Euclidean distance between them; S23. Based on the distance values ​​in the distance matrix as the matching cost, with the goal of minimizing the total matching cost, the Hungarian algorithm is used to solve for an optimal bijective matching result between the predicted layout element position information set and the elements in the actual labeled layout element position information set. S24. Based on the optimal bijective matching result, calculate the layout similarity sub-loss of the current frame; wherein, the layout similarity loss of the current frame... The calculation formula is: in, Indicates the result of the optimal bijective matching with the first... The index of the real element that matches each predicted element. The L2 norm of a vector; S25. Summate the layout similarity sub-loss of all frames in the sequence to obtain the layout similarity loss of the current training sample. .

4. The method according to claim 2, characterized in that, The calculation steps for the motion smoothing loss include: S31. Based on the complete token sequence predicted by the autoregressive language model for the intermediate frame filling task, decode to obtain the generated intermediate frame layout sequence; The intermediate frame layout sequence includes the joint coordinates of the character pose sequence and the corner coordinates of the object bounding box sequence in each frame; the intermediate frame layout sequence is represented as follows: , The number of intermediate frames. Indicates the first All of the intermediate frame layout sequences of the frames A set of coordinate vectors of elements. ; S32, for the first frame in the intermediate frame layout sequence Frame, based on the set of coordinate vectors, calculates the motion smoothing error of each element in the intermediate frame layout sequence; wherein, , No. Motion smoothing error of each element The calculation formula is: in, Indicates the first The set of coordinate vectors of the frame The Middle A vector of coordinates of elements; S33, regarding the first The motion smoothing errors of all elements in the intermediate frame layout sequence of the frame are summed to obtain the frame-level smoothing loss of the corresponding frame; wherein, the frame-level smoothing loss The calculation formula is ; S34. Layout the intermediate frame sequence Frame-level smoothing loss for all intermediate frames Summing is performed to obtain the overall motion smoothing loss for the current training samples. The formula for calculating the overall motion smoothing loss is as follows: .

5. The method according to any one of claims 1 to 4, characterized in that, The step of concatenating the user-inputted creative description text with the storyboard generation task identifier and inputting it into the autoregressive language model, and generating the corresponding initial continuous shot storyboard layout sequence through autoregressive decoding, includes: S41. Tokenize the creative description text to obtain a text token sequence; concatenate the predefined storyboard generation task identifier token with the text token sequence to form the initial decoding input sequence; S42. Input the initial decoded input sequence into the autoregressive language model, use the forward propagation calculation of the autoregressive language model to predict the next token based on all tokens in the initial decoded input sequence, and output the probability distribution of the first predicted token; S43. Sample according to the probability distribution of the first predicted token to obtain the first generated token, and append the first generated token to the end of the initial decoding input sequence to form a new decoding input sequence; S44. Input the new decoded input sequence into the autoregressive language model, predict the next token through the autoregressive language model, output the probability distribution of the next predicted token and sample it, and continue to append the newly sampled token to the input sequence; S45. Repeat step S44 to iteratively generate subsequent tokens until the generated token is a predefined sequence end token or the sequence length reaches the preset maximum value, and obtain a complete output token sequence. S46. The output token sequence is detoxified and decoded into a structured initial continuous shot layout sequence; wherein the initial continuous shot layout sequence includes camera parameters, character identification and posture coordinate sequence, object identification and bounding box coordinate sequence.

6. A video storyboard layout generation system based on an autoregressive model, used to implement the method of any one of claims 1 to 5, characterized in that, The system includes: The video semantic annotation construction module is used to extract information and track targets from the original video, and generate video annotation data including shot text descriptions, camera parameters, human pose sequences and object bounding box sequences; based on the video annotation data, a multi-task training dataset is constructed, including storyboard generation tasks, interactive editing tasks, intermediate frame filling tasks and storyboard verification tasks. The multi-task model training module is used to train the parameters of a Transformer-based autoregressive language model by minimizing the multi-task joint loss function using the multi-task training dataset, so that the autoregressive language model can obtain a unified layout sequence processing capability. The creative-driven storyboard generation module is used to concatenate the creative description text input by the user with the storyboard generation task identifier and input it into the autoregressive language model, and generate the corresponding initial continuous shot storyboard layout sequence through autoregressive decoding; The interactive storyboard editing module is used to respond to user editing commands. It concatenates the historical shot storyboard layout sequence before the frame to be edited, the editing command, and the interactive editing task identifier, and inputs them into the autoregressive language model. It then outputs the edited subsequent shot storyboard layout sequence through a conditional generation method. The storyboard consistency verification module is used to respond to user verification commands, concatenate the storyboard layout sequence of the shot to be verified and its content description text with the storyboard verification task identifier, input it into the autoregressive language model, and output the consistency judgment result; wherein, when the consistency judgment result indicates that the storyboard layout sequence of the shot to be verified conforms to the content description text. The intermediate frame intelligent filling module is used to respond to user filling instructions, and inputs the specified first and last frame shot layout, filling frame number and intermediate frame filling task identifier into the autoregressive language model, and outputs the intermediate frame shot layout sequence through the model. The shot sequence fusion module is used to integrate the initial continuous shot layout sequence, the subsequent shot layout sequence whose consistency judgment result is qualified, and the intermediate frame shot layout sequence to generate the final shot layout sequence.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images