Method and system for generating video of movie and television character guided by text sketch based on AIGC

Abstract

This invention provides a text sketch-guided video generation method and system for film and television characters based on AIGC (AIGC), relating to the fields of computer vision, natural language processing, and AIGC video generation technology. The method includes: acquiring text sketches to obtain the static outline of each character, initial character description, and initial scene description; performing semantic event coarse decomposition on the initial scene description and performing action parsing and duration estimation, and multi-granular semantic parsing on each first sub-description to obtain a guidance set; decomposing the initial character description to construct an outline component graph, generating pose parameters and rendering parameter sequences through a graph conditional diffusion model, and generating initial dynamic change lines for the corresponding character for processing, filtering to obtain fine decomposition points; encoding point-level semantic features for point-guided descriptions, performing multi-character spatiotemporal joint alignment to obtain a complete sequence, inputting the video generation diffusion model to output a film and television character video. This improves the overall automation level and generation effect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, natural language processing, and AIGC video generation technology, and particularly to a method and system for generating film and television character videos based on AIGC text sketches. Background Technology

[0002] With the development of AIGC video generation technology, the demand for converting static comic page content into dynamic film and television character videos is increasing.

[0003] Existing technologies for converting comics to video mainly fall into two categories: 3D reconstruction-driven and end-to-end 2D generation. 3D reconstruction-driven solutions require first constructing a 3D model of the character and then binding motion data to drive generation. While the posture and motion logic is relatively regular, the output often loses the original 2D hand-drawn art style. Furthermore, the production chain is long and labor-intensive, making it difficult to adapt to the rapid generation needs of batch comic content. Some end-to-end 2D generation solutions can directly output video based on static images, preserving the original visual style relatively well. However, the generation process generally lacks clear human joint topological constraints and multi-character collaborative control mechanisms. The output character postures often exhibit motion logic that does not conform to human anatomy, and distorted component connections. In multi-character interaction scenarios, the consistency of action timing, spatial position, and semantic logic is also difficult to guarantee, and the overall generation quality cannot meet the presentation requirements of film-level character animation.

[0004] Therefore, how to simultaneously preserve the original art style, ensure the rationality of character postures and movements, and improve the consistency of multi-character actions under a pure 2D generation system is a core technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention provides a text sketch-guided video generation method and system for film and television characters based on AIGC, in order to solve the aforementioned technical problems.

[0006] This invention provides a text sketch-guided video generation method for film and television characters based on AIGC, comprising: Step 1: Obtain text sketches and use multimodal large models to obtain the static outline of each character, initial character description, and initial scene description; Step 2: Perform semantic event coarse decomposition on the initial scene description to obtain the first sub-description. Perform action parsing and duration estimation on each first sub-description. At the same time, perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, and combine them into a guidance set. Step 3: The initial character description is split into a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, a shape component diagram with joint connection constraints is constructed. The posture parameter and rendering parameter sequence are generated through graph condition diffusion model, and the initial dynamic change line of the corresponding character is generated. Step 4: Process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak and end of a single action. At the same time, detect the local peak points and valley points of the intensity curve, filter out the fine-grained points, and generate corresponding point guidance descriptions. Step 5: Encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. Input the video to generate a diffusion model and output the film and television character video.

[0007] Preferably, multi-granular semantic parsing is performed on each first sub-description to obtain corresponding preset dimension screen guidance tags, including: For each first sub-description, perform multi-granularity semantic parsing to extract semantic units at the semantic level, modification attribute level, action granularity level, and scene association level in sequence. Input the semantic units at each level into a pre-trained language model to encode them into semantic vectors, calculate the cosine similarity with the corresponding level's baseline semantic prototype vector, and obtain a hierarchical semantic matching score with a value range of [0,1]. Concatenate the scores of the four levels in a fixed order to obtain a description granularity sequence set with a length L=4. Obtain a pre-constructed reference granularity sequence set that maps one-to-one with each preset dimension, wherein the reference granularity sequence set is a numerical vector of length 4, obtained by clustering typical samples under the corresponding dimension; The cosine similarity between the description granularity sequence set and the reference granularity sequence set for each preset dimension is calculated to obtain the matching confidence value for each preset dimension. Based on the matching confidence value corresponding to each preset dimension, interval matching is performed with the preset dimension-feature-label table to obtain several candidate guide labels for the corresponding preset dimension. Discover the semantic mutual exclusion relationship and attribute linkage relationship among several candidate guidance tags in the same preset dimension, eliminate semantically conflicting tags and retain tags with consistent attribute logic as the corresponding screen guidance tags for the preset dimension.

[0008] Preferably, for each first sub-description, action parsing and duration estimation are performed, including: For each first sub-description, perform action parsing to identify the included action entities and their order to obtain an action sequence, wherein the action sequence contains K actions; For each action, the prior average duration and prior standard deviation of the action are obtained from the pre-built action type-duration table based on the action type. Extract two visual features from the action area corresponding to the current text sketch: grid ratio and action line density. Then, calculate the action duration observation value and the standard deviation of the observation noise by weighting the preset linear fitting coefficients. Under the assumption of normal distribution, the estimated duration of each action is calculated by combining the prior distribution and the observed values ​​through the closed-form solution of the maximum a posteriori estimation. The sum of the estimated durations of all actions in the action sequence is taken as the estimated screen duration, and the length K of the action sequence is taken as the estimated number of actions in the screen. For each action, the action type feature vector of the action is concatenated with the matching confidence value of the corresponding preset dimension, mapped to a fusion feature through a fully connected layer, and input into a pre-trained text generator to generate the action guidance description corresponding to the action.

[0009] Preferably, generating the initial dynamic change line for the corresponding character includes: Multi-scale curvature analysis is performed on the static outline, and the body is segmented into N closed outline components based on human topological priors. The Hu moment geometric invariant features of each closed outline component are extracted. Construct a shape component diagram with joint connection constraints, using each closed shape component as a node and the joint connection relationship between components as an edge; The basic shape attribute description and dynamic action attribute description in the split description pair are input into the text encoder to obtain the character shape semantic vector and action text vector. Combined with the scene semantic vector of the first sub-description, the character-scene coupled semantic features are obtained. The estimated screen duration, estimated screen action number, and action guidance description for each estimated screen action in the guidance set are encoded as dynamic temporal condition features, and the scene requirements are encoded as scene requirement semantic features. The dynamic temporal condition features, scene requirement semantic features and role-scene coupling semantic features are concatenated and mapped through a fully connected layer to obtain dynamic constraint features of a unified dimension. The dynamic constraint features are embedded with a time-length embedding vector, which is used to control the time length of the output sequence of the graph conditional diffusion model. The graph features of the external component graph and the dynamic constraint features are used as input to the graph conditional diffusion model. The training loss function of the graph conditional diffusion model includes a noise prediction loss term, a temporal smoothing regularization term, and a joint relative distance constraint term. This generates a sequence of external component pose parameters for T video time steps. Each video time step includes N component x-axis offset, y-axis offset, and planar rotation angle. After the inference output, a joint overlap verification is performed, and position correction is performed on components with deviations exceeding the limit. Based on the posture parameters of each component, a two-dimensional affine transformation is performed on the corresponding component segments of the static outline with the corresponding joint point as the rotation center, and cubic spline interpolation is performed on the connection area of ​​adjacent components to construct the intermediate character outline. Extract the position index of each shape component on the middle character outline, and use it as the parameter position; Extract the rendering list corresponding to the parameter position from the pre-built position-lookup table. The rendering list includes position attributes, layout attributes and adjacent position transition attribute groups. The execution priority of rendering attributes follows the rule of layout attributes first, position attributes second, and transition attribute groups last. In case of attribute conflict, the higher priority attribute shall prevail. The posture parameter sequences of the external components are grouped according to temporal continuity and component correlation, and a rendering event is set for each group; According to the order of rendering events, for each parameter position in the corresponding group, the rendering attributes are set sequentially according to the attribute priority of the rendering list to obtain the rendering parameter sequence for each video time step. The sequence of pose parameters of the external parts at the same video time step is combined with the sequence of rendering parameters to generate the initial dynamic change line of the corresponding character.

[0010] Preferably, the estimated screen duration, estimated screen action number, and action guidance description for each estimated screen action in the guidance set are encoded as dynamic temporal conditional features, including: Based on the estimated number of actions in the scene, the guidance description for each action is input into the text encoder to obtain an action semantic vector of dimension D; The estimated duration of each action is normalized and mapped to a time span vector of dimension D through a learnable linear layer; the action semantic vector and the time span vector of the same action are concatenated and converted into an action event feature vector of dimension D through a linear mapping layer. A learnable temporal location code is added to each action event feature vector, and the action event feature sequence is obtained by arranging them in the order of the actions. The action event feature sequence is input into a two-layer Transformer encoder for temporal self-attention encoding, and the output is a dynamic temporal conditional feature with dimension D.

[0011] Preferably, the process involves filtering to obtain detailed breakdown points and generating corresponding point guidance descriptions, including: Calculate the relative rate of change of each pose parameter and the relative rate of change of each rendering parameter between adjacent video time steps in the initial dynamic change line; The relative rates of change of all parameters are uniformly mapped to the [0,1] interval to complete the normalization process; Based on preset weights, the average relative change rate of the normalized attitude parameters and the average relative change rate of the rendering parameters are weighted and summed to construct a parameter change intensity curve. The first-order difference of the intensity curve is calculated to identify all local extreme points, including local peak points in the corresponding peak phase of the action amplitude and local valley points in the corresponding steady-state phases at the beginning and end of the action; at the same time, the second-order difference of the intensity curve is calculated, and inflection points whose absolute value of the second-order difference exceeds the preset curvature threshold are added as candidate subdivision points, and finally a set of candidate subdivision points is obtained. Based on the rule that each action corresponds to three types of keyframes—start, peak, and end—determine the target number of subdivision points; If the number of candidate subdivision points is greater than the target number, the highest intensity peak point is selected as the action peak point, and the two valley points with the closest distance before and after the peak point on the time axis are selected as the action start point and end point respectively, to obtain the final subdivision points of the target number; If the number of candidate subdivision points is less than the target number, retain all candidate subdivision points and evenly supplement the sampling points of the difference number on the time axis at the corresponding action stage, so that the final number of subdivision points is consistent with the target number. For each final subdivision point, extract the pose parameter features and rendering parameter features of the two video time steps before and after it, and generate a point guidance description for that subdivision point using a preset text template.

[0012] Preferably, performing multi-role spatiotemporal joint alignment and inputting the video to generate a diffusion model to output film and television character videos includes: Combining the three types of constraints in the scenario requirements—temporal synchronization, spatial interaction, and action coordination—temporal synchronization alignment, spatial position alignment, and action semantic coordination correction are performed sequentially to obtain aligned point-level semantic features and aligned spatial parameters. Among them, temporal synchronization alignment and spatial position alignment are hard constraints, while action semantic coordination correction is a soft constraint. The constraint priority is spatial interaction constraint > temporal synchronization constraint > action coordination constraint. Based on the point-level semantic features and spatial parameters of each aligned subdivision point, the screen generation parameters for each character at the corresponding subdivision point are generated. Perform linear interpolation of parameters on non-key frames between key frames to obtain a complete sequence of image generation parameters for the timeline, which is considered a complete sequence. Input the complete sequence of all characters into a 2D video generation diffusion model in chronological order to generate film and television character videos with corresponding text sketches.

[0013] This invention provides a text sketch-guided video generation system for film and television characters based on AIGC, comprising: The contour extraction and description acquisition module is used to acquire text sketches and obtain the static outline, initial character description and initial scene description of each character through a multimodal large model; The coarse segmentation and guidance set generation module is used to perform semantic event coarse segmentation on the initial scene description to obtain the first sub-description, perform action parsing and duration estimation on each first sub-description, and perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, which are combined into a guidance set. The initial dynamic change line generation module is used to split the initial character description to obtain a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, a shape component diagram with joint connection constraints is constructed. The posture parameter and rendering parameter sequence are generated through the graph condition diffusion model, and the initial dynamic change line of the corresponding character is generated. The fine-splitting module is used to process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak, and end of a single action. At the same time, it detects local peak points and valley points of the intensity curve, filters out fine-splitting points, and generates corresponding point guidance descriptions. The spatiotemporal alignment and video generation module is used to encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and to perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. The input video generation diffusion model outputs a video of film and television characters.

[0014] Compared with the prior art, the beneficial effects of this application are as follows: 1. It can generate in-plane pose transformations based solely on two-dimensional static contours without the need for three-dimensional reconstruction. The transformation method with joints as the rotation center conforms to the laws of human movement. Combined with smooth interpolation of connected regions and post-inference verification, it can effectively avoid contour breakage and better preserve the original art style. The consistency of character appearance is significantly better than conventional end-to-end generation solutions.

[0015] 2. By simultaneously constraining temporal synchronization, spatial location, and action logic through a spatiotemporal joint alignment mechanism, and combining hierarchical pixel-physical scale mapping rules and constraint priority mechanisms, the action matching accuracy in multi-role interaction scenarios is significantly improved compared to pure temporal alignment schemes. Orthogonal decomposition-based semantic correction can achieve action coordination while ensuring the character's ontological attributes, avoiding excessive semantic shift.

[0016] 3. A generation mechanism using three types of keyframes—start, peak, and end of a single action—combined with parameter interpolation is adopted. By jointly detecting peak and valley values, the keyframes are accurately located, which can completely depict the entire process of the action and effectively improve the naturalness of the action transition.

[0017] 4. Local repair based on the character's inherent texture template solves the problem of style drift in general algorithms, and significantly improves the style similarity between the repaired area and the original image.

[0018] 5. The maximum a posteriori estimation, which integrates prior statistics and local visual features of comics, can effectively reduce the error in action duration prediction and make the rhythm of the scene more in line with the expression of the original work.

[0019] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a text sketch-guided video generation method for film and television characters based on AIGC, as described in an embodiment of the present invention. Figure 2 This is a structural diagram of a text sketch-guided video generation system for film and television characters based on AIGC, as described in an embodiment of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] Example 1: This example provides a text sketch-guided video generation method for film and television characters based on AIGC, such as... Figure 1 As shown, it includes: Step 1: Obtain the text sketch, and use the multimodal large model to obtain the static outline of each character, the initial character description, and the initial scene description.

[0024] In this embodiment, the text sketch is a static image containing character images and text information, serving as the input material for the solution. It is in JPEG / PNG format with a resolution of at least 512×512 pixels, such as a static comic page image. The scene requirements are user-inputted multi-character interaction constraints, including cross-character temporal synchronization relationships, spatial interaction relationships, and action coordination relationships. The multimodal large model is a pre-trained neural network model capable of simultaneously processing image and text inputs, possessing visual understanding and language generation capabilities, used to uniformly extract contours, character attributes, and scene semantic information from the comic. The static outline is the closed boundary of the character, composed of a set of two-dimensional pixel coordinate points arranged clockwise. The static outline is uniformly resampled to a fixed number of points M. Uniform resampling is an algorithm that samples evenly along the total perimeter of the outline. The steps are: calculating the total arc length of the outline, calculating the sampling interval = total arc length / M, accumulating the arc length point by point along the outline, recording one coordinate point at each interval, and finally obtaining M ordered coordinate points. The sampling starting point is fixed at the extreme point at the top of the outline, and the first and last points of the closed outline are not counted repeatedly.

[0025] Step 2: Perform semantic event coarse decomposition on the initial scene description to obtain the first sub-description. Perform action parsing and duration estimation on each first sub-description. At the same time, perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, which are combined into a guidance set.

[0026] In this embodiment, the coarse semantic event segmentation adopts a large language model-driven event segmentation method. The segmentation granularity follows the principles of single continuous action, single spatial scene, and single semantic event. The segmentation prompt template is: Please split the following scene description into multiple first sub-descriptions according to independent semantic events. Each first sub-description corresponds to a continuous action segment. It cannot be segmented too finely, nor can it cross spatial scenes. The output format is one description per line: Initial Scene Description. After segmentation, the estimated duration of each first sub-description is controlled within the range of 0.5-3 seconds by default. If it exceeds this range, it will be segmented again or merged. After the coarse semantic event segmentation, two levels of verification are performed: First, action number verification: the number of actions parsed in a single first sub-description should be controlled within 1-3. If it exceeds this range, it will be segmented again. If it is insufficient, it will be merged with adjacent sub-descriptions. Second, duration verification: the estimated total duration of a single first sub-description should be controlled within the range of 0.5-3 seconds. If it exceeds this range, it will be segmented according to the action boundary. If it is insufficient, it will be merged with adjacent sub-descriptions to ensure uniform segmentation granularity.

[0027] The guidance set is a structured set of guidance parameters for a single first sub-description, including the estimated screen duration, the estimated number of screen actions, the action guidance description list, and the screen guidance tag set.

[0028] Step 3: Decompose the initial character description to obtain a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, construct a shape component diagram with joint connection constraints. Generate posture parameters and rendering parameter sequences through graph condition diffusion model, and generate the initial dynamic change line of the corresponding character.

[0029] Step 4: Process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak, and end of a single action. At the same time, detect the local peak and valley points of the intensity curve, filter out the subdivision points, and generate corresponding point guidance descriptions.

[0030] Step 5: Encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. Input the video to generate a diffusion model and output the film and television character video.

[0031] Text sketches are static 2D materials that include hand-drawn character images, storyboard layouts, and accompanying explanatory text. They are in JPEG / PNG format with a resolution of at least 512×512 pixels and are typically single-page comic storyboards or character design drafts.

[0032] The first sub-description is a single scene fragment text obtained by splitting the initial scene description into independent semantic events. Each sub-description corresponds to a continuous action fragment, and the estimated duration of a single sub-description is controlled between 0.5 and 3 seconds.

[0033] The split description pair is a binary text group obtained by splitting the initial character description of a single character according to the attribute dimension. It consists of basic shape attribute description and dynamic action attribute description. The former is used to construct the static shape component diagram of the character, and the latter is used to generate dynamic posture constraints.

[0034] The external component graph is a graph structure constructed with closed external components of the character as nodes and joint connections between components as edges. The node features are the geometrically invariant features of the component region, and the edge features are the joint constraint weights and topological connection attributes between components.

[0035] The initial dynamic change line is a parameterized dynamic representation of a single character in the complete time sequence. It is composed of the sequence of pose parameters of the shape parts and the sequence of rendering parameters corresponding to each video time step, and serves as an intermediate parameter layer from the static outline to the final video.

[0036] The fine-grained split points are key time nodes selected from the initial dynamic change line, corresponding to the key frame positions of the action; a single complete action corresponds to three types of fine-grained split points: start, peak, and end, which correspond to the three stages of action initiation, maximum action amplitude, and action convergence, respectively.

[0037] Point guidance descriptions are natural language descriptions of the character's pose and rendering state at a single detailed split point, with a length controlled between 10 and 20 characters, used to encode the semantic features of keyframes.

[0038] Spatiotemporal joint alignment is a joint constraint process that sequentially performs temporal synchronization alignment, spatial position alignment, and action semantic collaborative correction for multi-role scenarios. Among them, temporal and spatial are hard constraints, while semantic collaboration is a soft constraint.

[0039] In this embodiment, the point-level semantic features are 512-dimensional vectors obtained by encoding point-guided descriptions, used for multi-role alignment and screen parameter generation. Spatiotemporal joint alignment is a hard constraint, with higher priority than soft constraints within the diffusion model; in case of conflict, the alignment result prevails. The priority of the three types of constraints within the hard constraints is: spatial interaction constraints > temporal synchronization constraints > action coordination constraints. When multiple constraints cannot be satisfied simultaneously, they are guaranteed sequentially according to priority. The joint solution of the three types of constraints follows a priority progression rule: first, the coordinate offset of the spatial interaction constraints is executed; then, the timestamp adjustment of the temporal synchronization constraints is executed; finally, the semantic weighting of the action coordination constraints is executed. If two types of constraints conflict, the execution result of the higher-priority constraint prevails, and the lower-priority constraint is locally adjusted within the remaining degrees of freedom without overturning the higher-priority result.

[0040] The 2D video generation diffusion model adopts the 2DUNet architecture, with an output frame rate of 25fps and a spatial resolution similar to the input text. Figure 1 Conditional injection employs a combination of structured parameter encoding and cross-attention fusion, specifically implemented as follows: Image generation parameter split encoding: Four types of heterogeneous parameters are encoded separately and then fused into a unified conditional feature: Character outline coordinates: encoded into a 512-dimensional outline feature vector through 2 layers of 1D convolution; Rendering attributes and lighting parameters: concatenated and encoded into a 256-dimensional attribute feature vector through 2 layers of fully connected layers; Spatial position parameters: encoded into a 256-dimensional position embedding vector; The three types of features are concatenated and mapped into a 1024-dimensional single-character conditional feature through a fully connected layer.

[0041] Multi-role parameter fusion: The conditional features of all roles are concatenated in spatial order and fused into global frame conditional features through a cross-attention layer, maintaining a dimension of 1024. At the same time, learnable frame position encoding is added to distinguish conditional signals at different time steps.

[0042] Conditional injection method: Global frame conditional features are injected into each residual block of UNet through adaptive layer normalization (AdaLN). Simultaneously, conditional cross-attention layers are added to the three downsampling and three upsampling layers of UNet to implement structured parameters that constrain the entire generation process, ensuring consistency in character outline, attributes, and position. Specifically, the character outline coordinates are a closed outline sequence composed of M two-dimensional pixel coordinates, consistent with the static shape outline format; rendering attributes include three types of parameters: line thickness, fill color level, and shadow intensity; spatial position parameters are the character's global offset coordinates (x_offset, y_offset) in the image, in pixels; lighting parameters include ambient light intensity and principal light direction angle, with ambient light intensity being a normalized value of 0-1 and principal light direction angle being 0-2π radians.

[0043] Take a comic page featuring a blonde girl and a blue-haired boy as an example. The scene requirements are: the two are running towards the river at the same time, with the girl in front and the boy chasing behind. The two are 1 meter apart, and both characters are standing with their bodies fully upright and in a slightly sideways frontal view.

[0044] Input the comic page into the multimodal large model. The prompt instructions are: extract the closed contour coordinates of each character in the image, character appearance description, and overall scene description. The model outputs two static outlines of 200 points each for the two characters, as well as the corresponding initial character description and initial scene description. The pixel-physical scale mapping coefficient is calculated to be 200 pixels / meter.

[0045] The initial scene description is split into two sub-descriptions: running in the forest and slowing down upon reaching the river.

[0046] The first sub-description is parsed to obtain a running action with an estimated duration of 1.15 seconds. The corresponding screen guidance label and action guidance description are generated and combined into a guidance set.

[0047] The external component diagrams of the two characters are constructed and fused to obtain dynamic constraint features. The input graph conditional diffusion model generates the external component pose parameters for 29 video time steps. After connection and smoothing, the initial dynamic change lines of each character are obtained.

[0048] Following the rule of three keyframes for a single action, three detailed sub-points are selected from the initial dynamic change line, corresponding to the start, peak, and end states of the action, respectively, and corresponding point guidance descriptions are generated.

[0049] The point guidance description is encoded into point-level semantic features, and the spatial distance is converted according to a coefficient of 200 pixels / meter. Spatiotemporal joint alignment is performed: the time stamps of the two people's actions are aligned, the girl's x-coordinate is 200 pixels ahead, key frame image generation parameters are generated, the intermediate frames are interpolated and then input into the 2D video to generate a diffusion model, and the output is a 25fps character animation video.

[0050] Example 2: In this example, multi-granular semantic parsing is performed on each first sub-description to obtain corresponding preset dimension screen guidance tags. The specific steps are as follows: Step 21: Multi-level semantic unit extraction and vectorization: Perform multi-granularity semantic parsing on each first sub-description to extract semantic units at the semantic level, modification attribute level, action granularity level, and scene association level in sequence; input the semantic units of each level into the pre-trained language model, encode them into semantic vectors of the corresponding dimension, and then calculate the cosine similarity with the baseline semantic prototype vector of the same level to obtain the hierarchical semantic matching score with a value range of [0,1]; concatenate the scores of the four levels in a fixed order to obtain a description granularity sequence set (four-dimensional numerical vector) of length L=4.

[0051] Step 22: Reference Granularity Sequence Matching: Obtain a pre-constructed set of reference granularity sequences that are mapped one-to-one with each preset dimension; the set of reference granularity sequences is a numerical vector of length 4, which is obtained by K-means clustering of more than 1,000 labeled typical samples in the corresponding dimension to obtain the cluster center vector, which serves as the semantic matching template for that dimension.

[0052] Step 23: Matching confidence calculation: Calculate the cosine similarity between the description granularity sequence set and the reference granularity sequence set for each preset dimension to obtain the matching confidence value for each preset dimension, with a value range of [-1, 1].

[0053] Cosine similarity calculation formula: ,in, To describe the granularity sequence set vector, A reference granularity sequence set vector of a certain preset dimension. Let L2 norm be the vector. This represents the matching confidence score of the output.

[0054] Step 24: Candidate label interval matching: Based on the matching confidence value corresponding to each preset dimension, interval matching is performed with the preset dimension-feature-label table to obtain several candidate guiding labels for the corresponding preset dimension; the dimension-feature-label table is divided into 5 numerical intervals according to the matching confidence, each interval corresponds to 2 to 3 candidate guiding labels, and is constructed by more than 500 labeled samples after expert verification, as shown in Table 1.

[0055] Table 1 contains a partial list of the dimensions, features, and labels for the action amplitude dimension.

[0056] Step 25: Tag semantic verification and filtering: Mine the semantic mutual exclusion relationship and attribute linkage relationship between several candidate guidance tags in the same preset dimension, remove semantically conflicting tags (such as "large amplitude" and "small amplitude" cannot coexist in the same dimension) and retain tags with consistent attribute logic as the corresponding preset dimension screen guidance tags.

[0057] In this embodiment, multi-granularity semantic parsing is a process of decomposing text into semantic units at four levels: semantic level, modifier attribute level, action granularity level, and scene association level. This process is used to characterize the semantic features of the text from different dimensions. The description granularity sequence set is a fixed-length one-dimensional numerical vector obtained by concatenating the semantic matching scores of the four levels after multi-granularity semantic parsing. Its length L=4, representing the multi-level semantic distribution of the text. The reference granularity sequence set is a standard granularity vector obtained by clustering more than 1000 typical samples under a single preset dimension. Its length is consistent with the description granularity sequence set, serving as the semantic matching template for that dimension. The matching confidence score is the cosine similarity value between the description granularity sequence set and the reference granularity sequence set, ranging from -1 to 1, representing the degree of matching between the current text and the typical semantics of the corresponding dimension. The dimension-feature-label table is a pre-constructed mapping data table. Under each preset dimension, it is divided into 5 intervals according to the numerical range of the matching confidence score, with each interval corresponding to several candidate guiding labels.

[0058] For example, for the first sub-description "running in the forest", multi-granularity semantic parsing is performed: the semantic level is action, the modifier attribute level is fast, the action granularity level is the overall action, and the scene association level is the outdoor environment. After encoding and matching degree calculation, the matching degree scores of the four levels are 0.8, 0.9, 0.7, and 0.6 respectively, forming a description granularity sequence set [0.8, 0.9, 0.7, 0.6].

[0059] Cosine similarity was calculated with reference vectors for four preset dimensions: movement amplitude, emotional intensity, spatial range, and temporal rhythm. The matching confidence scores were 0.92, 0.85, 0.73, and 0.88, respectively. The corresponding candidate labels were obtained by querying the dimension-feature-label table. After semantic mutual exclusion verification, the final output was: "large stride" for movement amplitude, "tense" for emotional intensity, "open" for spatial range, and "rapid" for temporal rhythm.

[0060] Example 3: This example involves parsing and estimating the execution action and duration for each first sub-description, specifically including: For each first sub-description, perform action parsing to identify the included action entities and their order to obtain an action sequence, wherein the action sequence contains K actions; For each action, the prior average duration and prior standard deviation of the action are obtained from the pre-built action type-duration table based on the action type. Extract two visual features from the action area corresponding to the current text sketch: grid ratio and action line density. Then, calculate the action duration observation value and the standard deviation of the observation noise by weighting the preset linear fitting coefficients. Under the assumption of normal distribution, the estimated duration of each action is calculated by combining the prior distribution and the observed values ​​through the closed-form solution of the maximum a posteriori estimation. The sum of the estimated durations of all actions in the action sequence is taken as the estimated screen duration, and the length K of the action sequence is taken as the estimated number of actions in the screen. For each action, the action type feature vector of the action is concatenated with the matching confidence value of the corresponding preset dimension, mapped to a fusion feature through a fully connected layer, and input into a pre-trained text generator to generate the action guidance description corresponding to the action.

[0061] In this embodiment, action parsing employs a semantic extraction method driven by a large language model. Recognition is performed based on a pre-defined action classification system, which includes 12 primary categories and 48 secondary categories, covering common human actions such as standing, walking, running, raising an arm, and turning, as shown in Table 2. The parsing output is a standardized action sequence. Each action includes three fields: action type code, action start semantic position, and action end semantic position, ensuring consistency between action boundaries and type determination.

[0062] Table 2 Action System Table

[0063] The Action Type-Duration Table is a pre-built statistical data table that stores the prior average duration and prior standard deviation for each action type. It is derived from statistics of over 1200 action clips from films and comics, with the time unit being seconds, as shown in Table 3. Maximum a posteriori (MAP) estimation is a statistical method that integrates prior distribution and observed data to solve for the optimal parameter estimate; this scheme is used for accurate prediction of action duration. The Action Guidance Description is a detailed description of the action type and visual attributes, used to guide the semantic direction of subsequent pose generation.

[0064] Table 3 contains partial content of the Action Type-Duration Table.

[0065] Formula for calculating action duration observations: ,in, This represents the percentage of the page area corresponding to the current action, dimensionless, and its value ranges from [0,1]. The density of motion lines within the corresponding area, expressed in lines per pixel². , The preset linear fitting coefficients are, where, The unit is seconds. The unit is seconds·pixels² / strip; the two coefficients were obtained by least-squares linear regression fitting of 1000 sets of labeled samples; This is the output action duration observation value, in seconds.

[0066] Observation noise error propagation formula: ,in, The detection error for the percentage of cells divided into sections is typically 0.02. This represents the detection error for motion line density, typically 0.0005 lines / pixel². This represents the noise standard deviation of the observations, in seconds. A typical calculation result is approximately 0.05 seconds, which can cover more than 95% of the observation error.

[0067] Maximum a posteriori estimation closed-form solution formula: ,in, The prior average duration for the corresponding action type, in seconds; This represents the prior standard deviation of the duration of the corresponding action type, in seconds. The noise standard deviation of the observations, in seconds; The estimated duration of the output action is given in seconds. The prior distribution and observation likelihood of the action duration both follow a normal distribution. This assumption is an engineering approximation, and after sample calibration, it can meet the estimation accuracy requirements.

[0068] For example, for the action "running", the prior average duration is read from the action type-duration table. =1.2 seconds, prior standard deviation =0.3 seconds. Extract the grid proportion S=0.3 from the corresponding area of ​​the current text sketch, and the action line density D=0.002 lines / pixel², and substitute them into the observation formula to calculate. =1.0 seconds.

[0069] Substituting the prior parameters and observed values ​​into the maximum a posteriori estimation formula, the estimated duration is calculated to be 1.15 seconds. The action type feature vector and the matching confidence scores of each dimension are fused, and after dimensionality reduction by a fully connected layer, they are input into a text generator to generate the action guidance description "Run fast with big strides".

[0070] Example 4: This example generates the initial dynamic change line for the corresponding character, specifically including: Step 31: Component segmentation and feature extraction: Perform multi-scale curvature analysis on the static outline and segment it into N closed outline components based on human topological prior (typically the head, torso, left arm, right arm, left leg, and right leg, a total of 6 components), each component being an independent closed outline region; extract the Hu moment geometric invariant features of each closed outline component as the shape feature vector of the node.

[0071] Multi-scale component segmentation is performed according to the following standardized process: (1) Multi-scale curvature extremum extraction: The contour point set is smoothed by Gaussian kernels of three scales: σ=2, σ=5, and σ=8, and the curvature value of each point is calculated respectively. Points whose absolute curvature values ​​at the three scales are all greater than 1.5 times the average curvature are selected as candidate segmentation points, and nearby redundant points with a spacing of less than 2% of the total contour length are removed.

[0072] (2) Joint matching: Based on the 6-part structure of the standard human body topology, the candidate segmentation points are sorted from top to bottom according to their positions and matched in turn as 9 key joint points: neck, left and right shoulders, left and right elbows, left and right hips, and left and right knees. If the number of candidate points is insufficient, the corresponding joint points are filled by interpolation along the contour according to the standard human body proportion.

[0073] (3) Topological constraint verification: After segmentation, the length ratio of adjacent parts is verified in turn. The allowable range for the length ratio of upper arm to forearm is 0.8~1.2, the allowable range for the length ratio of thigh to calf is 0.8~1.2, and the allowable range for the head height to the proportion of the whole body is 1 / 7~1 / 9. If the range is exceeded, the position of the segmentation point is finely adjusted until the proportional constraint is met.

[0074] (4) Abnormal contour tolerance: If there is a local missing contour (missing length ≤ 10% of the total length), then linear completion is performed based on the extension trend of adjacent parts before segmentation is performed; if the missing length exceeds 10%, it is marked as an abnormal contour, and the part segmentation result of the complete frame of the same role is used for mapping and adaptation.

[0075] Step 32: Construction of the outer shape component graph: Using each closed outer shape component as a node and the joint connection relationship between components as an edge, construct an outer shape component graph with joint connection constraints; the node features are 7-dimensional Hu moment geometric invariant features, and the edge features are the relative angle between components, normalized distance and joint connection constraint weight, representing the topological relationship of the character components.

[0076] This method generates basic poses suitable for standard humanoid characters from a frontal to a 45° micro-side view. Within this range, parts only undergo in-plane translation and rotation, with no obvious depth perspective distortion, resulting in the highest generation accuracy.

[0077] Step 33: Generating character-scene coupled semantic features: Input the basic shape attribute description and dynamic action attribute description in the split description pair into the text encoder to obtain the character shape semantic vector and action text vector. Combine the scene semantic vector of the first sub-description and perform concatenation and fully connected mapping to obtain the character-scene coupled semantic features.

[0078] Step 34: Dynamic Constraint Feature Generation: The estimated screen duration, estimated number of screen actions, and action guidance descriptions for each estimated screen action in the guidance set are encoded as dynamic temporal condition features, and the scene requirements are encoded as scene requirement semantic features. The dynamic temporal condition features, scene requirement semantic features, and role-scene coupling semantic features are concatenated and mapped through a fully connected layer to obtain dynamic constraint features of a unified dimension. The dynamic constraint features are embedded with a duration embedding vector, which is used to control the duration of the output sequence of the graph conditional diffusion model.

[0079] Step 35: Graph Conditional Diffusion Model Inference: The graph features of the external component graph and the dynamic constraint features are used as input graph conditional diffusion models to generate a sequence of external component attitude parameters for T video time steps. Each video time step contains N components' x-axis offset, y-axis offset, and planar rotation angle.

[0080] The training loss function of the graph conditional diffusion model includes a noise prediction loss term, a temporal smoothing regularization term, and a joint relative distance constraint term. The total loss function is: ,in, This is the standard noise prediction mean square error loss; The L2 regularization term is the difference in pose parameters between adjacent video time steps, which is a time-series smoothing constraint. This is a constraint term for the relative distance between joints of adjacent components; =0.2、 =0.3 is the weighting coefficient. Cross-validation has determined that this value provides the best overall performance in terms of attitude continuity and connectivity stability.

[0081] The formula for calculating the joint relative distance constraint is: Where T is the total number of video time steps. The total number of key points. For the j-th joint in The actual pixel distance between the two endpoints at time 1. The static reference length of the distal component corresponding to the j-th joint is in pixels. This constraint penalizes the separation of the joint endpoints by a relative length ratio to prevent component connection breakage.

[0082] During training, all pose parameters are normalized to min-max: the x / y pixel offset ranges from ±200 pixels, and the plane rotation angle ranges from ±π / 2 radians, all mapped to the interval [-1,1]. During inference, the output parameters are denormalized to restore them to the true pixel and angle values.

[0083] The model inference employs the DDIM deterministic sampling algorithm with a sampling step count of 50, which improves inference efficiency while ensuring generation quality. Conditional injection utilizes a cross-attention mechanism, where external component graph features are encoded into graph feature vectors via a graph neural network and interact with noise features at each attention layer of the U-Net. Dynamic constraint features serve as global conditions, injected into each residual block of the U-Net through adaptive layer normalization (AdaLN) to achieve global semantic constraints. The output sequence length is precisely controlled by duration embedding vectors, which are mapped one-to-one with the estimated frame duration. Training is supervised by duration classification loss, and during inference, inputting the embedding vector corresponding to the target duration outputs a pose sequence corresponding to the number of video time steps, with the error controlled within one video time step.

[0084] The feature encoding of the external component image employs a two-layer Graph Attention Network (GAT) with four attention heads, outputting a 256-dimensional feature vector for each node. After global pooling, a 512-dimensional global graph feature vector is obtained. The graph features are concatenated with the dynamic constraint features and then injected into each downsampling and upsampling module of the UNet through cross-attention layers, for a total of six layers, to constrain the pose generation by the graph topology. The temporal embedding vector is a learnable 512-dimensional vector, which is element-wise added to the dynamic constraint features and used as a global condition, injected into the residual block via AdaLN.

[0085] Step 36: Joint Overlap Verification: After the inference output, a joint overlap verification is performed. Components with deviations exceeding limits are corrected for their positions. The specific rules are as follows: Calculate the pixel distance between the corresponding joint endpoints of adjacent components at the current moment, and calculate the deviation rate between this distance and the reference length of the corresponding component. The calculation formula is: If the deviation rate If it exceeds 5%, a correction will be triggered.

[0086] The correction adopts a linear scaling method along the joint line, taking the endpoints of the component far from the torso as the adjustment object, and scaling the length of the component along the joint line to bring the distance between the two endpoints of the joint back to the allowable range. During the correction process, the rotation angle of the component remains unchanged, and only the position offset is adjusted to avoid disrupting the motion logic of the posture. At the same time, the correction amount of the three adjacent video time steps is smoothly transitioned to avoid inter-frame jumps.

[0087] Step 37: Contour Transformation and Smoothing: Based on the posture parameters of each component, perform a two-dimensional affine transformation on the corresponding component segments of the static outline with the corresponding joint point as the rotation center, and perform cubic spline interpolation smoothing on the connection area of ​​adjacent components to construct the intermediate character outline.

[0088] Two-dimensional affine transformation formula (joints are centers of rotation): , where (x,y) are the original pixel coordinates of a single point within a component in the static contour; This refers to the pixel coordinates of the joint point corresponding to the component, such as the shoulder joint point for the left arm. This represents the planar rotation angle of the component, in radians. , These are the pixel offsets along the x and y axes of this component; These are the pixel coordinates of the point after the transformation.

[0089] Cubic spline interpolation smoothing is an interpolation method that achieves smooth curve transitions through piecewise cubic polynomials. The overall function is second-order continuous and differentiable, and is used to eliminate breaks and sharp corners in the component connection area. For the discrete contour point sequence of the connection area, the interpolation function S(t) satisfies that each piecewise interval is a cubic polynomial, the overall function is second-order continuous and differentiable, and passes through all discrete control points. The boundary condition adopts natural splines, that is, the second derivative of the two endpoints is 0.

[0090] Step 38: Rendering Parameter Generation: Extract the position index of each shape component on the middle character outline as the parameter position; extract the rendering list corresponding to the parameter position from the pre-built position-lookup table, wherein the rendering list contains position attributes, layout attributes and adjacent position transition attribute groups. The execution priority of rendering attributes follows the rule of layout attributes first, position attributes second, and transition attribute groups last. In case of attribute conflict, the higher priority attribute shall prevail.

[0091] The posture parameter sequences of the external components are grouped according to temporal continuity and component correlation. A rendering event is set for each group. According to the order of the rendering events, the rendering attributes are set sequentially for each parameter position in the corresponding group according to the attribute priority of the rendering list, so as to obtain the rendering parameter sequence for each video time step.

[0092] The rendering list includes three positional attributes: line thickness, fill level, and shadow intensity; two layout attributes: layer priority and occlusion relationship; and two sets of transition attributes: line transition smoothness with adjacent components and color transition coefficient. Specifically, line thickness ranges from 1 to 8 pixels; fill level ranges from 0 to 255 grayscale levels, or the corresponding RGB three-channel values; shadow intensity ranges from 0 to 1 (normalized value); layer priority ranges from 1 to 5 (integer), with larger values ​​indicating higher layer priority; line transition smoothness ranges from 0 to 1 (larger values ​​indicate smoother transitions); and color transition coefficient ranges from 0 to 1 (larger values ​​indicate better color blending).

[0093] The position-reference table is built based on the standard two-dimensional humanoid character component system. It stores the list of rendering attributes corresponding to the position of each component parameter. The complete table covers all 6 shape components and provides a mapping basis for the generation of rendering parameters. Some contents are shown in Table 4.

[0094] Table 4 - Partial Content of the Location Reference Table

[0095] Step 39: Initial dynamic change line generation: Combine the posture parameter sequence of the external parts and the rendering parameter sequence at the same video time step to generate the initial dynamic change line of the corresponding character.

[0096] For example, the static outline of a girl can be divided into six closed parts: head, torso, left arm, right arm, left leg, and right leg. The shoulder joint, elbow joint, hip joint, and knee joint can be used as joint connection points to construct an outline component diagram with joint connection constraints.

[0097] A dynamic constraint feature input graph conditional diffusion model with embedded duration embedding vectors is used. Employing the DDIM sampling algorithm and 50 sampling steps, the model outputs a sequence of orientation parameters for the external components over 29 video time steps. Each time step includes the x-offset, y-offset, and rotation angle of six components. Post-joint validation after inference reveals a relative deviation rate of 3% for the left arm joint, which is within acceptable limits and requires no correction.

[0098] For the left arm component, a two-dimensional affine transformation is performed with the shoulder joint as the rotation center; for the shoulder connection area, five contour points are taken at the front and five at the back as control points, and cubic spline interpolation is performed to smooth the surface, eliminating the breaks and sharp corners caused by independent transformation of the component, and obtaining a continuous middle character contour.

[0099] The location lookup table is used to obtain the rendering list of each component. The rendering parameters are set according to the rule of prioritizing layout attributes, and finally combined to obtain the initial dynamic change line of the girl.

[0100] Example 5: This example encodes the estimated screen duration, estimated screen action number, and action guidance description for each estimated screen action in the guidance set into dynamic temporal condition features, specifically including: Based on the estimated number of actions in the scene, the guidance description for each action is input into the text encoder to obtain an action semantic vector of dimension D; The estimated duration of each action is normalized and mapped to a time span vector of dimension D through a learnable linear layer; the action semantic vector and the time span vector of the same action are concatenated and converted into an action event feature vector of dimension D through a linear mapping layer. A learnable temporal location code is added to each action event feature vector, and the action event feature sequence is obtained by arranging them in the order of the actions. The action event feature sequence is input into a two-layer Transformer encoder for temporal self-attention encoding, and the output is a dynamic temporal conditional feature with dimension D.

[0101] In this embodiment, the action semantic vector is a feature vector obtained by a pre-trained text encoder for action guidance instructions, with a dimension D=512, representing the semantic connotation and attributes of the action. The time span vector is a feature vector obtained by normalizing and linearly mapping the estimated duration of the action, with the same dimension as the action semantic vector, representing the time scale attribute of the action. The temporal position encoding is a learnable position embedding vector used to distinguish the temporal position of different actions in the sequence, ensuring that the model perceives the order of actions. The temporal self-attention encoding is an encoding process that captures the temporal dependencies within the action sequence through the Transformer self-attention mechanism, enabling the generated temporal features to have logical correlation between actions. The dynamic constraint feature is a global constraint vector with a dimension of 512 that integrates three types of information: temporal conditions, role-scene coupling, and scene requirements, serving as the global condition input for the graph conditional diffusion model.

[0102] If the action guidance description "running fast with big strides" is input into the BERT-base encoder, a 512-dimensional action semantic vector is obtained; the estimated duration of 1.15 seconds is normalized to the [0,1] interval and mapped to a 512-dimensional time span vector through a learnable linear layer.

[0103] Two vectors of the same action are concatenated into a 1024-dimensional vector, which is then mapped to a 512-dimensional action event feature vector through a fully connected layer. The corresponding time position encoding is added, and the features of other actions are combined into a sequence in chronological order.

[0104] The sequence is input into a two-layer Transformer encoder for temporal self-attention encoding, outputting a 512-dimensional dynamic temporal conditional feature; then concatenated with the 512-dimensional role-scene coupled semantic feature and the 512-dimensional scene requirement semantic feature to form a 1536-dimensional vector, which is then mapped to a 512-dimensional dynamic constraint feature through a fully connected layer.

[0105] Example 6: This example filters out subdivision points and generates corresponding point guidance descriptions. The specific steps are as follows: Step 41: Calculation of relative change rate of parameters: Calculate the relative change rate of each pose parameter and the relative change rate of each rendering parameter between adjacent video time steps in the initial dynamic change line. The formula for calculating the relative change rate of a single parameter is as follows: ,in, For the i-th parameter at video time step The relative rate of change of a single parameter at point i is dimensionless; i is the parameter index number. Index for the current video time step; For the i-th parameter at time step The possible values ​​of ; For the i-th parameter at the previous time step The possible values ​​of ; It is the maximum value of the i-th parameter across the entire time series; It is the minimum value of the i-th parameter across the entire time series.

[0106] Step 42: Constructing the parameter change intensity curve: Normalize all parameters by mapping their relative change rates to the [0,1] interval; construct the parameter change intensity curve by weighted summation of the normalized average relative change rate of the attitude parameters and the average relative change rate of the rendering parameters according to preset weights, using the following formula: ,in, for The intensity of the combined change of parameters at any given time, with values ​​ranging from [0,1]. for The average relative rate of change of all attitude parameters at time t; for The average relative rate of change of all rendering parameters at any given time; This is the attitude parameter weight hyperparameter, with a value of 0.6; The rendering parameter weight hyperparameter is set to 0.4, satisfying the constraints. .

[0107] Step 43: Candidate subdivision point detection: Calculate the first-order difference of the intensity curve to identify all local extreme points, including local peak points (intensity maxima) corresponding to the peak phase of the corresponding action amplitude and local valley points (intensity minima) corresponding to the start and end steady-state phases of the action; at the same time, calculate the second-order difference of the intensity curve, and add inflection points whose absolute value of the second-order difference exceeds a preset curvature threshold as candidate subdivision points, and finally obtain the candidate subdivision point set; the curvature threshold is used to filter nodes where the action state changes significantly, and in this embodiment, the value is 0.5.

[0108] Step 44: Adapt the number of subdivision points: Determine the target number of subdivision points according to the rule that each action corresponds to the three types of keyframes: start, peak, and end (the target number for a single action is 3).

[0109] If the number of candidate subdivision points is greater than the target number, the following rules are used for selection: the highest intensity peak point is selected first as the action peak point, and then the two valley points with the closest distance before and after the peak point on the time axis are selected as the action start point and end point respectively, and finally the target number of subdivision points are obtained.

[0110] If the number of candidate subdivision points is less than the target number, all candidate subdivision points are retained, and sampling points of the difference number are evenly added to the corresponding action stage on the time axis so that the final number of subdivision points is consistent with the target number.

[0111] Step 45: Point guidance description generation: For each final subdivision point, extract the pose parameter features and rendering parameter features of the two video time steps before and after it, and generate the point guidance description for the subdivision point through a preset text template; the starting point corresponds to the "action start pose" category description, the peak point corresponds to the "maximum amplitude pose" category description, and the ending point corresponds to the "action convergence pose" category description.

[0112] In this embodiment, the relative rate of change is the ratio of the difference between parameters in adjacent video time steps to the parameter's own value range. It is a dimensionless indicator used to measure the drastic change of the parameter itself, eliminating the influence of differences in the dimensions of different parameters. The parameter change intensity curve is a time-series curve obtained by weighted summation of the average relative rates of change of the pose parameters and rendering parameters, with a value range of [0,1], uniformly representing the overall drastic change of the image. Fine-grained split points are key time nodes selected from the initial dynamic change line, corresponding to the keyframe positions of the action. Each action corresponds to three split points in chronological order: start, peak, and end. The earliest valley corresponds to the start of the action, the peak to the peak of the action, and the latest valley to the end of the action. The point guidance description is a natural language description of the pose and rendering state at a single fine-grained split point, 10-20 characters in length, used to encode and generate keyframe images.

[0113] For example, for the first sub-description containing one action, the total number of target sub-decomposition points is 3. Calculate the relative rate of change of the parameters of the initial dynamic change line, and obtain the intensity change curve by weighted summation.

[0114] The first-order difference of the curve is used to identify extreme points, resulting in two valley points and one peak point, for a total of three candidate points, which exactly match the target quantity; in chronological order, they correspond to the start, peak, and end states of the action.

[0115] Extract the parameter features of two video time steps before and after each subdivision point, and generate point guidance descriptions using preset text templates, namely "running starting posture", "maximum stride posture" and "stride fall posture".

[0116] Example 7: This example performs multi-role spatiotemporal joint alignment and inputs a video to generate a diffusion model to output film and television character videos, specifically including: Combining the three types of constraints in the scenario requirements—temporal synchronization, spatial interaction, and action coordination—temporal synchronization alignment, spatial position alignment, and action semantic coordination correction are performed sequentially to obtain aligned point-level semantic features and aligned spatial parameters. Among them, temporal synchronization alignment and spatial position alignment are hard constraints, while action semantic coordination correction is a soft constraint. The constraint priority is spatial interaction constraint > temporal synchronization constraint > action coordination constraint.

[0117] In this embodiment, time synchronization alignment is achieved by unifying the timeline of each character's actions and adjusting the timestamps of each character's subdivision points, so that the start, peak, and end times of the collaborative actions strictly correspond.

[0118] Timing alignment calculation: ,in, This is the time offset for the i-th character, set according to the synchronization relationship.

[0119] In this embodiment, spatial alignment is achieved by adjusting the global coordinate offset of each character based on the pixel-physical scale mapping coefficient and the spatial interaction relationship between characters, ensuring that the relative positions of the characters meet the requirements of the scene.

[0120] Spatial alignment calculation: , ,in, These are the original global space pixel coordinates of the i-th character; The global space pixel coordinates of the i-th character after alignment; This represents the horizontal pixel offset, in pixels. This represents the vertical pixel offset, expressed in pixels.

[0121] In this embodiment, action semantic collaborative correction is performed: based on the action collaborative relationship description, the point-level semantic features of each role are reweighted and corrected to make the action semantics of the role closer to the collaborative requirements.

[0122] Based on the point-level semantic features and spatial parameters of each aligned sub-segmentation point, the image generation parameters for each character at the corresponding sub-segmentation point are generated; linear interpolation of parameters is performed on non-key frames between key frames to obtain the image generation parameter sequence of the complete timeline, which is regarded as the complete sequence.

[0123] Keyframe linear interpolation formula (non-keyframe completion): , where t is the target time to be interpolated; This refers to the previous adjacent keyframe moment; This refers to the next adjacent keyframe moment; for The parameters for generating the image at any given moment; for The parameters for generating the image at any given moment; The parameters for generating the image are obtained by interpolation at the target time t.

[0124] Input the complete sequence of all characters into a 2D video generation diffusion model in chronological order to generate film and television character videos with corresponding text sketches.

[0125] The attitude validity check is an optional optimization step. It can be performed after alignment, and the check rules are as follows: Joint angle constraints: ,in, Radius, which is a preset constraint angle; For joint angle; Component proportion constraints: ,in, This serves as a reference scale for components within a static profile. The actual proportions of the components in the static outline; If both conditions are met, the result is considered valid; otherwise, parameter correction is performed.

[0126] Example 8: In this example, the action semantic collaborative correction adopts a feature space orthogonal decomposition semantic correction mechanism. The specific steps are as follows: The action coordination relationship description is input into the text encoder to obtain a coordination semantic vector of dimension D, and then L2 normalization is performed on it to obtain a normalized coordination semantic vector c.

[0127] Point-level semantic features of the i-th character Calculate the cosine similarity with c. : .

[0128] The character features are orthogonally decomposed into cooperative directional components and vertical components: ,in, This refers to the projection component of the original semantic features onto the direction of the co-semantic vector; The inherent semantic components of roles orthogonal to the cooperative direction remain unchanged during the correction process; If similarity Below the preset threshold =0.7, scaling correction is performed on the cooperative direction component, and the corrected cooperative component is: ,in, The directional component is the result of the correction; α is the directional adjustment strength coefficient, with a value range of [0, 0.5], and a typical value of 0.3. is the L2 norm of the original eigenvectors.

[0129] The corrected point-level semantic features are as follows: ,in, To correct the completed role-level semantic feature vector.

[0130] Perform semantic legality verification: Calculate the cosine similarity between the corrected features and the original features. If it is less than 0.8, reduce the adjustment intensity α proportionally until the similarity constraint is met, ensuring that the character's ontological attributes do not undergo essential shift.

[0131] Example 9: In this example, spatial alignment is performed based on a hierarchical pixel-to-physical scale mapping system. The pixel-to-physical scale mapping coefficient is used to establish the correspondence between pixel distances in the image and actual physical distances, with the unit being pixels / meter, serving as the dimensional conversion benchmark for spatial alignment. A hierarchical mapping scheme is adopted for different perspective scenes: For level / slightly side-view scenes without perspective: a globally unified mapping coefficient is used, calculated as the ratio of the main character's total height in pixels to the standard human height (1.7 meters). The formula is as follows: ,in, The character's total height in pixels. This is a standard human reference height.

[0132] Single-point perspective scene: Establish depth-related dynamic mapping coefficients based on the pinhole imaging perspective projection model, the steps are as follows: (1) Extract the horizon position and vanishing point coordinates from the text sketch to establish the perspective coordinate system of the screen; take the horizon as the depth reference plane, and the closer the vertical coordinate of the bottom of the character is to the bottom of the screen, the smaller the depth value (closer to the observer).

[0133] (2) Calculate the normalized depth value of the i-th character. The value range is [0,1], where 0 corresponds to the foreground of the image and 1 corresponds to the horizon.

[0134] (3) Calculate the mapping coefficient for this character based on perspective scaling rules: ,in, This is the scaling factor between the foreground and background, typically set to 3.0, meaning the pixel size of the character at the horizon is 1 / 3 of that in the foreground, conforming to the standard perspective ratio in comics.

[0135] Physical distance between characters during spatial alignment The corresponding pixel offset is: ,in, , These are the mapping coefficients for close-up and distant characters, respectively. The average coefficient is used to approximate the pixel offset corresponding to the horizontal distance, balancing accuracy and computational efficiency.

[0136] Non-full-body image correction: For non-full-body images such as half-body and close-ups, the torso height is used as a reference to estimate the full-body height: According to standard human proportions, the torso height is approximately 0.4 times the full-body height, and the full-body height is then calculated using this to estimate the full-body height before calculating the mapping coefficient. For height fluctuations in dynamic movements, inter-frame correction is also performed using the torso height as a reference.

[0137] For scenes with inconsistent perspectives among multiple characters, the mapping coefficient of the main character's color is used as the benchmark, and other characters are scaled proportionally according to their depth position in the scene. The depth position is calculated from the vertical coordinates of the character in the scene and the position of the scene horizon.

[0138] Example 10: The basic pose generation method of this solution is suitable for standard humanoid characters with a frontal to 45° micro-side view. For large horizontal view scenes with a frontal to 90° side view, the adaptation can be optimized through view deformation factor compensation, as follows: Analyze the character's horizontal rotation angle from the initial character description. The unit is radians, and the value range is [0, π / 2]. 0 corresponds to the frontal view, and π / 2 corresponds to the side view.

[0139] Calculate the perspective scaling factor for each external component: ,in, The perspective scaling factor for the horizontal viewpoint; The horizontal rotation angle of the character, in radians, with a range of values. Corresponding to the front orientation, Corresponding to the front and side orientation.

[0140] This factor is based on the perspective principle of objects appearing larger when closer and smaller when farther away, and it represents the compression ratio of the visual width of a component along the depth direction.

[0141] Before performing a two-dimensional affine transformation, first adjust the horizontal width of the component outline according to... Scaling is applied while keeping the rotation center and transformation rules unchanged; a larger scaling factor is used for parts closer to the observer, and a smaller factor is used for parts farther away, to simulate the perspective effect in the depth direction.

[0142] After correction, the rendering parameters of the components are updated synchronously, and the shadow intensity on the compressed side is enhanced accordingly to improve the visual realism.

[0143] The scope of application of this solution is clearly defined: This solution is a pure 2D pose generation solution and is not applicable to rear view, pitch view greater than 30°, or non-humanoid character scenes; the above-mentioned scenes require supplementary static contour input for the corresponding view or to be combined with a 3D reconstruction solution, which are not covered by this solution.

[0144] Example 11: This example performs rendering quality optimization after step 5. The specific steps are as follows: Perform a rendering quality check on the aforementioned film and television character videos to determine if there are any areas that need rendering improvement. If there are no defects, directly output the video of the aforementioned film and television character; If defects exist, the video of the film and television character is split into frames, and the area to be rendered is located by combining the semantic segmentation results of each frame. The standard texture of the corresponding part of the area to be rendered is matched from the pre-built character texture template library. The optimal texture mapping parameters are calculated by combining the lighting parameters and spatial position of the area to be rendered in three consecutive frames before and after. The character texture template library is constructed by extracting line style, fill texture and color reference features from the character image area of ​​the text sketch. Determine the type of rendering failure based on the spatial position changes and occlusion relationships of the area to be rendered; Based on the type of rendering failure, select the corresponding optimization algorithm and combine it with the optimal texture mapping parameters to perform pixel-level repair on the area to be rendered. Temporal smoothing constraints are applied to the repaired frames to ensure visual coherence with adjacent frames, ultimately outputting optimized video of film and television characters.

[0145] In this embodiment, the rendering quality check employs a two-level detection mechanism: rule-based judgment and lightweight CNN scoring. First, the character component regions are located through semantic segmentation, and three indicators—gradient variance, edge sharpness, and texture similarity—are calculated within these regions. A gradient variance higher than a threshold of 0.8 is considered texture stretching, an edge sharpness lower than a threshold of 0.3 is considered edge blurring, and an intersection-union ratio (IUU) between adjacent component regions higher than 0.4 is considered occlusion distortion. The region to be rendered is located on a component-by-component basis, with precision down to the pixel-level component boundaries.

[0146] The optimal texture mapping parameters are solved with the objective function of "maximizing texture matching similarity". The constraints include part boundary alignment error ≤ 2 pixels, rotation angle deviation ≤ 5°, and scaling ratio deviation ≤ 5%. The least squares method is used to solve the optimal solutions for translation, rotation and scaling mapping parameters.

[0147] The optimization algorithms corresponding to the three types of rendering failures are as follows: Occlusion failure type: A pixel-based filling algorithm based on part texture templates is adopted. The same part texture in the unoccluded frame is used as the reference, and color correction is performed in combination with the lighting parameters of the current frame. The occluded area is filled pixel by pixel, and the filling boundary is feathered with a width of 3 pixels. Texture stretching failure type: A texture remapping algorithm is adopted. The standard texture is transformed into a thin plate spline (TPS) according to the outline shape of the current part and remapped to the stretching area to replace the original stretching texture. Edge blurring failure type: A part-specific edge enhancement algorithm is adopted. The part edge gradient in the template library is used as the reference to perform adaptive sharpening processing on the blurred area. The sharpening intensity is dynamically adjusted by the blur degree.

[0148] The temporal smoothing constraint adopts a weighted average mechanism of a 3-frame sliding window with a window weight of [0.2, 0.6, 0.2]. It only applies to pixels in the repair area, while the non-repair area retains its original value, thus eliminating inter-frame flicker while avoiding overall image blurring.

[0149] Texture matching similarity calculation formula: ,in, This represents the texture matching similarity score, with a value range of [-1, 1]. The texture feature vector of the area to be rendered; This is the standard texture feature vector of the template component.

[0150] If the rendering quality is checked frame by frame in the generated video, it is found that there is a blurry edge defect in the area of ​​the boy's armor, which is located as the area to be rendered.

[0151] Standard texture features of each component are extracted from the character texture template library, and cosine similarity is calculated between these features and the features of the area to be rendered. The optimal matching result is the standard texture of the armor component. The optimal texture mapping parameters are calculated by combining the lighting parameters and spatial positions of the preceding and following three frames.

[0152] The rendering failure type was determined to be edge blurring, and a targeted texture enhancement algorithm was used for pixel-level repair. Temporal smoothing constraints were applied to the repaired frame and adjacent frames to avoid inter-frame flickering. The final output video was repaired, and the repaired area maintained the same style as the original.

[0153] In this embodiment, the dimension-feature-label table is constructed as follows: more than 500 scene description samples labeled with dimension labels are collected, the description granularity sequence set of each sample is extracted, clustered according to the matching confidence interval, and the 2 to 3 labels with the highest frequency are selected for each interval. After verification by domain experts, the final mapping table is formed.

[0154] In this embodiment, the action type-duration table is constructed by collecting more than 1,200 standard action clips from movies and comics, labeling the action type and actual duration, and calculating the mean and standard deviation of the duration of each secondary sub-action to form a mapping table. The action system includes 12 primary categories and 48 secondary sub-actions, covering common action types for humanoid characters.

[0155] In this embodiment, the position-lookup table is constructed as follows: based on the 6-part structure of a standard 2D humanoid character, three types of positional attributes are defined for each part: line thickness, fill color level, and shadow intensity; two types of layout attributes are defined: layer priority and occlusion relationship; and transition attributes with adjacent parts are defined. The attribute values ​​are calibrated by art experts and conform to the rendering specifications of conventional hand-drawn comics.

[0156] In this embodiment, the character texture template library is constructed as follows: from the character area of ​​the input text sketch, the line texture, fill color, and shadow style features of each closed part are extracted according to the part segmentation results, and after normalization processing, they are stored as part standard texture templates; only one standard template is retained for the same character and part, which serves as the benchmark for subsequent rendering and repair.

[0157] Example 12: This example provides a text sketch-guided video generation system for film and television characters based on AIGC, such as... Figure 2 As shown, it includes: The contour extraction and description acquisition module is used to acquire text sketches and obtain the static outline, initial character description and initial scene description of each character through a multimodal large model; The coarse segmentation and guidance set generation module is used to perform semantic event coarse segmentation on the initial scene description to obtain the first sub-description, perform action parsing and duration estimation on each first sub-description, and perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, which are combined into a guidance set. The initial dynamic change line generation module is used to split the initial character description to obtain a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, a shape component diagram with joint connection constraints is constructed. The posture parameter and rendering parameter sequence are generated through the graph condition diffusion model, and the initial dynamic change line of the corresponding character is generated. The fine-splitting module is used to process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak, and end of a single action. At the same time, it detects local peak points and valley points of the intensity curve, filters out fine-splitting points, and generates corresponding point guidance descriptions. The spatiotemporal alignment and video generation module is used to encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and to perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. The input video generation diffusion model outputs a video of film and television characters.

[0158] The beneficial effects of the above technical solution are: it realizes the automatic conversion from comics to dynamic videos based on a two-dimensional system without the need for three-dimensional reconstruction, can preserve the original art style, relies on a multi-modal large model to extract multi-modal information in a unified manner, has a simple process and high consistency, ensures the collaboration of multiple characters through spatiotemporal joint alignment, and balances generation efficiency and motion quality through a two-level generation architecture, and significantly improves the overall automation level and generation effect.

[0159] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An AIGC-based text sketch guided video generation method for a movie and television character, characterized in that, include: Step 1: Obtain text sketches and use multimodal large models to obtain the static outline of each character, initial character description, and initial scene description; Step 2: Perform semantic event coarse decomposition on the initial scene description to obtain the first sub-description. Perform action parsing and duration estimation on each first sub-description. At the same time, perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, and combine them into a guidance set. Step 3: The initial character description is split into a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, a shape component diagram with joint connection constraints is constructed. The posture parameter and rendering parameter sequence are generated through graph condition diffusion model, and the initial dynamic change line of the corresponding character is generated. Step 4: Process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak and end of a single action. At the same time, detect the local peak points and valley points of the intensity curve, filter out the fine-grained points, and generate corresponding point guidance descriptions. Step 5: Encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. Input the video to generate a diffusion model and output the film and television character video.

2. The AIGC-based text sketch guided video generation method of a movie and television character according to claim 1, characterized in that, Perform multi-granular semantic parsing on each first sub-description to obtain corresponding preset-dimensional screen guidance tags, including: For each first sub-description, perform multi-granularity semantic parsing to extract semantic units at the semantic level, modification attribute level, action granularity level, and scene association level in sequence. Input the semantic units at each level into a pre-trained language model to encode them into semantic vectors, calculate the cosine similarity with the corresponding level's baseline semantic prototype vector, and obtain a hierarchical semantic matching score with a value range of [0,1]. Concatenate the scores of the four levels in a fixed order to obtain a description granularity sequence set with a length L=4. Obtain a pre-constructed reference granularity sequence set that maps one-to-one with each preset dimension, wherein the reference granularity sequence set is a numerical vector of length 4, obtained by clustering typical samples under the corresponding dimension; The cosine similarity between the description granularity sequence set and the reference granularity sequence set for each preset dimension is calculated to obtain the matching confidence value for each preset dimension. Based on the matching confidence value corresponding to each preset dimension, interval matching is performed with the preset dimension-feature-label table to obtain several candidate guide labels for the corresponding preset dimension. Discover the semantic mutual exclusion relationship and attribute linkage relationship among several candidate guidance tags in the same preset dimension, eliminate semantically conflicting tags and retain tags with consistent attribute logic as the corresponding screen guidance tags for the preset dimension.

3. The AIGC-based text sketch guided video generation method of a movie and television character according to claim 1, characterized in that, For each first sub-description, perform action parsing and duration estimation, including: For each first sub-description, perform action parsing to identify the included action entities and their order to obtain an action sequence, wherein the action sequence contains K actions; For each action, the prior average duration and prior standard deviation of the action are obtained from the pre-built action type-duration table based on the action type. Extract two visual features from the action area corresponding to the current text sketch: grid ratio and action line density. Then, calculate the action duration observation value and the standard deviation of the observation noise by weighting the preset linear fitting coefficients. Under the assumption of normal distribution, the estimated duration of each action is calculated by combining the prior distribution and the observed values ​​through the closed-form solution of the maximum a posteriori estimation. The sum of the estimated durations of all actions in the action sequence is taken as the estimated screen duration, and the length K of the action sequence is taken as the estimated number of actions in the screen. For each action, the action type feature vector of the action is concatenated with the matching confidence value of the corresponding preset dimension, mapped to a fusion feature through a fully connected layer, and input into a pre-trained text generator to generate the action guidance description corresponding to the action.

4. The AIGC-based text sketch guided video generation method of a movie and television character according to claim 1, characterized in that, Generate the initial dynamic change line for the corresponding character, including: Multi-scale curvature analysis is performed on the static outline, and the body is segmented into N closed outline components based on human topological priors. The Hu moment geometric invariant features of each closed outline component are extracted. Construct a shape component diagram with joint connection constraints, using each closed shape component as a node and the joint connection relationship between components as an edge; The basic shape attribute description and dynamic action attribute description in the split description pair are input into the text encoder to obtain the character shape semantic vector and action text vector. Combined with the scene semantic vector of the first sub-description, the character-scene coupled semantic features are obtained. The estimated screen duration, estimated screen action number, and action guidance description for each estimated screen action in the guidance set are encoded as dynamic temporal condition features, and the scene requirements are encoded as scene requirement semantic features. The dynamic temporal condition features, scene requirement semantic features and role-scene coupling semantic features are concatenated and mapped through a fully connected layer to obtain dynamic constraint features of a unified dimension. The dynamic constraint features are embedded with a time-length embedding vector, which is used to control the time length of the output sequence of the graph conditional diffusion model. The graph features of the external component graph and the dynamic constraint features are used as input to the graph conditional diffusion model. The training loss function of the graph conditional diffusion model includes a noise prediction loss term, a temporal smoothing regularization term, and a joint relative distance constraint term. This generates a sequence of external component pose parameters for T video time steps. Each video time step includes N component x-axis offset, y-axis offset, and planar rotation angle. After the inference output, a joint overlap verification is performed, and position correction is performed on components with deviations exceeding the limit. Based on the posture parameters of each component, a two-dimensional affine transformation is performed on the corresponding component segments of the static outline with the corresponding joint point as the rotation center, and cubic spline interpolation is performed on the connection area of ​​adjacent components to construct the intermediate character outline. Extract the position index of each shape component on the middle character outline, and use it as the parameter position; Extract the rendering list corresponding to the parameter position from the pre-built position-lookup table. The rendering list includes position attributes, layout attributes and adjacent position transition attribute groups. The execution priority of rendering attributes follows the rule of layout attributes first, position attributes second, and transition attribute groups last. In case of attribute conflict, the higher priority attribute shall prevail. The posture parameter sequences of the external components are grouped according to temporal continuity and component correlation, and a rendering event is set for each group; According to the order of rendering events, for each parameter position in the corresponding group, the rendering attributes are set sequentially according to the attribute priority of the rendering list to obtain the rendering parameter sequence for each video time step. The sequence of pose parameters of the external parts at the same video time step is combined with the sequence of rendering parameters to generate the initial dynamic change line of the corresponding character.

5. The AIGC-based text sketch guided video generation method of claim 4, wherein, The estimated screen duration, estimated number of screen actions, and action guidance descriptions for each estimated screen action in the guidance set are encoded as dynamic temporal conditional features, including: Based on the estimated number of actions in the scene, the guidance description for each action is input into the text encoder to obtain an action semantic vector of dimension D; The estimated duration of each action is normalized and mapped to a time span vector of dimension D through a learnable linear layer; the action semantic vector and the time span vector of the same action are concatenated and converted into an action event feature vector of dimension D through a linear mapping layer. A learnable temporal location code is added to each action event feature vector, and the action event feature sequence is obtained by arranging them in the order of the actions. The action event feature sequence is input into a two-layer Transformer encoder for temporal self-attention encoding, and the output is a dynamic temporal conditional feature with dimension D.

6. The AIGC-based text sketch guided video generation method of a movie and television character according to claim 1, characterized in that, The process involves filtering to obtain detailed breakdown points and generating corresponding point guidance descriptions, including: Calculate the relative rate of change of each pose parameter and the relative rate of change of each rendering parameter between adjacent video time steps in the initial dynamic change line; The relative rates of change of all parameters are uniformly mapped to the [0,1] interval to complete the normalization process; Based on preset weights, the average relative change rate of the normalized attitude parameters and the average relative change rate of the rendering parameters are weighted and summed to construct a parameter change intensity curve. The first-order difference of the intensity curve is calculated to identify all local extreme points, including local peak points in the corresponding peak phase of the action amplitude and local valley points in the corresponding steady-state phases at the beginning and end of the action; at the same time, the second-order difference of the intensity curve is calculated, and inflection points whose absolute value of the second-order difference exceeds the preset curvature threshold are added as candidate subdivision points, and finally a set of candidate subdivision points is obtained. Based on the rule that each action corresponds to three types of keyframes—start, peak, and end—determine the target number of subdivision points; If the number of candidate subdivision points is greater than the target number, the highest intensity peak point is selected as the action peak point, and the two valley points with the closest distance before and after the peak point on the time axis are selected as the action start point and end point respectively, to obtain the final subdivision points of the target number; If the number of candidate subdivision points is less than the target number, retain all candidate subdivision points and evenly supplement the sampling points of the difference number on the time axis at the corresponding action stage, so that the final number of subdivision points is consistent with the target number. For each final subdivision point, extract the pose parameter features and rendering parameter features of the two video time steps before and after it, and generate a point guidance description for that subdivision point using a preset text template.

7. The AIGC-based text sketch guided video generation method of a movie and television character according to claim 1, characterized in that, Perform multi-role spatiotemporal joint alignment and input video to generate a diffusion model to output film and television character videos, including: Combining the three types of constraints in the scenario requirements—temporal synchronization, spatial interaction, and action coordination—temporal synchronization alignment, spatial position alignment, and action semantic coordination correction are performed sequentially to obtain aligned point-level semantic features and aligned spatial parameters. Among them, temporal synchronization alignment and spatial position alignment are hard constraints, while action semantic coordination correction is a soft constraint. The constraint priority is spatial interaction constraint > temporal synchronization constraint > action coordination constraint. Based on the point-level semantic features and spatial parameters of each aligned subdivision point, the screen generation parameters for each character at the corresponding subdivision point are generated. Perform linear interpolation of parameters on non-key frames between key frames to obtain a complete sequence of image generation parameters for the timeline, which is considered a complete sequence. Input the complete sequence of all characters into a 2D video generation diffusion model in chronological order to generate film and television character videos with corresponding text sketches.

8. An AIGC-based text sketch guided video generation system for a movie character, characterized in that, include: The contour extraction and description acquisition module is used to acquire text sketches and obtain the static outline, initial character description and initial scene description of each character through a multimodal large model; The coarse segmentation and guidance set generation module is used to perform semantic event coarse segmentation on the initial scene description to obtain the first sub-description, perform action parsing and duration estimation on each first sub-description, and perform multi-granular semantic parsing on each first sub-description to obtain the corresponding preset dimension screen guidance tags, which are combined into a guidance set. The initial dynamic change line generation module is used to split the initial character description to obtain a split description pair consisting of basic shape attribute description and dynamic action attribute description. Based on the closed shape component obtained by static shape contour segmentation, a shape component diagram with joint connection constraints is constructed. The posture parameter and rendering parameter sequence are generated through the graph condition diffusion model, and the initial dynamic change line of the corresponding character is generated. The fine-splitting module is used to process the initial dynamic change line based on the parameter change intensity curve and the three types of keyframe rules for the start, peak, and end of a single action. At the same time, it detects local peak points and valley points of the intensity curve, filters out fine-splitting points, and generates corresponding point guidance descriptions. The spatiotemporal alignment and video generation module is used to encode the point-level semantic features of each subdivided point in the initial dynamic change line of each character, and to perform spatiotemporal joint alignment of multiple characters based on scene requirements to obtain a complete sequence. The input video generation diffusion model outputs a video of film and television characters.

Images