An AI-based animation generation method

Abstract

This application discloses an AI-based animation generation method, relating to the field of AI technology, comprising: generating a structured constraint dataset based on a 2D character sketch to obtain generated image data; establishing a temporal consistency function to evaluate the generated image data and determine a dedicated model file; controlling rendering based on the structured constraint dataset and the dedicated model file to generate a rendered character image sequence; rendering the animation background according to a preset background rendering model to obtain a background image; performing image fusion processing on the character image sequence and the background image to obtain the target animation; by extracting structured constraint data from a 2D hand-drawn sketch and optimizing image generation based on a temporal consistency function, high consistency and dynamic rationality of character movement are achieved; simultaneously, through multi-scale feature injection and layered rendering, the problems of character face-changing and stiff movements in traditional animation are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an animation generation method based on artificial intelligence. Background Technology

[0002] In the field of animation production, traditional techniques mainly rely on manual drawing or 3D modeling software (such as Maya and Blender) for character design and rendering. While existing AI animation generation technologies (such as text-based video and video-to-image) can improve efficiency, significant problems still exist: First, character consistency is poor, and characters are prone to "changing faces" or discontinuous movements in long videos; second, emotional expression is insufficient, as AI models struggle to capture the emotional details of artists' hand-drawn sketches, resulting in stiff movements and a lack of expressiveness; third, rendering is disconnected from motion, and generated images are often overly smoothed, losing the exaggerated beauty and dynamism of hand-drawn animation.

[0003] Existing technologies do not deeply bind emotional data with motion parameters, and the rendering process lacks an adaptive adjustment mechanism. They do not achieve full-process optimization of animation generation and cannot enhance the emotional expressiveness and visual realism of characters while ensuring efficiency. Summary of the Invention

[0004] This application provides an AI-based animation generation method that solves the problems of poor character consistency, insufficient emotional expression, and disconnect between rendering and motion in existing technologies. It achieves efficient generation of animations with emotional drive and physical realism, and significantly enhances artistic expression.

[0005] This application provides an AI-based animation generation method, including:

[0006] S1: Generate a structured constraint dataset based on the two-dimensional draft of each character, construct a character-specific generator to obtain generated image data, establish a temporal consistency function to evaluate the generated image data, and determine the specific model file;

[0007] S2: Control rendering based on the structured constraint dataset and dedicated model file to generate a sequence of rendered character images; render the animation background based on the preset background rendering model to obtain the background image; perform image fusion processing on the character image sequence and the background image to obtain the target animation;

[0008] The structured constraint dataset includes edge feature tensors, key point coordinate matrices, and motion trajectory vectors;

[0009] Specifically, for the character's key point sequence, feature vectors of each key point are extracted based on the changing performance; a continuity criterion is set based on the feature vectors, and key points that meet the continuity criterion are connected to form trajectory segments; the merging compliance of any two trajectory segments is calculated according to a pre-set merging index, and if both meet the merging conditions, they are merged into a long trajectory.

[0010] An initial trajectory vector is formed by the displacement vector of each frame in each long trajectory and its corresponding time node; based on the sentiment tags of the trajectory segments, the long trajectory is divided into continuous sentiment segments, sentiment motion mapping rules are established and a scaling factor set is generated; the initial trajectory vector is segmented and adjusted according to the scaling factor set, and combined to form a motion trajectory vector.

[0011] Furthermore, the method also includes: acquiring several emotional segments and their corresponding durations based on sentiment analysis results and time series data; setting core nodes at 0.1-second intervals within each emotional segment, assigning a target sentiment coefficient to each core node, and outputting a sequence of target sentiment coefficients with timestamps; generating a rhythm curve based on the target sentiment coefficient sequence; calculating the energy value of each core node in the emotional segment and performing fitting analysis to obtain an energy curve; and correcting the motion trajectory vector based on the rhythm curve and the energy curve to obtain a corrected motion trajectory vector.

[0012] Furthermore, the rendering parameter set for each frame is determined based on the rhythm curve and energy curve, and the rendered character image sequence is generated based on the rendering parameter set for each frame and the corrected motion trajectory vector; wherein, the time axis of the rhythm curve and energy curve is aligned with the rendering frame rate, and the emotion coefficient and energy value corresponding to each frame are regenerated; color saturation, line intensity and contrast are determined based on the rhythm curve; motion blur intensity, sharpness and light and shadow contrast are determined based on the energy curve; and the rendering parameter set for each frame is generated by combining the above parameters.

[0013] Furthermore, the rhythm curve is a smooth curve used to describe the rate of change of the emotion coefficient over time, with the horizontal axis representing time and the duration of the emotion segment, and the vertical axis representing the emotion coefficient; the energy curve reflects the temporal distribution of motion energy, with the horizontal axis representing time and the vertical axis representing energy, and the energy value is based on the motion kinetic energy formula to measure the motion intensity of each core node.

[0014] Furthermore, peak points in the energy curve are identified, and several peak points are selected as exaggeration points based on the variation rules. The exaggeration ratio is set according to the energy values ​​of the neighboring points to the left and right of the exaggeration point, and the energy value of the exaggeration point is amplified according to the exaggeration ratio. The variation rules are selected based on peak significance, neighboring point change rate and time interval rules.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0016] By extracting structured constraint data from 2D hand-drawn drafts, a character-specific generator was constructed, and image generation was optimized based on a temporal consistency function, achieving high consistency and dynamic rationality of character movement. At the same time, through multi-scale feature injection and layered rendering, the problems of character face-changing and stiff movements in traditional animation were solved.

[0017] By introducing an emotional rhythm curve and energy curve driving mechanism, and through precise mapping of emotional coefficients to the time axis, the amplitude of motion trajectory and rendering parameters are dynamically adjusted, enabling the animation to respond to emotional fluctuations and energy changes in real time, thus achieving a deep integration of emotion and motion. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an animation generation method based on artificial intelligence in an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] To address the issues of poor character consistency and insufficient emotional expression in existing technologies, this application employs structured constraint data generation based on two-dimensional drafts and character-specific model training, thereby achieving the effects of consistent character movement and dynamic rationality; it also utilizes motion trajectory correction and adaptive rendering driven by emotional rhythm curves and energy curves, thereby enhancing emotional expression and improving the realism of the visuals.

[0022] Example 1: As Figure 1 As shown, an animation generation method based on artificial intelligence includes:

[0023] S1: Generate structured constraint data based on the two-dimensional draft of each character, construct a character-specific generator to obtain generated image data, establish a temporal consistency function to evaluate the generated image data, and determine the exclusive model file.

[0024] Specifically, the outline of the two-dimensional draft is extracted based on the edge detection algorithm to obtain a binary edge map, and the edge feature vector is extracted; character pose annotation data is generated, key points of the character and their corresponding coordinates are detected, and a key point coordinate matrix is ​​generated; temporal information is extracted, and motion trajectory vectors are calculated based on frame sequences; the structured constraint dataset includes edge feature tensors, key point coordinate matrices and motion trajectory vectors.

[0025] In some embodiments, the artist's hand-drawn character sketch is obtained as a two-dimensional draft. The outline of the draft is extracted based on the Canny edge detection algorithm to generate a binary edge map. This map is used to detect edge information in the image, clearly separate the character and scene outlines in the draft, separate the main lines of the hand-drawn sketch from the background, retain key structural information, and reduce image complexity.

[0026] The OpenPose model is used to detect keypoints of characters. Based on a convolutional neural network (CNN), it outputs two sets of feature maps: a confidence map and a body part affinity field. The confidence map refers to a keypoint type where each channel corresponds to a keypoint (e.g., left shoulder, right elbow), and each pixel value in the map represents the probability that the location belongs to the corresponding keypoint. The body part affinity field is a set of 2D vector fields, where each vector encodes the position and orientation information of the corresponding limb (e.g., left arm), used to associate keypoints.

[0027] Local maxima are extracted from the confidence map using non-maximum suppression (NMS) as candidate points. For each candidate point, those with a confidence score higher than a confidence threshold are retained as keypoints, and their coordinates and confidence scores are recorded. For example, if the confidence map of the left shoulder reaches its maximum value at pixel position (x, y), then that position is the coordinate of the left shoulder. Non-maximum suppression (NMS) is a commonly used post-processing algorithm in computer vision to eliminate redundant detection boxes or keypoints, ensuring that only one optimal detection result is retained for each target. Its core principle is to suppress non-maximum values ​​and retain local maxima, thereby avoiding the same target being detected multiple times.

[0028] Preferably, a skeletal joint mapping relationship is established, and skeletal connection relationships are predefined to form a skeletal tree. Each key point corresponds to a node in the skeletal tree. Key points belonging to the same limb are connected to form skeletal segments. All key point pairs are traversed to construct a complete skeletal joint mapping relationship. The human pose estimation model based on deep learning can accurately detect the position of human joints. By establishing the skeletal joint mapping relationship, the pose information of the character is converted into digital signals that can be processed by a computer.

[0029] Extracting time-series information, which refers to the dynamic characteristics of data changing over time, requires time-domain analysis, frequency-domain analysis, and joint time-frequency analysis. Time-domain analysis involves extracting basic indicators, including mean, variance, and extreme values. By calculating the variance of keypoint coordinates, the amplitude of character movement is quantified. Fourier transform is used to convert the time-domain signal into a frequency-domain signal, extracting features such as power spectral density and median frequency. For example, high-frequency components correspond to fast movement, while low-frequency components correspond to slow movement. After performing a Fourier transform on the signal, taking the logarithm, and then performing an inverse transform, the cepstral coefficients are obtained to analyze periodic movement patterns, such as the gait cycle of a character walking. Joint time-frequency analysis uses wavelet transform to decompose the signal into multi-scale components, extracting the energy distribution in different time-frequency segments, such as analyzing the take-off and landing phases of a character's jumping action.

[0030] A frame sequence is generated, which refers to a set of consecutive image frames arranged in chronological order. The core characteristics are: temporal continuity (the time interval between adjacent frames is constant or variable); spatial correlation (the position and posture of the same character in different frames must satisfy physical laws (such as parabolic motion); and data redundancy (a large amount of repetitive information exists between consecutive frames, which is compressed through differential coding).

[0031] For the positional analysis of each keypoint, identification is based on the form of change. Specifically, the form of change includes spatial changes: the absolute displacement vector of the keypoint coordinates (e.g., moving from (100,200) to (150,250)); temporal changes: the rate of change of displacement over time (e.g., uniform motion, accelerated motion); morphological changes: changes in the relative distance between keypoints (e.g., an arm going from extended to bent); and emotional changes. Based on a pre-trained sentiment analysis model, sentiment change analysis is performed on the two-dimensional manuscript, outputting sentiment labels and sentiment coefficients. The sentiment coefficients are used to measure the intensity of emotional expression.

[0032] The key point coordinate data of a continuous frame sequence is obtained and standardized to obtain the key point sequence. The standardization process includes: converting the two-dimensional coordinates into offsets relative to the character's centroid to eliminate global displacement interference; ensuring a constant frame rate, and if the frame rate is variable, using interpolation (such as linear interpolation) to resample to a fixed frame rate.

[0033] For each keypoint in the keypoint sequence, feature vectors are extracted based on the change manifestation, including: absolute displacement vector, rate of change of motion, and relative distance change. Based on the feature vectors, a continuity criterion is set. Each keypoint is traversed, starting from the starting frame. If W consecutive frames (W>3, to ensure the integrity of each segment, the length of the frame cannot be too short) satisfy all continuity criteria, the keypoints that satisfy the continuity criteria are connected to form a trajectory segment. Each trajectory segment is labeled with the associated keypoints and time range.

[0034] Specifically, the key point coordinate data is as follows: ,in, This represents the two-dimensional coordinates of the k-th keypoint, where K is the total number of keypoints; after standardization, the centroid coordinates are: The key points of normalization are: ;

[0035] Calculate the offset between adjacent frames: As an absolute displacement vector, the motion start frame is identified based on the displacement amplitude threshold (e.g., pixel distance > 5).

[0036] Calculate the rate of change of motion: determine the type of motion based on instantaneous velocity and instantaneous acceleration, including uniform motion, acceleration, and deceleration;

[0037] Calculate the relative distance change rate: Based on a predefined skeleton tree, calculate the Euclidean distance between keypoint pairs (e.g., the distance between the left elbow and the left wrist) to obtain the distance change rate, quantifying the degree of stretching or bending, as follows: ,in, It is the rate of change of relative distance. It is the Euclidean distance of the k-th keypoint in frame t+1. It is the Euclidean distance of the k-th keypoint in frame t;

[0038] Calculate the sentiment coefficient: Using the two-dimensional draft and content text description as input for sentiment analysis, a pre-trained sentiment analysis model (such as BERT for text, and audio spectrum analysis for music) is used to process the input data, outputting sentiment labels (such as "happy", "angry", "calm") and corresponding sentiment intensity coefficients (value range [0.1, 1]). For example, the coefficient for a high-intensity happy scene is 0.9, and the coefficient for a low-intensity sad scene is 0.3.

[0039] The feature vector of each keypoint in the corresponding frame t is obtained as follows: ,in, It is the absolute displacement vector of the k-th keypoint in frame t. It is the instantaneous velocity of the k-th key point in frame t. It is the instantaneous acceleration of the k-th key point in frame t. It is the rate of change of the relative distance of the k-th keypoint in frame t. It is the sentiment coefficient of the kth key point in frame t. The sentiment coefficient needs to be obtained by analyzing a longer segment. First, the overall sentiment of a segment is determined based on the two-dimensional draft and the content text, and then the sentiment category and sentiment coefficient of each frame are implemented.

[0040] Specifically, the sentiment analysis model aims to automatically identify emotional tendencies and quantify emotional intensity from associated textual descriptions of two-dimensional artwork (such as character backgrounds and scene atmosphere descriptions provided by artists). Based on a pre-trained natural language processing model, it outputs sentiment labels and sentiment coefficients through text classification and regression tasks. Alternatively, it can be based on manually labeled sentiment labels and intensity coefficients; for example, animators can score 1000 text samples according to emotional intensity, with labels categorized as "joyful," "angry," "sad," and "calm."

[0041] Based on the pre-trained sentiment analysis model, a classification layer (for sentiment labels) and a regression layer (for sentiment coefficients) are added. The classification layer outputs the probability of the sentiment category (e.g., P(happiness) = 0.7, P(sadness) = 0.2); the regression layer outputs continuous values, i.e., sentiment coefficients (e.g., 0.85); the sentiment label with the highest probability is selected as the final output (e.g., if P(happiness) = 0.7 > threshold 0.5, then the label is "happiness").

[0042] The continuity criteria include spatial continuity, temporal continuity, morphological continuity, and emotional consistency. Requirements for each criterion are pre-set, such as: adjacent frame displacement being less than a displacement amplitude threshold (e.g., pixel distance > 5); motion change rate thresholds for velocity and acceleration are set to avoid abrupt changes; relative distance change rate is less than 0.1 to prevent abnormal deformation; emotional consistency refers to whether the emotional category and emotional coefficient of two segments are consistent. If the emotional category is the same and the difference in emotional coefficient is less than 0.2 (this needs to be dynamically adjusted according to actual requirements), then the emotional consistency requirement is met. Therefore, the emotional category of each frame in each trajectory segment is the same and the difference in emotional coefficient is less than 0.2.

[0043] The merging compliance of any two trajectory segments is calculated based on pre-set merging indices. If both segments meet the merging conditions, they are merged into a longer trajectory. The merging indices include endpoint velocity direction, morphological features, and inter-frame difference.

[0044] Specifically, the difference in the direction of the endpoint velocity, for the segment End frame fragments starting frame The calculation of the velocity vector angle measures the difference in velocity direction between endpoints; if it is less than 20 degrees, candidates are merged; the calculation of morphological features (such as limb angles) at the endpoints of segments is performed, with a difference threshold set to 10 degrees; the frame difference between endpoints is also considered. If the number of frames is less than 2, the candidates will be merged.

[0045] In this application, both the continuity criterion and the merging index evaluate multi-dimensional factors between any frames to generate trajectory segments and long trajectories. The continuity criterion analyzes the correlation between frames, so the threshold setting should be lower than the threshold of the merging index.

[0046] The continuity of long trajectories is verified using color histograms or feature point matching. Preferably, Kalman filtering is applied to the generated long trajectories to suppress noise, preserving high-frequency motion details while removing jitter interference. Pre-set analysis metrics are used to analyze the generated long trajectories. These metrics include: motion amplitude (trajectories length, reflecting the overall range of motion); motion direction (main motion directions are statistically analyzed using angle histograms, e.g., 0° represents horizontal to the right); motion frequency (number of vibrations at key points per unit time, e.g., the cycle of a waving gesture); motion stability (standard deviation of the trajectory vector, a smaller value indicates smoother motion); and motion symmetry (similarity of the motion trajectories of key points on the left and right limbs, e.g., the symmetry of arm swings during walking). This ensures that the movements conform to the laws of mechanics (e.g., inertia, symmetry), detects consistency in frequency and direction, avoids mechanical-sounding movements, and the analysis results are directly fed back to the generator, automatically adjusting parameters. Furthermore, the analysis time is reduced from hours to seconds, making it suitable for batch generation of long videos.

[0047] The initial trajectory vector is formed by the displacement vector of each frame in each long trajectory and its corresponding time node. The long trajectory is composed of multiple trajectory segments. Each segment must meet the continuity criterion. Based on the sentiment tags of the trajectory segments, the long trajectory is divided into continuous sentiment segments. Sentiment motion mapping rules are established and a scaling factor set is generated. The initial trajectory vector is segmented and adjusted according to the scaling factor set and combined to form the motion trajectory vector.

[0048] An emotion-motion mapping rule is established to map emotion categories and emotion coefficients to specific motion parameters. The value range of the emotion coefficient is [0.1, 1]. A scaling factor is generated. Specifically, this application takes three emotion categories (joy, sadness, and anger) as examples, and sets three motion dimensions to describe each emotion category. The motion dimensions include motion amplitude, motion frequency, and motion smoothness, and each motion dimension corresponds to a personality factor. A scaling factor set is formed based on the emotion category and the corresponding three personality factors. According to the emotion category of each segment in the initial trajectory vector, the corresponding scaling factor set is selected for adjustment.

[0049] The specific adjustment steps are as follows: The emotion coefficient is set to Q. For the happy emotion category, the motion amplitude corresponds to the displacement scaling factor α (α=1.0+0.8×Q, the baseline is 1.0, when Q=1, α=1.8, motion amplification is 80%), the motion frequency corresponds to the frequency factor β (β=1.0+0.5×Q, when Q=1, β=1.5, motion speed is increased by 50%), and the motion smoothness corresponds to the jitter factor J (J=0.05×(1-Q), the higher Q is, the smaller the jitter, when Q=1, J=0 (no jitter)).

[0050] For the emotion of sadness, the amplitude of motion corresponds to the displacement scaling factor α (α=1.0-0.6×Q, with a baseline of 1.0, and α=0.4 when Q=1, resulting in a 60% reduction in motion), the motion frequency corresponds to the frequency factor β (β=1.0-0.4×Q, and β=0.6 when Q=1, resulting in a 40% reduction in motion, which can be adjusted by adjusting the frame interval), and the motion smoothness corresponds to the delay factor D (D=0.1×Q, with higher Q indicating greater delay, simulating a sense of heaviness).

[0051] For the emotion of anger, the amplitude of movement corresponds to the displacement scaling factor α (α=1.0+1.0×Q, when Q=1 α=2.0, the movement is magnified by 100%), the frequency of movement corresponds to the frequency factor β (β=1.0+0.8×Q, when Q=1 β=1.8, the high frequency component is prominent), and the smoothness of movement corresponds to the abrupt change allowance M (M=0.2×Q, the higher Q is, the more abrupt the movement abrupt changes are allowed, and when M>0.1, the smoothness constraint is ignored).

[0052] Preferably, the displacement vector of each frame of each trajectory segment is multiplied by a displacement scaling factor to determine a new displacement vector; within 5 frames before and after the boundary of the emotion segment, the old displacement vector is transitioned to the new displacement vector by linear interpolation to avoid motion jumps. Preferably, an upper limit of morphological features is set to prevent deformation distortion.

[0053] Traditional animation generation methods express characters' emotions through voice-over dialogue and facial expressions, without combining the character's movement state, range of motion, and degree of change with emotional changes, or quantifying the relationship between the two. This results in an inability to fully reflect the artist's emotions and artistic expression in character portrayal.

[0054] In this implementation, emotion tags are obtained based on an emotion analysis model to determine the emotion category and emotion coefficient. The emotion coefficient is checked to see if it falls within the range [0.1, 1]; otherwise, it is cropped to the boundary value. The long trajectory is divided into emotion segments, and the starting frame, emotion category, and emotion coefficient of each segment are recorded. For each emotion segment, the corresponding scaling factor set is calculated based on its category and emotion coefficient Q. The morphological change rate and acceleration constraints are checked in real time, and parameters are adjusted if they are violated. The precise quantitative relationship between the artist's emotions, artistic expression, and multi-dimensional parameters of the motion trajectory is obtained through experiments and then presented.

[0055] A structured constraint dataset is constructed based on the edge feature tensor, keypoint coordinate matrix, and motion trajectory vector. The keypoint coordinate matrix is ​​used to record the coordinate positions of all joint keypoints in the image.

[0056] Based on the character concept design, a large-scale text-based image model (such as StableDiffusion) is used to generate a sample set for the character through prompt word engineering. This sample set includes: a multi-angle sample set (covering front, side, and back views), a multi-action sample set (such as walking, jumping, and facial expression changes), and a multi-scene adaptation sample set (such as the character's appearance under different lighting and background environments). The number of samples is no less than 500, ensuring full coverage of the character's features. Low-rank adaptation techniques are used to iteratively optimize the large-scale text-based image model, learning the character's unique features (such as facial feature proportions, clothing details, and hair color). A loss function is introduced during training to enhance the similarity of features among similar characters (such as the same character in different poses) and suppress interference from non-target features, forming a character-specific model.

[0057] In some embodiments, character concept art consists of hand-drawn sketches by artists that define the visual appearance of animated characters; it includes the character's overall shape, clothing details, facial features, color scheme, etc., serving as a blueprint for character design. It ensures that the generated characters conform to the artist's original intent. For example, a concept art of a fox character might include details such as red fur, large eyes, and specific clothing. Cue word engineering refers to guiding the text-based image generation model to generate images with specific content through carefully designed text descriptions (cues). It involves selecting keywords and adjusting grammatical structure to precisely control the model's output. It ensures that the generated samples cover all aspects of the character's features (such as frontal, side, and walking animations). For example, cue words might be "fox character, red fur, large eyes, frontal view, natural light." The text input used to generate the training dataset creates diverse samples by combining keywords such as perspective, action, and scene. The text-based image generation model is a deep learning-based generative model capable of generating high-quality images from text descriptions, using a diffusion process to progressively denoise and generate images from random noise. The model comprises a text encoder (such as CLIP) that converts cue words into vectors, and a diffusion model that generates images through iterative denoising. These serve as the base model, which learns character-specific features through fine-tuning. Low-rank adaptation is a parameter-efficient fine-tuning technique that adapts to new tasks by injecting a trainable low-rank matrix into a pre-trained model without updating all parameters. Specifically, low-rank decomposition matrices are added to the linear layers of the model (such as attention layers), optimizing only a small number of parameters. Low-rank adaptation is used to fine-tune large text-to-image models, enabling the model to learn character-specific features (such as facial proportions and clothing details). Further details are omitted here.

[0058] Specifically, a large-scale image processing model (such as Stable Diffusion) is used to generate multi-angle, multi-action, and multi-scene sample sets, with a total number of images not less than 500. Rotation, scaling, and color jitter are applied to the generated images to increase diversity. A trainable low-rank matrix is ​​inserted into the cross-attention layer to initialize the model parameters. A loss function is set to minimize the loss, and the low-rank matrix is ​​updated through gradient descent. Specifically, for the input image and noise, the noise prediction error is calculated, using the mean squared error and the contrastive loss function as the total loss function, as shown in the formula:

[0059]

[0060] in, For the total loss function, It is a weighting coefficient (with a value range of [0.4, 0.7]), used to balance the two losses; The mean squared error loss function is given by the following formula:

[0061]

[0062] in, It is the expectation operator, which means taking the average of the data distribution; These are sample images, which can be original images from the training set; It's Gaussian noise. It is the diffusion time step (the value range is [1, T], where T is the total number of time steps, the time length of all long trajectories is obtained, and the largest one is set as T); It is the noise predicted by the model, and the parameter θ contains LoRA matrices A and B; It is the noise dispatch coefficient, which controls the noise intensity. It is defined by the diffusion process and the formula is: in, It is a variance table, which contains predefined parameters for the diffusion process. It represents the noise variance at step s, and usually increases with s to ensure that the final result is close to pure noise. It is a chain multiplication operator, representing the cumulative product from s=1 to s=t.

[0063] It is the contrastive loss function, and the formula is:

[0064]

[0065] in, It is the feature embedding (positive sample pair) of the same character in different poses; is the feature embedding (negative sample) of the kth distinct role; N is the total number of distinct roles; It is a similarity function; This is a temperature parameter that controls the distribution sharpness (set to a baseline value of 0.1). During training, each batch contains both positive and negative samples, while simultaneously optimizing noise prediction and feature similarity.

[0066] Calculate the gradient of the loss with respect to A and B, and the update formula is:

[0067]

[0068] in, is the learning rate, which controls the step size of gradient descent; A is the low-rank matrix A in the LoRA model, with a size of m×r, where m is the size of the base model layer and r is the rank (e.g., r=64); A is initialized to a zero matrix to ensure that the fused weights remain unchanged at the start of training; B is the low-rank matrix B in the LoRA model, with a size of r×n, where n is the size of the base model layer; B is initialized to a random small value to introduce small perturbations; It is the gradient of the loss function with respect to matrix A, representing The rate of change in direction A is calculated through backpropagation; It is the gradient of the loss function with respect to matrix B; the update rule adjusts A and B along the direction of loss reduction, minimizing... .

[0069] Strengthening the similarity of similar characters involves: extracting image pairs of the same character in different poses from the training set (e.g., front and side views of a fox), encoding them into feature vectors through the model, and setting the loss to minimize the distance between positive sample pairs, making the similarity of the feature vectors close to 1. For example, in the loss, the numerator maximizes the similarity of positive samples. It also involves learning the invariance of character features, ensuring that core features (such as facial proportions) remain consistent even as poses change.

[0070] Suppressing interference from non-target features includes: extracting images of different characters or irrelevant backgrounds from the dataset as negative samples (e.g., a fox character against a tree background). The denominator includes all negative samples, maximizing the distance between the character feature embedding and the background or different character feature embeddings, making their similarity close to 0. This prevents the model from overfitting to irrelevant features such as background or lighting, ensuring that attention is focused on the character itself.

[0071] Multi-scale features are injected into the trained LoRA model to form a character-specific generator. During the video generation stage, character features are injected into each frame of the generation process to ensure that the appearance, proportions, and material features of the character remain consistent across different frames, thus solving the problem of character face distortion in traditional video transcribing.

[0072] In some embodiments, the multi-scale features include high-level feature control: maintaining the overall structure and proportions of the character; mid-level feature refinement: clothing texture and material representation; and low-level feature optimization: facial details and color consistency. Image data is generated based on a character-specific generator, and a temporal consistency function is established to evaluate the quality of the generated image data. Based on the evaluation results, a specific model file is determined. The temporal consistency function includes feature similarity loss, structural similarity loss, appearance consistency loss, and motion consistency loss; the specific formula is as follows:

[0073]

[0074] in, It is the overall assessment score; These are the weighting coefficients for the corresponding losses, which sum to 1 and are preset based on historical experimental data (e.g., 0.2, 0.2, 0.3, 0.3).

[0075]

[0076] in, It is feature similarity loss; It is a feature extraction function; These are image features of adjacent frames; Represents the norm;

[0077]

[0078] in, It is structural similarity loss. It is a character mask, representing the character region; It involves multiplying elements one by one, focusing on the differences between the roles and regions;

[0079]

[0080] in, It is a loss of appearance consistency. It is a color histogram feature;

[0081]

[0082] in, It is a loss of motor consistency. It is an optical flow estimation function that ensures smooth motion.

[0083] The image data with the lowest overall evaluation score is selected as the dedicated model file. Rendering is controlled based on the structured constraint dataset and the dedicated model file to generate a sequence of rendered character images. In this embodiment, layered rendering data is implemented, including a character layer, a background layer, and an effects layer. Structured constraints are input into the generator as conditions. For example, edge maps are injected through a ControlNet network to control the character's outline; keypoint matrices are injected through position encoding into the cross-attention layer to guide pose generation.

[0084] For each frame in the sequence, the structured constraint dataset of the current frame is used as a condition vector for noise initialization and denoising. After denoising, a rendered single-frame image is obtained according to a preset character rendering model. The constraint and denoising process is repeated to generate the entire character image sequence. For example, the edge map and keypoint matrix are converted into latent space representation by an encoder. Local metrics for each frame (e.g., PSNR > 30dB) are calculated in real time. If the metrics are abnormal, the frame is regenerated. Optical flow is used to check the continuity of motion between frames to ensure no jumps. For example, to generate a fox character walking sequence (60 seconds, 1440 frames), for the t-th frame, the input structured constraint dataset, such as the edge map (defining the contour) and keypoints (defining the walking posture), is used. The generator outputs a color image that meets the constraints. The rendered character image sequence is a series of static images arranged in chronological order (e.g., 1440 PNG images), with each frame representing a moment in the animation. Optical flow is used to interpolate the motion between frames to reduce stuttering.

[0085] Based on the two-dimensional draft, the corresponding animation background material is pre-set, and the animation background is rendered according to the preset background rendering model to obtain the background image; the character image sequence and the background image are image fusion processed to obtain the target animation.

[0086] In this embodiment, based on the conversion of two-dimensional drafts into constraint data, the LoRA model is trained to render the character, and the model timing is optimized to generate the final animation; this achieves synergy between artistic creation and technical implementation; the character-specific LoRA model is combined with line drawing constraints to form a dual control system of feature anchoring and structural constraints: the LoRA model learns the character's exclusive visual features (such as facial proportions and clothing details) through pre-training, while the line drawing defines the character's dynamic structure and spatial relationships. The two work together to ensure the dual accuracy of the generated content in terms of character consistency (appearance, proportions) and dynamic rationality (actions, postures), breaking through the technical bottleneck of losing focus or dynamics due to a single constraint method.

[0087] Example 2: In Example 1, a basic framework for animation generation was constructed, transforming the artist's hand-drawn intentions into a digital sequence of character movements, establishing a precise mapping from the original sketch to the animation. This example further improves upon the above.

[0088] The method further includes: acquiring several emotional segments and their corresponding durations based on sentiment analysis results and time series data; setting core nodes at 0.1-second intervals within each emotional segment; for example, for a 2-second happy segment, time points include t=0.0s, 0.1s, 0.2s…2.0s. Each core node is assigned a target sentiment coefficient, outputting a sequence of target sentiment coefficients with timestamps; generating a rhythm curve based on the target sentiment coefficient sequence; calculating the energy value of each core node within the emotional segment and performing fitting analysis to obtain an energy curve; and correcting the motion trajectory vector based on the rhythm curve and energy curve to obtain a corrected motion trajectory vector. The time series data includes a sequence of keypoint coordinates and time nodes.

[0089] The target emotional coefficient is generated based on linear interpolation of the overall intensity of the emotional segment. Linear interpolation is used to generate a smooth transition emotional coefficient within the emotional segment, determining the start time, end time, initial emotional coefficient, and end emotional coefficient of the segment. Within the emotional segment, core nodes and corresponding emotional coefficients are set at 0.1-second intervals. For example, if the initial Q of a joyful segment is 0.7 and the ending Q is 0.9, then Q is 0.75 at t=0.5s. Linear interpolation ensures the continuity and smoothness of the emotional coefficient over time, avoiding sudden jumps; it simulates the gradual change of real emotions, making the emotionally driven movement adjustments more natural. Each core node represents the emotional intensity at that moment, serving as the basis for generating the rhythm curve and controlling the rate of change of the movement rhythm. Linear interpolation is a well-known technique, and will not be elaborated upon further here.

[0090] In some embodiments, the rhythm curve is a smooth curve describing the rate of change of the emotion coefficient over time, and its shape determines the acceleration or deceleration of the rhythm. Generation is based on Bézier curve interpolation to ensure the curve is continuous and conforms to the characteristics of emotion intensity. The horizontal axis represents time t (seconds), indicating the duration of an emotional segment; the vertical axis represents the emotion coefficient Q (unitless), indicating the emotion intensity. The curve reflects the rate of change of the emotion coefficient over time. High-intensity emotion segments (Q>0.8) have steep curves, indicating rapid changes (such as the accelerated rhythm of a joyful segment); low-intensity emotion segments (Q<0.4) have flat curves, indicating slow changes (such as the decelerated rhythm of a sorrowful segment).

[0091] To ensure the smoothness of the rhythm curve, interpolation is required between any two adjacent core nodes, as follows: For each pair of adjacent core nodes... and Calculate the control points (i.e., interpolation points): Control point one: Control point two: ; Generating continuous curves using Bézier's formula:

[0092]

[0093] in, Used to calculate the rate of change over time. Indicates the start and end points of time. This represents the time corresponding to the i-th core node; This represents the sentiment coefficient of the i-th core node; and These are the ordinates of control point one and control point two, respectively.

[0094] In some embodiments, the energy value is based on the kinetic energy formula. The motion intensity of each core node is quantified. Gaussian fitting is used to smooth the energy values ​​of the core nodes, generating energy curves. For core nodes... Key point speed The energy curve is calculated by dividing the displacement difference by the time interval, which is 0.1 seconds. The horizontal axis of the energy curve represents time (seconds), indicating the timeline of the emotional segment; the vertical axis represents energy E (a unitless scalar value), indicating the intensity of the movement. It reflects the temporal distribution of movement energy: peaks correspond to high-intensity movements (such as the apex of a jump), and troughs correspond to stillness or slow movement. The smoothness of the curve ensures natural movement and avoids abrupt changes in energy.

[0095] By controlling the amplitude of movement (displacement scaling) through rhythm curves and constraining the intensity of movement through energy curves, the movement trajectory is made to conform to the emotional rhythm while maintaining physical rationality; ensuring that the movement rhythm matches the emotional fluctuations and that the energy distribution is natural.

[0096] Peak points in the energy curve are identified, and several peak points are selected as exaggeration points based on variation rules. An exaggeration ratio is set according to the energy values ​​of the neighboring points to the left and right of each exaggeration point, and the energy value of the exaggeration point is amplified according to the exaggeration ratio. The variation rules are selected based on peak significance, the rate of change of neighboring points, and time interval rules. A sliding window method is used to identify all local extreme points in the energy curve. A dynamic exaggeration ratio is calculated based on the energy values ​​of the neighboring points to the left and right of each exaggeration point to ensure smooth energy changes after amplification. The exaggeration ratio is applied to adjust the energy of the exaggeration point, and the energy curve is updated synchronously.

[0097] Specifically, record the set of all peak points that meet the conditions. ,in, Peak time, The peak energy is the energy value. If the peak energy is higher than the sum of the global energy average and one standard deviation, it is marked as peak significance. The energy change rate between the peak point and its left and right neighboring points is calculated. If it is greater than the preset threshold (0.1), it is marked. The minimum time interval between exaggerated points is 0.3 seconds, i.e., 3 sampling points. If it meets the criteria, it is marked. Points that meet all the rules are marked as exaggerated points.

[0098] For each exaggerated point, a dynamic exaggeration ratio is calculated. This ratio is set based on the energy values ​​of the neighboring points to the left and right of the exaggerated point, ensuring a natural transition between the amplified peak and the energy gradient of neighboring points. Specifically, one neighboring point is selected on each side, with a base exaggeration ratio of 1.3 (default amplification 30%). The average energy of the neighboring points is calculated, and the gradient factor is also calculated. ,in, It's an exaggerated energy value. It is the average energy of neighboring points; the exaggeration ratio is... ,in, The exaggeration coefficient is set to 0.35; the exaggeration ratio ranges from [1.3, 2.0]. The exaggeration ratio is applied to adjust the energy at the exaggeration point, and the energy curve is updated.

[0099] In this embodiment, by amplifying the peak energy value, the intensity of the character's movement at high-energy moments (such as jumping or striking) is significantly enhanced, making key actions more visually impactful, simulating the dynamic feeling of high-speed movement, and enhancing the rhythm and layering of the image. The exaggerated operation of the energy curve peak, through reverse fine-tuning, produces multi-dimensional positive effects in animation generation: visually enhancing key actions, artistically improving expressiveness, and technically ensuring high efficiency and controllability.

[0100] Traditional rendering processes often overlook the dynamic influence of emotional rhythm and motion energy, leading to a disconnect between the visuals and the artistic expression of character movements. To address this issue, this solution introduces an emotion-driven mechanism into the rendering stage based on rhythm and energy curves, proposing an adaptive rendering scheme. By dynamically adjusting rendering parameters (such as color, texture, and lighting) for each frame, the sequence of character images visually and accurately reflects emotional fluctuations and energy changes, thereby enhancing the immersiveness and expressive depth of the animation and forming an integrated motion-rendering pipeline.

[0101] The rendering parameter set for each frame is determined based on the rhythm curve and energy curve. A rendered character image sequence is then generated based on this parameter set and the corrected motion trajectory vector. Specifically, the time axes of the rhythm curve and energy curve are aligned with the rendering frame rate, and the emotion coefficient and energy value for each frame are regenerated. Color saturation, line intensity, and contrast are determined based on the rhythm curve. Specifically, color saturation is as follows: ,in, Based on saturation, The scaling factor is set to 0.3; This is the emotional coefficient; the higher the emotional coefficient, the higher the saturation and the more vibrant the image.

[0102] Line strength is: ,in, For line strength, Based on strength, The control coefficient is set to 0.5; the higher the sentiment coefficient, the stronger the line intensity, i.e., the more pronounced the line jitter.

[0103] Contrast ratio: ,in, For contrast, Based on contrast, To adjust the coefficient, set it to 0.2; the contrast of high-intensity emotional segments is enhanced to highlight the tension of the image.

[0104] Motion blur intensity, sharpness, and light and shadow contrast are determined based on the energy curve; specifically, the motion blur intensity is: ,in, The fuzzy coefficient is set to 0.1. The energy value represents the number of frames. High-energy frames (such as jump peaks) have strong blur intensity and simulate fast motion. Low-energy frames (Mf<0.02) have no blur.

[0105] Sharpening level: ,in, Base sharpness, The sharpening factor is set to 0.2; sharpening is enhanced during peak energy (Ef>0.7) to highlight character details; sharpening is reduced during low energy to avoid noise.

[0106] The light and shadow contrast is: ,in, Based on contrast, The contrast factor is set to 0.25; high-energy frames increase the highlight area to enhance the impact of motion. The scaling factor, control factor, adjustment factor, blur factor, sharpening factor, and contrast factor mentioned above are all optimal values ​​obtained through analysis of experimental data. Due to differences in animation style and personal experience, these factor values ​​can be dynamically adjusted, and this application does not impose specific restrictions on them.

[0107] The above parameters are combined to generate the rendering parameter set for each frame. As a conditional vector, the character image sequence is re-rendered. For each frame, the corresponding motion trajectory data, background description text, and rendering parameter set are input. The preset character base rendering model generates the image according to the above conditions, realizing color adjustment, stylization processing, and unique motion effects, and outputs a single frame image whose visual characteristics are synchronized with the emotion coefficient and energy value.

[0108] This embodiment establishes a dynamic mapping relationship between emotional coefficients, energy peaks, and rendering parameters to achieve real-time adaptive adjustments to visual style, lighting effects, and material representation, ultimately outputting an animation sequence with emotional expressiveness and physical realism. The core lies in breaking the traditional rendering model of separating motion and image processing, deeply binding the visual expression of each frame to the character's movement state. This achieves emotion-driven visuals, making the animation not only logically sound but also emotionally engaging. Rendering parameters are dynamically adjusted over time, avoiding the mechanical feel of traditional static rendering, resulting in an organically changing visual style reminiscent of hand-drawn illustrations.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An animation generation method based on artificial intelligence, characterized in that, include: S1: Generate a structured constraint dataset based on the two-dimensional draft of each character, construct a character-specific generator to obtain generated image data, establish a temporal consistency function to evaluate the generated image data, and determine the specific model file; S2: Control the rendering based on the structured constraint dataset and the dedicated model file to generate a sequence of rendered character images; The animation background is rendered according to the preset background rendering model to obtain the background image; The target animation is obtained by fusing the character image sequence and the background image. The structured constraint dataset includes edge feature tensors, key point coordinate matrices, and motion trajectory vectors; Specifically, for the character's key point sequence, feature vectors of each key point are extracted based on the changing performance; a continuity criterion is set based on the feature vectors, and key points that meet the continuity criterion are connected to form trajectory segments; the merging compliance of any two trajectory segments is calculated according to a pre-set merging index, and if both meet the merging conditions, they are merged into a long trajectory. An initial trajectory vector is formed by the displacement vector of each frame in each long trajectory and its corresponding time node; based on the sentiment tags of the trajectory segments, the long trajectory is divided into continuous sentiment segments, sentiment motion mapping rules are established and a scaling factor set is generated; the initial trajectory vector is segmented and adjusted according to the scaling factor set, and combined to form a motion trajectory vector.

2. The method as described in claim 1, characterized in that, The changes manifest in various forms, including spatial changes, temporal changes, morphological changes, and emotional changes. Spatial changes are reflected by the absolute displacement vector of key point coordinates, temporal changes are reflected by the rate of change of displacement over time, and morphological changes are reflected by the change in relative distance between key points. Based on a pre-trained sentiment analysis model, sentiment change analysis is performed on the two-dimensional manuscript, and sentiment tags and sentiment coefficients are output as sentiment analysis results to reflect the emotional changes. Based on the analysis of the changes in the manifestation, the feature vector of each key point k in the corresponding frame t is obtained as follows: ,in, It is an absolute displacement vector. It is instantaneous speed. It is instantaneous acceleration. It is the rate of change of relative distance. It is the sentiment coefficient.

3. The method as described in claim 2, characterized in that, The method further includes: acquiring several emotional segments and their corresponding durations based on sentiment analysis results and time series data; setting core nodes at 0.1-second intervals within each emotional segment, assigning a target sentiment coefficient to each core node, and outputting a sequence of target sentiment coefficients with timestamps; generating a rhythm curve based on the target sentiment coefficient sequence; calculating the energy value of each core node in the emotional segment and performing fitting analysis to obtain an energy curve; and correcting the motion trajectory vector based on the rhythm curve and the energy curve to obtain a corrected motion trajectory vector.

4. The method as described in claim 3, characterized in that, The rendering parameter set for each frame is determined based on the rhythm curve and energy curve. The rendered character image sequence is generated based on the rendering parameter set for each frame and the corrected motion trajectory vector. Specifically, the time axis of the rhythm curve and energy curve is aligned with the rendering frame rate, and the emotion coefficient and energy value corresponding to each frame are regenerated. Color saturation, line intensity, and contrast are determined based on the rhythm curve. Motion blur intensity, sharpness, and light and shadow contrast are determined based on the energy curve. The rendering parameter set for each frame is generated by combining the above parameters.

5. The method as described in claim 1, characterized in that, The process of forming a trajectory segment includes: setting continuity criteria based on feature vectors, traversing each key point, starting from the initial frame, and if W consecutive frames satisfy all continuity criteria, then a trajectory segment is formed, and each trajectory segment is labeled with associated key points and time range; The continuity criteria include spatial continuity, temporal continuity, morphological continuity, and emotional unity, with requirements for each criterion pre-set. Emotional unity refers to whether the emotional categories and emotional coefficients of two segments are consistent. If the emotional categories are the same and the difference in emotional coefficients is less than 0.2, then the emotional unity requirement is met.

6. The method as described in claim 1, characterized in that, The merging metrics include endpoint velocity direction, morphological features, and inter-frame difference; among which, for segments... End frame fragments Start frame Calculate the angle between the velocity vectors of the two segments to measure the difference in velocity direction at the endpoints; if it is less than 20 degrees, the candidates are merged; calculate the morphological features at the segment endpoints, with a difference threshold set to 10 degrees; and calculate the frame difference between endpoints. If the number of frames is less than 2, the candidates will be merged.

7. The method as described in claim 1, characterized in that, Establishing emotion motion mapping rules and generating a scaling factor set includes: mapping emotion categories and emotion coefficients to specific motion parameters, with the emotion coefficients ranging from [0.1, 1], and generating scaling factors; wherein, three motion dimensions are set for each emotion category, including motion amplitude, motion frequency, and motion smoothness, and each motion dimension corresponds to a personality factor; a scaling factor set is formed based on the emotion category and the corresponding three personality factors, and the corresponding scaling factor set is selected for adjustment according to the emotion category of each segment in the initial trajectory vector.

8. The method as described in claim 3, characterized in that, The rhythm curve is a smooth curve used to describe the rate of change of the emotion coefficient over time. The horizontal axis represents time and indicates the duration of the emotional segment; the vertical axis represents the emotion coefficient. The energy curve reflects the temporal distribution of motion energy, with the horizontal axis representing time. The vertical axis represents energy, and the energy value is based on the kinetic energy formula to measure the motion intensity of each core node.

9. The method as described in claim 1, characterized in that, The peak points in the energy curve are identified, and several peak points are selected as exaggeration points based on the variation rules. The exaggeration ratio is set according to the energy values ​​of the neighboring points to the left and right of the exaggeration point, and the energy value of the exaggeration point is amplified according to the exaggeration ratio. The variation rules are selected based on the peak significance, the rate of change of neighboring points, and the time interval rules.

10. The method as described in claim 1, characterized in that, Constructing a character-specific generator includes: obtaining a sample set based on the character concept design diagram, training a character-specific model, and injecting multi-scale features to form a character-specific generator; the sample set includes a multi-angle sample set, a multi-action sample set, and a multi-scene adaptation sample set; the multi-scale features include high-level feature control, mid-level feature refinement, and low-level feature optimization; the temporal consistency function includes feature similarity loss, structural similarity loss, appearance consistency loss, and motion consistency loss.

Images