An AI cartoon production pipeline construction method based on multi-agent cooperation

Abstract

The application discloses an AI cartoon production pipeline construction method based on multi-agent cooperation, which comprises the following steps: a role consistency agent generates a first candidate frame based on an identity maintaining condition; a scene coherence agent performs motion detection and adaptive smoothing processing to output a second candidate frame; a director scheduling agent identifies the model of a current shot, calculates the feature similarity between the first candidate frame and a reference standard image, and calculates the structural similarity between the second candidate frame and the latest frame; the two results are weighted and summed, and the candidate frame with a higher score is selected and output; and the score difference between the two candidate frames is selected and output. Through the three-module cooperative architecture of the role consistency agent, the scene coherence agent and the director scheduling agent, the three technical problems of role identity drift, scene frame flicker and target conflict of the AI cartoon generation are solved simultaneously in the same framework.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer animation production technology, and in particular to a method for constructing an AI animation production pipeline based on multi-agent collaboration. Background Technology

[0002] In recent years, with the rapid development of generative artificial intelligence technology, deep learning-based image and video generation technologies have been widely applied in the field of animation production. As a video content format with high market demand, the automated production of animated series has become a hot topic of interest in both academia and industry. Currently, various technical solutions have been proposed to address various problems in AI-generated animated series. One existing technology is an animation image generation method based on diffusion models and ControlNet. For example, Chinese patent CN120339457A discloses an AI-based comic book generation device and method. It uses a LoRA model training method combined with the Stable Diffusion framework, and leverages Regional Prompter, ControlNet, and Adetailer plugins to achieve multi-dimensional control over character positioning, actions, and scene style, while also using a local redraw module to refine details. This approach can generate high-quality animation images in single frames or short sequences, but it primarily targets the generation of comic strips or single images. For the generation of continuous multi-frame, multi-shot animation series, this approach lacks explicit modeling of cross-frame character identity consistency and is prone to character feature drift issues in long sequence generation. Furthermore, this approach does not involve a multi-agent collaborative architecture and cannot handle the target conflict between character generation and scene generation.

[0003] The second existing technology is an animation generation method based on multi-agent symbolic reasoning. For example, Chinese patent CN121788677A discloses an automated AI animation generation method and system based on multi-agent collaboration. It constructs a scripting agent, a visual agent, a motion effects agent, and a synthesis agent. Through symbolic reasoning, it generates scene description information with constraints, verifies frame-level logical consistency, and drives iterative adjustments. This solution has advantages in controlling the logical consistency of animation production, but its core relies on the construction of a symbolic reasoning system and a logical rule base. It belongs to logical-level constraints rather than pixel-level visual consistency control, making it difficult to directly solve the problem of maintaining character identities at the image synthesis level, and it lacks a processing mechanism for pixel-level temporal smoothing between generated frames.

[0004] The third existing technology is a video generation method based on multi-agent task decomposition. For example, Chinese patent CN120151560A discloses a video generation method, apparatus, and agent based on multi-agent collaboration. This method uses a first sub-agent to determine sub-task execution elements, and a second sub-agent executes the target task based on these elements, thus achieving the decomposition and collaboration of the video generation task. This solution provides a general multi-agent collaboration framework, but it does not address techniques for preserving character identity features over time, nor does it design a specific inter-frame smoothing mechanism for animation / drama generation scenarios. Furthermore, the optimization goals of multiple agents in this solution are consistent or pre-aligned, failing to consider potential conflicts between character generation and scene generation goals, and lacking a corresponding conflict arbitration mechanism.

[0005] In summary, the following technical problems remain unresolved in the field of automated AI animation production: First, there's the issue of character identity consistency across shots. When generating animated series spanning multiple frames and shots, existing methods often exhibit frame-by-frame changes in facial features, hairstyles, clothing, accessories, and other identifying characteristics of the same character—a phenomenon known as "character drift." While some existing technologies use ControlNet or LoRA for control, these methods lack explicit identity preservation constraints, making it difficult to guarantee consistency in long sequence generation.

[0006] Secondly, there is the issue of scene frame continuity. Existing generation methods typically process each frame or each short sequence independently, leading to abrupt changes or flickering in scene background, lighting, shadows, and object poses between adjacent frames. Although there are existing frame interpolation methods based on optical flow estimation, these methods are mostly general video post-processing techniques, computationally intensive, and require pre-trained models, lacking deep integration with animation generation pipelines.

[0007] Next, there is the conflict between the goals of character generation and scene generation. In animation generation, maintaining character identity consistency may require preserving character edge details and texture features, while maintaining scene frame coherence may require image smoothing. There is an inherent conflict between these two. For example, when a character moves quickly, character consistency requires accurately preserving the character's clothing texture, while scene coherence may blur character edges due to smoothing. In existing multi-agent systems, multiple agents typically serve the same optimization goal, and no solution has yet identified and resolved this conflict between character generation and scene generation.

[0008] Furthermore, the existing solutions face high barriers to implementation. The aforementioned existing technologies generally employ training-intensive methods, including LoRA fine-tuning, reinforcement learning training, and optical flow network pre-training, which require a large amount of labeled data and computing resources. This results in long deployment cycles and high costs, making it difficult to achieve rapid commercialization in resource-constrained scenarios.

[0009] Therefore, how to provide a method for constructing an AI animation production pipeline that can simultaneously solve the problems of character identity consistency, scene frame continuity, and the conflict between the two, while having a low barrier to implementation, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0010] The technical problem solved by this invention is to provide a method for constructing an AI animation production pipeline based on multi-agent collaboration.

[0011] This application provides a method for constructing an AI animation production pipeline based on multi-agent collaboration, including the following steps: S1: Through the role consistency agent, based on the identity preservation conditional generative adversarial network, generate image frames that maintain the consistency of identity features in multiple shots for the input role, and output the first candidate frame; S2: Through the scene coherence agent, motion detection and adaptive smoothing are performed on the continuous historical frame sequence maintained by the director scheduling agent to generate the current frame that is coherent with the continuous historical frame sequence and output the second candidate frame. The scene coherence is implemented using a deterministic algorithm that does not require neural network training. S3: Through the director scheduling agent, the script information is parsed to identify the type of the current shot. The character weight and scene weight are determined according to the preset correspondence table between shot type and weight. A continuous historical frame sequence is maintained. The continuous historical frame sequence is initially empty. After each frame of the final output result is generated, it is stored in the continuous historical frame sequence. S4: Through the director scheduling agent, the feature similarity between the first candidate frame and the reference standard image of the input character is calculated based on the pre-trained visual language model, as a character consistency score; and the structural similarity between the second candidate frame and the latest frame in the continuous historical frame sequence is calculated, as a scene coherence score. S5: Through the director scheduling agent, the role consistency score and the scene coherence score are weighted and summed according to the role weight and scene weight, and the candidate frame with the higher score is selected as the current frame output and the current frame is stored in the continuous historical frame sequence maintained by the director scheduling process. S6: When the score difference between the first candidate frame and the second candidate frame is less than the score difference threshold, the first candidate frame or the second candidate frame is selected and output according to the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence, and the selected frame is stored in the continuous historical frame sequence maintained by the director scheduling process.

[0012] This application also provides an AI animation drama production pipeline construction system based on multi-agent collaboration, comprising the following parts: The role-consistent agent has a built-in identity-preserving conditional generative adversarial network, which receives role identifiers and posture control parameters, generates image frames that maintain the consistency of the role's identity features, and outputs the first candidate frame. The director scheduling device, which is set up in parallel with the role consistency agent, is implemented using a deterministic algorithm that does not require neural network training. It is used to receive the continuous historical frame sequence maintained by the director scheduling device, perform motion detection and adaptive smoothing on the continuous historical frame sequence, generate the current frame that is coherent with the historical frame sequence, and output the second candidate frame. The director scheduling agent, which is communicatively connected to the role consistency agent and the scene coherence agent, performs the following functions: parsing script information to identify the type of the current shot; determining the role weight and scene weight according to a preset correspondence table between shot type and weight; maintaining a continuous historical frame sequence, which is initially empty and is stored in the continuous historical frame sequence after each final output frame is generated; calculating the feature similarity between the first candidate frame and the standard reference image as a role consistency score based on a pre-trained visual language model, and calculating the structural similarity between the second candidate frame and the latest frame in the continuous historical frame sequence as a scene coherence score; weighted summing of the two scores according to the role weight and scene weight, and selecting the candidate frame with the higher score as the current frame output; when the score difference between the two candidate frames is less than the score difference threshold, selecting one of the candidate frames to output based on the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence; and storing the output candidate frame or the current frame in the continuous historical frame sequence.

[0013] Compared to existing technologies, this invention, through a three-module collaborative architecture of a character consistency agent, a scene coherence agent, and a director scheduling agent, solves for the first time, simultaneously addressing three major technical challenges in AI animation generation: character identity drift, scene frame flickering, and conflict between their objectives. Specifically, the character consistency agent, based on an identity-preserving conditional generative adversarial network and a multi-scale discriminator, achieves stable character identity preservation across shots; the scene coherence agent employs a training-free deterministic smoothing algorithm to achieve inter-frame structural coherence with extremely low computational cost; and the director scheduling agent, through shot type-driven weighted scheduling and motion amplitude-driven conflict mitigation mechanisms, proactively arbitrates output conflicts between the two agents in scenarios such as close-ups versus long shots, and details versus smoothness, transforming their mutual constraints into collaborative complementarity. The three agents form an organic whole of "character detail preservation, scene smoothness preservation, and director coordination preservation," with significantly better overall performance than using any single agent or its simple superposition. Compared to existing training-intensive solutions, this invention requires training only one module, has virtually zero data annotation needs, and shortens the deployment cycle from 6-12 months to 1-3 months, significantly lowering the barrier to entry for AI animation production. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0015] Figure 1 This is a flowchart of the AI ​​animation production pipeline construction method based on multi-agent collaboration of the present invention; Figure 2 This is a structural block diagram of the AI ​​animation drama production pipeline construction system based on multi-agent collaboration of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Example 1

[0017] like Figure 1 As shown, the AI ​​animation production pipeline construction method based on multi-agent collaboration of the present invention includes the following steps: S1: Through the role consistency agent, based on the Identity-preserving Conditional Generative Adversarial Network (IPcGAN), generate image frames that maintain consistent identity features for the input role in multiple shots, and output the first candidate frame; S2: Through the scene coherence agent, motion detection and adaptive smoothing are performed on the continuous historical frame sequence maintained by the director scheduling agent to generate the current frame that is coherent with the continuous historical frame sequence and output the second candidate frame. The scene coherence is implemented using a deterministic algorithm that does not require neural network training. S3: Through the director scheduling agent, the script information is parsed to identify the type of the current shot. The character weight and scene weight are determined according to the preset correspondence table between shot type and weight. A continuous historical frame sequence is maintained. The continuous historical frame sequence is initially empty. After each frame of the final output result is generated, it is stored in the continuous historical frame sequence. S4: Through the director scheduling agent, the feature similarity between the first candidate frame and the reference standard image of the input character is calculated based on the pre-trained visual language model, as a character consistency score; and the structural similarity between the second candidate frame and the latest frame in the continuous historical frame sequence is calculated, as a scene coherence score. S5: Through the director scheduling agent, the role consistency score and the scene coherence score are weighted and summed according to the role weight and scene weight, and the candidate frame with the higher score is selected as the current frame output and the current frame is stored in the continuous historical frame sequence maintained by the director scheduling process. S6: When the score difference between the first candidate frame and the second candidate frame is less than the score difference threshold, the first candidate frame or the second candidate frame is selected and output according to the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence, and the selected frame is stored in the continuous historical frame sequence maintained by the director scheduling process.

[0018] The specific implementation methods of the three intelligent agents are described in detail below.

[0019] I. Specific Implementation of Role-Consistent Intelligent Agents The core of the role-consistent agent is an identity-preserving conditional generative adversarial network (IPcGAN), whose training and generation process is as follows: (1) Character Registration and Embedding Extraction: Collect anime-style image datasets, such as the Danbooru dataset, which contains approximately 3 million labeled images. For each new character, input K standard reference images (including different angles such as front, side, and back views), where K is a preset positive integer ranging from 3 to 5. Map the standard reference images to a fixed-dimensional identity embedding vector using a pre-trained identity embedding network, denoted as: ; in, Embedding identity into the network, This represents the i-th standard reference image. R represents the real number field. The dimension of the identity embedding vector is 128 or 256, where i represents the digit of the embedding vector. Zhang's standard reference image, with values ​​ranging from 1 to... .Will Store in the character knowledge base.

[0020] (2) Conditional Generation: The generator adopts the StyleGAN2 architecture and contains 18 convolutional layers. The generator's input consists of three parts: a random noise vector sampled from a standard normal distribution. Attitude control vector (One-hot encoding or continuous encoding can be used to represent action types and expression types, such as actions like "running" and "jumping" and expressions like "happy" and "angry"), as well as identity embedding vectors extracted from the character knowledge base. ,in This represents a normal distribution (also known as a Gaussian distribution). The text in parentheses... This represents a vector with zero mean. The unit covariance matrix (i.e., each dimension is independent and has a variance of 1) is concatenated and input into a mapping network. The intermediate hidden code is output, and then a synthesis network generates an output image with a resolution of 512×512. The generator outputs candidate character images: ; in, For generator functions, The generated candidate character image is the first candidate frame output by the character consistency agent, denoted as . .

[0021] (3) Multi-scale discriminator: The discriminator consists of M sub-discriminators, where M is a preset positive integer, with a value of 3, to process images at the original resolution (512×512), 1 / 2 resolution (256×256), and 1 / 4 resolution (128×128), respectively. Each sub-discriminator contains 5 convolutional layers and outputs two branches: a true / false discrimination branch. The output range is True or False Scores Identity matching branch The output range is Identity matching score The m-th sub-discriminator (m=1,2,…, M The output of ) can be represented as: ; in, The input image is either a real image or a candidate character image output by the generator. The final score determines whether the image is real or fake. Identity matching score Take the average value of each sub-discriminator: .

[0022] (4) Joint Loss Function: The Adam optimizer was used with a learning rate of 0.0002, a batch size of 16, and 500,000 training steps. During training, the parameters of the generator and multi-scale discriminator were updated using the following joint loss function: ; The definitions of each loss term are as follows: For the adversarial loss in generative adversarial networks, a hinged loss function is used: ; in, Represents a real image Take the expected value, which is the average value over all real image samples; Indicates the candidate character image Take the expected value, which is the average value of the candidate character image samples. The true / false score output by the discriminator for a real image. The true / false score is the score output by the discriminator for the candidate character image. Indicates the true / false discrimination branch of the discriminator, subscript Derived from "adversarial," it is used to determine whether an input image is a real image or a generated image; Loss due to identity matching: ; in, Indicates the identity matching branch of the discriminator, subscript Derived from "identity," it is used to determine whether the character in the input image is consistent with the character in the standard reference image.

[0023] This loss forces the generator to output images with higher identity matching scores.

[0024] For pixel reconstruction loss, adopt Loss function: ; in, The real image corresponding to the input conditions (if it exists).

[0025] To perceive the loss, the mean squared error is calculated after extracting feature maps using a pre-trained VGG19 network: ; in, For the VGG19 network Feature maps extracted from layers, These represent the number of channels, height, and width of the feature map, respectively. The real target image corresponding to the input conditions. This represents the square of the Euclidean norm.

[0026] The weighting coefficients for each loss term range from 0.1 to 10.0, and are preferably set to... .

[0027] After training, the generator is able to generate high-quality images that maintain consistency between the character's identity and the input identity embedding vector and pose control vector.

[0028] (5) Consistency calibration during inference: During the generation phase, for every preset number generated, i.e. After the first consecutive candidate frame, the newly generated frame is sent back to the multi-scale discriminator to calculate the identity matching score. .like ,in If the identity matching threshold is set to a value between 0.6 and 0.8, preferably 0.7, then the calibration process is triggered: the most recent reliable output result is retrieved from the continuous historical frame sequence. Re-extract the identity embedding vector This allows for fine-tuning of the generator's output, ensuring identity consistency in subsequent frames.

[0029] II. Specific Implementation of Scene-Coherent Intelligent Agents The scene coherence agent employs a deterministic algorithm that does not require neural network training to achieve inter-frame smoothing, specifically including the following sub-steps: (1) Motion detection: Calculate the current frame to be generated Compare with the latest frame (i.e., the output of the previous frame) in the continuous historical frame sequence. Mean pixel difference between: ; in, and These represent the row and column indices of pixels in the image, respectively. The value range is 1 to , The value range is 1 to , and These are the height and width of the image, respectively (in pixels). Indicates the current position of the frame to be generated. Pixel value at that location, Indicates the position of the output result of the previous frame. Pixel value at; subscript To indicate difference. This is the average pixel difference, used to characterize the motion amplitude between two adjacent frames. Taking frame t=35 as an example, H and W are both 512, calculated as follows: =0.42.

[0030] Mean With the preset high threshold and low threshold Compare and determine the scene type: like It is determined to be a static area; like This is classified as slow motion; like This is determined to be rapid movement or scene switching.

[0031] in, The value ranges from 0.03 to 0.08, preferably 0.05; The value range is from 0.25 to 0.35, preferably 0.30; at this time, because This is determined to be rapid movement.

[0032] (2) Smoothing strategy selection: Select the corresponding smoothing strategy according to the determined scene type to obtain the smoothed current frame. : ; in, The linear mixing coefficient has a value ranging from 0.5 to 0.9, preferably 0.7; since the above determination involves fast motion, the current frame is kept unchanged. .

[0033] (3) Adaptive Hybridization and Temporal Filtering: A temporal decay weight is introduced to perform weighted median filtering on the output results of the current frame and the previous two frames. Let the weight of the current frame be... According to the range of motion Dynamic adjustment: ; in, This represents the maximum weight value, which ranges from 0.8 to 0.95, preferably 0.9; This is an offset constant, ranging from 0.1 to 0.3, preferably 0.2; taking frame t=35 as an example, Output frame after weighted median filtering for: ; in, This indicates a median-taking operation. The current frame, which is the one processed by the scene coherence agent, is output as the second candidate frame to the director agent, denoted as... ; Taking frame t=35 as an example, the output result of the previous frame Output results from the previous two frames Given that weighted median filtering is performed according to the above formula: .

[0034] (4) Flicker Detection and Repair: When alternating changes in the value of the same pixel position are detected in three consecutive frames, flickering is determined, and median filtering of the three frames is initiated. The flicker determination condition is: the current position is in and The pixel values ​​are similar, but with The pixel values ​​within the range vary considerably. Specifically, for pixel positions... If the following conditions are met simultaneously: ; Then the pixel is determined to be a blinking pixel. The upper limit threshold for the difference ranges from 5 to 15 (pixel value range from 0 to 255), and is preferably 10. The difference threshold is set to a value ranging from 20 to 40, preferably 30. Pixel positions that satisfy the above conditions... These pixels are marked as flickering pixels. For all flickering pixels, perform three-frame median filtering: ; For non-flickering pixels, keep After flicker repair Ultimately selected as the second candidate frame Output.

[0035] For example, during frames 200 to 205, detection was performed. The location is flashing: Calculate the difference: ,and In fact, in this example The blinking condition is not met. Assuming the condition is met at another location, then the above formula will be applied: .

[0036] III. Specific Implementation of the Director-Scheduling Intelligent Agent The director agent is the scheduling core of the entire pipeline, responsible for parsing the script, determining weights, arbitrating scores, and maintaining a continuous sequence of historical frames.

[0037] (1) Script parsing and shot type identification: The director agent receives the script text input and extracts shot annotation information through natural language processing technology. If the script already contains shot type annotations, it is read directly; if there are no annotations, it is identified through keyword matching rules. The preset correspondence between shot types and weights is as follows:

[0038] The preferred value range for each weight is as follows: , The constraints are satisfied. ,in, and These are the character weights and scene weights determined by the director's scheduling agent. Different shot types correspond to different weight allocations. For example, close-ups focus more on character details, so they have higher character weights; long shots focus more on scene continuity, so they have higher scene weights.

[0039] (2) Lightweight scoring: The director agent calls the pre-trained visual language model (contrastive language image pre-trained model, i.e. CLIP model) to calculate the first candidate frame. Standard reference image for the character The cosine similarity between them is used as a character consistency score: ; in, The feature vector output by the image encoder of the CLIP model. .

[0040] Simultaneously, the structural similarity index is used to calculate the second candidate frame. Compared with the previous frame output result The structural similarity between them is used as a score for scene coherence: ; The calculation formula is: ; in, and These are the two images to be compared. and Images and The average pixel value, and Images and pixel variance For image and covariance, and This is a stability constant used to avoid the denominator being zero. .

[0041] (3) Weighted voting and arbitration: Calculate the total score of the two candidate frames, and the total score of the first candidate frame. Total score of the second candidate frame They are respectively: ; Among them, superscript This indicates that the score corresponds to the first candidate frame (generated by the role consistency agent), with the superscript indicating the first candidate frame. This indicates that the score corresponds to the second candidate frame (generated by the scene coherence agent). The first candidate frame received a high score for character consistency. The scene coherence score for the first candidate frame is low. The second candidate frame received a low score for role consistency. The scene coherence score for the second candidate frame is (high).

[0042] Calculate the score difference: ; Will The difference threshold between the score and the preset score (Values ​​ranging from 0.03 to 0.08, preferably 0.05) are compared: Scenario 1 (Normal Arbitration): If The candidate frame with the higher score is directly selected as the current frame for output. ; Scenario 2 (Conflict Coverage): If This indicates that the two candidate frames are of similar quality, and they enter the conflict fallback process, based on the motion amplitude. Making decisions: ; in, The first motion threshold is defined as 0.35 to 0.45, preferably 0.40. The second motion threshold is defined as having a value ranging from 0.08 to 0.15, preferably 0.10; and satisfies the following conditions: .

[0043] The meaning of the above decision-making rules is as follows: When the range of motion is large ( Select the second candidate frame output by the scene coherence agent to avoid image ghosting; When the range of motion is small ( Select the first candidate frame output by the role consistency agent to preserve role details; When the motion amplitude is between the two, the average pixel value of the two candidate frames is taken as a compromise.

[0044] (5) Maintenance of continuous historical frame sequence: The director agent maintains a queue of fixed length as a continuous historical frame sequence. : ; in, The sequence length, ranging from 5 to 20, is preferably 10. Each frame outputs the final result. The sequence is stored at the tail of the queue, and the oldest frame is removed (if the queue is full). This sequence is used by the scene coherence agent for motion detection and temporal filtering.

[0045] IV. Multi-agent cooperative mechanism The collaboration between the three agents is achieved in the following ways: The director agent sends the current shot's type information and weight parameters to the character consistency agent and the scene coherence agent. This serves as a guiding signal for the two production agents. The role consistency agent and the scene coherence agent work in parallel, each returning the first candidate frame. Second candidate frame The director agent receives two candidate frames, scores and arbitrates them, and selects the final output frame. The director agent will select the result (i.e., the identifier of the selected candidate frame). Feedback is sent to the two production agents for adjustments to the generation of subsequent frames.

[0046] This invention, through a three-module collaborative architecture of a role consistency agent, a scene coherence agent, and a director scheduling agent, solves for the first time, within the same framework, three major technical challenges in AI animation drama generation: character identity drift, scene frame flickering, and target conflict between the two. Among them, the character consistency agent, based on an identity-preserving conditional generative adversarial network and a multi-scale discriminator, achieves stable maintenance of character identity across shots, and the same character can maintain facial keypoint errors of less than 3 pixels in continuous shots of more than 50 frames, which is significantly better than existing methods based on diffusion models. The scene coherence agent adopts a deterministic smoothing algorithm that does not require training, which greatly reduces the light and shadow jumps and structural flicker caused by independently generating each frame, and achieves inter-frame structural coherence with extremely low computational cost. The director scheduling agent actively arbitrates the output conflicts between the two agents in scenarios such as close-up and long shot, detail and smoothness through shot type-driven weight scheduling and motion amplitude-driven conflict fallback mechanism, so that the two agents are transformed from mutual constraints to synergistic complementarity. In addition, only the character consistency agent requires training, while the scene coherence agent and the director agent both adopt deterministic algorithms or pre-trained models that do not require training, reducing the overall training cost by about 67% compared with existing end-to-end solutions. The three agents form an organic whole of "character detail preservation, scene smoothness preservation, and director coordination preservation", and the comprehensive effect is significantly better than using any one agent alone or its simple superposition. Compared to existing training-intensive solutions, the entire pipeline of this invention can run on a single NVIDIA RTX 4090 graphics card, generating a one-minute video takes about 3 minutes, and only one module needs to be trained. The data annotation requirement is almost zero, and the deployment cycle is shortened from 6-12 months to 1-3 months, significantly reducing the threshold for the implementation of AI animation production.

[0047] Example 2: Complete Generation of the Animated Short Series "Forest Adventure" This embodiment uses the generation of a 3-minute animated short drama containing approximately 4,500 frames as an example to describe the complete implementation process of the present invention.

[0048] 1. Preliminary preparations: (1) Character registration: Input three standard reference images of the protagonist "Little Deer" (front, left side, and right side), and extract a 128-dimensional identity embedding vector through the identity embedding network. Store the information in the character knowledge base. Register supporting characters such as "Squirrel" and "Rabbit" using the same method.

[0049] (2) Script Input: Input the script "Forest Adventure". The script includes storyboard descriptions, for example: Shot 1: Wide shot, panoramic view of the forest, a deer walks into the frame from the left. Scene 2: Medium shot, the deer looks up at the fruit on the tree. Scene 3: Close-up, the deer's facial expression (happy). 2. Frame-by-frame generation process: Initialization: The director agent creates an empty queue as a continuous sequence of historical frames. The initial length is frame.

[0050] Process the first frame (long shot): The director's AI analyzes the script, identifies the current shot as a "wide shot / long shot," and queries the weight table to obtain... .

[0051] The role-consistent agent receives the role ID of "Little Deer" (corresponding to...) (and the posture of "walking"), generating the first candidate frame. .

[0052] Scene coherence intelligent agent receives and connects historical frame sequences (Currently empty, using the default black frame), generate the second candidate frame. .

[0053] Director agent computation: ; Calculate the total score: ; choose As the first frame output.

[0054] Store the first frame. .

[0055] Processing frame 20 (close-up): Recognized as "character close-up". .

[0056] generate The character's features are highly consistent with the standard reference image. .

[0057] generate The scene is smooth, but the character details are slightly blurry. .

[0058] Calculate the total score: ; choose Output.

[0059] Processing frame 35 (fast-moving shot - deer running): Recognized as "chasing / fast movement" .

[0060] generate .

[0061] generate motion detection .

[0062] Calculate the total score: ; Difference Then, proceed with the conflict resolution process.

[0063] Conflict resolution process: ,choose Output.

[0064] 3. Role Consistency Calibration: When generating the 100th frame, the role-consistent agent had completed the generation of 10 frames, triggering calibration: The newly generated frame is sent back to the multi-scale discriminator to calculate the identity matching score. Calibration process initiated: Frame 95 is retrieved from the historical frame sequence. Re-extract the identity embedding vector After fine-tuning the generator parameters, the generation of frame 101 and subsequent frames continues.

[0065] Example 3: Comparative Test with Existing Technology This embodiment compares the present invention with the following prior art solutions (i.e., the prior art referenced in Prior Art 1 and Prior Art 2 in the background art) under the same hardware environment (single NVIDIA RTX 4090 graphics card) and the same script input conditions.

[0066]

[0067] Experimental data show that the present invention outperforms existing technical solutions in four dimensions: role consistency, scene coherence, training cost, and inference speed.

[0068] Example 4: Adaptation to different application scenarios This embodiment describes the adaptation method of the present invention in different application scenarios.

[0069] 1. Production of long-running animated series Maintain the length of the historical frame sequence It can stably generate long sequences of hundreds of frames or more. For long series exceeding 1000 frames, the sequence can be reset periodically or a sliding window mechanism can be used, making it suitable for the automated production of seasonal anime series.

[0070] 2. Interactive animation content generation The director's AI agent can respond to user-inputted shot adjustment commands in real time and dynamically update the weight table. and This value enables collaborative creation between users and AI. For example, when a user emphasizes "maintaining character details," the AI ​​will... Temporarily increased to above 0.8.

[0071] 3. Cross-style anime adaptations By changing the training dataset for the character consistency agent (such as a cyberpunk style or ink painting style dataset) and retraining the generator parameters, it is possible to adapt to the production of animation dramas with different art styles. The scene coherence agent and the director agent do not need to be modified, maintaining their universality.

[0072] Example 5 like Figure 2 As shown, this embodiment provides an AI animation production pipeline construction system based on multi-agent collaboration, which consists of three core agents: a character consistency agent, a scene coherence agent, and a director agent. The three work together to form an end-to-end animation production pipeline.

[0073] The role-consistent agent comprises the following sub-modules: (1) Role Knowledge Base Module: This module stores the identity ID, standard reference image, and corresponding identity embedding vector for each role. The knowledge base uses a key-value pair storage structure, where the key is the role ID and the value is a data structure containing the standard reference image and the identity embedding vector.

[0074] (2) Identity Embedding Extraction Module: Built-in pre-trained identity embedding network (based on ArcFace architecture, fine-tuned on Danbooru anime dataset). Input standard reference image, output 128-dimensional or 256-dimensional identity embedding vector.

[0075] (3) Conditional generation module: The generator adopts the StyleGAN2 architecture, takes random noise vector, pose control vector and identity embedding vector as input, and outputs the first candidate frame. The generator contains 18 convolutional layers and the output resolution is 512×512.

[0076] (4) Multi-scale discrimination module: It contains 3 sub-discriminators, which process images with resolutions of 512×512, 256×256 and 128×128 respectively. Each sub-discriminator outputs a true / false score and an identity matching score.

[0077] (5) Calibration module: After every 10 frames generated, the latest frame is sent back to the multi-scale discrimination module to calculate the identity matching score. If it is lower than the threshold (0.7), the most recent reliable output is obtained from the director agent, the identity embedding vector is re-extracted, and the generator parameters are fine-tuned.

[0078] The scene coherence agent employs a deterministic algorithm that requires no training and includes the following sub-modules: (1) Motion detection module: Calculate the average pixel difference between the current frame to be generated and the output result of the previous frame, compare it with the low motion threshold (0.05) and the high motion threshold (0.30), and determine the scene type (static area / slow motion / fast motion or scene switching).

[0079] (2) Smoothing strategy selection module: Select a smoothing strategy according to the scene type. Static areas reuse the previous frame, slow motion uses linear blending (blending coefficient 0.7), and fast motion keeps the current frame unchanged.

[0080] (3) Adaptive filtering module: Calculate dynamic weights (range 0.6 to 0.9) and perform weighted median filtering on the output results of the current frame and the previous two frames.

[0081] (4) Flicker Repair Module: Detects flickering pixels (pixel values ​​at the same position change alternately in three consecutive frames) and performs median filtering on the flickering pixels over three frames.

[0082] The director agent is the scheduling core of the entire system and includes the following sub-modules: (1) Script parsing module: Receives script text input and extracts shot type information through natural language processing technology. If the script contains annotations, it is read directly; otherwise, it is identified through keyword matching rules.

[0083] (2) Weight determination module: Query the preset weight table according to the shot type and output the character weight. and scene weight ,satisfy .

[0084] (3) Historical frame sequence maintenance module: Maintain a queue of length 10 to store the continuous historical frame sequence that has been output. Each time a frame is output, it is stored at the end of the queue and the oldest frame is removed.

[0085] (4) Scoring module: Call the pre-trained CLIP model to calculate the cosine similarity between the first candidate frame and the standard reference image as the role consistency score; call the SSIM calculation function to calculate the structural similarity between the second candidate frame and the output result of the previous frame as the scene coherence score.

[0086] (5) Arbitration module: Calculate the weighted total score of the two candidate frames and calculate the difference in total score. When the score difference is greater than or equal to 0.05, the one with the higher score is selected for output. When the score difference is less than 0.05, the conflict fallback process is entered: when the motion amplitude is greater than 0.40, the second candidate frame is selected; when it is less than 0.10, the first candidate frame is selected; when it is between the two, the pixel average value is taken.

[0087] (6) Feedback module: Feedback the arbitration result (the identifier of the selected candidate frame) to the role consistency agent and the scene coherence agent for the generation and adjustment of subsequent frames.

[0088] The system workflow is as follows: After the system starts, the director agent initializes the historical frame sequence to an empty queue. For each frame: (1) The director agent analyzes the current shot type and determines the weight. The weights are then sent to the role-consistent agent and the scene-coherent agent.

[0089] (2) The role-consistent agent generates the first candidate frame based on the role ID and the attitude control vector. Return to the director's intelligent agent.

[0090] (3) Scene coherence agent generates second candidate frames based on historical frame sequence Return to the director's intelligent agent.

[0091] (4) The director agent calculates the scores of the two candidate frames, performs weighted voting and arbitration, and outputs the final frame. .

[0092] (5) The director agent will The historical frame sequence is stored, and the arbitration result is fed back to the two production agents.

[0093] (6) Repeat steps (1) to (5) until all frames are generated.

[0094] This system can be deployed in the following hardware environments: Minimum requirements: Single NVIDIA RTX 3060 graphics card, 16GB RAM, Intel Core i7 processor.

[0095] Recommended configuration: Single NVIDIA RTX 4090 graphics card, 32GB RAM, Intel Core i9 processor.

[0096] Generation speed: With the recommended configuration, the video generation time is approximately 3 minutes.

[0097] Training environment: Under the recommended configuration, the training time for the role-consistent agent is approximately 4 hours.

[0098] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0099] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0100] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for constructing an AI animation drama production pipeline based on multi-agent collaboration, characterized in that, Includes the following steps: S1: Through the role consistency agent, based on the identity preservation conditional generative adversarial network, generate image frames that maintain the consistency of identity features in multiple shots for the input role, and output the first candidate frame; S2: Through the scene coherence agent, motion detection and adaptive smoothing are performed on the continuous historical frame sequence maintained by the director scheduling agent to generate the current frame that is coherent with the continuous historical frame sequence and output the second candidate frame. The scene coherence is implemented using a deterministic algorithm that does not require neural network training. S3: Through the director scheduling agent, the script information is parsed to identify the type of the current shot. The character weight and scene weight are determined according to the preset correspondence table between shot type and weight. A continuous historical frame sequence is maintained. The continuous historical frame sequence is initially empty. After each frame of the final output result is generated, it is stored in the continuous historical frame sequence. S4: Through the director scheduling agent, the feature similarity between the first candidate frame and the reference standard image of the input character is calculated based on the pre-trained visual language model, as a character consistency score; and the structural similarity between the second candidate frame and the latest frame in the continuous historical frame sequence is calculated, as a scene coherence score. S5: Through the director scheduling agent, the role consistency score and the scene coherence score are weighted and summed according to the role weight and scene weight, and the candidate frame with the higher score is selected as the current frame output and the current frame is stored in the continuous historical frame sequence maintained by the director scheduling process. S6: When the score difference between the first candidate frame and the second candidate frame is less than the score difference threshold, the first candidate frame or the second candidate frame is selected and output according to the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence, and the selected frame is stored in the continuous historical frame sequence maintained by the director scheduling process.

2. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 1, characterized in that, The identity-preserving conditional generative adversarial network in step S1 includes the following training and generation steps: Extract a standard reference image for each character from the character knowledge base, and map the standard reference image into a fixed-length identity embedding vector through a pre-trained identity embedding network; The random noise vector, the attitude control vector, and the identity embedding vector are input together into the generator in the identity-preserving conditional generative adversarial network to output candidate character images. Using a multi-scale discriminator containing at least two sub-discriminators with different resolutions, a true / false discrimination score and an identity matching score are simultaneously output for the input real image or the candidate character image output by the generator. The identity matching score is used to measure the identity similarity between the character in the input image and the reference character corresponding to the identity embedding vector. The parameters of the generator and the multi-scale discriminator are updated by a joint loss function, the value of which includes at least four terms: adversarial loss, identity matching loss, pixel reconstruction loss, and perceptual loss.

3. The method for constructing an AI animation production pipeline based on multi-agent collaboration as described in claim 2, characterized in that, Step S1 also includes a consistency calibration step: after generating a preset number of consecutive first candidate frames, the latest generated first candidate frame is sent back to the multi-scale discriminator to calculate the identity matching score. If the identity matching score is lower than the identity matching threshold, the calibration process is triggered, the most recent reliable output result is taken from the consecutive historical frame sequence, the identity embedding vector is re-extracted, and the output of the multi-scale generator is adjusted.

4. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 3, characterized in that, Step S2 involves motion detection and adaptive smoothing using a scene coherence agent, specifically including the following steps: Calculate the average pixel difference between the current frame to be generated and the latest frame in the continuous historical frame sequence, and determine the scene type based on the comparison result of the average value with the preset high threshold and low threshold. The scene type includes static area, slow motion and fast motion or scene switching. The corresponding smoothing strategy is selected based on the determined scene type: for static areas, the latest frame in the continuous historical frame sequence is directly reused; for slow motion, a linear mixture of the current frame and the latest frame in the continuous historical frame sequence is used; for fast motion or scene switching, the current frame is kept unchanged. A temporal decay weight is introduced to perform weighted median filtering on the current frame and the two most recent frames in the continuous historical frame sequence. The weight of the current frame is dynamically adjusted according to the motion amplitude; the greater the motion amplitude, the higher the weight of the current frame.

5. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 4, characterized in that, Step S2 also includes: when pixel flickering is detected in multiple consecutive frames, three-frame median filtering is started to remove isolated pixels.

6. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 5, characterized in that, The preset correspondence table between lens type and weight in step S3 includes at least the following entries: Close-up shot of a character: The character weight is set to the first preset value, and the scene weight is set to the second preset value; Close-up facial shot: Character weight is the third preset value, scene weight is the fourth preset value; Dialogue with two people: Character weight is the fifth preset value, and scene weight is the sixth preset value; Medium shot action: Character weight is the seventh preset value, scene weight is the eighth preset value; Panoramic or long shot: Character weight is the ninth preset value, scene weight is the tenth preset value; Scene transition shots: Character weight is the eleventh preset value, and scene weight is the twelfth preset value; For chase or fast-moving shots: character weight and scene weight are equal; 7. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 6, characterized in that, In step S4: The pre-trained visual language model is a contrastive language image pre-trained model, used to calculate the cosine similarity between the first candidate frame and the standard reference image of the character as a character consistency score. The structural similarity is calculated using a structural similarity index.

8. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in claim 7, characterized in that, The specific rules for selecting the first or second candidate frame in step S6 based on the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence are as follows: When the motion amplitude is greater than the first motion threshold, the second candidate frame is selected for output; When the motion amplitude is less than the second motion threshold, the first candidate frame is selected for output; The first motion threshold is greater than the second motion threshold.

9. A system for constructing an AI animation drama production pipeline based on multi-agent collaboration, characterized in that, Includes the following parts: The role-consistent agent has a built-in identity-preserving conditional generative adversarial network, which receives role identifiers and posture control parameters, generates image frames that maintain the consistency of the role's identity features, and outputs the first candidate frame. The director scheduling device, which is set up in parallel with the role consistency agent, is implemented using a deterministic algorithm that does not require neural network training. It is used to receive the continuous historical frame sequence maintained by the director scheduling device, perform motion detection and adaptive smoothing on the continuous historical frame sequence, generate the current frame that is coherent with the historical frame sequence, and output the second candidate frame. The director scheduling agent communicates with the character consistency agent and the scene coherence agent respectively, and performs the following functions: parsing script information to identify the type of the current shot, and determining the character weight and scene weight according to the preset correspondence table between shot type and weight; A continuous historical frame sequence is maintained, initially empty. Each time a final output frame is generated, it is stored in this sequence. Based on a pre-trained visual language model, the feature similarity between the first candidate frame and the standard reference image is calculated as a role consistency score. The structural similarity between the second candidate frame and the latest frame in the continuous historical frame sequence is calculated as a scene coherence score. The two scores are weighted and summed according to the role weight and scene weight, and the candidate frame with the higher score is selected as the current frame output. When the score difference between two candidate frames is less than a score difference threshold, one candidate frame is selected and output based on the motion amplitude between the current frame and the latest frame in the continuous historical frame sequence. The output candidate frame or the current frame is then stored in the continuous historical frame sequence.

10. The method for constructing an AI animation drama production pipeline based on multi-agent collaboration as described in any one of claims 1 to 8, characterized in that, The method is applicable to at least one of the following scenarios: Automated production of long-running animated series; Batch generation of short video animation series; Anime adaptation based on a text script; Interactive animation content generation.

Images