Multi-agent character video generation method and device based on latent diffusion model, equipment and storage medium

Abstract

The application discloses a kind of multi-agent character video generation method, device and equipment based on latent diffusion model and storage medium, the method includes: original video data is processed, obtains multimodal input set and multimodal feature, generates the state triple corresponding to each time step of multi-agent based on multimodal feature, state triple is input to pre-constructed hierarchical intention decomposition model and is decomposed to intention, output structure constraint sequence, structure constraint sequence is input to target rendering model, to execute denoising in latent space, obtain latent video sequence, latent video sequence is decoded, and target character video is obtained;Since the cooperation strategy of the application is optimized by reinforcement learning multi-agent, high-quality character video is generated by combining diffusion model, so as to effectively improve the behavior consistency, emotional expression and interaction coordination of multi-agent character video, so as to greatly improve the spatiotemporal consistency and natural fidelity degree of multi-character video.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a multi-agent character video generation method and device based on a latent diffusion model, equipment and a storage medium. BACKGROUND

[0002] Existing video generation techniques mainly rely on single generation models (such as generative adversarial networks, variational autoencoders, diffusion models, etc.) to model image sequences. However, these methods have significant shortcomings in handling multi-character interactions, emotional expression, and language coordination. Traditional single generation models can only generate the actions or expressions of a single character, lack of interaction and emotional consistency between characters, and are prone to problems such as unnatural character actions, out-of-sync expressions and voices, and lack of logic in interactions in generated videos.

[0003] Especially in multi-character scenarios, the performance of the generation model will be significantly reduced. The lack of consistency in character behavior and emotional coherence often leads to generated videos that cannot effectively express the interactions and emotional changes between characters. For example, when multiple characters appear in a dialogue scenario at the same time, traditional single-agent models cannot effectively simulate dynamic interactions between characters, resulting in a lack of coordination in virtual character behavior and an inability to present natural dialogue flow and emotional changes. SUMMARY

[0004] The main purpose of the present application is to provide a multi-agent character video generation method and device based on a latent diffusion model, which aims to solve the technical problem that existing technologies cannot handle multi-character videos in complex interactive scenarios, resulting in a lack of structure fidelity, spatiotemporal consistency, and long-term behavior in multi-character videos.

[0005] To achieve the above purpose, the present application provides a multi-agent character video generation method based on a latent diffusion model, which comprises the following steps:

[0006] Processing the original video data to obtain candidate input data, the candidate input data including a multi-modal input set and multi-modal features, the multi-modal input set including latent representations, speech signals, and text description information, and the multi-modal features including visual features, emotional features, and language features;

[0007] Generating a state triple corresponding to each time step of the multi-agent based on the multi-modal features, the agent corresponding to a virtual character, and the state triple including a fused emotional state, a task goal, and an interaction state;

[0008] inputting the state triple into a pre-constructed hierarchical intention decomposition model for intention decomposition, outputting a structure constraint sequence, the hierarchical intention decomposition model being constructed based on a Transformer architecture and a language model planner;

[0009] inputting the structure constraint sequence into a pre-constructed target rendering model for processing, obtaining a latent video sequence, the target rendering model taking a denoising UNet network of a latent diffusion model as a backbone and introducing a ControlNet control branch to realize structure condition control, the target rendering model being trained based on a multi-modal input set and multi-modal features;

[0010] decoding the latent video sequence to obtain a target character video.

[0011] Optionally, the processing of the original video data to obtain candidate input data comprises:

[0012] extracting a frame sequence from the original video data and pre-processing the frame sequence, the pre-processing comprising cropping, scaling, brightness and color normalization processing;

[0013] mapping the pre-processed frame sequence to a latent space to obtain latent representation;

[0014] time-aligning the latent representation with a speech signal and text description information in the original video data to obtain a multi-modal input set;

[0015] extracting visual features, emotional features and language features of each frame based on the multi-modal input set to generate multi-modal features.

[0016] Optionally, the task target comprises high-level intention information of the agent; and the generating of the state triple corresponding to each time step of the multi-agent based on the multi-modal features comprises:

[0017] performing feature fusion based on the multi-modal features to generate a fused emotional feature corresponding to each time step of the multi-agent;

[0018] optimizing a collaboration strategy of the multi-agent at each time step to obtain high-level intention information corresponding to each time step;

[0019] constructing a dynamic interaction graph based on the visual features and dynamic relationships between the agents, nodes in the dynamic interaction graph being configured as the agents in each frame scene, initial features of the nodes being the visual features, and edges in the dynamic interaction graph being configured to encode the dynamic relationships between the agents;

[0020] performing information propagation and aggregation on the dynamic interaction graph through a pre-constructed graph attention network to obtain interaction information corresponding to each time step of the multi-agent.

[0021] Optionally, the cooperation strategy of the plurality of agents at each time step is optimized to obtain high-level intention information corresponding to each time step, including:

[0022] The historical state data of the plurality of agents is encoded to map a structural triple in the historical state data to a latent space, to obtain a latent representation sequence and high-level intention information corresponding to each latent representation in the latent representation sequence;

[0023] A dynamic transition model is constructed, which is configured to learn a state transition function in the latent space to predict a latent representation and a reward parameter at a next time based on a current latent representation and high-level intention information of the agent;

[0024] A dataset is constructed based on the latent representation sequence and the high-level intention information, and the dynamic transition model is trained based on the dataset to obtain a world model;

[0025] A hierarchical policy network is constructed, including a Critic network and an Actor network;

[0026] The hierarchical policy network is optimized based on an output result of the world model to obtain an optimized policy network;

[0027] The cooperation strategy of the plurality of agents at each time step is optimized based on the optimized policy network to obtain high-level intention information corresponding to each time step.

[0028] Optionally, the hierarchical policy network is optimized based on an output result of the world model to obtain an optimized policy network, including:

[0029] A composite reward is generated based on a composite reward function, and the composite reward is processed based on a virtual trajectory output by the world model to obtain a discounted cumulative reward, the composite reward function referring to the following formula:

[0030]

[0031] wherein, represents a composite reward value of a time step, represents a task matching reward, represents a spatiotemporal distribution balance reward, represents an action activation sparsity reward, and respectively represent weight coefficients;

[0032] The hierarchical policy network is optimized based on the discounted cumulative reward to obtain an optimized policy network, the optimized policy network referring to the following formula:

[0033]

[0034] wherein, denotes the optimized policy, denotes the policy network parameters, denotes the parameter optimization operation, denotes the expectation on the policy generated trajectory, denotes the time step index, denotes the discount factor, denotes the trajectory termination time step.

[0035] Optionally, before the structure constraint sequence is input into the pre-constructed target rendering model for processing to obtain a latent video sequence, the method further includes:

[0036] adding Gaussian noise to the latent representation to obtain a noisy latent representation;

[0037] taking a denoising UNet network of a latent diffusion model as a skeleton, and generating an original rendering model based on a ControlNet architecture;

[0038] generating a semantic condition based on a state triple and a language feature of a video frame, and generating a structure condition based on the structure constraint sequence;

[0039] inputting the noisy latent representation, the semantic condition and the structure condition into the original rendering model to train a denoising network of the original rendering model to obtain a target rendering model, and the training process of the denoising network refers to the following formula:

[0040]

[0041] wherein, denotes the predicted noise, denotes the denoising network, denotes the noisy latent representation, denotes the time step, denotes the semantic condition, denotes the structure condition, denotes the temporal consistency condition, which is generated by a temporal consistency module embedded in a UNet structure of the original rendering model;

[0042] the loss function of the denoising network includes:

[0043]

[0044] wherein, denotes the minimized loss function of the denoising network, denotes the mathematical expectation, denotes the real noise, represents a condition set, the condition set comprising a semantic condition, a structure condition and a timing consistency condition, represents a latent representation.

[0045] Optionally, the decoding of the latent video sequence to obtain the target character video comprises:

[0046] decoding the latent video sequence by a decoder of a pre-trained variational autoencoder to obtain a candidate video frame sequence;

[0047] weighting and fusing the candidate video frame sequence based on the contribution degree of each agent to obtain a target video frame sequence;

[0048] applying timing filtering to the target video frame sequence;

[0049] globally correcting the target video frame sequence after the timing filtering based on a global correction strategy to obtain the target character video, the global correction strategy comprising character ID correction and background stabilization correction.

[0050] In addition, to achieve the above-mentioned purpose, the application further provides a multi-agent character video generation device based on a latent diffusion model, comprising:

[0051] a data processing module configured to process original video data to obtain candidate input data, the candidate input data comprising a multi-modal input set and multi-modal features, the multi-modal input set comprising a latent representation, a voice signal and text description information, and the multi-modal features comprising visual features, emotional features and language features;

[0052] a multi-agent behavior structuring module configured to generate a state triple corresponding to each time step of a multi-agent based on the multi-modal features, the agent corresponding to a virtual character, and the state triple comprising a fused emotional state, a task target and an interaction state;

[0053] an intention decomposition module configured to input the state triple into a pre-constructed hierarchical intention decomposition model for intention decomposition to output a structure constraint sequence, the hierarchical intention decomposition model being constructed based on a Transformer architecture and a language model planner;

[0054] a rendering module configured to input the structure constraint sequence into a pre-constructed target rendering model for processing to obtain a latent video sequence, the target rendering model taking a denoising UNet network of a latent diffusion model as a backbone and introducing a ControlNet control branch to realize structure condition control, and the target rendering model being obtained by training based on the multi-modal input set and the multi-modal features;

[0055] a video decoding module configured to decode the latent video sequence to obtain a target character video.

[0056] In addition, to achieve the above object, the present application also provides a multi-agent character video generation device based on a latent diffusion model, comprising a memory, a processor and a multi-agent character video generation program based on a latent diffusion model stored in the memory, wherein the processor is configured to run the multi-agent character video generation program based on a latent diffusion model, and the computer program is configured to implement the steps of the multi-agent character video generation method based on a latent diffusion model as described above.

[0057] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-agent character video generation method based on a latent diffusion model as described above.

[0058] In addition, to achieve the above object, the present application also provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of the multi-agent character video generation method based on a latent diffusion model as described above.

[0059] The present application processes the original video data to obtain candidate input data, which includes a multi-modal input set and multi-modal features, the multi-modal input set includes latent representation, speech signal and text description information, and the multi-modal features include visual features, emotional features and language features. Based on the multi-modal features, a state triple corresponding to each time step of the multi-agent is generated, the agent corresponds to a virtual character, and the state triple includes a fusion emotional state, a task target and an interaction state. The state triple is input into a pre-constructed hierarchical intention decomposition model for intention decomposition, and a structure constraint sequence is output. The hierarchical intention decomposition model is constructed based on a Transformer architecture and a language model planner. The structure constraint sequence is input into a pre-constructed target rendering model for processing to obtain a latent video sequence. The target rendering model takes a denoising UNet network of a latent diffusion model as a backbone and introduces a ControlNet control branch to realize structure condition control. The target rendering model is trained based on the multi-modal input set and the multi-modal features. The latent video sequence is decoded to obtain a target character video. Since the present application optimizes the collaboration strategy of the multi-agent through reinforcement learning and generates high-quality character videos combined with the diffusion model, the behavior consistency, emotional expression and interaction coordination of the multi-agent character video are effectively improved, and the spatio-temporal consistency and natural fidelity of the multi-character video are greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0061] Figure 1 is the structural schematic diagram of the multi-agent character video generation device based on the latent diffusion model of the hardware running environment related to the embodiment of the present application.

[0062] Figure 2 is the flowchart of the first embodiment of the multi-agent character video generation method based on the latent diffusion model of the present application.

[0063] Figure 3 is the flowchart of the second embodiment of the multi-agent character video generation method based on the latent diffusion model of the present application.

[0064] Figure 4 is the flowchart of the third embodiment of the multi-agent character video generation method based on the latent diffusion model of the present application.

[0065] Figure 5 is the structural block diagram of the first embodiment of the multi-agent character video generation device based on the latent diffusion model of the present application.

[0066] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0068] Reference Figure 1 , Figure 1 is the structural schematic diagram of the multi-agent character video generation device based on the latent diffusion model of the hardware running environment related to the embodiment of the present application.

[0069] As Figure 1As shown, the multi-agent character video generation device based on the latent diffusion model can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection communication between the components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and an optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.

[0070] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the multi-agent character video generation device based on the latent diffusion model, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0071] As Figure 1 As shown, the memory 1005 as a computer readable storage medium can include an operating system, a network communication module, a user interface module, and a multi-agent character video generation program based on the latent diffusion model.

[0072] In Figure 1 In the multi-agent character video generation device based on the latent diffusion model, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the multi-agent character video generation device based on the latent diffusion model can be arranged in the multi-agent character video generation device based on the latent diffusion model. The multi-agent character video generation device based on the latent diffusion model calls the multi-agent character video generation program based on the latent diffusion model stored in the memory 1005 through the processor 1001, and executes the multi-agent character video generation method based on the latent diffusion model provided by the embodiment of the application.

[0073] The embodiment of the application provides a multi-agent character video generation method based on a latent diffusion model, which is described with reference to Figure 2 , Figure 2A flowchart of a first embodiment of the multi-agent character video generation method based on a latent diffusion model of the present application is shown.

[0074] In this embodiment, the multi-agent character video generation method based on a latent diffusion model comprises the following steps:

[0075] Step S10: Process the original video data to obtain candidate input data.

[0076] It should be noted that the present embodiment is applied to multi-character video generation, specifically by using reinforcement learning to improve the multi-agent collaboration ability, and combining a diffusion model to generate multi-character videos with consistent structure. Reinforcement learning can optimize the behavior strategy of the agent through a reward mechanism, and enable it to collaborate in a dynamic environment. However, most existing multi-agent reinforcement learning algorithms are applied to control and game scenarios, and are difficult to apply to multi-agent character video generation in complex interactive scenarios, and are difficult to achieve high-fidelity and high-consistency natural generation and interactive performance in video generation tasks. Therefore, the current multi-agent character video generation in complex interactive scenarios faces the following challenges:

[0077] 1. Multi-agent interaction problem: how to establish a reasonable and natural interaction relationship between multiple virtual characters, and ensure that the behavior of each agent is consistent with others;

[0078] 2. Emotional consistency problem: how to ensure the consistency of emotional expression, language and behavior of the characters in the generation process;

[0079] 3. Generation quality problem: how to generate high-quality, multi-character natural videos through appropriate models and optimization strategies.

[0080] To solve these problems, the present embodiment proposes a generation framework combining multi-agent reinforcement learning and diffusion model. This framework models the state of multi-agent, task allocation and collaborative learning, so that the virtual characters corresponding to the multi-agent can achieve natural, continuous and coordinated performance in video generation. Specifically, reinforcement learning is used to optimize the collaboration strategy of multi-agent, and a diffusion model is used for high-quality character video generation, thereby effectively solving the problems of behavior consistency, emotional expression and interaction coordination in the prior art, enhancing the adaptive ability of multi-agent in interactive scenarios, and generating more natural and expressive virtual character videos.

[0081] It should be understood that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a terminal electronic device capable of realizing the above functions. The following takes a multi-agent character video generation device based on a latent diffusion model (referred to as a video generation device) as an example to illustrate the embodiment and the following embodiments.

[0082] It should be noted that the candidate input data includes a multi-modal input set and multi-modal features, the multi-modal input set includes latent representations, speech signals and text description information, and the multi-modal features include visual features, emotional features and language features.

[0083] It should be noted that in the character video generation task, the input video frame often contains multiple characters, complex background and light difference. In order to ensure that the model can learn stable and generalizable spatio-temporal features, the embodiment maps the video, speech and text to a unified learnable representation by standardizing and feature extracting the original video data and mapping it to a low-dimensional latent space.

[0084] Step S20: generating a state triple corresponding to each time step of the multi-agent based on the multi-modal features.

[0085] It should be noted that the agent corresponds to a virtual character, and the state triple includes a fusion emotional state, a task target and an interaction state.

[0086] It can be understood that the video generation device regards each virtual character as an independent agent, and each agent performs different generation tasks (such as action, expression or language) according to the scene state, realizing the structured modeling of complex group interaction.

[0087] In specific implementation, in order to realize the behavior modeling of the agent, the present application defines the state of each agent at each time step as a state triple:

[0088]

[0089] Among them, is a fusion emotional state, is a task target, is interaction information (i.e. interaction state).

[0090] Step S30: inputting the state triple into a pre-constructed hierarchical intention decomposition model for intention decomposition, and outputting a structure constraint sequence.

[0091] It should be noted that the hierarchical intention decomposition model is constructed based on a Transformer architecture and a language model planner.​

[0092] It is understood that this embodiment decomposes the abstract intent output by the agent into the precise structural constraints required by the model by using a pre-built hierarchical intent decomposition model.

[0093] In the specific implementation, given a state triple Next, the agent needs to generate video content. This embodiment employs a hierarchical intent decomposition system to transform abstract intents into precise physical constraints. A large language model planner (LLM_Planner) based on the Transformer architecture is introduced as a task decomposer from high-level intents to low-level constraints. This planner receives the state... And break it down into a future-oriented A frame-specific, precise, executable sequence of structural constraints. :

[0094]

[0095] in, Each element in Both are high-density structural constraints designed to provide fine geometric information to achieve sub-pixel level precise attitude control. Indicates the planning step size.

[0096] The structural constraint sequence will serve as a strong conditional and precise control signal input to the potential diffusion model based on the ControlNet architecture, ensuring that the generated video frames achieve high fidelity and accuracy in pose, structure, and shape.

[0097] Step S40: Input the structural constraint sequence into the pre-built target rendering model for processing to obtain the potential video sequence.

[0098] It should be noted that the target rendering model uses the denoising UNet network of the latent diffusion model as the backbone and introduces the ControlNet control branch to realize structural condition control. The target rendering model is trained based on the multimodal input set and multimodal features.

[0099] It is understood that this embodiment constructs a target rendering model based on the ControlNet architecture and a latent diffusion model (LDM) denoising network to achieve high-fidelity and precise structural control in core generation. The target rendering model operates in an efficient latent space and is built based on the LDM framework. Its main objective is to utilize the ControlNet architecture to constrain the generated high-density structural sequences. Perform precise rendering.

[0100] Further, in order to improve the coherence of the video in the time axis and improve the rendering performance, in an embodiment, before the step S40, the method can further include:

[0101] Step S41: Gaussian noise is added to the latent representation to obtain a noisy latent representation;

[0102] Step S42: a denoising UNet network of a latent diffusion model is taken as a skeleton, and a ControlNet architecture is generated based on the original rendering model;

[0103] Step S43: a semantic condition is generated based on the state triple and the language features of the video frame, and a structure condition is generated based on the structure constraint sequence;

[0104] Step S44: the noisy latent representation, the semantic condition and the structure condition are input into the original rendering model to train the denoising network of the original rendering model, and a target rendering model is obtained.

[0105] It should be noted that the generation process of the target rendering model starts from the forward diffusion, and the model generation process is as follows:

[0106] First, the obtained video frame latent representation is gradually added with Gaussian noise . This forward process is a Markov chain, which defines the sampling formula of the noisy latent representation at any step :

[0107]

[0108] wherein, , is a predefined noise scheduling coefficient.

[0109] In this embodiment, the denoising UNet network of the pre-trained LDM is taken as a skeleton, and the ControlNet architecture is integrated. The ControlNet is a zero-initialized trainable encoder branch, which specially receives a high-density structure constraint sequence (as a structure condition ), and through a spatial feature connection mechanism, accurate structure features are injected into each layer.

[0110] Multiple conditioning and temporal consistency reverse denoising: in the inference and training phase, the goal of the reverse denoising process is to gradually recover the clear latent representation of the target video frame from the noisy . In order to ensure the high fidelity and multi-modal consistency of the generated results, the denoising network Three types of key conditions must be received simultaneously Guidance:

[0111]

[0112] Among them, the semantic condition The agent state And the language vector are injected through the cross-attention mechanism; the structural condition The high-density constraint is injected through the ControlNet branch; represents the predicted noise, represents the denoising network, represents the noisy latent representation, represents the time step, represents the temporal consistency condition, which is generated by the temporal consistency module embedded in the UNet structure of the original rendering model.

[0113] To solve the common motion jitter and picture flicker problems in video generation, the invention particularly embeds a temporal consistency module in the UNet structure. This module contains a temporal self-attention layer that aggregates latent features along the time axis to ensure motion coherence, and a cross-frame attention layer that maintains the appearance consistency of the generated characters. The training objective of the denoising network is to minimize the error between the predicted noise and the real noise , and the minimization loss function is:

[0114]

[0115] where, represents the minimization loss function of the denoising network, represents the mathematical expectation, represents the real noise, represents the condition set, which includes the semantic condition, the structural condition and the temporal consistency condition, represents the latent representation.

[0116] Backward iterative sampling and latent video sequence output: in the inference stage, the denoising process is performed through multiple iterations to recover the final latent video sequence from pure noise. This iterative process is strongly constrained by the ControlNet and coordinated by the temporal consistency module, ensuring that the generated latent representation not only conforms to the abstract state of the agent, but also accurately follows the structural guidance of the high-density constraint at the pixel level. The final output is The sequence will be sent to the VAE decoder for pixel space reconstruction.

[0117] Step S50: Decode the potential video sequence to obtain the target person video.

[0118] Understandably, after undergoing the ControlNet conditional denoising process, the system obtains a final, high-fidelity, and structurally precisely controlled potential video sequence from the structurally constrained sequence. The latent video sequence resides in a low-dimensional latent space and must be decoded to be restored to video frames viewable by the human eye. This embodiment decodes the latent video sequence, restoring it from the low-dimensional latent space to a high-dimensional pixel space, thus obtaining the final video of the target person.

[0119] Furthermore, to ensure the visual quality and temporal consistency of the final output, in one embodiment, step S50 may include:

[0120] Step S501: Decode the potential video sequence using a pre-trained variational autoencoder decoder to obtain a candidate video frame sequence;

[0121] Step S502: The candidate video frame sequence is weighted and fused based on the contribution of each agent to obtain the target video frame sequence;

[0122] Step S503: Apply time-series filtering to the target video frame sequence;

[0123] Step S504: Perform global consistency correction on the target video frame sequence after applying temporal filtering based on the global correction strategy to obtain the target person video. The global correction strategy includes person ID correction and background stabilization correction.

[0124] It should be noted that this embodiment can use a decoder based on a pre-trained variational autoencoder. For potential video sequences Decode it from the low-dimensional latent space Restored to high-dimensional pixel space The final video frame sequence is obtained. .

[0125] In multi-agent scenarios, each agent may be responsible for generating different elements in the video. Therefore, the decoded video frame sequence needs to be weighted and fused to integrate the contributions of different agents or modules to obtain the final video output. :

[0126]

[0127] in, For the first An agent at a time step decoded output frame, as a weighted fusion coefficient.

[0128] It can be understood that although the target rendering model has introduced a timing consistency module, after pixel space reconstruction and fusion, there may still be slight high-frequency artifacts or texture flicker. Therefore, the present application carries out strict post-processing before the final output. The video sequence after fusion Temporal filtering is applied to ensure the visual coherence and motion naturalness of the video sequence in time.

[0129] Finally, a global consistency correction module is applied, which pays special attention to the consistency of the character ID and the stability of the background, ensuring that the character identity and scene appearance do not drift within a long time sequence. Through the above accurate reconstruction, fusion and post-processing, the final output target character video has high fidelity, precise controllability and excellent spatio-temporal coherence.

[0130] The embodiment obtains candidate input data by processing the original video data, the candidate input data including a multi-modal input set and multi-modal features, the multi-modal input set including latent representations, speech signals and text description information, and the multi-modal features including visual features, emotional features and language features. Based on the multi-modal features, a state triple corresponding to each time step of the multi-agent is generated, the agent corresponding to a virtual character, and the state triple including a fused emotional state, a task target and an interaction state. The state triple is input into a pre-constructed hierarchical intent decomposition model for intent decomposition, and a structure constraint sequence is output. The hierarchical intent decomposition model is constructed based on a Transformer architecture and a language model planner. The structure constraint sequence is input into a pre-constructed target rendering model for processing to obtain a latent video sequence. The target rendering model takes a denoising UNet network of a latent diffusion model as a backbone and introduces a ControlNet control branch to realize structure condition control. The target rendering model is trained based on the multi-modal input set and the multi-modal features. The latent video sequence is decoded to obtain a target character video. Since the embodiment optimizes the collaboration strategy of the multi-agent through reinforcement learning and generates high-quality character videos in combination with the diffusion model, the behavior consistency, emotional expression and interaction coordination of the multi-agent character video are effectively improved, thereby greatly improving the spatio-temporal consistency and natural fidelity of the multi-character video.

[0131] Reference Figure 3 , Figure 3 is a flowchart of the second embodiment of the present application for generating a multi-agent character video based on a latent diffusion model.

[0132] Based on the first embodiment described above, in this embodiment, step S10 further includes:

[0133] Step S101: Extract frame sequences from the original video data and preprocess the frame sequences, including cropping, scaling, brightness and color normalization.

[0134] In the specific implementation, frame sequences are extracted from the original video. The system identifies key figures using human detection and pose estimation algorithms, then adaptively crops and scales the images to maintain consistent scale and pose across frames. Finally, it normalizes the brightness and color of the cropped video frames.

[0135]

[0136] in, and These represent the image mean and standard deviation, respectively.

[0137] Step S102: Map the preprocessed frame sequence to the latent space to obtain the latent representation.

[0138] Understandably, to address the issue of the massive computational cost of the pixel spatial diffusion model, this embodiment employs a pre-trained variational autoencoder. Standardized pixel frames Mapping to a low-dimensional latent space To obtain potential characterization At the same time, for The sequence is processed by temporal smoothing to make adjacent frames more consistent in color and texture, thereby reducing the impact of temporal jitter and inter-frame discontinuity on the training of the generative model.

[0139] Step S103: Time-align the latent representation with the speech signal and text description information in the original video data to obtain a multimodal input set.

[0140] It should be understood that, in the time dimension, this embodiment will standardize the potential representation of video frames. Synchronized voice signal and text description Perform time alignment to form a unified multimodal input set:

[0141]

[0142] After the above processing, a standardized video latent representation sequence with balanced illumination, smooth temporal sequence, clear subject and multimodal synchronization is obtained. This sequence provides a unified and high-quality input foundation for subsequent visual feature extraction, emotion recognition and language modeling.

[0143] Step S104: Extract visual features, emotional features and language features of each frame based on the multi-modal input set, and generate multi-modal features.

[0144] It should be noted that the embodiment uses a pre-trained VisionTransformer (ViT) to extract high-level visual semantic features of each frame on the basis of potential representation

[0145]

[0146] Among them, represents the visual representation corresponding to the i-th frame, which contains information such as human posture, expression, background and action, and is used for subsequent agent state modeling. It can be understood that the behavior of the person in the video is often driven by emotion and language. Therefore, in addition to visual features, additional emotional features and language features are introduced. The extraction of emotional and language features includes:

[0147] Emotional vector

[0148] : extracted by a facial expression recognition model or a speech emotion recognition model, representing the instantaneous emotional features of the person at time step t:

[0149] Language vector

[0150] : encoded by a pre-trained language model (such as BERT or GPT) on the text segment, representing the semantic content and tone features of the person at that moment:

[0151] Finally, the visual features , the emotional vector

[0152] and the language vector together serve as multi-modal input, providing a feature basis for subsequent agent state construction:

[0153]

[0154] ​​​​The embodiment extracts a frame sequence from original video data, and pre-processes the frame sequence, the pre-processing including cropping, scaling, brightness and color normalization processing, maps the pre-processed frame sequence to a latent space to obtain a latent representation, time-aligns the latent representation with a voice signal and text description information in the original video data to obtain a multi-modal input set, extracts visual features, emotional features and language features of each frame based on the multi-modal input set, and generates multi-modal features, so as to guarantee data standardization and consistency, integrate three core features of vision, emotion and language, generate a comprehensive and closely related multi-modal feature set, improve the emotional expression performance of multi-agent interaction, and improve the rationality of interaction behavior.

[0155] Reference Figure 4 , Figure 4 The flowchart of the third embodiment of the multi-agent character video generation method based on the latent diffusion model of the application is shown.

[0156] Based on the above embodiments, in the embodiment, the task target includes high-level intention information of the agent, and the step S20 further includes:

[0157] Step S201: performing feature fusion based on the multi-modal features to generate fusion emotional features corresponding to each time step of the multi-agent.

[0158] It should be noted that the embodiment can perform feature fusion on the multi-modal features through a feature fusion function or a mapping function:

[0159]

[0160] Among them, represents a feature fusion or mapping function, represents multi-modal features, represents fusion emotional features.

[0161] Step S202: optimizing the cooperation strategy of the multi-agent at each time step to obtain high-level intention information corresponding to each time step.

[0162] It should be noted that the task target in the state triple in the embodiment represents the high-level intention of the current agent (for example, “walking”, “smiling”, “talking”), the high-level intention can be generated by the MARL strategy module as the input of the large language model planner (LLM_Planner) in the subsequent step.

[0163] Further, in order to optimize the cooperation strategy of the agent, give it the ability of long-term prediction and behavior planning, and ensure the logical coherence of multi-character interaction behavior, in an embodiment, the above step S202 can include:

[0164] Step S2021: encode the historical state data of the multi-agent to map the structural triplets in the historical state data to a latent space, obtain a latent representation sequence and high-level intention information corresponding to each latent representation in the latent representation sequence;

[0165] Step S2022: construct a dynamic transition model configured to learn a state transition function in the latent space to predict a latent representation at a next time and a reward parameter based on a current latent representation and high-level intention information of the agent;

[0166] Step S2023: construct a dataset based on the latent representation sequence and the high-level intention information, and train the dynamic transition model based on the dataset to obtain a world model;

[0167] Step S2024: construct a hierarchical policy network including a Critic network and an Actor network;

[0168] Step S2025: optimize the hierarchical policy network based on an output result of the world model to obtain an optimized policy network;

[0169] Step S2026: optimize a collaboration policy of the multi-agent at each time step based on the optimized policy network to obtain high-level intention information corresponding to each time step.

[0170] It should be noted that the embodiment is based on a MARL long-time sequence policy optimization of a world model (World Models). By using a hierarchical multi-agent reinforcement learning (HMARL) framework, the collaboration policy of the agent is optimized in combination with the world model (World Models), and the agent is endowed with long-term prediction and behavior planning capabilities to ensure the logical coherence of the multi-person interactive behavior.

[0171] It should be noted that the traditional MARL algorithm is mostly a Model-Free framework, which has the disadvantages of low data efficiency and lack of long-time sequence planning capability. The embodiment upgrades the policy optimization framework to a Model-Based RL, and by constructing a world model, the agent can perform policy rehearsal in the latent space.

[0172] It can be understood that the core component of constructing the world model is a dynamic transition model which learns a state transition function in the latent space to predict a given current latent representation and high-level intention of the agent (as an action ), a latent representation at a next time and a reward :

[0173]

[0174] It should be understood that the embodiment strategy optimization adopts a hierarchical strategy , wherein is a structured state triple, is a high-level intention of the agent . The hierarchical strategy network realizes centralized training and decentralized execution through a shared Critic network and a private Actor network to balance collaboration efficiency and individual differences.

[0175] Further, in order to improve the quality and coordination of multi-agent interactive behavior, in an embodiment, the above step S2025 can include:

[0176] Step S20251: generating a composite reward based on a composite reward function, and processing the composite reward based on the virtual trajectory output by the world model to obtain a discounted cumulative reward;

[0177] Step S20252: optimizing the hierarchical strategy network based on the discounted cumulative reward to obtain an optimized strategy network.

[0178] It should be noted that the goal of strategy optimization is to maximize long-term cumulative reward, and to guide the agent to generate high-quality and coordinated interactive behavior. In this embodiment, a composite reward function is designed, which refers to the following formula:

[0179]

[0180] wherein, represents the composite reward value of the time step, represents the task matching reward, represents the spatiotemporal distribution balance reward, represents the action activation sparsity reward, and respectively represent weight coefficients.

[0181] It can be understood that the PPO strategy optimization algorithm is used to maximize the long-term cumulative reward in the virtual trajectory generated by the world model, thereby realizing the optimization of the strategy network. The optimized strategy network refers to the following formula:

[0182]

[0183] wherein, represents the optimized strategy, represents the strategy network parameter, represents the parameter optimization operation, represents the expectation on the strategy generated trajectory, denotes a time step index, denotes a discount factor, denotes a trajectory termination time step.

[0184] In a specific implementation, in the inference phase of the policy network, to ensure the intention of the agent and the structural constraint sequence of the LLM are fully executed, a decoupled classifier-independent guidance mechanism is adopted, as shown in the following formula:

[0185]

[0186] wherein, denotes a guided output representation, denotes a predicted latent representation of a time step , weights and respectively enhance the guidance strength of the structural constraint and the semantic condition.

[0187] Step S203: constructing a dynamic interaction graph based on the visual features and the dynamic relationships between the agents.

[0188] It should be noted that the nodes in the dynamic interaction graph are configured as the agents in each frame scene, the initial features of the nodes are the visual features, and the edges in the dynamic interaction graph are configured to encode the dynamic relationships between the agents.

[0189] It should be noted that in the state triple, is interaction information, but its representation is relatively fuzzy and difficult to accurately describe complex multi-agent dynamics. To solve this problem, the embodiment introduces a graph neural network to explicitly and structurally model .

[0190] Step S204: performing information propagation and aggregation on the dynamic interaction graph through a pre-constructed graph attention network to obtain the interaction information of the multi-agent at each time step.

[0191] In a specific implementation, the video generation device constructs a dynamic interaction graph at each time step , the nodes are all the agents in the scene, the initial features of the nodes are , and the edges encode the dynamic relationships between the agents. The graph is used to perform information propagation and aggregation using a graph attention network. The agent updates its interaction state by aggregating neighbor information to obtain . Therefore, the state is accurately refined as:

[0192]

[0193] The embodiment is based on the multi-modal feature to perform feature fusion, generate a fusion emotion feature corresponding to each time step of the multi-agent, optimize the cooperation strategy of the multi-agent at each time step, obtain high-level intention information corresponding to each time step, construct a dynamic interaction graph based on a visual feature and a dynamic relationship between the agents, nodes in the dynamic interaction graph are configured as the agents in each frame scene, an initial feature of the nodes is the visual feature, edges in the dynamic interaction graph are configured to encode the dynamic relationship between the agents, information propagation and aggregation of the dynamic interaction graph are performed through a pre-constructed graph attention network, and interaction information corresponding to each time step of the multi-agent is obtained, so that precise, dynamic and multi-dimensional modeling of the multi-agent state is realized, the abstract intention of the agent is converted into precise structural constraints, and the modeling capability and realism of the multi-agent for a complex group interaction scene are greatly improved, so that the behavior consistency of the multi-person video is improved.

[0194] In addition, the embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a multi-agent person video generation program based on a latent diffusion model. The multi-agent person video generation program based on the latent diffusion model is executed by a processor to realize the steps of the multi-agent person video generation method based on the latent diffusion model.

[0195] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0196] The computer readable storage medium can be included in the multi-agent character video generation device based on the latent diffusion model, or can exist independently and not be assembled into the multi-agent character video generation device based on the latent diffusion model.

[0197] In addition, the embodiment of the present application also provides a computer program product, which comprises a multi-agent character video generation program based on a latent diffusion model. When the multi-agent character video generation program based on the latent diffusion model is executed by a processor, the steps of the multi-agent character video generation method based on the latent diffusion model are implemented.

[0198] The computer program product embodiment of the present application is basically the same as the above-mentioned multi-agent character video generation method based on the latent diffusion model, and will not be repeated here.

[0199] Reference Figure 5 , Figure 5 is a structural block diagram of the first embodiment of the multi-agent character video generation device based on the latent diffusion model of the present application.

[0200] As Figure 5 shown, the multi-agent character video generation device based on the latent diffusion model of the embodiment of the present application comprises:

[0201] The data processing module 10 is configured to process the original video data to obtain candidate input data, wherein the candidate input data comprises a multi-modal input set and multi-modal features, the multi-modal input set comprises latent representations, speech signals and text description information, and the multi-modal features comprise visual features, emotional features and language features.

[0202] The multi-agent behavior structuring module 20 is configured to generate a state triple corresponding to each time step of a multi-agent based on the multi-modal features, wherein the agent corresponds to a virtual character, and the state triple comprises a fusion emotional state, a task target and an interaction state.

[0203] The intention decomposition module 30 is configured to input the state triple into a pre-constructed hierarchical intention decomposition model for intention decomposition, and output a structure constraint sequence, wherein the hierarchical intention decomposition model is constructed based on a Transformer architecture and a language model planner.

[0204] The rendering module 40 is configured to input the structure constraint sequence into a pre-constructed target rendering model for processing to obtain a latent video sequence, wherein the target rendering model takes a denoising UNet network of a latent diffusion model as a backbone, and introduces a ControlNet control branch to realize structure condition control, and the target rendering model is trained based on the multi-modal input set and the multi-modal features.

[0205] The video decoding module 50 is configured to decode the potential video sequence to obtain a target character video.

[0206] The embodiment obtains candidate input data by processing original video data, the candidate input data including a multi-modal input set and multi-modal features, the multi-modal input set including potential representation, speech signals and text description information, the multi-modal features including visual features, emotional features and language features, a state triple corresponding to each time step of the multi-agent is generated based on the multi-modal features, the agent corresponds to a virtual character, the state triple includes a fusion emotional state, a task target and an interaction state, the state triple is input into a pre-constructed hierarchical intent decomposition model for intent decomposition, and a structure constraint sequence is output, the hierarchical intent decomposition model is constructed based on a Transformer architecture and a language model planner, the structure constraint sequence is input into a pre-constructed target rendering model for processing, a potential video sequence is obtained, the target rendering model takes a denoising UNet network of a potential diffusion model as a backbone, and a ControlNet control branch is introduced to realize structure condition control, the target rendering model is trained based on the multi-modal input set and the multi-modal features, the potential video sequence is decoded, and a target character video is obtained; since the embodiment optimizes the collaboration strategy of the multi-agent through reinforcement learning, and generates a high-quality character video in combination with a diffusion model, the behavior consistency, emotional expression and interaction coordination of the multi-agent character video are effectively improved, and the spatiotemporal consistency and natural fidelity of the multi-character video are greatly improved.

[0207] The multi-agent character video generation device based on a potential diffusion model provided in the application adopts the multi-agent character video generation method based on a potential diffusion model in the above embodiment, and can solve the technical problem of multi-agent character video generation based on a potential diffusion model. Compared with the prior art, the multi-agent character video generation device based on a potential diffusion model provided in the application has the same beneficial effects as the multi-agent character video generation method based on a potential diffusion model provided in the above embodiment, and other technical features in the multi-agent character video generation device based on a potential diffusion model are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0208] It should be understood that the above is only an example, and the technical solutions of the application do not constitute any limitation, and in specific applications, those skilled in the art can set it up as needed, and the application does not limit it.

[0209] It should be noted that the above-described workflow is merely illustrative and does not limit the scope of protection of the present application. In actual applications, a person skilled in the art can select part or all of the above-described workflow to achieve the purpose of the embodiment according to actual needs, which is not limited herein.

[0210] In addition, technical details not described in detail in the present embodiment can be found in the multi-agent character video generation method based on a latent diffusion model provided by any embodiment of the present application, which will not be described here.

[0211] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or system that includes the element.

[0212] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0213] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk) and includes a number of instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0214] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A multi-agent character video generation method based on a latent diffusion model, characterized in that, The method comprises: processing the original video data to obtain candidate input data, the candidate input data comprising a multimodal input set and multimodal features, the multimodal input set comprising latent representations, speech signals and text description information, and the multimodal features comprising visual features, emotional features and language features; generating state triplets corresponding to each time step of multiple agents based on the multimodal features, the agents corresponding to virtual characters, the state triplets comprising fused emotional states, task goals and interaction states; inputting the state triplets into a pre-constructed hierarchical intent decomposition model for intent decomposition, and outputting a structure constraint sequence, the hierarchical intent decomposition model being constructed based on a Transformer architecture and a language model planner; inputting the structure constraint sequence into a pre-constructed goal rendering model for processing to obtain a latent video sequence, the goal rendering model taking a denoising UNet network of a latent diffusion model as a backbone and introducing a ControlNet control branch to realize structure condition control, the goal rendering model being trained based on the multimodal input set and the multimodal features; decoding the latent video sequence to obtain a target character video.

2. The multi-agent character video generation method based on a latent diffusion model according to claim 1, wherein, The processing of the original video data to obtain the candidate input data comprises: extracting a frame sequence from the original video data and pre-processing the frame sequence, the pre-processing comprising cropping, scaling, brightness and color normalization processing; mapping the pre-processed frame sequence to a latent space to obtain latent representations; time-aligning the latent representations with speech signals and text description information in the original video data to obtain a multimodal input set; extracting visual features, emotional features and language features of each frame based on the multimodal input set to generate multimodal features.

3. The multi-agent character video generation method based on a latent diffusion model according to claim 2, wherein, The task goal comprises high-level intent information of the agents; The generation of the state triplets corresponding to each time step of the multiple agents based on the multimodal features comprises: performing feature fusion based on the multimodal features to generate fused emotional features of the multiple agents corresponding to each time step; optimizing a collaboration strategy of the multiple agents at each time step to obtain high-level intent information corresponding to each time step; constructing a dynamic interaction graph based on visual features and dynamic relationships between the agents, nodes in the dynamic interaction graph being configured as the agents in each frame of scene, initial features of the nodes being the visual features, and edges in the dynamic interaction graph being configured to encode the dynamic relationships between the agents; performing information propagation and aggregation on the dynamic interaction graph through a pre-constructed graph attention network to obtain interaction information of the multiple agents corresponding to each time step.

4. The multi-agent character video generation method based on a latent diffusion model according to claim 3, wherein, The optimization of the collaboration strategy of the multiple agents at each time step to obtain the high-level intent information corresponding to each time step comprises: encoding historical state data of the multiple agents to map structure triplets in the historical state data to a latent space to obtain a latent representation sequence and high-level intent information corresponding to each latent representation in the latent representation sequence; A dynamic transition model is constructed, which is configured to learn a state transition function in a latent space to predict a latent representation and a reward parameter at a next time based on a current latent representation and high-level intention information of an agent; A dataset is constructed based on the latent representation sequence and the high-level intention information, and the dynamic transition model is trained based on the dataset to obtain a world model; A hierarchical policy network is constructed, which includes a Critic network and an Actor network; The hierarchical policy network is optimized based on an output result of the world model to obtain an optimized policy network; The collaborative strategy of the multi-agent at each time step is optimized based on the optimized policy network to obtain the high-level intention information corresponding to each time step.

5. The multi-agent character video generation method based on a latent diffusion model according to claim 4, wherein, The hierarchical policy network is optimized based on an output result of the world model to obtain an optimized policy network, including: A composite reward is generated based on a composite reward function, and the composite reward is processed based on a virtual trajectory output by the world model to obtain a discounted cumulative reward, the composite reward function referring to the following formula: wherein, represents a composite reward value of a time step, represents a task matching reward, represents a spatiotemporal distribution balance reward, represents an action activation sparsity reward, and respectively represent weight coefficients; The hierarchical policy network is optimized based on the discounted cumulative reward to obtain an optimized policy network, the optimized policy network referring to the following formula: wherein, represents the optimized policy, represents the policy network parameters, represents the parameter optimization operation, represents the expectation on the policy generated trajectory, represents the time step index, represents the discount factor, represents the trajectory termination time step.

6. The multi-agent character video generation method based on a latent diffusion model according to any one of claims 1 to 5, wherein, Before the structure constraint sequence is input into a pre-constructed target rendering model for processing to obtain a latent video sequence, the method further includes: The latent representation is added with Gaussian noise to obtain a noisy latent representation; A denoising UNet network of a latent diffusion model is taken as a skeleton, and an original rendering model is generated based on a ControlNet architecture; A semantic condition is generated based on a state triple and a language feature of a video frame, and a structure condition is generated based on a structure constraint sequence; The noisy latent representation, the semantic condition, and the structure condition are input into the original rendering model to train a denoising network of the original rendering model to obtain a target rendering model, a training process of the denoising network referring to the following formula: wherein, denotes a predicted noise, denotes a denoising network, denotes a noisy latent representation, denotes a time step, denotes a semantic condition, denotes a structural condition, denotes a temporal consistency condition, the temporal consistency condition being generated by a temporal consistency module embedded in a UNet structure of the original rendering model; The loss function of the denoising network includes: wherein, denotes a minimization loss function of the denoising network, denotes a mathematical expectation, denotes a real noise, denotes a condition set, the condition set comprising a semantic condition, a structural condition, and a temporal consistency condition, denotes a latent representation.

7. The multi-agent character video generation method based on a latent diffusion model according to any one of claims 1 to 5, wherein, The latent video sequence is decoded to obtain a target character video, including: The latent video sequence is decoded by a decoder of a pre-trained variational autoencoder to obtain a candidate video frame sequence; The candidate video frame sequence is weighted and fused based on a contribution degree of each agent to obtain a target video frame sequence; Temporal filtering is applied to the target video frame sequence; The target video frame sequence subjected to the temporal filtering is globally consistent modified based on a global modification strategy to obtain a target character video, the global modification strategy including character ID modification and background stability modification.

8. An apparatus for generating multi-agent character videos based on a latent diffusion model, comprising: The multi-agent character video generation device based on the latent diffusion model includes: A data processing module is configured to process original video data to obtain candidate input data, the candidate input data including a multi-modal input set and multi-modal features, the multi-modal input set including a latent representation, a voice signal, and text description information, and the multi-modal features including visual features, emotional features, and language features; The multi-agent behavior structuring module is configured to generate a state triple corresponding to a multi-agent at each time step based on the multi-modal feature, the agent corresponding to a virtual character, and the state triple including a fused emotional state, a task goal, and an interaction state. The intention decomposition module is configured to input the state triple into a pre-built hierarchical intention decomposition model for intention decomposition, and output a structure constraint sequence, the hierarchical intention decomposition model being built based on a Transformer architecture and a language model planner. The rendering module is configured to input the structure constraint sequence into a pre-built target rendering model for processing, and obtain a latent video sequence, the target rendering model taking a denoising UNet network of a latent diffusion model as a backbone, and introducing a ControlNet control branch to realize structure condition control, the target rendering model being trained based on a multi-modal input set and multi-modal features. The video decoding module is configured to decode the latent video sequence, and obtain a target character video.

9. A multi-agent character video generation device based on a latent diffusion model, characterized by, The multi-agent character video generation device based on a latent diffusion model includes a memory, a processor, and a multi-agent character video generation program based on a latent diffusion model stored in the memory, the processor being configured to run the multi-agent character video generation program based on a latent diffusion model, and the multi-agent character video generation program based on a latent diffusion model being configured to implement the multi-agent character video generation method based on a latent diffusion model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a multi-agent character video generation program based on a latent diffusion model, and the multi-agent character video generation program based on a latent diffusion model is executed by the processor to implement the multi-agent character video generation method based on a latent diffusion model according to any one of claims 1 to 7.

Images