Method for generating a listener video based on a multi-modal unified embedding space guidance

Abstract

The application discloses a method for generating a listener video based on a multi-modal unified embedding space guide, which comprises the following steps: obtaining a first real video of a speaker and extracting corresponding video features and audio features as initiator user features; performing emotion prediction on the initiator user features by using a pre-trained emotion prediction model to obtain receiver emotion features of a listener; fusing the initiator user features and the receiver emotion features by using a modal self-adaptive hybrid expert encoder to obtain fused features; generating corresponding action data of the listener based on the fused features by using a pre-trained action generation model based on a cross-attention layer; and rendering the action data to generate a head video corresponding to the listener. The method focuses on generating core action data instead of directly generating original pixel videos, and the subsequent rendering step is relatively independent, which reduces the calculation complexity of real-time generation and improves the real-time response potential.

Description

Technical Field

[0001] This application relates to the field of video processing, specifically to a method for generating listener videos based on multimodal unified embedding space guidance. Background Technology

[0002] A virtual digital human, or simply a digital human, is a digital virtual character created using computer technology (such as 3D modeling, artificial intelligence, animation-driven technology, etc.) that has an anthropomorphic appearance and behavior. It is a virtual entity that exists in a virtual environment and is designed to simulate human appearance, expressions, movements, and even thought patterns.

[0003] In a real-world scenario, one party (referred to as the speaker for ease of description) initiates an interactive action such as speaking, and the other party (referred to as the listener for ease of description) may respond based on that interactive action through actions, facial expressions, etc.

[0004] When simulating the real environment in a virtual environment, it is necessary to generate a digital human head video of the listener. By simulating the listener's expressions, movements, etc., the listener's state in the real scene can be reflected.

[0005] In traditional solutions, when generating a video of the listener's head in a virtual environment, a representation method based on three-dimensional deformable face parameters can be used, along with a direct end-to-end model or a diffusion model, to address the listener's response actions. However, this solution involves a computationally intensive process, and the model struggles to provide real-time feedback to the speaker and to explicitly model the listener's dynamic emotional shifts. Summary of the Invention

[0006] To address the aforementioned issues, this application proposes a listener video generation method based on multimodal unified embedding space guidance, comprising:

[0007] Obtain the speaker's first real video and extract the corresponding video and audio features as the initiating user features;

[0008] By using a pre-trained sentiment prediction model, sentiment prediction is performed based on the characteristics of the initiating user to obtain the sentiment characteristics received by the listener.

[0009] The initiating user features and the received sentiment features are fused within a unified embedding space using a modality-adaptive hybrid expert encoder to obtain fused features.

[0010] The listener's action data is generated by a pre-trained action generation model based on a cross-attention layer and according to the fusion features.

[0011] The motion data is rendered to generate a head video corresponding to the listener.

[0012] In one example, the speaker's first real video is obtained, and the corresponding video and audio features are extracted as the initiating user features, specifically including:

[0013] Obtain the speaker's first authentic video;

[0014] Obtain each first frame image of the first real video, and for each first frame image, extract the two-dimensional coordinates of the corresponding facial key points to construct the corresponding video feature sequence;

[0015] The audio of each frame of the first real video is obtained, and for each frame audio, the corresponding Mel frequency cepstral coefficients, zero crossover rate and root mean square energy are extracted to construct the corresponding audio feature sequence.

[0016] In one example, a pre-trained sentiment prediction model is used to predict sentiment based on the characteristics of the initiating user, thereby obtaining the listener's received sentiment characteristics, specifically including:

[0017] The video features are input into a video sentiment prediction model based on Transformer coding layers, and the first predicted probability of the sentiment category is output.

[0018] The audio features are input into an audio sentiment prediction model based on the Whisper model, and the second predicted probability of the sentiment category is output.

[0019] Based on the first prediction probability and the second prediction probability, the listener's emotional characteristics are obtained.

[0020] In one example, the first constraint is defined in advance using the real emotions of listeners in the training samples as supervision, and the emotion prediction model is trained by minimizing the first constraint.

[0021] In one example, during the training of the sentiment prediction model using the first constraint, the loss function is: ;in, For the number of emotion categories, A unique and popular tag for the category of genuine emotions. To predict the probability of sentiment category.

[0022] In one example, a temporal motion sequence based on a 3D deformable face representation is extracted from a second real video of the listener in advance. The temporal motion sequence of the listener in the training sample is used as a supervision definition for a second constraint. The modality adaptive hybrid expert encoder and the action generation model are trained by minimizing the second constraint.

[0023] In one example, the training process of the modality adaptive hybrid expert encoder by means of the second constraint includes:

[0024] The user features are randomly masked to mask at least some of the video features and / or at least some of the audio features;

[0025] The masked user features are input into a modality-adaptive hybrid expert encoder to map the masked user features to a unified embedding space.

[0026] The received emotion features obtained by encoding the emotion category through the embedding module are concatenated with the user features after masking in the unified embedding space to obtain the fused features;

[0027] The fused features are input into the action generation model, and the modality adaptive hybrid expert encoder is trained using the second constraint.

[0028] In one example, during the training of the modality adaptive hybrid expert encoder and the action generation model using the second constraint, the loss function is: ;in, These are weight parameters;

[0029] ;in, The actual motion parameters for frame t. Let be the predicted motion parameters for frame t. The motion change function between adjacent frames. The weights are for the smoothing term, and T is the number of frames in the temporal motion sequence.

[0030] C represents the number of emotion categories, and T represents the number of frames in the temporal motion sequence. For frame t, the unique hot label of the sentiment category of the i-th class. The predicted probability of the sentiment category of the i-th class when the frame is t;

[0031] ; For the emotion classification model based on motion parameters, the model parameters of the modal adaptive hybrid expert encoder, the embedding module, and the action generation model are updated by minimizing the loss function.

[0032] In one example, a temporal motion sequence based on a 3D deformable face representation is extracted from a second real video of the listener, specifically including:

[0033] Obtain the second real video of the listener, and obtain the second frame image of each of the second real video;

[0034] For the second frame image, principal component analysis is used to obtain the corresponding facial expression features, pose features, camera rotation features, and camera translation features, which are then used as motion parameters for the second frame image.

[0035] By combining the motion parameters in chronological order, a temporal motion sequence is obtained.

[0036] In one example, rendering is performed based on the motion data to generate a head video corresponding to the listener, specifically including:

[0037] Through 3D reconstruction, the temporal motion sequence corresponding to the motion data is converted into a head mesh;

[0038] Obtain the facial image corresponding to the listener, extract the corresponding Gaussian sphere rendering attributes, and generate multiple corresponding Gaussian spheres;

[0039] Bind the Gaussian sphere to the vertex position of the head mesh;

[0040] The head mesh is refined using the face image to increase the number of Gaussian spheres;

[0041] The head video corresponding to the listener is generated by Gaussian splash rendering and neural network rendering.

[0042] On the other hand, this application also proposes a listener video generation device based on multimodal unified embedded spatial guidance, including:

[0043] At least one processor; and,

[0044] A memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the listener video generation method based on multimodal unified embedding space guidance as described in any of the above examples.

[0046] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions configured to implement the listener video generation method based on multimodal unified embedded space guidance as described in any of the above examples.

[0047] The listener video generation method based on multimodal unified embedding space guidance proposed in this application can bring the following benefits:

[0048] 1. By using a specially trained emotion prediction model, the listener's emotional state is directly predicted and used as input for subsequent generation. This explicitly models and drives the dynamic emotional responses and transitions of the receiving node, overcoming the limitation of traditional solutions that struggle to explicitly control emotions.

[0049] 2. Employing a modality-adaptive hybrid expert encoder to fuse the initiator's multimodal features (including video and audio) with the predicted receiver's emotional features enables more effective capture of complex interaction information. The subsequent action generation model, based on a cross-attention mechanism, focuses on generating temporal motion sequences representing precise action data based on the fused features, avoiding the computational burden of end-to-end or diffusion models.

[0050] 3. The training process is decoupled into two clear objectives: the first constraint corresponds to sentiment prediction, ensuring that the sentiment prediction model accurately predicts the real sentiment of the receiving nodes; the second constraint corresponds to action generation, ensuring that the action generation model can accurately output motion sequences based on 3D reconstruction that conform to real videos. This phased, goal-oriented optimization strategy improves the reliability and controllability of the overall model performance.

[0051] 4. Focusing on generating core motion data (corresponding to temporal motion sequences) rather than directly generating raw pixel video, subsequent rendering steps are relatively independent. This separation design, combined with a lightweight motion generation model, reduces the computational complexity of real-time generation and enhances the real-time response potential compared to end-to-end pixel-level generation models. Attached Figure Description

[0052] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0053] Figure 1 This is a flowchart illustrating the listener video generation method based on multimodal unified embedding space guidance in the embodiments of this application;

[0054] Figure 2 This is a schematic diagram of a listener video generation device based on multimodal unified embedded space guidance in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, this application provides a method for generating listener videos based on multimodal unified embedding space guidance, including:

[0058] S101: Obtain the speaker's first real video and extract the corresponding video and audio features as the initiating user features.

[0059] When the interaction involves speaking, the speaker can be referred to as the speaker (or speaker, speaker, etc.), and the listener can be referred to as the listener (or listener, listener, etc.). When the speaker speaks in a real-world scenario, a camera device captures the corresponding video (referred to here as the first real video) and extracts user characteristics.

[0060] Of course, when the interactive action is a hand or head movement or facial expression, the speaker can also be the initiator of the action or expression. In this case, the listener is the recipient of the action or expression. This article mainly uses interactive actions as an example for explanation.

[0061] Specifically, the first real video of the speaker is acquired, and the first frame images of the first real video are acquired. Here, "first frame image" refers to each frame image in the first real video. "First" is only used to distinguish it from "second frame image" below. The first frame image contains some or all of the frame images in the first real video, which usually contains multiple frame images, and does not only contain the "first frame" image of the first real video.

[0062] For the first frame image, the two-dimensional coordinates of the corresponding facial key points are extracted to construct the corresponding video feature sequence. For example, for each first frame image in the first actual video, a key point detection method is used to obtain all facial key points. Then, 104 key points corresponding to the facial contour, eye contour, and mouth contour are retained. The two-dimensional coordinates of the key points are transformed into 208-dimensional video features. The features of each frame are combined to form a 208-dimensional video feature sequence with the same length as the first actual video.

[0063] Audio from each frame of the first real video is acquired, and for each frame's audio, the corresponding Mel-frequency cepstral coefficients, zero-crossing rate, and root-mean-square energy are extracted to construct the corresponding audio feature sequence. For the audio in the acquired first real video, the audio corresponding to the first frame of each frame is obtained. For each frame's audio, 39-dimensional Mel-frequency cepstral coefficients, 1-dimensional zero-crossing rate, and 1 root-mean-square energy are extracted. After concatenation, the features of each frame are combined to form a speaker audio feature sequence with the same length as the video feature sequence.

[0064] S102: Using a pre-trained sentiment prediction model, sentiment prediction is performed based on the characteristics of the initiating user to obtain the recipient's sentiment characteristics.

[0065] Specifically, video features are input into a video sentiment prediction model based on Transformer coding layers, and the model outputs the first predicted probability of the sentiment category. Sentiment categories can include happiness, sadness, anger, calmness, etc.

[0066] Similarly, audio features are input into an audio sentiment prediction model based on the Whisper model, which outputs a second predicted probability of the sentiment category.

[0067] Based on the first and second predicted probabilities, the listener's emotional characteristics are obtained. For example, the first and second predicted probabilities are weighted and summed to obtain the final predicted probability of the emotional category.

[0068] In this process, the first constraint is defined in advance using the real emotions of listeners in the training samples as supervision, and the emotion prediction model is trained by minimizing the first constraint.

[0069] When training video and audio sentiment prediction models, the training process involves minimizing the first constraint to continuously update the model parameters. The loss function is as follows: ;in, For the number of emotion categories, A unique and popular tag for the category of genuine emotions. To predict the predicted probability of the sentiment category, this loss function constrains the sentiment prediction model to make the predicted probability of the sentiment category as close as possible to the true sentiment label, so that the sentiment prediction model can more accurately predict the listener's sentiment category based on the speaker's information.

[0070] S103: The initiating user features and the received sentiment features are fused in a unified embedding space using a modal adaptive hybrid expert encoder to obtain fused features.

[0071] Traditional solutions typically assume an ideal environment where both the speaker's audio and video can be acquired simultaneously, neglecting the potential modality loss problem in real-world scenarios, which may cause them to fail in real-world applications.

[0072] Based on this, the training process of the modality-adaptive hybrid expert encoder through the second constraint (which is used to train the modality-adaptive hybrid expert encoder and the action generation model mentioned below, and the second constraint will be described in detail below) includes:

[0073] User features are randomly masked to mask at least some video features and / or at least some audio features. During masking, at least a preset length of video or audio features must be retained to simulate potential modality loss in the application scenario. A modality-adaptive hybrid expert encoder is then used to encode the masked speaker information to obtain user features.

[0074] The masked user features are input into a modality-adaptive hybrid expert encoder to map them to a unified embedding space. The received sentiment features, encoded by the sentiment category, are then concatenated with the masked user features in the unified embedding space to obtain the fused features.

[0075] To simulate the potential modality loss problem in real-world application scenarios, the speaker's video or audio features are randomly masked with a certain probability. At this point, the input user features have three states: only audio features, only video features, and both audio and video features. These three types of user features are input into a modality adaptive hybrid expert encoder, which maps all states of user features to a unified embedding space to obtain the final user features. Then, an embedding module is used to encode the listener's predicted sentiment category into the listener's sentiment features. Finally, the speaker's user features and the listener's sentiment features are concatenated to obtain the complete conditional features.

[0076] The fused features are input into the action generation model, and the modality-adaptive hybrid expert encoder is trained using a second constraint. This second constraint is described in detail below.

[0077] S104: Using a pre-trained action generation model, based on a cross-attention layer, generate action data corresponding to the listener according to the fusion features.

[0078] In this process, a temporal motion sequence based on a three-dimensional deformable face representation is extracted from a second real video of the listener. The temporal motion sequence of the listener in the training sample is used as a supervision definition for the second constraint. The modal adaptive hybrid expert encoder and action generation model are trained by minimizing the second constraint.

[0079] A second real video of the listener is acquired in advance, and each second frame of the second real video is acquired. Similar to the first frame, the second frame may also include multiple frames. The "second" in the second frame is only used to distinguish it from the "first" in the first frame above, and is not used to refer to the image of the "second frame".

[0080] For the second frame image, principal component analysis is used to obtain the corresponding facial expression features, pose features, camera rotation features, and camera translation features, which are then used as motion parameters for the second frame image. For example, 100-dimensional facial expression features, 9-dimensional pose features, 3-dimensional camera rotation features, and 3-dimensional camera translation features are obtained from each second frame image of the second real video to obtain the temporal motion sequence of the listener corresponding to the second real video.

[0081] By combining motion parameters in chronological order, a temporal motion sequence is obtained. At this point, the listener's temporal motion sequence has the same length as the speaker's user feature sequence.

[0082] In addition, when setting up the cross-attention layer, the fused features are input into the lightweight action generation model as keys and values, and the temporal motion sequence of the listener corresponding to the first frame image is input into the lightweight action generation model as a query of the cross-attention layer. Through the cross-attention mechanism, the fused features guide the action generation model to generate the listener's action sequence, and guide the model to generate subsequent action sequences.

[0083] During the training of the modal adaptive hybrid expert encoder and the action generation model through the second constraint, the real temporal motion sequence represented by the second real video is used, and the sentiment category of the listener's temporal action sequence obtained by the sentiment prediction model is used as sentiment supervision to reduce the gap between the sentiment category corresponding to the generated temporal action sequence and the real sentiment label, thereby minimizing the second constraint for training the modal adaptive hybrid expert encoder and the action generation model. Of course, it is also possible to choose to train the embedding module together.

[0084] During training, the loss function is: ;in, These are weight parameters;

[0085] ;in, The actual motion parameters for frame t. Let be the predicted motion parameters for frame t. The motion change function between adjacent frames. The weights are for the smoothing term, and T is the number of frames in the temporal motion sequence.

[0086] C represents the number of emotion categories, and T represents the number of frames in the temporal motion sequence. For frame t, the unique hot label of the sentiment category of the i-th class. The predicted probability of the sentiment category of the i-th class when the frame is t;

[0087] ; For a motion parameter-based emotion classification model, the model parameters of the modal adaptive hybrid expert encoder and the embedded module action generation model are updated by minimizing the loss function.

[0088] S105: Render the video based on the motion data to generate a head video corresponding to the listener.

[0089] Specifically, through 3D reconstruction, the temporal motion sequence corresponding to the motion data is converted into a head mesh. This head mesh can be used to describe the 3D representation of the temporal motion sequence in 3D space.

[0090] Obtain the facial image corresponding to the listener. This facial image can be captured from a second real video, or provided additionally by the listener.

[0091] Extract the corresponding Gaussian sphere rendering attributes and generate multiple corresponding Gaussian spheres. The rendering attributes of the Gaussian sphere can include color, transparency, scale, and rotation attributes. The number of Gaussian spheres is fixed and is referred to as the first number.

[0092] Bind the Gaussian sphere to the vertex positions of the head mesh. Generally, the Gaussian sphere and vertex position are bound one-to-one. However, in some special cases (such as in flat regions), there may be multiple vertex positions corresponding to one Gaussian sphere.

[0093] The head mesh is refined by using a face image with the number of Gaussian spheres. The number of Gaussian spheres is the second largest, which is larger than the first, thus enabling a more detailed description of the head mesh.

[0094] By using Gaussian splash rendering and neural network rendering, a head video corresponding to the listener is generated, thus realizing the generation of the listener's head video and providing feedback through a digital human.

[0095] like Figure 2 As shown in the embodiments of this application, a listener video generation device based on multimodal unified embedded spatial guidance is also proposed, including:

[0096] At least one processor; and,

[0097] A memory communicatively connected to the at least one processor; wherein,

[0098] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the listener video generation method based on multimodal unified embedding space guidance as described in any of the above embodiments.

[0099] This application also proposes a non-volatile computer storage medium storing computer-executable instructions configured to implement the listener video generation method based on multimodal unified embedded space guidance as described in any of the above embodiments.

[0100] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0101] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0108] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for speech-driven video generation based on multi-modal unified embedding space guidance, characterized in that, The method comprises: obtaining a first real video of a speaker, and extracting corresponding video features and audio features as initiator user features; performing emotion prediction on the initiator user features by a pre-trained emotion prediction model to obtain receiving emotion features of a listener; fusing the initiator user features and the receiving emotion features based on a unified embedding space by a modal adaptive hybrid expert encoder to obtain fused features; generating action data corresponding to the listener based on the fused features by a pre-trained action generation model based on a cross-attention layer; rendering according to the action data to generate a head video corresponding to the listener; The method further comprises: previously extracting a time sequence of motion sequence based on a three-dimensional deformable face representation from a second real video of the listener, and defining a second constraint by taking the time sequence of motion sequence of the listener in the training sample as supervision, and training the modal adaptive hybrid expert encoder and the action generation model by minimizing the second constraint; the training process of the modal adaptive hybrid expert encoder by the second constraint comprises: randomly masking the user features to mask at least part of the video features and / or at least part of the audio features; inputting the masked user features into the modal adaptive hybrid expert encoder to map the masked user features to the unified embedding space; splicing the receiving emotion features encoded by the emotion category and the masked user features in the unified embedding space by an embedding module to obtain fused features; inputting the fused features into the action generation model to train the modal adaptive hybrid expert encoder by the second constraint.

2. The method of claim 1, wherein, obtaining a first real video of a speaker, and extracting corresponding video features and audio features as initiator user features, specifically comprising: obtaining a first real video of a speaker; obtaining each first frame image of the first real video, and extracting the two-dimensional coordinates of the corresponding facial key points for the first frame image to construct a corresponding video feature sequence; obtaining each frame of audio of the first real video, and extracting the mel-frequency cepstrum coefficient, zero-crossing rate and root mean square energy corresponding to the frame of audio to construct a corresponding audio feature sequence.

3. The method of claim 1, wherein, performing emotion prediction on the initiator user features by a pre-trained emotion prediction model to obtain receiving emotion features of a listener, specifically comprising: inputting the video features into a video emotion prediction model based on a Transformer encoding layer to output first prediction probabilities of emotion categories; inputting the audio features into an audio emotion prediction model based on a Whisper model to output second prediction probabilities of emotion categories; obtaining receiving emotion features of a listener according to the first prediction probabilities and the second prediction probabilities.

4. The method of claim 1, wherein, The method further comprises: previously defining a first constraint by taking the real emotion of the listener in the training sample as supervision, and training the emotion prediction model by minimizing the first constraint.

5. The method of claim 4, wherein, During the training process of the emotion prediction model by the first constraint, a loss function is: ; wherein, is the number of emotion categories, is a one-hot label of a true emotion category, is a predicted probability of a predicted emotion category.

6. The method of claim 1, wherein, In the training process of the modal adaptive hybrid expert encoder and the action generation model by the second constraint, a loss function is: ; wherein, is a weight parameter; ; wherein, is a true motion parameter of a frame, is a predicted motion parameter of a frame, is a neighboring frame motion change function, is a smoothing term weight, T is the number of frames of the temporal motion sequence; ; C is the number of emotion categories, T is the number of frames of the time-series motion sequence, is the one-hot label of the i-th emotion category at frame t, is the predicted probability of the i-th emotion category at frame t. ; For the motion parameter-based emotion classification model, model parameters of the modal adaptive hybrid expert encoder, the embedding module, and the action generation model are updated by minimizing a loss function.

7. The method of claim 1, wherein, The time sequence motion sequence based on the three-dimensional deformable face representation is extracted through the second real video of the listener, and specifically includes: A second real video of a listener is obtained, and each second frame image of the second real video is obtained; For the second frame image, the corresponding expression feature, posture feature, camera rotation feature and camera translation feature are obtained through principal component analysis as the motion parameters of the second frame image; The motion parameters are combined in time sequence to obtain a time sequence motion sequence.

8. The method of claim 1, wherein, According to the action data, a head video corresponding to the listener is generated, specifically including: Through three-dimensional reconstruction, the time sequence motion sequence corresponding to the action data is converted into a head mesh; A face image corresponding to the listener is obtained, and corresponding Gaussian sphere rendering attributes are extracted to generate a plurality of Gaussian spheres; The Gaussian spheres are bound to the vertex positions of the head mesh; The head mesh is refined through the face image to increase the number of Gaussian spheres; The head video corresponding to the listener is generated through Gaussian splashing rendering and neural network rendering.

Images