Listener video generation method based on multi-modal unified embedding space guidance

Abstract

The invention discloses a listener video generation method based on multi-modal unified embedding space guidance, and the method comprises the steps: obtaining a first real video of a speaker, and extracting corresponding video features and audio features as initiating user features; performing emotion prediction according to the characteristics of the initiating user through a pre-trained emotion prediction model to obtain receiving emotion characteristics of the listener; fusing the initiating user features and the receiving emotion features through a modal self-adaptive hybrid expert encoder to obtain fused features; generating motion data corresponding to the listener according to the fusion features based on the cross attention layer through a pre-trained motion generation model; and rendering according to the action data to generate a head video corresponding to the listener. The method focuses on generating core action data instead of directly generating an original pixel video, and subsequent rendering steps are relatively independent, so that the calculation complexity of real-time generation is reduced, and the real-time potential of response is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing, in particular to a method for generating a listener video based on a multi-modal unified embedding space guide. BACKGROUND

[0002] Virtual Digital Human, referred to as Digital Human for short, is a digital virtual character with anthropomorphic appearance and behavior created through computer technology (such as 3D modeling, artificial intelligence, animation driving, etc.), which is a virtual entity existing in a virtual environment and aims to simulate the appearance, expression, action and even thinking mode of a human being.

[0003] In a real environment, one party (for convenience, it is referred to as a speaker) initiates an interaction behavior such as speaking, and the other party (for convenience, it is referred to as a listener) may respond through actions, expressions, etc. based on the interaction behavior.

[0004] When simulating the real environment in a virtual environment, a head video of a Digital Human of the listener needs to be generated to reflect the state of the listener in the real scene through the simulation of the expressions and actions of the listener.

[0005] In a conventional scheme, when generating a head video of a listener in a virtual environment, a representation method based on three-dimensional deformable face parameters can be used, and a direct end-to-end model or a diffusion model can be used to solve the response actions of the listener. However, the calculation process of this scheme is heavy, and the model is difficult to provide feedback for the speaker in real time, and it is also difficult to explicitly model the dynamic emotional changes of the listener. SUMMARY

[0006] To solve the above problems, the present application provides a method for generating a listener video based on a multi-modal unified embedding space guide, comprising: obtaining a first real video of a speaker and extracting corresponding video features and audio features as initiating user features; performing emotion prediction on the initiating user features by a pre-trained emotion prediction model to obtain receiving emotion features of a listener; fusing the initiating user features and the receiving emotion features based on a unified embedding space by a modal adaptive hybrid expert encoder to obtain fused features; generating action data corresponding to the listener based on the fused features by a pre-trained action generation model based on a cross-attention layer; rendering according to the action data to generate a head video corresponding to the listener.

[0007] In one example, a first real video of a speaker is obtained, and corresponding video features and audio features are extracted as initiator user features, specifically including: obtaining a first real video of a speaker; obtaining each first frame image of the first real video, and for the first frame image, extracting the two-dimensional coordinates of the corresponding facial key points to construct a corresponding video feature sequence; obtaining each frame audio of the first real video, and for the frame audio, extracting the mel-frequency cepstrum coefficient, zero-crossing rate and root mean square energy to construct a corresponding audio feature sequence.

[0008] In one example, a sentiment prediction model is pre-trained, and a receiving sentiment feature of a listener is obtained by performing sentiment prediction on the initiator user features, specifically including: inputting the video features into a video sentiment prediction model based on a Transformer encoding layer group to output a first prediction probability of the sentiment category; inputting the audio features into an audio sentiment prediction model based on a Whisper model to output a second prediction probability of the sentiment category; obtaining the receiving sentiment feature of the listener according to the first prediction probability and the second prediction probability.

[0009] In one example, a first constraint is defined by taking the real sentiment of the listener in the training sample as supervision, and the sentiment prediction model is trained by minimizing the first constraint.

[0010] In one example, during the training process of the sentiment prediction model by the first constraint, the loss function is: wherein, is the number of sentiment categories, is the one-hot label of the real sentiment category, is the prediction probability of the predicted sentiment category.

[0011] In one example, a second real video of the listener is pre-processed to extract a time series motion sequence based on a three-dimensional deformable face representation, and the time series motion sequence of the listener in the training sample is taken as supervision to define a second constraint, and the modal adaptive hybrid expert encoder and the action generation model are trained by minimizing the second constraint.

[0012] In one example, the training process of the modal adaptive hybrid expert encoder by the second constraint includes: randomly masking the user features to mask at least part of the video features and / or at least part of the audio features; input the shielded user features to a modal adaptive hybrid expert encoder to map the shielded user features to a unified embedding space; concatenate the received emotion features encoded by the emotion category encoding module and the shielded user features in the unified embedding space to obtain a fusion feature; input the fusion feature into the action generation model and train the modal adaptive hybrid expert encoder through the second constraint.

[0013] In one example, during the training process of the modal adaptive hybrid expert encoder and the action generation model through the second constraint, the loss function is: ; wherein, is a weight parameter; ; wherein, is the true motion parameter of frame t, is the predicted motion parameter of frame t, is the adjacent frame motion change function, is a smoothing term weight, and T is the number of frames of the time sequence motion sequence; ; C is the number of emotion categories, and T is the number of frames of the time sequence motion sequence, is the one-hot label of the i-th emotion category at frame t, is the predicted probability of the i-th emotion category at frame t; ; is an emotion classification model based on motion parameters, and the model parameters of the modal adaptive hybrid expert encoder, the embedding module and the action generation model are updated by minimizing the loss function.

[0014] In one example, the time sequence motion sequence based on the three-dimensional deformable face representation is extracted from the second real video of the listener, specifically including: obtain the second real video of the listener and obtain each second frame image of the second real video; For the second frame image, obtain the corresponding expression feature, pose feature, camera rotation feature and camera translation feature thereof through principal component analysis as the motion parameters of the second frame image; combine the motion parameters in time sequence to obtain a time sequence motion sequence.

[0015] In one example, the head video corresponding to the listener is generated by rendering the action data, specifically including: convert the time sequence motion sequence corresponding to the action data into a head mesh through three-dimensional reconstruction; obtain a face image of the listener, and extract corresponding Gaussian sphere rendering attributes to generate a plurality of corresponding Gaussian spheres; bind the Gaussian spheres to vertex positions of the head mesh; refine the head mesh through the face image to increase the number of Gaussian spheres; generate a head video of the listener through Gaussian splatting rendering and neural network rendering.

[0016] In another aspect, the present application also provides a listener video generation device based on multi-modal unified embedding space guidance, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the listener video generation method based on multi-modal unified embedding space guidance as described in any of the above examples.

[0017] In another aspect, the present application also provides a non-volatile computer storage medium storing computer executable instructions configured to implement the listener video generation method based on multi-modal unified embedding space guidance as described in any of the above examples.

[0018] The listener video generation method based on multi-modal unified embedding space guidance provided by the present application can bring the following beneficial effects: 1. Through a specially trained emotion prediction model, the emotional state of the listener is directly predicted, and this is used as the input for subsequent generation. This explicitly models and drives the dynamic emotional response and transformation of the receiving node, overcoming the limitations of traditional schemes that are difficult to explicitly control emotions.

[0019] 2. The modal adaptive hybrid expert encoder is used to fuse the multi-modal features (including video, audio) of the initiator and the predicted receiving node emotional features, which can more effectively capture complex interaction information. The subsequent action generation model is based on a cross-attention mechanism, focusing on generating time-series motion sequence representations that accurately represent actions based on the fused features, avoiding the computational burden of end-to-end models or diffusion models.

[0020] 3. The training process is decoupled into two clear objectives: the first constraint corresponds to emotion prediction, ensuring that the emotion prediction model accurately predicts the true emotions of the receiving node; the second constraint corresponds to action generation, ensuring that the action generation model can accurately output motion sequences based on three-dimensional reconstruction that conform to real videos. This phased and target-specific optimization strategy improves the reliability and controllability of the overall performance of the model.

[0021] 4. Focus on generating core action data (corresponding to the time sequence of motion sequence) instead of directly generating original pixel video, and the subsequent rendering step is relatively independent. This separation design, combined with a lightweight action generation model, reduces the computational complexity of real-time generation and improves the potential real-time response of the model compared with an end-to-end pixel-level generation model. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application in any way. In the drawings: Figure 1 A flowchart of the method for generating a listener video based on a multi-modal unified embedding space guide according to an embodiment of the application is shown. Figure 2 A schematic diagram of a device for generating a listener video based on a multi-modal unified embedding space guide according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the application clearer, the technical solutions of the application will be described below in conjunction with specific embodiments of the application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the application.

[0024] The technical solutions provided by the embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0025] As shown in Figure 1 The embodiments of the application provide a method for generating a listener video based on a multi-modal unified embedding space guide, which includes: S101: Obtain a first real video of a speaker and extract corresponding video features and audio features as initiating user features.

[0026] When the interaction action is speaking, the speaker can be referred to as a speaking party (also referred to as a speaker, a speaker, etc.), and the listener can be referred to as a listening party (also referred to as a listener, a listener, etc.). When the speaking party speaks in a real scene, a video corresponding to the speaking party (referred to as a first real video herein) is collected by a camera device, and user features are extracted.

[0027] Of course, when the interaction action is a hand or head action, facial expression, etc., the speaker can also be an action initiator or an expression initiator, and at this time, the listener is an action receiver or an expression receiver. In this paper, the interaction action is mainly explained and described as an example.

[0028] Specifically, a first real video of a speaker is obtained, and each first frame image of the first real video is obtained. The "first frame image" here refers to each frame image in the first real video. The first is only used to distinguish from the "second" in the "second frame image" below. The first frame image contains part or all of the frame images in the first real video, usually containing multiple frame images, not just the "first frame" image of the first real video.

[0029] For the first frame image, the two-dimensional coordinates of the corresponding face key points are extracted, and the corresponding video feature sequence is constructed. For example, for each first frame image in the first real video, a key point detection method is used to obtain all face key points, and then 104 key points corresponding to the face contour, eye contour and mouth contour are retained. The two-dimensional coordinates of the key points are converted into 208-dimensional video features, and the features of each frame are combined to form a 208-dimensional video feature sequence with the same length as the first real video.

[0030] Each frame of audio of the first real video is obtained, and for each frame of audio, the corresponding mel-frequency cepstrum coefficient, zero-crossing rate and root mean square energy are extracted to construct the corresponding audio feature sequence. For the audio of the obtained first real video, the audio corresponding to each first frame image is obtained. For each frame of audio, 39-dimensional mel-frequency cepstrum coefficients, 1-dimensional zero-crossing rates and 1 root mean square energies are extracted, and the features of each frame are combined to form a speaker audio feature sequence with the same length as the video feature sequence.

[0031] S102: According to the initiator user feature, the emotion prediction model is trained in advance to perform emotion prediction to obtain the receiving emotion feature of the listener.

[0032] Specifically, the video feature is input into a video emotion prediction model composed of a Transformer encoding layer group, and a first prediction probability of an emotion category is output. The emotion category can include happy, sad, angry, calm, etc.

[0033] Similarly, the audio feature is input into an audio emotion prediction model based on the Whisper model, and a second prediction probability of an emotion category is output.

[0034] According to the first prediction probability and the second prediction probability, the receiving emotion feature of the listener is obtained. For example, the first prediction probability and the second prediction probability are weighted and summed according to the weight to obtain the final prediction probability of the emotion category.

[0035] Wherein, the real emotion of the listener in the training sample is defined as the first constraint by pre-training, and the emotion prediction model is trained by minimizing the first constraint.

[0036] When the video emotion prediction model and the audio emotion prediction model are trained, that is, the training process of the emotion prediction model is constrained by the first constraint, the model is trained by minimizing the first constraint, and the model parameters of the emotion prediction model are updated continuously. The loss function is: ; wherein, is the number of emotion categories, is the one-hot label of the true emotion category, is the predicted probability of the predicted emotion category. The loss function constrains the predicted probability of the predicted emotion category of the emotion prediction model to be as close as possible to the true emotion label, so that the emotion prediction model can more accurately predict the emotion category of the listener according to the information of the speaker.

[0037] S103: Fuse the initiator user feature and the received emotion feature based on a unified embedding space through the modal adaptive hybrid expert encoder to obtain a fused feature.

[0038] In the traditional scheme, it is usually assumed that the ideal environment of the speaker audio and video can be obtained at the same time, and the potential modal missing problem in the real scene is ignored, which leads to the failure of the scheme in real applications.

[0039] Based on this, the training process of the modal adaptive hybrid expert encoder through the second constraint (which is used to train the modal adaptive hybrid expert encoder and the action generation model mentioned below, and the second constraint will be described in detail below) includes: Randomly mask the user feature to mask at least part of the video feature and / or at least part of the audio feature. When masking, at least more than a preset length of video features or audio features need to be retained, so as to simulate the potential modal missing condition in the application scenario, and use the modal adaptive hybrid expert encoder to encode the information of the speaker after masking to obtain the user feature.

[0040] The user feature after the masking process is input into the modal adaptive hybrid expert encoder to map the user feature after the masking process to the unified embedding space. The received emotion feature obtained by encoding the emotion category is spliced with the user feature after the masking process in the unified embedding space to obtain a fused feature.

[0041] To simulate the potential missing modalities in real application scenarios, the video features or audio features of the speaker are randomly masked with a certain probability, at this time, the input user features have three states, namely only audio features, only video features, and both audio features and video features. The three types of user features are input into the modal adaptive hybrid expert encoder, which maps all states of user features to a unified embedding space to obtain the final user features. Then, the listener's predicted emotion category is encoded into the listener's emotion features using the embedding module. Finally, the user features of the speaker and the emotion features of the listener are spliced to obtain complete conditional features.

[0042] The fusion features are input into the action generation model, and the modal adaptive hybrid expert encoder is trained through the second constraint. The second constraint is described in detail below.

[0043] S104: Based on the cross-attention layer, the action data corresponding to the listener is generated according to the fusion features through the pre-trained action generation model.

[0044] Wherein, the time sequence motion sequence based on three-dimensional deformable face representation is extracted in advance through the second real video of the listener, and the time sequence motion sequence of the listener in the training sample is defined as the second constraint as a supervision, and the modal adaptive hybrid expert encoder and the action generation model are trained by minimizing the second constraint.

[0045] The second real video of the listener is obtained in advance, and each second frame image of the second real video is obtained. Similar to the first frame image, the second frame image can also include multiple images, and the "second" is only used to distinguish from the "first" of the first frame image in the above, and is not used to represent the image of the "second frame".

[0046] For the second frame image, the corresponding expression features, posture features, camera rotation features, and camera translation features are obtained through principal component analysis as the motion parameters of the second frame image. For example, 100-dimensional expression features, 9-dimensional posture features, 3-dimensional camera rotation features, and 3-dimensional camera translation features in each second frame image of the second real video are obtained to obtain the time sequence motion sequence corresponding to the listener.

[0047] The motion parameters are combined in time sequence to obtain the time sequence motion sequence, at this time, the time sequence motion sequence of the listener has the same length as the sequence of the user features of the speaker.

[0048] In addition, when setting the cross-attention layer, the fusion feature is input into the lightweight action generation model as the key and value, and the listener's time sequence motion sequence corresponding to the first frame image is input into the lightweight action generation model as the query of the cross-attention layer, and the action generation model is guided by the fusion feature through the cross-attention mechanism to generate the listener's action sequence, and the model is guided to generate the subsequent action sequence.

[0049] In the training process of the modal adaptive hybrid expert encoder and the action generation model by the second constraint, the real time sequence motion sequence of the second real video is used as the emotion category of the listener's time sequence action sequence obtained by the emotion prediction model as the emotion supervision, so as to narrow the gap between the emotion category corresponding to the generated time sequence action sequence and the real emotion label, thereby minimizing the training of the modal adaptive hybrid expert encoder and the action generation model by the second constraint. Of course, the embedding module can also be trained at the same time.

[0050] In the training process, the loss function is: ; wherein, is a weight parameter; ; wherein, is the real motion parameter of t frame, is the predicted motion parameter of t frame, is the adjacent frame motion change function, is the smoothing term weight, and T is the number of frames of the time sequence motion sequence; ; C is the number of emotion categories, and T is the number of frames of the time sequence motion sequence, is the one-hot label of the i-th emotion category at t frame, is the predicted probability of the i-th emotion category at t frame; ; is the emotion classification model based on the motion parameter, and the model parameters of the modal adaptive hybrid expert encoder, the embedding module and the action generation model are updated by minimizing the loss function.

[0051] S105: rendering according to the action data to generate the head video corresponding to the listener.

[0052] Specifically, the time sequence motion sequence corresponding to the action data is converted into a head mesh through three-dimensional reconstruction. The head mesh can be used to describe the three-dimensional representation of the time sequence motion sequence in three-dimensional space.

[0053] Obtain the face image corresponding to the listener. The face image can be obtained by cutting the second real video, or by the listener providing additionally.

[0054] The corresponding Gaussian sphere rendering attributes are extracted to generate a corresponding plurality of Gaussian spheres, the rendering attributes of the Gaussian spheres can include color, transparency, scale, and rotation attributes, etc., and the number of the Gaussian spheres is fixed, referred to as a first number.

[0055] The Gaussian spheres are bound to the vertex positions of the head mesh. Generally, the Gaussian spheres are bound to the vertex positions one by one, and in some special cases (for example, in a flat area), there can be a plurality of vertex positions corresponding to one Gaussian sphere.

[0056] The head mesh is refined through a face image, and the number of Gaussian spheres is the second number, which is greater than the first number, so as to more finely describe the head mesh.

[0057] The head video of the listener is generated through Gaussian splatting rendering and neural network rendering, so as to realize the generation of the listener head video and feedback through the digital person.

[0058] As shown in Figure 2 The embodiment of the present application also provides a listener video generation device based on a multi-modal unified embedding space guide, which comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the listener video generation method based on the multi-modal unified embedding space guide according to any one of the above embodiments.

[0059] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to implement the listener video generation method based on the multi-modal unified embedding space guide according to any one of the above embodiments.

[0060] Each embodiment in the present application adopts a progressive manner for description, and the same or similar parts of each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0061] The device and medium provided by the embodiment of the present application are one-to-one corresponding to the method, so the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0062] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0063] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0064] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0066] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0067] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory. The memory can also include non-volatile memory, such as read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), flash memory, or a combination of non-volatile memories in different forms. The memory is an example of computer readable storage media.

[0068] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0069] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0070] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for speech-driven video generation based on multi-modal unified embedding space guidance, characterized in that, The method comprises the following steps: obtaining a first real video of a speaker, and extracting corresponding video features and audio features as initiator user features; performing emotion prediction on the initiator user features by a pre-trained emotion prediction model to obtain receiving emotion features of a listener; fusing the initiator user features and the receiving emotion features based on a unified embedding space by a modal adaptive hybrid expert encoder to obtain fused features; generating corresponding action data of the listener based on the fused features by a pre-trained action generation model based on a cross-attention layer; rendering according to the action data to generate a head video corresponding to the listener.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining a first real video of a speaker, and extracting corresponding video features and audio features as initiator user features, specifically comprising: obtaining a first real video of a speaker; obtaining each first frame image of the first real video, and extracting the two-dimensional coordinates of the corresponding facial key points for the first frame image to construct a corresponding video feature sequence; 3. The method of claim 1, wherein, obtaining each frame of audio of the first real video, and extracting the corresponding mel-frequency cepstrum coefficient, zero-crossing rate and root mean square energy for the frame of audio to construct a corresponding audio feature sequence. The method comprises the following steps: inputting the video features into a video emotion prediction model based on a Transformer encoding layer to output a first prediction probability of an emotion category; inputting the audio features into an audio emotion prediction model based on a Whisper model to output a second prediction probability of an emotion category; 4. The method of claim 1, wherein, obtaining receiving emotion features of a listener according to the first prediction probability and the second prediction probability. The method further comprises the following steps:

5. The method of claim 4, wherein, During the training process of the emotion prediction model by the first constraint, a loss function is: ; wherein, is the number of emotion categories, is a one-hot label of a true emotion category, is a predicted probability of a predicted emotion category.

6. The method of claim 1, wherein, previously defining a first constraint with the real emotion of the listener in the training sample as supervision, and training the emotion prediction model by minimizing the first constraint. The method further comprises the following steps:

7. The method of claim 6, wherein, previously extracting a time sequence motion sequence based on a three-dimensional deformable face representation from a second real video of a listener, and defining a second constraint with the time sequence motion sequence of the listener in the training sample as supervision, and training the modal adaptive hybrid expert encoder and the action generation model by minimizing the second constraint. The training process of the modal adaptive hybrid expert encoder through the second constraint comprises the following steps: randomly masking the user features to mask at least part of the video features and / or at least part of the audio features; inputting the masked user features into the modal adaptive hybrid expert encoder to map the masked user features to a unified embedding space; splicing the receiving emotion features encoded by the emotion category and the masked user features in the unified embedding space through an embedding module to obtain fused features; inputting the fused features into the action generation model to train the modal adaptive hybrid expert encoder through the second constraint.

8. The method of claim 7, wherein, In the training process of the modal adaptive hybrid expert encoder and the action generation model by the second constraint, a loss function is: ; wherein, is a weight parameter; ; wherein, is a true motion parameter of a frame, is a predicted motion parameter of a frame, is a neighboring frame motion change function, is a smoothing term weight, T is the number of frames of the temporal motion sequence; ; C is the number of emotion categories, T is the number of frames of the time-series motion sequence, is the one-hot label of the i-th emotion category at frame t, is the predicted probability of the i-th emotion category at frame t. ; For the motion parameter-based emotion classification model, model parameters of the modal adaptive hybrid expert encoder, the embedding module, and the action generation model are updated by minimizing a loss function.

9. The method of claim 1, wherein, The time sequence motion sequence based on the three-dimensional deformable face representation is extracted through the second real video of the listener, and specifically includes: A second real video of a listener is obtained, and each second frame image of the second real video is obtained; For the second frame image, the corresponding expression feature, posture feature, camera rotation feature and camera translation feature are obtained through principal component analysis as the motion parameters of the second frame image; The motion parameters are combined in time sequence to obtain a time sequence motion sequence.

10. The method of claim 1, wherein, According to the action data, a head video corresponding to the listener is generated, specifically including: Through three-dimensional reconstruction, the time sequence motion sequence corresponding to the action data is converted into a head mesh; A face image corresponding to the listener is obtained, and corresponding Gaussian sphere rendering attributes are extracted to generate a plurality of Gaussian spheres; The Gaussian spheres are bound to the vertex positions of the head mesh; The head mesh is refined through the face image to increase the number of Gaussian spheres; The head video corresponding to the listener is generated through Gaussian splashing rendering and neural network rendering.

Images