Digital human interaction method and device, electronic device, storage medium and program product

Abstract

The present disclosure provides a digital human interaction method and device, electronic equipment, storage medium and program product. The method comprises: obtaining original audio to be processed from an audio stream, the audio stream being used to respond to inquiry content of an interaction object; obtaining audio content features and audio emotion features of the original audio; generating a target video containing a digital human according to the audio content features, the audio emotion features and an identity identifier of the digital human; and performing a response interaction based on the target video and the interaction object. According to the embodiment of the present disclosure, the digital human video can be generated based on the audio stream, the response delay in the response interaction process of the digital human and the interaction object is reduced, and the interaction experience of the interaction object is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a digital human interaction method and apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of large language model technology, interactive digital humans are being gradually applied to more and more human-computer interaction scenarios to improve the interactive experience of interactive objects.

[0003] In related technologies, during the generation of digital human videos and the interaction between the digital human and interactive objects, the correspondence between the digital human's mouth movements and audio is usually learned based on the input speech signal and video frames, thereby generating lip-synced digital human videos for interaction with interactive objects.

[0004] Digital human videos generated by related technologies suffer from poor interactive experience due to stiff facial expressions and a lack of realism. Summary of the Invention

[0005] This disclosure provides a digital human interaction method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, this disclosure provides a digital human interaction method, which includes:

[0007] Obtain the raw audio to be processed from the audio stream, which is used in response to the query content of the interactive object;

[0008] Obtain the audio content features and audio emotional features of the original audio;

[0009] Based on the audio content features, the audio emotional features, and the digital human's identity, a target video containing the digital human is generated;

[0010] The interaction is based on the target video and the interactive object.

[0011] Secondly, this disclosure provides a digital human interaction device, which includes:

[0012] The first acquisition module is used to acquire the raw audio to be processed from the audio stream, which is used to respond to the query content of the interactive object;

[0013] The second acquisition module is used to acquire the audio content features and audio emotional features of the original audio.

[0014] The generation module is used to generate a target video containing the digital human based on the audio content features, the audio emotional features, and the digital human's identity identifier;

[0015] An interaction module is used to respond to and interact with the target video and the interaction object.

[0016] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the digital human interaction method described above.

[0017] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned digital human interaction method.

[0018] Fifthly, this disclosure provides a computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the aforementioned digital human interaction method.

[0019] The embodiments provided in this disclosure, for audio streams used to respond to inquiries from interactive objects, can process the raw audio to be processed in real time without waiting for the audio to be received completely, by obtaining the raw audio from the audio stream. This is achieved by simultaneously obtaining the audio content features and audio emotional features of the raw audio; and generating a target video containing the digital human based on the audio content features, the audio emotional features, and the identity of the digital human to be generated. The interaction with the interactive object is then based on this target video. Since the target video is generated based on the audio content features obtained simultaneously from the raw audio and the corresponding audio emotional features representing the audio emotion, the facial expressions of the generated digital human can be highly matched with the audio content and emotional emotion of the raw audio. This avoids the problems of stiff and unrealistic digital human expressions, thereby improving the vividness, realism, and overall video quality of the digital human in the target video. This reduces the response latency of the digital human while enhancing the interactive experience of the interactive object.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a digital human interaction method provided in an embodiment of this disclosure;

[0023] Figure 2 A schematic diagram of digital human interaction processing provided in an embodiment of this disclosure;

[0024] Figure 3 A flowchart for generating a target video provided in this embodiment of the disclosure;

[0025] Figure 4 A flowchart for generating an expression coefficient sequence provided in this embodiment of the disclosure;

[0026] Figure 5 A flowchart for obtaining the target flow field provided in this embodiment of the disclosure;

[0027] Figure 6 A first flowchart of audio content and emotion extraction processing provided in this embodiment of the disclosure;

[0028] Figure 7 This is a second flowchart of audio content and emotion extraction processing provided in an embodiment of the present disclosure;

[0029] Figure 8 A flowchart illustrating the target content extraction model provided in this embodiment of the disclosure;

[0030] Figure 9 A flowchart illustrating the content recognition model training process provided in this embodiment of the disclosure;

[0031] Figure 10 This is a schematic diagram illustrating the training process of the content recognition model provided in an embodiment of the present disclosure;

[0032] Figure 11 A block diagram of a digital human interaction device provided in an embodiment of this disclosure;

[0033] Figure 12 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0036] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0038] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0039] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example, appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely identifying specific individuals.

[0040] For ease of understanding, the terminology used in the embodiments of this disclosure will be explained below.

[0041] A digital human is a virtual character created using computer technology that possesses human appearance and behavioral characteristics. Digital human technology can be used in applications such as video generation, human-computer interaction, and virtual live streaming.

[0042] The expression coefficient is a set of parameters used to characterize the facial expressions of a digital human. The expression coefficient generally corresponds to parameters such as facial key points, muscle movements, or three-dimensional deformations of the digital human. Based on the expression coefficient, it can be used to drive the digital human to produce different expressions such as smiling, frowning, and opening its mouth.

[0043] An expression coefficient sequence is a sequence of multiple expression coefficients arranged in chronological order. Each expression coefficient in the sequence corresponds to a video frame. Based on this expression coefficient sequence, the rendering model can drive the changes in digital human facial expressions frame by frame to achieve expression presentation that is synchronized with the audio content and emotion type.

[0044] Emotion type tags are identifiers used to indicate the emotion type of audio. For example, they can be identifiers used to represent emotions such as joy, sadness, anger, seriousness, and gentleness.

[0045] An identity identifier, also known as a speaker tag or digital human identity ID, is a unique identifier used to distinguish different digital human identities. Different identity identifiers correspond to different digital human appearance characteristics and expression styles, ensuring the consistency of the generated digital humans and avoiding confusion in appearance across different scenarios. For example, identity identifiers can be used to indicate that a digital human is a live-streaming digital human, a customer service digital human, etc.

[0046] An appearance reference image is an image used to define the baseline appearance of a digital human. It can contain appearance information such as the digital human's facial contours, facial features, skin color, and hairstyle. This appearance reference image can also be called a reference image. It is used to provide appearance information of the digital human when generating video frames containing the digital human.

[0047] Audio content features refer to the features extracted from audio to represent the semantic and content information of the audio. Unlike simple acoustic features, audio content features can be used to represent the speech content and semantic information in audio.

[0048] An embedding vector is a vector representation obtained by mapping discrete category information to a continuous high-dimensional space, so that the model can recognize and process that type of information.

[0049] The velocity field is a velocity function used to describe the transformation process of data distribution. It can be used to characterize the direction and rate of continuous transformation from a simple prior distribution to a specific distribution.

[0050] A rendering model is a model used to convert facial expression coefficient sequences and appearance reference images into digital human video frames, which can be based on two-dimensional image rendering or three-dimensional modeling rendering. In the embodiments of this disclosure, the rendering model can be, for example, a PIRenderer, LAM, or other similar models.

[0051] Automatic Speech Recognition (ASR) is a technology used to convert user speech into a text sequence. It typically uses speech recognition models to automatically identify and transcribe speech content, and is a fundamental technology for voice interaction, intelligent assistants, and other similar scenarios. Common speech recognition models include RNN-T, Paraformer, and U2++. These models generally consist of an audio encoder and an audio decoder. The audio encoder encodes the acoustic features of the audio, while the audio decoder decodes these features to obtain the recognized text.

[0052] Acoustic features are feature vectors extracted from audio waveforms to characterize the acoustic properties of audio. Common acoustic features include Mel frequency cepstral coefficients and Mel filter bank features. Acoustic features are the core input of audio models.

[0053] Mel-frequency cepstral coefficients (MFCCs) are one of the most commonly used acoustic features in speech recognition. They simulate the human ear's perception of different frequencies of sound and are extracted through steps such as speech segmentation, Fourier transform, and Mel filtering. They can effectively characterize the acoustic properties of speech.

[0054] Mel-scale filter bank (FBank) features are a type of fundamental acoustic feature, typically obtained by performing a Fourier transform on speech followed by a Mel-scale filter bank. They preserve the spectral details of speech and are often used as the basic features of MFCC or directly as input to speech recognition models.

[0055] An audio frame is the basic processing unit in audio processing. An audio frame generally refers to a short segment obtained by dividing a continuous audio waveform into segments of fixed duration (e.g., 25ms). Frames usually overlap (e.g., 10ms frame shift) to ensure signal continuity.

[0056] A multilayer perceptron (MLP) generally refers to a neural network consisting of fully connected layers and nonlinear activation functions. In the embodiments of this disclosure, MLP can be used for nonlinear transformation and dimension mapping of features to extract high-order discriminative information.

[0057] As described in the background section, related technologies, when generating digital human videos for interaction with interactive objects, typically focus on lip-sync, for example, generally using the Wav2Lip model for digital human video generation. These technologies often only model the mouth area, making it difficult to generate detailed changes in head posture, facial expressions, etc. Furthermore, they often struggle to generate high-resolution videos and may suffer from mouth artifacts or blurring, resulting in low video quality and negatively impacting the interactive experience.

[0058] To address the aforementioned issues, the applicant discovered that a two-stage generation framework could be considered for generating digital human videos. Specifically, in the first stage, facial motion representations can be predicted from audio and the target face, for example, identity features and expression parameters can be obtained through 3D reconstruction or keypoint estimation; in the second stage, digital human video generation processing is performed based on the facial motion representations estimated in the first stage.

[0059] However, while digital human videos generated based on the above two-stage generation framework can improve video quality to some extent and thus enhance the interactive experience, this approach focuses on rendering the facial posture of digital humans. When a face expresses an emotion, there are often multiple forms of expression. That is, one emotion often corresponds to multiple facial expressions. Therefore, digital human videos generated based on this approach still suffer from stiff facial expressions, lack of realism, and low video quality, which still affect the interactive experience.

[0060] The applicant also found that, to enhance the expressive richness of the generated digital human videos, additional emotion type labels could be introduced into the digital human video processing for control. These emotion type labels could be manually specified or obtained through emotion recognition of the input audio by a large model. However, manually specifying emotion type labels often results in the generated digital human's expressions not being adaptively adjusted, meaning they may not match the original emotion of the input audio. On the other hand, using emotion recognition of the input audio often requires processing a complete audio segment; that is, recognition can only be performed after a complete audio segment is input. This often significantly increases the response latency of the interactive digital human, reducing the interactive experience.

[0061] In view of this, embodiments of this disclosure provide a digital human interaction method, which involves obtaining raw audio to be processed from an audio stream, the audio stream being used to respond to the query content of an interactive object; obtaining audio content features and audio emotional features of the raw audio; generating a target video containing the digital human based on the audio content features, audio emotional features, and the digital human's identity identifier; and performing response interaction with the interactive object based on the target video.

[0062] The embodiments provided in this disclosure, for audio streams used to respond to inquiries from interactive objects, can process the raw audio to be processed in real time without waiting for the audio to be received completely, by obtaining the raw audio from the audio stream. This is achieved by simultaneously obtaining the audio content features and audio emotional features of the raw audio; and generating a target video containing the digital human based on the audio content features, the audio emotional features, and the identity of the digital human to be generated. The interaction with the interactive object is then based on this target video. Since the target video is generated based on the audio content features obtained simultaneously from the raw audio and the corresponding audio emotional features representing the audio emotion, the facial expressions of the generated digital human can be highly matched with the audio content and emotional emotion of the raw audio. This avoids the problems of stiff and unrealistic digital human expressions, thereby improving the vividness, realism, and overall video quality of the digital human in the target video. This reduces the response latency of the digital human while enhancing the interactive experience of the interactive object.

[0063] The digital human interaction method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0064] Please refer to Figure 1 and Figure 2 These are, respectively, a flowchart of a digital human interaction method provided in an embodiment of this disclosure, and a schematic diagram of digital human interaction processing provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method may include the following steps S11-S14, which are described in detail below.

[0065] Step S11: Obtain the raw audio to be processed from the audio stream.

[0066] The audio stream is used to respond to the queries from the interactive object.

[0067] The original audio can be the audio to be processed at the current moment in the audio stream.

[0068] The original audio refers to the audio used to drive the generation of digital human videos. In this embodiment of the disclosure, in order to improve the efficiency of digital human interaction and reduce interaction latency, thereby enhancing the interactive experience of the interactive object, the original audio can be a frame or a preset number of audio frames from an audio stream generated in real time to respond to the query content of the interactive object. The preset number can be, for example, 1, 3, 5, etc., and this disclosure does not make any special limitation in this regard.

[0069] Understandably, the interaction target can be a user, such as an e-commerce user or a customer inquiring user in a smart customer service scenario. The inquiry content can be in various forms, including text, voice, images, or video. After receiving the inquiry content, a response matching the inquiry content can be generated through a preset question-and-answer system, a large language model, or a dialogue management module. Subsequently, an audio stream can be continuously generated using text-to-speech (TTS) technology.

[0070] Step S12: Obtain the audio content features and audio emotion features of the original audio.

[0071] In this embodiment of the disclosure, audio content features are used to represent the audio content of the original audio; audio emotion features are used to represent the audio emotion of the original audio, which can represent the emotional tendency contained in the original audio, such as any one or more of the emotion types like joy, anger, sorrow, and happiness.

[0072] Specifically, in order to improve the interactive experience of the interactive object during the interaction between the digital human and the interactive object, in this embodiment of the disclosure, instead of assigning an audio emotion to the original audio to be processed in the audio stream, feature extraction processing is performed on the original audio at the same time. While extracting its audio content features, audio emotion features that match the audio content features are also extracted. That is, the extracted audio emotion features are consistent with the audio emotion represented by the original audio and change in real time with the change of emotion of the original audio, thereby making the expression of the digital human in the generated target video more vivid and natural, and improving the interactive experience of the interactive object.

[0073] Step S13: Generate a target video containing a digital human based on audio content features, audio emotional features, and the digital human's identity identifier.

[0074] An identity identifier can be used to identify the identity of the digital human to be generated. For example, identity identifier "01" can be used to indicate that the digital human's identity is customer service, and identity identifier "02" can be used to indicate that the digital human's identity is a broadcaster.

[0075] That is, depending on the digital human's identity, the way a digital human expresses the same emotion often differs. For example, the facial expressions of a digital human who is a game streamer and a digital human who is an e-commerce customer service representative often show fundamental differences in expressing emotions such as excitement and anger. Therefore, in this embodiment of the disclosure, in order to enable the digital human to express expressions appropriate to its identity during interaction with the interactive object, thereby improving the user experience, this disclosure can also obtain the digital human's identity identifier to generate a target video for interaction.

[0076] Step S14: Conduct response interaction based on the target video and the interactive object.

[0077] After generating a target video containing a digital human based on the above steps, you can interact with the interactive object based on the target video.

[0078] For example, in human-computer interaction scenarios, electronic devices can continuously receive inquiries from the interacting object, generate corresponding response text based on the continuously received inquiries, and continuously perform audio conversion processing on the corresponding response text to generate an audio stream for real-time interaction. The original audio can be one frame or a few frames of audio in the real-time generated audio stream that are currently to be processed. Then, by simultaneously extracting the audio content features and audio emotional features of the original audio, a digital human video corresponding to the original audio, i.e., the target video, can be generated based on the audio content features, the audio emotional features, and the digital human's identity identifier, and timely response interaction can be performed with the interacting object based on the target video.

[0079] It is understood that, in some embodiments, the original audio can also be encapsulated into the target video to ensure that the visuals of the target video are synchronized with the original audio, so as to generate a complete and satisfactory target video in real time, thereby enabling the digital human to interact with the target object.

[0080] As can be seen, based on the method provided in this disclosure, for an audio stream used to respond to the query content of an interactive object, by obtaining the original audio to be processed from the audio stream, it is not necessary to wait for the audio to be received completely, but the original audio can be processed in real time. This is achieved by simultaneously obtaining the audio content features and audio emotional features of the original audio; and generating a target video containing the digital human based on the audio content features, the audio emotional features, and the identity of the digital human to be generated. The interaction with the interactive object is then based on this target video. Since the target video is generated based on the audio content features obtained simultaneously from the original audio and the audio emotional features corresponding to the audio content, the facial expressions of the generated digital human can be highly matched with the audio content and emotional features of the original audio, avoiding the problem of stiff and unrealistic digital human expressions. This improves the vividness, realism, and overall video quality of the digital human in the target video, reducing the response latency of the digital human and enhancing the interactive experience of the interactive object.

[0081] The digital human interaction method according to embodiments of this disclosure will now be described in detail.

[0082] Please refer to Figure 3 This is a flowchart illustrating the generation of the target video provided in an embodiment of this disclosure. For example... Figure 3 As shown, in some embodiments, the step S13, which involves generating a target video containing a digital human based on audio content features, audio emotional features, and the digital human's identity, may include the following steps S31-S32:

[0083] Step S31: Generate an expression coefficient sequence based on audio content features, audio emotional features, and identity identifiers.

[0084] Among them, the expression coefficient sequence is used to define the facial expression patterns of the digital human in the target video.

[0085] The expression coefficient sequence corresponds to multiple video frames in the target video, and each expression coefficient is used to define the facial expression of the digital human in the corresponding video frame.

[0086] In this embodiment of the disclosure, each expression coefficient in the expression coefficient sequence can be a set of parameters used to accurately define the facial expression morphology of the digital human in the corresponding video frame.

[0087] For example, one expression coefficient corresponds to a smiling expression with "upturned corners of the mouth and relaxed eyebrows and eyes", while another expression coefficient corresponds to a serious expression with "frowning and downturned corners of the mouth". By using expression coefficients frame by frame, the continuous and natural changes of digital human facial expressions can be achieved.

[0088] In this embodiment of the disclosure, the expression coefficient sequence can be determined based on a target flow field using the audio content features and emotional features of the original audio, as well as the identity of the digital human, as constraints. The target flow field can be a pre-learned velocity function, and the noise data sampled from a preset data distribution can be integrally transformed based on the velocity function to determine the expression coefficient sequence.

[0089] The expression coefficients can be represented using a 3D Morphable Model (3DMM). In some embodiments, during the pre-training process of learning the target flow field, the expression coefficient annotations in the expression coefficient annotation sequence of the training data can be extracted from the video frames corresponding to the sample audio using methods such as DECA or HRN. For example, during the training process of obtaining the target flow field, each expression coefficient annotation can be: a BFM (Basel Face Model) expression coefficient obtained from the video frames corresponding to the sample audio by using HRN as the expression coefficient extractor.

[0090] Step S32: Generate the target video based on the expression coefficient sequence and the digital human's appearance reference image.

[0091] This appearance reference image can be used to define the baseline appearance of the digital human to be generated, that is, as an appearance template for the digital human to be generated.

[0092] In some embodiments, step S32 may include: inputting an expression coefficient sequence and an appearance reference image into a rendering model to obtain a multi-frame video containing a digital human; and generating a target video based on the multi-frame video.

[0093] Specifically, in this embodiment of the disclosure, after obtaining the original audio to be processed from the audio stream, a target video containing a digital human can be generated based on a two-stage framework, so as to conduct response interaction with the interactive object based on the target video. In the first stage, the audio content features and audio emotional features of the original audio can be obtained simultaneously; then, based on the audio content features, audio emotional features, and the digital human's identity, an expression coefficient sequence is determined to define the facial expression morphology of the digital human in each video frame of the generated target video. Then, in the second stage, the expression coefficient sequence and the appearance reference image of the digital human can be input into the rendering model to execute the rendering process. For example, the appearance reference image is used as a rendering template, and the corresponding expression coefficients in the expression coefficient sequence are input frame by frame. The rendering model adjusts the facial key points and muscle movement parameters of the digital human based on the baseline appearance of the appearance reference image and the expression coefficients to generate corresponding multi-frame video frames; the multi-frame video frames are arranged in chronological order to form a video frame sequence containing continuous expression changes of the digital human. The resolution and frame rate of the video frames can be set according to actual needs to avoid problems such as video blurring and stuttering; then, as Figure 2 As shown, multiple video frames can be stitched together in chronological order to obtain the target video.

[0094] In some embodiments, step S32 may include generating a target video based on an expression coefficient sequence, an identity identifier, and an appearance reference image of the digital human. That is, the digital human's identity identifier may also be input into a rendering model for rendering processing to improve speaker consistency.

[0095] As can be seen, in this embodiment of the disclosure, by jointly determining the digital human's expression coefficient sequence based on the audio content features and audio emotional features of the original audio, as well as the identity identifier that can represent the digital human's identity, and generating a target video based on the expression coefficient sequence and the digital human's appearance reference image, the vividness and realism of the digital human's facial expressions in the target video can be improved, thereby enhancing the interactive experience of the interactive object during the response interaction process based on the target video and the interactive object.

[0096] Please refer to Figure 4 This is a flowchart illustrating the generation of an expression coefficient sequence provided in an embodiment of this disclosure. For example... Figure 4 As shown, in some embodiments, the step S31 of generating an expression coefficient sequence based on audio content features, audio emotional features, and identity identifiers may include the following steps S41-S43:

[0097] Step S41: The identity identifier is vectorized to obtain the first embedding vector.

[0098] The first embedding vector can be obtained by vectorizing the identity identifier based on the embedding layer or word embedding model, or it can be obtained in other ways. This disclosure does not make any special limitations on this.

[0099] Step S42: Sample noise data from a preset data distribution to obtain initial expression coefficients for defining the facial expression morphology of the digital human.

[0100] The preset data distribution can be selected as a Gaussian distribution, i.e. a standard normal distribution, or it can be set as needed. This disclosure does not impose any special limitations on this.

[0101] Step S43: Using audio content features, audio emotion features, and the first embedding vector as constraints, perform an integral transformation on the initial expression coefficients based on the target flow field to obtain the expression coefficient sequence.

[0102] The target flow field can be a pre-learned velocity function. Based on this velocity function, an integral transformation can be performed on the noise data sampled from the preset data distribution to determine the expression coefficient sequence.

[0103] In some embodiments, the target flow field can be an expression coefficient generation model trained based on a flow matching training strategy. Flow matching is a generative modeling method based on continuous-time normalizing flow (CNF).

[0104] In some embodiments, since the original audio is a single audio frame, the number of audio content features and audio emotion features extracted from the original audio is usually also one. Therefore, in the process of generating an expression coefficient sequence based on the target flow field using the audio content features, audio emotion features, and the first embedding vector as constraints, the audio content features, audio emotion features, and the first embedding vector can each be input into the target flow field separately; or, the audio content features, audio emotion features, and the first embedding vector can be concatenated before being input into the target flow field. The concatenation process can be set as needed, and this disclosure does not impose any special limitations on it.

[0105] Specifically, in related technologies, traditional regression methods are often used for facial expression prediction, which has the following problems: Problem 1: Traditional regression methods are essentially point estimations, which can only produce a unique and definite output for the same input. Therefore, models trained based on this type of method often only output a single definite facial expression parameter. The digital human videos generated based on this facial expression parameter often result in overly flat facial expressions. Problem 2: Traditional regression methods produce mode averaging when facing multimodal distributions, that is, the prediction result falls in the middle position of multiple modes. Therefore, the facial expression parameters predicted by models trained based on this type of method often do not correspond to any real and reasonable facial expressions, but are represented by an average of facial expressions, which leads to overly averaged facial expressions in digital humans.

[0106] To address this type of problem, in this embodiment of the disclosure, a flow matching strategy is used to train a target flow field. For example, an expression coefficient generation model representing the velocity function of the target flow field is trained. The target flow field performs an integral transformation on the initial expression coefficients obtained by sampling noise data from a preset data distribution, using the audio content features, audio emotion features, and the first embedding vector of the identity of the digital human to be generated as constraints. This determines the expression coefficient sequence corresponding to the audio content, audio emotion, and identity of the digital human in the original audio. Then, a target video with rich and varied expressions is generated based on the expression coefficient sequence. Finally, the target video is used to interact with the interactive object, improving the interactive experience of the interactive object.

[0107] Please refer to Figure 5 This is a flowchart of obtaining the target flow field provided in an embodiment of this disclosure. For example... Figure 5 As shown, in some embodiments, the target flow field in this disclosure can be obtained through the following processing:

[0108] Step S51: Obtain the first sample audio, the sample identity identifier of the sample digital human, and the expression coefficient annotation sequence.

[0109] The first sample audio can be a sample used for target flow field learning processing, and the first sample audio corresponds to m time points, where m > 0. That is, each first sample audio can include audio frames from m time points.

[0110] The sample identity identifier can be an identifier used to represent the identity of the sample digital human. For details, please refer to the relevant sections mentioned above, and it will not be repeated here.

[0111] This sequence of facial expression coefficient annotations can serve as the ground truth, or supervisory signal, in the process of learning the target flow field based on the initial flow field. The sequence includes multiple facial expression coefficient annotations, each of which can represent the facial expression of a real user in the sample video frame corresponding to the first sample audio.

[0112] In some embodiments, obtaining the expression coefficient annotation sequence in step S51 may include: obtaining the sample video frame sequence corresponding to the first sample audio; and performing expression coefficient extraction processing on each sample video frame in the sample video frame sequence to obtain the expression coefficient annotation sequence.

[0113] That is, in actual implementation, the video frames of the real video corresponding to the first sample audio (e.g., the video when the real user recorded the first sample audio) can be obtained as a sequence of sample video frames, and the expression coefficients can be extracted from each sample video frame to obtain the expression coefficient annotation sequence.

[0114] Specifically, techniques such as DECA and HRN can be used to extract facial expression coefficients from the sample video frames corresponding to the first sample audio. For example, HRN can be used as a facial expression coefficient extractor, and the BFM facial expression coefficients obtained from the sample video frames can be used as the facial expression coefficient annotation sequence corresponding to the first sample audio.

[0115] Step S52: Obtain the sample audio content features and sample audio emotion features at m time points of the first sample audio.

[0116] In this embodiment of the disclosure, the audio content and emotional features of the first sample audio can be extracted by performing audio content extraction processing on the first sample audio, thereby obtaining the sample audio content features and sample audio emotional features at m time points of the first sample audio.

[0117] Step S53: The sample identity identifier is vectorized to obtain the second embedding vector.

[0118] Step S54: The second embedding vector is concatenated with the audio content features and audio sentiment features corresponding to each time moment to obtain m sample content sentiment identity vectors corresponding to m time moments.

[0119] Specifically, given that the number of second embedding vectors obtained by vectorizing the sample identity of the sample digital human is 1, while the number of sample audio content features and sample audio emotion features of the first sample audio is m, in order to learn the target flow field based on the initial flow field, the initial flow field can be made to learn how to integrally transform the noise data sampled from the preset data distribution to the expression coefficient label at the corresponding time based on the sample audio content, sample audio emotion and sample digital human identity at each time step. In the process of learning the target flow field, the second embedding vector can be concatenated with the sample audio content features and sample audio emotion features corresponding to each time step.

[0120] For example, taking the audio content features of the first sample audio as (audio content feature 1 at time 1, audio content feature 2 at time 2) and the emotional features of the first sample audio as (emotional feature 1 at time 1, emotional feature 2 at time 2), we can concatenate "second embedding vector + audio content feature 1 at time 1 + emotional feature 1 at time 1" to obtain sample content emotional identity vector 1, and concatenate "second embedding vector + audio content feature 2 at time 2 + emotional feature 2 at time 2" to obtain sample content emotional identity vector 2. Based on these two sample content emotional identity vectors as constraints, we can learn the target flow field from the initial flow field. It should be noted that in actual implementation, this concatenation process can be set as needed, and this disclosure does not impose any special limitations on it.

[0121] Step S55: Using m sample content emotion identity vectors as constraints, the initial flow field learning will be transformed from the noise data sampled from the preset data distribution to the expression coefficient annotation sequence to obtain the target flow field.

[0122] In this embodiment, given that the DIT (Diffusion Transformer) model has characteristics such as stable training, high generation accuracy, and good temporal consistency, the initial flow field can be implemented based on the DIT (Diffusion Transformer) model to ensure the accuracy of the target flow field prediction results, thereby achieving continuous and natural changes in digital human facial expressions. Of course, in actual implementation, the initial flow field can also be other models that can be used to perform flow matching velocity field prediction processing, and this disclosure does not impose any special limitations on this.

[0123] Specifically, in order to address the problem of overly uniform digital human expressions when predicting expression coefficients based on traditional regression methods in related technologies, in this embodiment of the disclosure, a flow matching strategy can be used to learn the initial flow field to obtain the target flow field.

[0124] This embodiment of the present disclosure concatenates the second embedded vector of the sample identity identifier with the sample content features and sample audio features at the corresponding time, and uses the resulting m sample content emotion identity vectors as constraints to inject them into the learning process of the target flow field. This allows the learned target flow field to simultaneously adapt to the needs of the three dimensions of audio content, audio emotion, and digital human identity, avoiding expression prediction bias caused by a single constraint and solving the problem of stiff and unrealistic digital human expressions in related technologies.

[0125] That is, based on this constraint, the target flow field is modeled as a complete probability distribution of conditional probability distribution (expression coefficients | audio, emotion, digital human identity), rather than a deterministic mapping. This allows the learned target flow field to generate diverse and reasonable expression coefficient sequences based on different emotional conditions, digital human identity, and sampled noise data, given the same audio input. This makes the digital human expressions in the generated digital human videos richer and more natural, avoiding the problem of overly uniform expressions.

[0126] It is understandable that in the process of learning the target flow field based on the initial flow field, the predicted expression coefficient sequence of the initial flow field output can be obtained, and the first loss value can be determined based on the predicted expression coefficient sequence and the expression coefficient annotation sequence. Then, the parameters of the initial flow field are adjusted based on the first loss value, and the converged initial flow field is determined as the target flow field.

[0127] The first loss value can be obtained, for example, by processing the predicted expression coefficient sequence and the expression coefficient labeled sequence based on the mean squared error loss function, or it can be obtained through other loss functions. This disclosure does not impose any special limitations on this.

[0128] After obtaining the first loss value, the parameters of the initial flow field can be adjusted based on the first loss value until they meet the preset convergence condition. The preset convergence condition can be, for example, that the error between the predicted expression coefficient sequence and the expression coefficient annotation sequence is less than a preset threshold, or it can be set as needed, without special limitation here.

[0129] In addition, in some embodiments, the step of using m sample content emotion identity vectors as constraints to transform the initial flow field learning from the noise data sampled from the preset data distribution into an expression coefficient labeling sequence to obtain the target flow field may include: interpolating the noise data sampled from the preset data distribution according to the expression coefficient labeling sequence to obtain an intermediate state expression coefficient sequence; and using m sample content emotion identity vectors as constraints to transform the initial flow field learning from the intermediate state expression coefficient sequence into an expression coefficient labeling sequence to obtain the target flow field.

[0130] That is, in order to improve the convergence speed of the initial flow field, the noise data sampled from the preset data distribution can be interpolated according to the expression coefficient annotation sequence to obtain the expression coefficient sequence of the intermediate state, and the initial flow field learning can be transformed into the flow field of the expression coefficient annotation sequence based on the expression coefficient annotation sequence of the intermediate state.

[0131] As can be seen, the method provided by the embodiments of this disclosure enables the target flow field learned from the initial flow field to generate diverse and reasonable expression sequences based on different emotional conditions, identity identifiers and sampling noise, given the same audio input. This enhances the vividness of the digital human's expressions in the generated target video, making them more compatible with the audio content of the original audio, thereby improving the interactive experience of the interactive object in the digital human interaction process.

[0132] Please refer to Figure 6 This is a first flowchart of audio content and emotion extraction processing provided in the embodiments of this disclosure. Figure 6 As shown, in some embodiments, the acquisition of audio content features and audio emotional features of the original audio in step S12 may include the following steps S61-S62:

[0133] Step S61: Perform audio content encoding processing on the original audio to obtain audio content features.

[0134] In this step, the audio content encoding process of the original audio to obtain audio content features may include: extracting acoustic features from the original audio to obtain acoustic features of the original audio; and performing audio content encoding process on the acoustic features to obtain the audio content features.

[0135] The acoustic feature extraction can be any acoustic feature extracted from the original audio, such as MFCC or FBank.

[0136] Step S62: Obtain audio emotional features based on audio content features.

[0137] That is, in this embodiment of the disclosure, in order to make the expression of the digital human in the target video correspond to the audio emotion represented by the original audio to be processed in the audio stream, that is, to make the digital human reflect the emotional changes of the input speech, thereby making the expression of the digital human more vivid and natural, this embodiment of the disclosure obtains the audio content features of the original audio, and then obtains the audio emotion features representing the audio emotion of the original audio based on the audio content features.

[0138] Please refer to Figure 7 This is a second flowchart of the audio content and emotion extraction processing provided in the embodiments of this disclosure. Figure 7As shown, in some embodiments, obtaining audio emotional features based on audio content features in step S62 may include the following steps S71-S73:

[0139] Step S71: Obtain the multi-frame historical content vector corresponding to the original audio.

[0140] In this embodiment, the multi-frame historical content vector corresponds to multiple historical audio segments, which include multiple audio segments in the audio stream that precede the original audio. It is understood that, corresponding to the original audio, each historical audio segment may include one or a few audio frames (e.g., a 25ms audio clip). In this embodiment, unless otherwise specified, each historical audio segment is illustrated as consisting of one audio frame.

[0141] Step S72: Pool the audio content features and multi-frame historical content vectors to obtain segment-level content vectors;

[0142] Step S73: Perform audio sentiment prediction processing on the segment-level content vector to obtain audio sentiment features.

[0143] The segment-level content vector represents the audio content of the original audio within a preset time period. This preset time period can be set as needed; for example, it can be the time period corresponding to 50 audio frames.

[0144] The multi-frame historical content vector, for example, can be an M-frame (M can be 50) historical content vector. That is, since this embodiment of the disclosure performs streaming processing on the audio stream, that is, it obtains the real-time, unprocessed original audio and performs digital human video generation processing to generate a target video containing a digital human and interacting with interactive objects. The original audio is usually one or a few audio frames (e.g., a 25ms audio segment) in the audio stream that are currently unprocessed. Since one or a few audio frames often cannot accurately represent the audio emotion, that is, the emotion is often continuous, in order to accurately extract the audio emotion of the original audio in this embodiment of the disclosure, the historical content vector of M frames, such as 50 frames, can be taken from the current audio frame of the original audio in the audio stream. In the process of obtaining the audio emotion features representing the audio emotion of the original audio, the audio content features of the original audio and the multi-frame historical content vector can be pooled to obtain a segment-level content vector that can represent the baseline emotion and emotion intensity of the original audio within a preset time period. The audio emotion features of the original audio can be obtained by performing audio emotion prediction processing on the segment-level content vector.

[0145] In some embodiments, the pooling process of the audio content features and the multi-frame historical content vectors in step S72 to obtain the segment-level content vector may include: performing mean pooling on the audio content features and the multi-frame historical content vectors to obtain a first content vector; performing standard deviation pooling on the audio content features, the multi-frame historical content vectors, and the first content vector to obtain a second content vector; and obtaining the segment-level content vector based on the first content vector and the second content vector.

[0146] Specifically, the first content vector can be calculated using the following formula 1:

[0147] Formula 1;

[0148] in, Let M represent the first content vector, and M represent the total number of audio content features of multiple historical content vectors and the original audio. This represents the frame-level content vector (also known as frame-level content feature, i.e., the audio content feature of the j-th audio frame) of the audio frame. , That is, in Formula 5, t represents the raw audio to be processed at the current moment, which is obtained in real time from the audio stream. This raw audio corresponds to one audio frame at the current moment, i.e., time t.

[0149] For example, if the original audio is an audio segment consisting of the 10th audio frame in the audio stream, then when M is 3, it can start from the current 10th frame in the audio stream, and then take the historical audio of the 9th and 8th audio frames in the audio stream to process, and obtain the first content vector and the subsequent second content vector and segment-level content vector.

[0150] The second content vector can be calculated using the following formula 2:

[0151] Formula 2;

[0152] in, This represents the second content vector. Represents the first content vector. Let represent the frame-level content vector of the j-th audio frame. , .

[0153] This segment-level content vector can be obtained by combining the first and second content vectors using the following formula 3, for example, by concatenating them:

[0154] Formula 3;

[0155] in, Represents a segment-level content vector. This represents the second content vector. This represents the first content vector.

[0156] As can be seen, in the embodiments of this disclosure, in the process of obtaining the audio content features and audio emotion features of the original audio, the audio content features of the original audio are obtained first, and then a segment-level content vector corresponding to the original audio, which can simultaneously represent the baseline emotion and emotion intensity of the original audio within a preset time period, is obtained based on the audio content features. The audio emotion features of the original audio are then obtained based on the segment-level content vector. Subsequently, the expression coefficient sequence of the digital human is determined according to the audio content features, the audio emotion features, and the digital human's identity identifier. The target video is generated according to the expression coefficient sequence and the digital human's appearance reference image. This allows the expression of the digital human in the target video to change with the emotion changes of the original audio, thereby better adapting to the interactive scenario and improving the interactive experience of the interactive object.

[0157] In some embodiments, the acquisition of audio content features and audio sentiment features of the original audio in step S12 can be achieved by inputting the acoustic features of the original audio into a target content extraction model, and then performing audio content encoding and audio sentiment prediction processing on the original audio based on the target content extraction model to obtain the audio content features and audio sentiment features.

[0158] The target content extraction model may include a first audio encoder, a first pooling layer, and a first sentiment predictor; wherein, the first audio encoder can be used to encode and generate content vectors, and the first pooling layer can be used to perform pooling processing on multiple content vectors.

[0159] For example, the acoustic features of the original audio can be input into the first audio encoder to obtain the audio content features of the original audio; and, based on the first pooling layer, the audio content features of the original audio and the multi-frame historical content vector corresponding to the original audio are pooled to obtain the above-mentioned segment-level content vector; and the segment-level content vector is processed by the first sentiment predictor to obtain the audio sentiment features of the original audio.

[0160] Please refer to Figure 8 This is a flowchart of the target content extraction model provided in the embodiments of this disclosure. For example... Figure 8 As shown, the target content extraction model can be obtained through the following steps S81-S83:

[0161] Step S81: Obtain the second sample audio and its text and sentiment annotations.

[0162] The second sample audio can be any audio clip, and its duration can be set as needed, for example, it can be a 30-second audio clip. It is understood that the second sample audio can include multiple audio frames. In this embodiment, to ensure that the trained target content extraction model has strong generalization ability, the second sample audio can be audio corresponding to various application scenarios, such as e-commerce customer service, game voice-over, etc.

[0163] This text annotation is the annotation information corresponding to the audio content of the second sample audio.

[0164] This sentiment annotation is the annotation information corresponding to the audio sentiment of the second sample audio.

[0165] Step S82: Train the content recognition model based on the second sample audio, text annotation, and sentiment annotation to obtain a converged content recognition model.

[0166] This content recognition model is used to perform audio content recognition and emotion recognition processing.

[0167] Step S83: Based on the converged content recognition model, obtain the target content extraction model.

[0168] That is, after the content recognition model is trained based on step S82, the second audio decoder and the output layer used to output predicted audio sentiment in the converged content recognition model can be removed to obtain the target content extraction model in this application. The second audio decoder can be used to decode the audio content features to output predicted text representing the audio content.

[0169] Please refer to Figure 9 and Figure 10 These are, respectively, a flowchart of the content recognition model training process provided in the embodiments of this disclosure, and a schematic diagram of the content recognition model training process provided in the embodiments of this disclosure. The following, in conjunction with... Figure 9 and Figure 10 This further explains how to train a convergent content recognition model, and then, based on this convergent content recognition model, obtain a target content extraction model that can be used to simultaneously acquire audio content features and audio sentiment features of the original audio. For example... Figure 9 As shown, the content recognition model may include a second audio encoder, a second audio decoder, a second pooling layer, and a second sentiment predictor; the second sample audio may include multiple audio frames.

[0170] Specifically, the content recognition model can be constructed by adding a branch processing layer for sentiment prediction after the audio encoder to the speech recognition model (e.g., the U2++ model) consisting of an audio encoder and an audio decoder. The branch processing layer can include a second pooling layer and a second sentiment predictor. The second pooling layer can be used to pool multiple frame-level content vectors, and the second sentiment predictor can be used to perform sentiment prediction processing to output the predicted audio sentiment of the second sample audio. Here, the frame-level content vector is a feature vector used to represent the audio content of each audio frame.

[0171] like Figure 9 As shown, step S82, which involves training the content recognition model based on the second sample audio, text annotation, and sentiment annotation to obtain a converged content recognition model, may include the following steps S91-S96.

[0172] Step S91: Input the acoustic features of the second sample audio into the second audio encoder to obtain multiple sample frame-level content vectors corresponding to multiple audio frames.

[0173] The acoustic features of this sample can be the feature data obtained after acoustic feature extraction from the second sample audio. The extraction method is explained in the relevant sections above and will not be repeated here. For example, the acoustic features of this sample can be the MFCC or FBank features of the second sample audio.

[0174] The second audio encoder can be used to encode the acoustic features of the samples to output sample frame-level content vectors corresponding to each audio frame in the second sample audio. Each sample frame-level content vector (i.e., sample frame-level content features) is used to represent the audio content of the corresponding audio frame.

[0175] like Figure 10 As shown, given the acoustic features of the second sample audio corresponding to 0-T audio frames (understandably, each audio frame corresponds to a different time), the sample acoustic features input to the second audio encoder can be... Figure 10 The output of the second audio encoder, given x0, x1, x2, ..., xT, can be a frame-level content vector e0, e1, e2, ..., eT corresponding to the acoustic features of each sample, where T > 0.

[0176] Step S92: Input multiple sample frame-level content vectors into the second audio decoder to obtain the predicted text of the second sample audio.

[0177] The second audio decoder can decode multiple sample frame-level content vectors, converting the frame-level content vectors into corresponding text, and forming a sequence such as... Figure 10The predicted text is composed of the text sequences t0, t1, t2, ..., tN, where N > 0. It should be noted that since not every audio frame represents different audio content, the number of words N in the text sequence obtained by recognizing the second sample audio is usually not consistent with the number of audio frames T in the sample audio.

[0178] Step S93: Input multiple sample frame-level content vectors into the second pooling layer to obtain sample segment-level content vectors.

[0179] The sample segment-level content vector is used to represent the entire audio content of the second sample audio.

[0180] The second pooling layer can be used to convert multiple sample frame-level content vectors into sample segment-level content vectors, so as to represent the overall content information of the second sample audio based on the sample segment-level content vector. For example, the second pooling layer can use mean pooling, max pooling, or statistical pooling to obtain the sample segment-level content vector.

[0181] That is, since the emotions in audio are continuous, and a single audio frame corresponding to a single sample frame-level content vector often cannot represent the audio emotions, this embodiment of the present disclosure obtains a segment-level content vector that can represent the entire audio content of the second sample audio based on the second pooling layer, so as to predict the audio emotions of the second sample audio based on the segment-level content vector.

[0182] Step S94: Input the sample segment-level content vector into the second sentiment predictor to obtain the predicted audio sentiment.

[0183] The second sentiment predictor can be used to perform sentiment classification or regression processing based on the sample segment-level content vector to obtain predicted audio sentiment.

[0184] Step S95: Obtain a second loss value based on the predicted text and text annotations; and obtain a third loss value based on the predicted audio sentiment and sentiment annotations.

[0185] This second loss value can be used to measure the difference between the predicted text and the text annotation, and can be calculated using, for example, the cross-entropy loss function or other loss functions.

[0186] This third loss value can be used to represent the difference between the predicted audio sentiment and the sentiment label, and can be calculated using, for example, the cross-entropy loss function or the mean squared error loss function.

[0187] Step S96: Adjust the parameters of the second audio encoder, the second audio decoder, the second pooling layer, and the second sentiment predictor based on the second and third loss values ​​to obtain a converged content recognition model.

[0188] After obtaining the second and third loss values, the total loss value of the model can be determined based on the second and third loss values. Based on the total loss value, the parameters of the second audio encoder, the second audio decoder, the second pooling layer, and the second sentiment predictor in the content recognition model can be adjusted to obtain a converged content recognition model.

[0189] The total loss value can be calculated based on the following formula 1:

[0190] Formula 4;

[0191] in, This represents the total loss value. This represents the second loss value. This represents the third loss value. This is the weight value for the second loss value, and its value can be set as needed.

[0192] In some embodiments, step S93, which involves inputting multiple sample frame-level content vectors into a second pooling layer to obtain sample segment-level content vectors, may include: performing mean pooling on the multiple sample frame-level content vectors based on the second pooling layer to obtain a first sample content vector; performing standard deviation pooling on the multiple sample frame-level content vectors and the first sample content vector based on the second pooling layer to obtain a second sample content vector; and obtaining a segment-level sample content vector based on the first sample content vector and the second sample content vector.

[0193] Specifically, the first sample content vector can be calculated, for example, using the following formula 5:

[0194] Formula 5;

[0195] in, This represents the content vector of the first sample. The second sample audio represents the first... Sample frame-level content vectors of audio frames. T represents the total number of audio frames of the second sample audio.

[0196] The second sample content vector can be calculated using the following formula 6:

[0197] Formula 6;

[0198] in, This represents the content vector of the second sample. This represents the content vector of the first sample. The second sample audio represents the first... Sample frame-level content vectors of audio frames. T represents the total number of audio frames of the second sample audio.

[0199] In some embodiments, the segment-level sample content vector is obtained based on the first sample content vector and the second sample content vector, for example, by concatenating the first sample content vector and the second sample content vector. Specifically, the segment-level sample content vector can be calculated using the following formula 7:

[0200] Formula 7;

[0201] in, This represents a segment-level sample content vector. This represents the content vector of the second sample. This represents the content vector of the first sample.

[0202] It should be noted that the reason for pooling multiple sample frame-level content vectors based on the second pooling layer during the prediction of sample audio sentiment is that the sentiment of audio is often not accurately identified in a single frame, i.e., sentiment is usually persistent. Therefore, in this embodiment, the multiple sample frame-level content vectors are first subjected to mean pooling through the second pooling layer to obtain a first sample content vector that can represent the baseline sentiment of the sample audio segment. Then, considering that even for the same sentiment, the sentiment intensity, i.e., the sentiment fluctuation, is different at different times in the audio, this embodiment also obtains the second sample content vector by acquiring the second pooling layer of the multiple frames to represent the sentiment intensity of the audio at different times. This allows the obtained segment-level sample content vector to simultaneously represent the baseline sentiment and sentiment intensity at different times in the second sample audio. Based on the segment-level content vector, sentiment type prediction can be performed, enabling the second audio predictor to accurately obtain the sample sentiment features that can represent the audio sentiment of the second sample audio, thereby improving the accuracy of the predicted audio sentiment based on the sample sentiment features.

[0203] In some embodiments, such as Figure 10 The second emotion predictor can be a multilayer perceptron, and the output layer of the multilayer perceptron can be a softmax layer. The multilayer perceptron can perform nonlinear feature transformation and dimensionality transformation on the segment-level sample content vector to obtain sample emotion features with dimensions consistent with the dimension of the sample frame-level content vector based on its hidden layers. Then, the sample emotion features are classified based on the softmax layer to output the sample audio emotion.

[0204] In this embodiment, the converged content recognition model may include a second audio encoder, a second pooling layer, and a multilayer perceptron for performing sentiment feature prediction processing; the step S83 of obtaining the target content extraction model based on the converged content recognition model may include: determining the second audio encoder in the converged content recognition model as a first audio encoder, determining the second pooling layer as a first pooling layer, and determining the multilayer perceptron with the output layer removed as a first sentiment predictor.

[0205] That is, after obtaining the converged content recognition model, its first audio decoder can be removed, and the output layer of the multilayer perceptron used to perform sentiment prediction, such as the softmax layer, can be removed to obtain the target content extraction model in this embodiment of the present disclosure. Based on the target content extraction model, the audio content features and audio sentiment features of the original audio can be obtained simultaneously, thereby generating a target video containing a digital human. Based on the target video, the interactive object can respond and interact, thereby improving the interactive experience of the interactive object.

[0206] It should be noted that, in some embodiments, in step S81, obtaining the sentiment annotation of the second sample audio may include: inputting the second sample audio and preset prompt words into a language model to obtain sentiment annotation; wherein, the preset prompt words are used to prompt the language model to perform sentiment annotation prediction processing.

[0207] This language model can be a large language model used for multimodal processing. The preset prompt could be, for example, "You are an audio annotation assistant, outputting strict JSON based on the received audio content."

[0208] By inputting the second sample audio and the preset prompt word into a language model for sentiment annotation prediction, sentiment annotations can be obtained in the following form:

[0209] {

[0210] "speaker_gender": "male|female|unknown", / / speaker's gender

[0211] "emotion_primary": "neutral|happy|sad|angry|tense|excited|other", / / emotion category

[0212] "emotion_intensity": "low|medium|high", / / emotional intensity

[0213] "confidence": … / / confidence level

[0214] }

[0215] Of course, the above sentiment labeling is only an example. In actual implementation, the sentiment labeling may only include the sentiment type of the audio. This disclosure does not impose any special limitations on this.

[0216] As can be seen, the method provided in this disclosure, which automatically annotates the second sample audio with sentiment based on a preset prompting language model, can save a lot of manpower, improve the convenience of model training, and also ensure accuracy.

[0217] The above provides an illustrative description of how the target content extraction model is obtained according to the embodiments of this disclosure. It should be noted that in actual implementation, this target content extraction model can also be based on other training methods and other models that can simultaneously extract audio content features and audio emotional features of audio; no special limitations are made here.

[0218] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0219] In addition, this disclosure also provides digital human interaction devices, electronic devices, and computer-readable storage media, all of which can be used to implement any of the digital human interaction methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0220] Figure 11 This is a block diagram of a digital human interaction device provided in an embodiment of the present disclosure.

[0221] Reference Figure 11 This disclosure provides a digital human interaction device, which includes: a first acquisition module 110, a second acquisition module 111, a generation module 112, and an interaction module 113.

[0222] The first acquisition module 110 is used to acquire the raw audio to be processed from the audio stream, which is used to respond to the query content of the interactive object.

[0223] The second acquisition module 111 is used to acquire the audio content features and audio emotional features of the original audio.

[0224] The generation module 112 is used to generate a target video containing the digital human based on the audio content features, the audio emotional features, and the digital human's identity identifier.

[0225] The interaction module 113 is used to respond to the interaction object based on the target video.

[0226] In some embodiments, when the second acquisition module 111 acquires the audio content features and audio emotional features of the original audio, it may be used to: perform audio content encoding processing on the original audio to obtain the audio content features; and obtain the audio emotional features based on the audio content features.

[0227] In some embodiments, when the second acquisition module 111 obtains the audio sentiment feature based on the audio content feature, it may be used to: acquire a multi-frame historical content vector corresponding to the original audio; wherein the multi-frame historical content vector corresponds to a plurality of historical audios, and the plurality of historical audios includes a plurality of audios in the audio stream preceding the original audio; perform pooling processing on the audio content feature and the multi-frame historical content vector to obtain a segment-level content vector; wherein the segment-level content vector represents the audio content of the original audio within a preset time period; and perform audio sentiment prediction processing on the segment-level content vector to obtain the audio sentiment feature.

[0228] In some embodiments, when the second acquisition module 111 performs pooling processing on the audio content features and the multi-frame historical content vector to obtain a segment-level content vector, it can be used to: perform mean pooling processing on the audio content features and the multi-frame historical content vector to obtain a first content vector; perform standard deviation pooling processing on the audio content features, the multi-frame historical content vector, and the first content vector to obtain a second content vector; and obtain the segment-level content vector based on the first content vector and the second content vector.

[0229] In some embodiments, when generating a target video containing the digital human based on the audio content features, the audio emotional features, and the digital human's identity, the generation module 112 may be used to: generate an expression coefficient sequence based on the audio content features, the audio emotional features, and the identity; wherein the expression coefficient sequence is used to define the facial expression morphology of the digital human in the target video; and generate the target video based on the expression coefficient sequence and the digital human's appearance reference image.

[0230] In some embodiments, when generating an expression coefficient sequence based on the audio content features, the audio emotion features, and the identity identifier, the generation module 112 may be used to: perform vectorization processing on the identity identifier to obtain a first embedding vector; sample noise data from a preset data distribution to obtain initial expression coefficients for defining the facial expression morphology of the digital human; and perform an integral transformation on the initial expression coefficients based on the target flow field, using the audio content features, the audio emotion features, and the first embedding vector as constraints, to obtain the expression coefficient sequence.

[0231] In some embodiments, the device further includes a target flow field acquisition module, which can be used to: acquire a first sample audio, a sample identity identifier of a sample digital human, and an expression coefficient annotation sequence, wherein the first sample audio corresponds to m time points, m>0; acquire sample audio content features and sample audio emotion features at the m time points of the first sample audio; perform vectorization processing on the sample identity identifier to obtain a second embedding vector; concatenate the second embedding vector with the audio content features and audio emotion features corresponding to each time point to obtain m sample content emotion identity vectors corresponding to the m time points; and use the m sample content emotion identity vectors as constraints to integrate and transform the noise data sampled from the preset data distribution to the expression coefficient annotation sequence to obtain the target flow field.

[0232] In some embodiments, when generating the target video based on the expression coefficient sequence and the appearance reference image of the digital human, the generation module 112 may be used to: input the expression coefficient sequence and the appearance reference image into a rendering model to obtain multiple video frames containing the digital human; and generate the target video based on the multiple video frames.

[0233] Figure 12 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0234] Reference Figure 12 This disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702; and one or more I / O interfaces 703 connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 to enable the at least one processor 701 to perform the above-described digital human interaction method.

[0235] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned digital human interaction method. The computer-readable storage medium may be volatile or non-volatile.

[0236] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described digital human interaction method.

[0237] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0238] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0239] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0240] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0241] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0242] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0243] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0244] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0245] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0246] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A digital human interaction method, characterized in that, include: Obtain the raw audio to be processed from the audio stream, which is used in response to the query content of the interactive object; Obtain the audio content features and audio emotional features of the original audio; Based on the audio content features, the audio emotional features, and the digital human's identity, a target video containing the digital human is generated; The interaction is based on the target video and the interactive object.

2. The method according to claim 1, characterized in that, The acquisition of audio content features and audio emotional features of the original audio includes: The original audio is subjected to audio content encoding processing to obtain the audio content features; The audio emotional features are obtained based on the audio content features.

3. The method according to claim 2, characterized in that, The step of obtaining the audio emotional features based on the audio content features includes: Obtain the multi-frame historical content vector corresponding to the original audio; wherein, the multi-frame historical content vector corresponds to multiple historical audios, and the multiple historical audios include multiple audios in the audio stream that are preceding the original audio; The audio content features and the multi-frame historical content vectors are pooled to obtain a segment-level content vector; wherein, the segment-level content vector represents the audio content of the original audio within a preset time period; The segment-level content vector is subjected to audio sentiment prediction processing to obtain the audio sentiment features.

4. The method according to claim 3, characterized in that, The step of pooling the audio content features and the multi-frame historical content vectors to obtain segment-level content vectors includes: The audio content features and the multi-frame historical content vector are subjected to mean pooling to obtain the first content vector. The audio content features, the multi-frame historical content vector, and the first content vector are subjected to standard deviation pooling to obtain the second content vector; The segment-level content vector is obtained based on the first content vector and the second content vector.

5. The method according to claim 1, characterized in that, The step of generating a target video containing the digital human based on the audio content features, the audio emotional features, and the digital human's identity includes: Based on the audio content features, the audio emotional features, and the identity identifier, an expression coefficient sequence is generated; wherein, the expression coefficient sequence is used to define the facial expression morphology of the digital human in the target video; The target video is generated based on the expression coefficient sequence and the appearance reference image of the digital human.

6. The method according to claim 5, characterized in that, The step of generating an expression coefficient sequence based on the audio content features, the audio emotional features, and the identity identifier includes: The identity identifier is vectorized to obtain a first embedding vector; Noise data is sampled from a preset data distribution to obtain initial expression coefficients for defining the facial expression morphology of the digital human; Using the audio content features, the audio emotion features, and the first embedding vector as constraints, the initial expression coefficients are integrally transformed based on the target flow field to obtain the expression coefficient sequence.

7. The method according to claim 6, characterized in that, The target flow field is obtained based on the following processing: Obtain the first sample audio, the sample identity identifier of the sample digital human, and the expression coefficient annotation sequence, wherein the first sample audio corresponds to m time points, m>0; Obtain the sample audio content features and sample audio emotion features at m time points of the first sample audio; The sample identity identifier is vectorized to obtain a second embedding vector; The second embedding vector is concatenated with the audio content feature and audio sentiment feature corresponding to each time moment to obtain m sample content sentiment identity vectors corresponding to the m time moments; Using the m sample content emotion identity vectors as constraints, the initial flow field learning will integrally transform the noise data sampled from the preset data distribution to the expression coefficient annotation sequence to obtain the target flow field.

8. The method according to claim 5, characterized in that, The step of generating the target video based on the expression coefficient sequence and the digital human's appearance reference image includes: The expression coefficient sequence and the appearance reference image are input into the rendering model to obtain a multi-frame video containing the digital human. The target video is generated based on the multiple video frames.

9. A digital human interaction device, characterized in that, include: The first acquisition module is used to acquire the raw audio to be processed from the audio stream, which is used to respond to the query content of the interactive object; The second acquisition module is used to acquire the audio content features and audio emotional features of the original audio. The generation module is used to generate a target video containing the digital human based on the audio content features, the audio emotional features, and the digital human's identity identifier; An interaction module is used to respond to and interact with the target video and the interaction object.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-8.

Images