Voice interaction method and device, medium and equipment

Abstract

The embodiment of the invention provides a voice interaction method and device, a medium and equipment, and the method comprises the steps: responding to an input voice, recognizing an input text corresponding to the input voice, generating an interaction text and an interaction emotion label which are matched with the input text through a large language model, and carrying out the recognition of the interaction emotion label based on a tone code extractor; and extracting a target timbre feature from the target timbre audio appointed by the user. And selecting a reference emotion audio matched with the interaction emotion label from a preset emotion audio library, extracting a target emotion feature corresponding to the reference emotion audio from the reference emotion audio through an emotion code extractor, fusing the target tone feature and the target emotion feature into a target interaction feature, and generating interaction voice in combination with the interaction text. By constructing feature decoupling extraction and feature fusion of the timbre and the emotion, flexible combination of the timbre specified by the user and any fine-grained emotion can be realized, so that the generated interactive voice can accurately restore the specified timbre, and can also transmit the specific emotion to meet the emotion resonance requirement.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a voice interaction method, device, medium and equipment. Background Technology

[0002] With the rapid development of artificial intelligence and digital media technologies, human-computer interaction has been widely applied in various fields such as intelligent customer service, elderly care, and virtual live streaming, becoming a key carrier connecting users with digital services. Meanwhile, in the process of interaction between users and digital avatars, users' needs for interactive experiences are no longer limited to basic functional implementation, but are gradually moving towards emotional resonance.

[0003] Currently, interactive speech is generally synthesized based on a fixed timbre library. However, the limited timbre selection can only meet the basic requirements for functional implementation and is insufficient to meet the needs for emotional resonance. Summary of the Invention

[0004] This application provides a voice interaction method, apparatus, medium, and device that can flexibly combine timbre and emotion to meet the needs of emotional resonance.

[0005] In a first aspect, embodiments of this application provide a voice interaction method, the method comprising: In response to received voice input, the input text of the voice input is identified; Using a large language model, interactive text and interactive sentiment tags corresponding to the input text are generated; Receive the target timbre audio specified by the user, and extract the target timbre features of the target timbre audio through a preset timbre encoder extractor; Select reference emotional audio that matches the interactive emotional tag from the preset emotional audio library; The target emotional features of the reference emotional audio are extracted using a preset emotional encoding extractor. The target emotional features and target timbre features are fused to obtain the target interaction features; Based on the interactive text and the target interactive features, generate interactive speech.

[0006] In one feasible implementation, based on the input text, an interactive text and interactive sentiment tags corresponding to the input text are generated using a large language model, including: The input speech is subjected to feature extraction to obtain input speech features; Based on the input text and the input speech features, determine the input emotion label of the input speech; Based on the input text and the input sentiment tag, the input text is semantically understood and sentiment adapted using a large language model to generate the interactive text and the interactive sentiment tag.

[0007] In one feasible implementation, the method further includes: Acquire multiple timbre audio samples and determine the real speaker label and real emotion label for each timbre audio sample; The sample timbre features of the timbre audio samples are extracted using a preset speaker encoder. Based on the timbre features of the sample, the predicted speaker label and the first emotion label of the timbre audio sample are determined by using a preset speaker classifier and a first emotion classifier, respectively. The speaker classification loss is determined based on the difference between the predicted speaker labels and the actual speaker labels; Based on the difference between the first sentiment label and the real sentiment label, a first sentiment classification loss is determined, and the gradient direction of the first sentiment classification loss is reversed to obtain the sentiment adversarial loss. With the goal of minimizing the speaker classification loss and the emotional adversarial loss, the speaker encoder is iteratively trained until a preset convergence condition is met. The speaker encoder that has been iteratively trained is then used as the timbre encoder extractor.

[0008] In one feasible implementation, the method further includes: The sample emotional features of the timbre audio samples are extracted using a preset initial emotion encoder. Based on the emotional features of the sample, a second emotional label for the timbre audio sample is determined using a preset second emotional classifier; The second sentiment classification loss is determined based on the difference between the second sentiment label and the true sentiment label; For each preset sentiment tag, the prototype vector of the preset sentiment tag is determined based on the sentiment features of the sample corresponding to the preset sentiment tag; The orthogonal prototype contrast loss is determined based on the difference between each prototype vector and the desired orthogonal state. With the goal of minimizing the second emotion classification loss and the orthogonal prototype contrast loss, the initial emotion encoder is iteratively trained until a preset convergence condition is met. The iteratively trained initial emotion encoder is then used as the emotion code extractor.

[0009] In one feasible implementation, the initial emotion encoder is iteratively trained with the objective of minimizing the second emotion classification loss and the orthogonal prototype contrast loss. The method further includes: For each of the timbre audio samples, the timbre features and the emotional features of the samples are fused to obtain a synthesized feature; Based on the preset reference text and the synthesis features, speech is synthesized, and preset watermark information is embedded in the synthesized speech to obtain a synthesized speech sample. The preset watermark information and the preset emotion tag have a one-to-one or one-to-many association relationship. The initial emotion encoder is used to extract the synthetic emotion features of the synthetic speech samples; Based on the difference between the synthesized emotional features and the sample emotional features, determine the emotional feature consistency loss; The initial emotion encoder is iteratively trained with the goal of minimizing the second emotion classification loss, the orthogonal prototype contrast loss, and the emotion feature consistency loss.

[0010] In one feasible implementation, the method further includes: Obtain the audio to be traced; The emotion encoding extractor is used to extract the source emotion features of the speech to be traced. The source-tracing emotional features are matched with at least one emotional feature corresponding to each of the preset emotional tags to obtain multiple matching degrees of the source-tracing emotional features; If any of the matching degrees is greater than a preset matching threshold, the speech to be traced is identified as historically generated speech.

[0011] In one feasible implementation, generating interactive speech based on the interactive text and the target interactive features includes: Based on the interactive text and the target interactive features, a Mel spectrogram is generated through pitch prediction, duration prediction, and spectrum decoding. The intersection frequency band between the Mel spectrogram and the preset hearing-insensitive frequency band is taken as the target frequency band; The preset watermark information corresponding to the interactive emotion tag is used as the target watermark information; The target watermark information is embedded in the target frequency band, and the interactive voice is obtained through a vocoder.

[0012] In one feasible implementation, after generating interactive speech based on the interactive text and the target interactive features, the method further includes: Based on the input text, interactive emoji tags and interactive action tags are generated using a large language model; The lip movement sequence is determined by predicting the key lip points of the interactive speech; The expressions corresponding to the interactive emoji tags and the actions corresponding to the interactive action tags are mapped onto digital avatars. The triggering time of the expressions, actions, and lip movement sequences is scheduled through a rhythm coordination scheduler to synchronously complete the interaction with the user.

[0013] Secondly, embodiments of this application provide a voice interaction device, the device comprising: The response module is used to identify the input text of the received input voice in response to the input voice. The interactive text module is used to generate interactive text and interactive sentiment tags corresponding to the input text through a large language model; The timbre encoding module is used to receive the target timbre audio specified by the user and extract the target timbre features of the target timbre audio through a preset timbre encoding extractor; The reference audio module is used to select reference emotional audio that matches the interactive emotional tag from a preset emotional audio library; The emotion encoding module is used to extract the target emotion features of the reference emotion audio through a preset emotion encoding extractor; The feature fusion module is used to fuse the target emotional features and the target timbre features to obtain the target interaction features; The interactive voice module is used to generate interactive voice based on the interactive text and the target interactive features.

[0014] Thirdly, embodiments of this application provide a voice interaction device, the device including: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement any of the above-mentioned voice interaction methods.

[0015] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement any one of the above-mentioned voice interaction methods.

[0016] Fifthly, the embodiments of this application provide a computer program product in which the instructions are executed by the processor of an electronic device, causing the electronic device to execute any one of the above-mentioned voice interaction methods.

[0017] This application discloses a voice interaction method, apparatus, medium, and device that can respond to input voice and recognize its corresponding input text. It generates interactive text and interactive emotion tags adapted to the input text using a large language model, and then extracts target timbre features from user-specified target timbre audio based on a timbre encoder extractor. Reference emotion audio matching the interactive emotion tags is selected from a preset emotion audio library, and the corresponding target emotion features are extracted from the reference emotion audio using an emotion encoder extractor. The target timbre features and target emotion features are fused into target interactive features, which are then combined with the interactive text to generate interactive voice. In other words, by constructing a feature decoupling extraction and feature fusion of timbre and emotion, a flexible combination of user-specified timbre and arbitrary fine-grained emotion can be achieved, enabling the generated interactive voice to accurately reproduce the specified timbre while conveying specific emotions, thus meeting the need for emotional resonance.

[0018] Furthermore, by embedding preset watermark information corresponding to interactive emotion tags during the generation of interactive speech, and combining it with consistency detection of emotion coding, accurate source tracing of synthesized speech can be achieved without affecting the authenticity of speech and emotional expression. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a voice interaction method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the principle of traceable speech synthesis provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of digital avatar voice interaction provided in the embodiments of this application; Figure 4 This is a schematic diagram of a voice interaction system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of this application. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] With the rapid development of artificial intelligence and digital media technologies, human-computer interaction has been widely applied in various fields such as intelligent customer service, elderly care, and virtual live streaming, becoming a key carrier connecting users with digital services. Meanwhile, in the process of interaction between users and digital avatars, users' needs for interactive experiences are no longer limited to basic functional implementation, but are gradually moving towards emotional resonance.

[0024] Currently, interactive speech is generally synthesized based on a fixed timbre library. However, the limited timbre selection can only meet the basic requirements for functional implementation and is insufficient to meet the needs for emotional resonance.

[0025] To address the problems in the prior art, embodiments of this application provide a voice interaction method, apparatus, device, and computer storage medium.

[0026] In practical applications, the execution subject of the voice interaction method in this application embodiment can be a terminal device, such as a desktop computer or laptop computer, or a remote device similar to a server. Of course, the execution subject in this application embodiment can also be a software entity, such as a client or software program installed on a terminal device. The specific type of execution subject corresponding to the technical solution provided in this application embodiment is not strictly limited here, and can be flexibly selected according to the actual application scenario and actual needs.

[0027] The following describes specific embodiments of a voice interaction method, apparatus, medium, and device provided in this application. First, a voice interaction method is introduced.

[0028] Figure 1 A flowchart illustrating a voice interaction method according to an embodiment of this application is shown. Figure 1 As shown, the method includes steps S100 to S106: S100: In response to the received input speech, identify the input text of the input speech.

[0029] In one or more embodiments of this application, in order to generate interactive text that conforms to the interactive context and user needs in subsequent steps, in this step, after receiving the input voice sent by the user, this application responds to the input voice by recognizing the input text of the input voice.

[0030] Specifically, this application uses speech recognition technology to recognize and process the input speech, converting the speech signal into a text sequence, thereby outputting input text that is consistent with the content of the input speech.

[0031] It should be noted that this application does not limit the specific method of speech recognition, which can be set according to actual needs, such as an end-to-end speech recognition model based on deep learning; or using convolutional neural networks to extract the spectral features of the input speech and combining them with recurrent neural networks or Transformer models to model the temporal features. This application does not limit the method of receiving the input speech, which can be set according to actual needs, such as using a high-sensitivity microphone array to acquire the input speech. Since the input speech may have accents, noise interference, or unclear pronunciation, in order to ensure the accuracy of the input text, this application can also perform preprocessing operations on the input speech after receiving it, such as noise reduction and voice enhancement; or introduce a text error correction model to correct, complete, and revise the recognition results, further improving the completeness and accuracy of the input text, and providing a reliable basis for subsequent large language models to generate interactive text and interactive sentiment tags.

[0032] S101: Generate interactive text and interactive sentiment tags corresponding to the input text using a large language model.

[0033] In one or more embodiments of this application, in order to generate interactive speech that conforms to the interactive context and can accurately convey emotions in subsequent steps, this application can generate interactive text and interactive emotion tags adapted to the input text in step S100 through a large language model, so as to provide a benchmark for the extraction of emotion features and speech synthesis.

[0034] Specifically, the input text identified in step S100 is input into a pre-trained large language model to understand the semantics, context, and intent / need of the input text, generating interactive text that matches the input text. Simultaneously, this application can also combine sentiment analysis of the input text (such as capturing sentiment words and modal particles in the input text through the large language model, or judging the dialogue scenario) to output interactive sentiment tags, ensuring the matching of interactive sentiment tags with the interactive text.

[0035] It should be noted that this application does not limit the specific type of large language model, which can be set according to actual needs, such as a dialogue model based on the Transformer architecture. Of course, when generating interactive sentiment tags, the large language model can support multi-dimensional sentiment classification, such as including sentiment intensity, to meet the fine-grained requirements of subsequent sentiment feature extraction. If the input text involves complex contexts (such as irony), the large language model can also correct sentiment tags through context (such as historical input text within a certain time period, or historical text of the current dialogue), avoiding misjudgment of sentiment and improving the accuracy of interactive sentiment tags. Since users' emotional expression is generally a combination of text semantics and speech prosody, this application can also determine the input speech features of the input semantics, and then identify the user's input sentiment tags based on these input speech features. That is, in one or more embodiments of this application, this application can also determine the input sentiment tags of the input speech, and use these input sentiment tags to assist the large language model in determining interactive sentiment tags, improving the accuracy of interactive sentiment tags, including: extracting features from the input speech to obtain input speech features, and then determining the input sentiment tags of the input speech based on the input text and input speech features. Based on the input text and the input sentiment tag, semantic understanding and sentiment adaptation are performed through a large language model to generate interactive text and interactive sentiment tags.

[0036] S102: Receive the target timbre audio specified by the user, and extract the target timbre features of the target timbre audio through a preset timbre encoder extractor.

[0037] In one or more embodiments of this application, in order to achieve the specified timbre setting of interactive voice in subsequent steps, and to generate interactive voice with a specified timbre and conveying the target emotional features through decoupling and fusion with the target emotional features, thereby meeting the user's emotional resonance needs, in this step, this application needs to separate and extract the target timbre features from the user-specified target timbre audio.

[0038] Specifically, this application can receive target timbre audio uploaded by users (such as local audio files, real-time recorded speech segments, or timbre audio selected from preset timbre templates). The received target timbre audio is input into a preset timbre encoding extractor, which can avoid interference from emotional information in the target timbre audio and extract the target timbre features.

[0039] It should be noted that this application does not limit the specific type of timbre encoding extractor, which can be set according to actual needs, such as ECAPA-TDNN. To ensure the accuracy of the timbre encoding extractor in extracting target timbre features from the target timbre audio and to avoid interference from emotional information in the target timbre audio, in one or more embodiments of this application, the timbre encoding extractor can be jointly trained based on speaker classification loss and emotional adversarial loss, as follows: First, multiple timbre audio samples are acquired, and the true speaker label and true emotion label for each timbre audio sample are determined. Then, the sample timbre features are extracted using a pre-defined speaker encoder. Based on the sample timbre features, the predicted speaker label and first emotion label for each timbre audio sample are determined using a pre-defined speaker classifier and a first emotion classifier, respectively. Second, the speaker classification loss is determined based on the difference between the predicted speaker label and the true speaker label; the first emotion classification loss is determined based on the difference between the first emotion label and the true emotion label, and the gradient direction of the first emotion classification loss is inverted to obtain the emotion adversarial loss. Finally, the speaker encoder is iteratively trained with the goal of minimizing the speaker classification loss and the emotion adversarial loss until a pre-defined convergence condition is met. The iteratively trained speaker encoder is then used as the timbre encoding extractor.

[0040] In this embodiment, this application optimizes the speaker encoder using a bidirectional constraint training logic through speaker classification loss and emotion adversarial loss, resulting in a timbre encoder extractor that combines timbre discernibility (identifying speaker identity) with emotion independence. During this bidirectional constraint training process, the speaker classification loss serves as a positive constraint, measuring the difference between predicted and actual speaker labels to constrain the speaker encoder to focus on timbre features representing speaker identity in the timbre audio, ensuring that the extracted timbre features can accurately distinguish different speakers. The emotion adversarial loss serves as a negative constraint, transforming emotion classification accuracy into an adversarial objective through gradient flipping. When the first emotion classifier predicts emotion labels based on timbre features, the inverted gradient causes the timbre encoder extractor to learn to strip away the emotional information carried in the timbre audio, achieving decoupling of timbre features and emotion features. Meanwhile, this application does not limit the specific method of determining the loss. It can be set according to actual needs. For example, the speaker classification loss can adopt cross-entropy loss, and the emotion adversarial loss can achieve gradient inversion based on the cross-entropy loss or mean square error loss of the first emotion classification loss. Furthermore, the training tendency of the identity recognition of timbre features and the emotion irrelevance can be balanced by adjusting the weight coefficients of the two losses (such as adopting a dynamic weight allocation strategy to adaptively adjust the weight ratio according to the loss convergence in the training phase).

[0041] This application employs a dual-loss collaborative training method for the timbre encoding extractor. During training, the timbre encoding extractor develops a bias that strengthens timbre features and weakens emotional characteristics. Speaker classification loss drives the timbre encoding extractor to continuously improve the speaker identity recognition of timbre features, while emotional adversarial loss inhibits the timbre encoding extractor's ability to capture emotional features. This dual-loss method ensures that the trained timbre encoding extractor can accurately extract target timbre features and filter out emotional information when processing user-specified target timbre audio. This avoids problems such as timbre interference with emotional information or emotional information masking timbre when subsequently fused with target emotional features.

[0042] S103: Select a reference emotional audio that matches the interactive emotional tag from the preset emotional audio library.

[0043] In one or more embodiments of this application, in order to accurately extract the target emotional features corresponding to the interactive emotional tags in subsequent steps, inject emotional expressions that fit the scene into the interactive voice, and ensure that the generated voice retains the target timbre while conveying specific emotions, this application first needs to select reference emotional audio that matches the interactive emotional tags in this step.

[0044] It should be noted that the preset emotional audio library stores audio with clear emotional tags (the granularity of the preset emotional tags can be set according to actual needs), and reference emotional audio that matches the emotional tag of the interaction can be filtered out from the emotional audio library.

[0045] S104: Extract the target emotional features of the reference emotional audio using a preset emotional encoding extractor.

[0046] In one or more embodiments of this application, in order to accurately fuse with the target timbre features in subsequent steps so that the generated interactive speech can convey an emotional tendency consistent with the interactive emotional label, it is necessary to extract the target emotional features from the reference emotional audio.

[0047] Specifically, the selected reference emotional audio is input into a preset emotion encoding extractor. This extractor extracts emotion-related features (such as intonation, speech rate, and fundamental frequency dynamic range) from the reference audio, captures and encodes them, removes interference from timbre information, and extracts the target emotional features that reflect the emotional information. For example, for reference emotional audio labeled "anger," the emotion encoding extractor focuses on emotional features such as fast speech rate, loud volume, and dramatic fundamental frequency fluctuations to ensure that these features accurately represent the intensity and style of anger.

[0048] It should be noted that this application does not limit the specific type of emotion encoding extractor, which can be set according to actual needs, such as the emotion2vec speech emotion encoding model. To ensure the accuracy of the emotion encoding extractor in extracting target emotion features from reference emotion audio and to avoid interference from timbre information in the reference emotion audio, in one or more embodiments of this application, the emotion encoding extractor can be jointly trained based on a second emotion classification loss and an orthogonal prototype contrast loss, as detailed below: First, the sample emotional features of the timbre audio samples are extracted through a preset initial emotional encoder. Then, based on the sample emotional features, the second emotional label of the timbre audio samples is determined through a preset second emotional classifier.

[0049] Secondly, based on the difference between the second sentiment label and the true sentiment label, the second sentiment classification loss is determined. For each preset sentiment label, based on the sentiment features of the samples corresponding to that preset sentiment label, the prototype vector of that preset sentiment label is determined, and based on the difference between each prototype vector and the expected orthogonal state, the orthogonal prototype contrast loss is determined.

[0050] Finally, with the goal of minimizing the second sentiment classification loss and the orthogonal prototype contrast loss, the initial sentiment encoder is iteratively trained until the preset convergence condition is met. The initial sentiment encoder that has been iteratively trained is then used as the sentiment code extractor.

[0051] In this embodiment, the application does not limit the number and types of preset emotion labels, which can be set according to actual needs. For example, emotions can be divided into ten types, and the ten preset emotion labels can be happiness, excitement, calmness, comfort, surprise, anger, doubt, expectation, tension, and seriousness. The application does not limit the determination method of the second emotion classification loss and the orthogonal prototype contrast loss, which can be set according to actual needs. For example, the second emotion classification loss can use cross-entropy loss or binary cross-entropy loss, which constrains the classification accuracy of emotion features by measuring the difference in probability distribution between the second emotion label and the real emotion label. The orthogonal prototype contrast loss can be determined by calculating the deviation between the cosine similarity between each prototype vector and the ideal orthogonal state (similarity of 0). Simultaneously, the proportion of the two losses can be dynamically adjusted according to the training stage. The weights can be set according to actual needs, such as increasing the weight of the second emotion classification loss in the early stages of training to ensure accurate basic emotion classification, and increasing the weight of the orthogonal prototype contrast loss in the later stages of training to improve the ability to distinguish fine-grained emotions and optimize the overall performance of the emotion encoding extractor.

[0052] Furthermore, in one or more embodiments of this application, an emotional feature consistency loss may be introduced to further improve the stability and reconstruction capability of the emotional feature extractor, ensuring that emotional features are not distorted during fusion, speech synthesis, and other processes, as detailed below: First, for each audio sample, the sample's timbre features and emotional features are fused to obtain synthesized features. Based on a preset reference text (which can be set according to actual needs) and the synthesized features, speech is synthesized, and preset watermark information is embedded in the synthesized speech to obtain a synthesized speech sample. The preset watermark information and preset emotional tags have a one-to-one or one-to-many association relationship.

[0053] Secondly, synthetic emotional features are extracted from the synthesized speech samples using an initial emotion encoder. Based on the difference between the synthetic emotional features and the sample emotional features, an emotion feature consistency loss is determined. Then, the initial emotion encoder is iteratively trained with the goal of minimizing the second emotion classification loss, the orthogonal prototype contrast loss, and the emotion feature consistency loss.

[0054] This application does not limit the method for determining the consistency of emotional features; it can be set according to actual needs, such as cosine similarity. The design logic of the emotion encoding extractor complements that of the timbre encoding extractor. The role of the emotion encoding extractor is to strip away timbre and preserve emotion, ensuring that the extracted target emotional features are not affected by the timbre in the reference emotional audio. That is, this application can constrain the emotional features extracted by the emotion encoding extractor to accurately match the real emotional labels through the second emotion classification loss, and force the feature vectors of different emotional labels to be far apart and have clear boundaries in the feature space through the orthogonal prototype contrast loss, thereby improving the emotional discrimination and stripping away the timbre features that represent the speaker's identity, thus achieving the decoupling of speaker features (timbre features) and emotional features to support subsequent fusion with the specified timbre. Meanwhile, by introducing emotional feature consistency loss, the emotion coder extractor can be forced to learn the ability to accurately extract the original emotional features from the fused synthesized speech. This ensures that the emotional features can still be stably recognized and extracted after being fused with timbre features, speech synthesis, watermark embedding, and other processing. It avoids the loss or distortion of emotional features in the transmission link, and ensures that the accuracy and consistency of emotional expression are not affected when the target emotional features are fused with any specified timbre features to generate speech.

[0055] S105: The target emotional features and target timbre features are fused to obtain the target interaction features.

[0056] S106: Generate interactive speech based on the interactive text and the target interactive features.

[0057] In one or more embodiments of this application, in order to generate interactive speech with a user-specified timbre that matches the emotion of the interactive emotion tag, it is necessary to combine timbre features and emotion features through feature fusion to obtain interactive features, and then perform speech synthesis based on the interactive features and interactive text to obtain interactive speech. This ensures that the interactive speech retains the specified timbre features and can convey the emotional tendency that matches the interactive emotion tag, thereby improving the naturalness of the interaction and meeting the need for emotional resonance.

[0058] Specifically, this application employs a feature fusion network (such as attention mechanism fusion, feature concatenation, or adaptive weighted fusion) to fuse target emotional features and target timbre features to obtain target interaction features. The interactive text and target interaction features are then input into a speech synthesis model (such as an end-to-end TTS model based on Transformer or a Tacotron2 speech synthesis model). The speech synthesis model determines the semantic content and pronunciation sequence of the interactive speech based on the interactive text, and adjusts the timbre parameters (such as fundamental frequency and formants) and emotional parameters (such as speech rate, pitch, and volume) based on the target interaction features to synthesize interactive speech that is semantically consistent with the interactive text, possesses the target timbre, and conveys accurate emotions.

[0059] It should be noted that, based on the decoupling of timbre features and emotional features, this application can generate natural and fluent interactive speech even when faced with complex combinations of emotions and timbre through feature fusion and speech synthesis. This application does not limit the specific method of feature fusion; it can be set according to actual needs, such as concatenating the target timbre features and target emotional features into a target interactive feature at the feature dimension. To achieve speech source tracing, in one or more embodiments of this application, watermark information can be embedded during the synthesis of interactive speech, including: generating a Mel spectrogram based on the interactive text and target interactive features through pitch prediction, duration prediction, and spectrum decoding; then using the intersection frequency band of the Mel spectrogram with a preset auditory insensitive frequency band as the target frequency band; using the preset watermark information corresponding to the interactive emotional tag as the target watermark information; then embedding the target watermark information in the target frequency band; and obtaining the interactive speech through a vocoder.

[0060] Meanwhile, this application does not limit the specific method and content of watermark embedding, which can be set according to actual needs. For example, watermarks can be embedded in the low-frequency region (hearing-insensitive frequency band, 0-200Hz) of the Mel spectrogram based on generative adversarial networks. That is, by adjusting the spectral amplitude or phase in the hearing-insensitive frequency band (such as applying periodic perturbations at specific frequency points), the concealment of the watermark can be ensured. The content of the watermark information can include customized identifiers bound to preset emotion tags. Since the audio in the preset emotion audio library is embedded with watermark information, the emotion encoding extractor trained based on this proprietary data format and emotion feature consistency can be used for speech source tracing. In one or more embodiments of this application, under the constraint of emotion feature consistency in the process of emotion encoding extractor, this application can realize source tracing detection of historically generated speech based on the emotion encoding extractor, including: acquiring the speech to be traced, which is the speech that needs to be verified as historically generated. The trained emotion encoding extractor extracts features from the speech to be traced to obtain source emotion features. Because this application forces the emotion encoding extractor to form a stable feature extraction logic for speech with preset watermark information through emotion feature consistency loss during training, the source emotion features can be matched with at least one emotion feature corresponding to a preset emotion label (such as the prototype vector formed by each preset emotion label during training, or the emotion feature library of historically generated speech), and the similarity between the two (such as cosine similarity) can be calculated to obtain multiple matching degrees for the source emotion features. If any matching degree is greater than a preset matching threshold (such as a matching threshold that can be set based on the feature consistency distribution in the training data), then the speech to be traced is identified as speech generated by this application.

[0061] In this embodiment, the emotion encoding extractor trained with emotion feature consistency loss has unique recognizability for the speech generated by this application, and the extracted source emotion features highly match the emotion features corresponding to preset emotion tags. Therefore, source tracing of historically generated speech can be achieved through simple feature matching, and the copyright ownership of the speech can also be verified, ensuring the consistency and traceability of emotional expression.

[0062] Figure 2 The diagram shown illustrates the principle of traceable speech synthesis provided in this application. Figure 2As shown, this application inputs the target timbre audio into the timbre encoding extractor 201 to obtain the target timbre features; selects reference emotional audio corresponding to the interactive emotional tag from the emotional audio library 202, wherein all audio in the emotional audio library is embedded with preset watermark information through the watermark injection unit 203; inputs the reference emotional audio into the emotional encoding extractor 204 to obtain the target emotional features; combines the target timbre features and the target emotional features through the emotional and timbre fusion unit 205 to obtain the target interactive features; based on the target interactive features and the input text, generates a Mel spectrogram through the speech codec 206; embeds preset watermark information into the Mel spectrogram through the watermark injection unit, and obtains the interactive speech through the vocoder 207. During the training phase, the synthetic emotional features of the synthesized speech samples are extracted by the emotional encoding extractor, and then the consistency loss of emotional features is determined by comparing the synthetic emotional features with the sample emotional features to constrain the emotional encoding extractor.

[0063] In the aforementioned voice interaction method, this application converts input speech into input text, captures the user's emotions to determine the input emotion tag, and then uses a large language model to generate interactive text and interactive emotion tags that match the input text. At the level of personalized timbre and emotion fusion, an innovative timbre encoder extractor and emotion encoder extractor are designed to ensure the decoupling of emotion features and timbre features, thereby generating interactive speech that possesses both a specified timbre and accurate emotion. A watermark embedding and source tracing mechanism is introduced. By embedding preset watermark information into the Mel spectrogram and combining it with a proprietary trained emotion encoder, source tracing detection of the generated speech is achieved, balancing the naturalness of the interaction and content security.

[0064] Furthermore, this application does not limit the presentation method of voice interaction, which can be set according to actual needs. For example, multimodal interaction with users can be achieved through the voice, facial expressions, and actions of digital avatars to meet the user's need for emotional resonance. That is, in one or more embodiments of this application, this application can also generate interactive emoticon tags and interactive action tags through a large language model. By predicting the lip key points corresponding to the interactive voice, the lip action sequence is determined; the expressions corresponding to the interactive emoticon tags and the actions corresponding to the interactive action tags are mapped onto the digital avatar, and the triggering time of the expressions, actions, and lip action sequences is scheduled through a rhythm coordination scheduler to synchronously complete the interaction with the user.

[0065] To ensure that the large language model can generate interactive text, interactive sentiment tags, interactive emoticon tags, and interactive action tags based on the input text (and input sentiment tags), this application employs a fine-tuning approach to ensure that the large language model can generate quadruples composed of interactive text, interactive sentiment tags, interactive emoticon tags, and interactive action tags. Of course, this application does not limit the specific method of fine-tuning the large language model; for example, using partial input cases as a reference and adding prompts to constrain the output quadruples of the large language model, it ensures that the large language model can output interactive text, interactive sentiment tags, interactive emoticon tags, and interactive action tags as required.

[0066] This application provides some examples of generating interactive text, interactive sentiment tags, interactive emoji tags, and interactive action tags based on input text and input sentiment tags, as follows: Case 1: Input text: "How could he stand me up!"; Input sentiment tag: anger. Interactive text: "He went too far! How could he break his promise!"; Interactive sentiment tag: anger; Interactive emoji tag: anger; Interactive action tag: hands on hips.

[0067] Case 2: Input text: "I'm going to Xi'an for a trip soon"; Input emotion tag: anticipation. Interactive text: "Wow! That sounds exciting! Have a great trip!"; Interactive emotion tag: anticipation; Interactive emoji tag: anticipation; Interactive action tag: clapping.

[0068] Case 3: Input text: "My puppy has passed away."; Input emotion tag: sadness. Interactive text: "I can feel your pain, let me give you a hug."; Interactive emotion tag: comfort; Interactive emoji tag: concern; Interactive action tag: reach out and hug.

[0069] This application does not limit the relationships between the emotional tags, vocal prosody, facial expression tags, and action tags of digital avatars; these can be set according to actual needs. This application provides ten preset, fine-grained emotional tags, along with the corresponding vocal prosody, facial expression, and action for each preset emotional tag, as shown in the table below: Table 1 Figure 3 This is a schematic diagram illustrating the structure of digital avatar voice interaction provided in an embodiment of this application. Figure 3 As shown, this application achieves synchronized lip movements, emotional facial expressions, and coordinated control of actions in digital avatars by integrating voice-driven, expression-driven, and motion-driven technologies, as detailed below: S300: Input includes interactive voice, interactive emoji tags, and interactive action tags.

[0070] S301: The lip action sequence is obtained through lip key point prediction and optimization.

[0071] In the technical implementation of lip-sync, neural network models (such as KeySync) can be used to convert speech signals into lip movement sequences. The first stage predicts the key lip points corresponding to the speech, and the second stage optimizes temporal continuity and occlusion robustness to ensure that lip movements remain synchronized with speech even when facial expressions are complex or the speech is occluded.

[0072] S302: Using facial bone binding technology, the facial expressions corresponding to interactive emoji tags are mapped to facial muscle control points.

[0073] In the implementation of facial expression-driven technology based on expression tags, facial bone binding technology is used to map expression tags to muscle control points of the digital human face, achieving precise linkage between emotions and expressions.

[0074] S303: By binding predefined interactive action libraries with animations, digital avatars can synchronously execute the actions corresponding to the interactive action tags.

[0075] In terms of the technical implementation of hand motion control, based on a predefined gesture library, the tags are converted into specific hand skeletal animations.

[0076] S304: Through the rhythm coordination scheduler, the triggering time of facial expressions, actions, and lip movements is scheduled to synchronously complete the interaction with the user.

[0077] Integrating all signals, the timing of facial expressions and gestures is scheduled through a rhythm coordinator to ensure that the actions are coherent and consistent with the emotional tone of the voice.

[0078] S305: Output includes digital avatar interactive video, including voice, lip movements, facial expressions, motion, and emotional coordination.

[0079] Based on the aforementioned voice interaction method, this application also provides a specific embodiment of a voice interaction system. For example... Figure 4 As shown, Figure 4This is a schematic diagram of a voice interaction system provided in an embodiment of this application. The system includes a user interaction layer 401, an intelligent processing layer 402, a speech synthesis layer 403, and a digital avatar driving layer 404. The user interaction layer receives input speech and user-specified target voice audio. The intelligent processing layer includes a speech recognition module, a multimodal emotion judgment module, and a large language model generation module. The speech recognition module converts input speech into input text. The multimodal emotion judgment module obtains input emotion tags based on the input text and input speech features. The large language model generation module generates interactive text, interactive emotion tags, interactive expression tags, and interactive action tags based on context and interactive emotion tags. The speech synthesis layer includes a voice-customized emotion speech synthesis module (voice and emotion decoupling) and a dual traceability mechanism (emotion feature consistency detection and preset watermark information embedding). The digital avatar driving layer includes a multimodal digital avatar driving module (jointly driven by multi-dimensional information of voice, expression, and action).

[0080] Based on the aforementioned voice interaction method, this application also provides a specific embodiment of a voice interaction device.

[0081] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of this application. The device 500 includes a response module 501, an interactive text module 502, a timbre encoding module 503, a reference audio module 504, an emotion encoding module 505, a feature fusion module 506, and an interactive voice module 507.

[0082] The response module 501 is used to identify the input text of the received input voice in response to the input voice. The interactive text module 502 is used to generate interactive text and interactive sentiment tags corresponding to the input text through a large language model; The timbre encoding module 503 is used to receive the target timbre audio specified by the user and extract the target timbre features of the target timbre audio through a preset timbre encoding extractor; The reference audio module 504 is used to select reference emotional audio that matches the interactive emotional tag from a preset emotional audio library; The emotion encoding module 505 is used to extract the target emotion features of the reference emotion audio through a preset emotion encoding extractor; The feature fusion module 506 is used to fuse the target emotional features and the target timbre features to obtain the target interaction features; The interactive voice module 507 is used to generate interactive voice based on the interactive text and the target interactive features.

[0083] In one feasible implementation, the interactive text module is specifically used to extract features from the input speech to obtain input speech features; determine the input sentiment tag of the input speech based on the input text and the input speech features; and perform semantic understanding and sentiment adaptation processing on the input text through a large language model based on the input text and the input sentiment tag to generate the interactive text and the interactive sentiment tag.

[0084] In one feasible implementation, the above-mentioned device further includes a training module, specifically configured to acquire multiple timbre audio samples and determine the real speaker label and real emotion label for each timbre audio sample; extract sample timbre features of the timbre audio samples using a preset speaker encoder; determine the predicted speaker label and first emotion label of the timbre audio samples based on the sample timbre features using a preset speaker classifier and a first emotion classifier, respectively; determine the speaker classification loss based on the difference between the predicted speaker label and the real speaker label; determine the first emotion classification loss based on the difference between the first emotion label and the real emotion label, and obtain the emotion adversarial loss by reversing the gradient direction of the first emotion classification loss; iteratively train the speaker encoder with the goal of minimizing the speaker classification loss and the emotion adversarial loss until a preset convergence condition is met, and use the iteratively trained speaker encoder as the timbre encoding extractor.

[0085] In one feasible implementation, the training module can also be used to extract sample emotional features of the timbre audio samples using a preset initial emotional encoder; determine a second emotional label of the timbre audio samples using a preset second emotional classifier based on the sample emotional features; determine a second emotional classification loss based on the difference between the second emotional label and the real emotional label; for each preset emotional label, determine the prototype vector of the preset emotional label based on the sample emotional features corresponding to the preset emotional label; determine an orthogonal prototype contrast loss based on the difference between each prototype vector and the expected orthogonal state; iteratively train the initial emotional encoder with the goal of minimizing the second emotional classification loss and the orthogonal prototype contrast loss; and use the iteratively trained initial emotional encoder as the emotional encoder extractor until a preset convergence condition is met.

[0086] In one feasible implementation, the training module can further be used to fuse the timbre features and the emotional features of each timbre audio sample to obtain synthetic features; synthesize speech based on a preset reference text and the synthetic features, and embed preset watermark information into the synthesized speech to obtain a synthesized speech sample, wherein the preset watermark information and the preset emotional label have a one-to-one or one-to-many association relationship; extract the synthetic emotional features of the synthesized speech sample through the initial emotional encoder; determine the emotional feature consistency loss based on the difference between the synthesized emotional features and the sample emotional features; and iteratively train the initial emotional encoder with the goal of minimizing the second emotional classification loss, the orthogonal prototype contrast loss, and the emotional feature consistency loss.

[0087] In one feasible implementation, the above-mentioned device further includes a source tracing module, specifically used to acquire the speech to be traced; extract the source tracing emotional features of the speech to be traced through the emotion coder; match the source tracing emotional features with at least one emotional feature corresponding to each of the preset emotional tags to obtain multiple matching degrees of the source tracing emotional features; if any of the matching degrees is greater than a preset matching threshold, the speech to be traced is identified as historically generated speech.

[0088] In one feasible implementation, the interactive voice module is specifically used to generate a Mel spectrogram based on the interactive text and the target interactive features through pitch prediction, duration prediction, and spectrum decoding; take the intersection frequency band of the Mel spectrogram with a preset auditory insensitive frequency band as the target frequency band; take the preset watermark information corresponding to the interactive emotion tag as the target watermark information; embed the target watermark information in the target frequency band, and obtain the interactive voice through a vocoder.

[0089] In one feasible implementation, the interactive voice module can also be used to generate interactive emoji tags and interactive action tags based on the input text using a large language model; determine the lip action sequence by predicting the lip key points of the interactive voice; map the emojis corresponding to the interactive emoji tags and the actions corresponding to the interactive action tags onto digital images; and schedule the triggering time of the emojis, actions, and lip action sequences through a rhythm coordination scheduler to synchronously complete the interaction with the user.

[0090] Figure 6 A schematic diagram of the hardware structure of a voice interaction device provided in an embodiment of this application is shown.

[0091] A voice interaction device may include a processor 601 and a memory 602 storing computer program instructions.

[0092] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0093] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one embodiment, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be non-volatile solid-state memory. Memory 602 may be internal or external to the integrated gateway disaster recovery device.

[0094] In one instance, memory 602 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0095] Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the voice interaction method according to one aspect of this application.

[0096] The processor 601 reads and executes computer program instructions stored in the memory 602 to achieve... Figure 1 The voice interaction method in the illustrated embodiment.

[0097] In one example, a voice interaction device may further include a communication interface 603 and a bus 604. Wherein, as... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 604 and complete communication with each other.

[0098] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0099] Bus 604 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0100] Furthermore, in conjunction with one of the voice interaction methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the voice interaction methods in the above embodiments.

[0101] This application also provides a computer program product, including a computer program, which, when executed, implements any of the voice interaction methods described in the above embodiments.

[0102] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0103] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0104] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0105] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0106] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A voice interaction method, characterized in that, The method includes: In response to received voice input, the input text of the voice input is identified; Using a large language model, interactive text and interactive sentiment tags corresponding to the input text are generated; Receive the target timbre audio specified by the user, and extract the target timbre features of the target timbre audio through a preset timbre encoder extractor; Select reference emotional audio that matches the interactive emotional tag from the preset emotional audio library; The target emotional features of the reference emotional audio are extracted using a preset emotional encoding extractor. The target emotional features and target timbre features are fused to obtain the target interaction features; Based on the interactive text and the target interactive features, generate interactive speech.

2. The method according to claim 1, characterized in that, Based on the input text, an interactive text and interactive sentiment tags corresponding to the input text are generated using a large language model, including: The input speech is subjected to feature extraction to obtain input speech features; Based on the input text and the input speech features, determine the input emotion label of the input speech; Based on the input text and the input sentiment tag, the input text is semantically understood and sentiment adapted using a large language model to generate the interactive text and the interactive sentiment tag.

3. The method according to claim 1, characterized in that, The method further includes: Acquire multiple timbre audio samples and determine the real speaker label and real emotion label for each timbre audio sample; The sample timbre features of the timbre audio samples are extracted using a preset speaker encoder. Based on the timbre features of the sample, the predicted speaker label and the first emotion label of the timbre audio sample are determined by using a preset speaker classifier and a first emotion classifier, respectively. The speaker classification loss is determined based on the difference between the predicted speaker labels and the actual speaker labels; Based on the difference between the first sentiment label and the real sentiment label, a first sentiment classification loss is determined, and the gradient direction of the first sentiment classification loss is reversed to obtain the sentiment adversarial loss. With the goal of minimizing the speaker classification loss and the emotional adversarial loss, the speaker encoder is iteratively trained until a preset convergence condition is met. The speaker encoder that has been iteratively trained is then used as the timbre encoder extractor.

4. The method according to claim 3, characterized in that, The method further includes: The sample emotional features of the timbre audio samples are extracted using a preset initial emotion encoder. Based on the emotional features of the sample, a second emotional label for the timbre audio sample is determined using a preset second emotional classifier; The second sentiment classification loss is determined based on the difference between the second sentiment label and the true sentiment label; For each preset sentiment tag, the prototype vector of the preset sentiment tag is determined based on the sentiment features of the sample corresponding to the preset sentiment tag; The orthogonal prototype contrast loss is determined based on the difference between each prototype vector and the desired orthogonal state. With the goal of minimizing the second emotion classification loss and the orthogonal prototype contrast loss, the initial emotion encoder is iteratively trained until a preset convergence condition is met. The iteratively trained initial emotion encoder is then used as the emotion code extractor.

5. The method according to claim 4, characterized in that, The initial sentiment encoder is iteratively trained with the objective of minimizing the second sentiment classification loss and the orthogonal prototype contrast loss. The method further includes: For each of the timbre audio samples, the timbre features and the emotional features of the samples are fused to obtain a synthesized feature; Based on the preset reference text and the synthesis features, speech is synthesized, and preset watermark information is embedded in the synthesized speech to obtain a synthesized speech sample. The preset watermark information and the preset emotion tag have a one-to-one or one-to-many association relationship. The initial emotion encoder is used to extract the synthetic emotion features of the synthetic speech samples; Based on the difference between the synthesized emotional features and the sample emotional features, determine the emotional feature consistency loss; The initial emotion encoder is iteratively trained with the goal of minimizing the second emotion classification loss, the orthogonal prototype contrast loss, and the emotion feature consistency loss.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the audio to be traced; The emotion encoding extractor is used to extract the source emotion features of the speech to be traced. The source-tracing emotional features are matched with at least one emotional feature corresponding to each of the preset emotional tags to obtain multiple matching degrees of the source-tracing emotional features; If any of the matching degrees is greater than a preset matching threshold, the speech to be traced is identified as historically generated speech.

7. The method according to claim 1, characterized in that, Based on the interactive text and the target interactive features, generate interactive speech, including: Based on the interactive text and the target interactive features, a Mel spectrogram is generated through pitch prediction, duration prediction, and spectrum decoding. The intersection frequency band between the Mel spectrogram and the preset hearing-insensitive frequency band is taken as the target frequency band; The preset watermark information corresponding to the interactive emotion tag is used as the target watermark information; The target watermark information is embedded in the target frequency band, and the interactive voice is obtained through a vocoder.

8. The method according to claim 1, characterized in that, After generating interactive speech based on the interactive text and the target interactive features, the method further includes: Based on the input text, interactive emoji tags and interactive action tags are generated using a large language model; The lip movement sequence is determined by predicting the key lip points of the interactive speech; The expressions corresponding to the interactive emoji tags and the actions corresponding to the interactive action tags are mapped onto digital avatars. The triggering time of the expressions, actions, and lip movement sequences is scheduled through a rhythm coordination scheduler to synchronously complete the interaction with the user.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the voice interaction method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the voice interaction method as described in any one of claims 1-8.

Images