Dialogue voice generation method and device based on causal audio coding

Abstract

The invention relates to the technical field of speech synthesis, and particularly discloses a dialogue speech generation method and device based on causal audio coding. According to the method and the device, the short-frame audio is subjected to streaming transmission and coding, so that waiting for complete input of user voice is not needed, the response time is shortened, voice generation delay is avoided, the style representation is decoupled into the timbre feature vector and the rhythm feature vector, and the user experience is improved. The problem that the existing single coupling style vector is easy to cause confusion of timbre and rhythm features is solved, the timbre of the target speaker can be stably kept unchanged in multiple rounds of dialogues, meanwhile, a reasonable rhythm is preset according to historical information, and the quality of synthesized voice is improved. The method is applied to a voice dialogue system of financial services such as intelligent investment advisers, telephone customer service and risk confirmation and medical services such as intelligent inquiry, chronic disease management and mental health guidance, high-quality reply voice can be generated in real time according to voice input by a user, voice generation delay is avoided, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a method and apparatus for generating dialogue speech based on causal audio coding. Background Technology

[0002] With breakthroughs in generative artificial intelligence technology, voice dialogue systems are evolving from traditional, mechanical command-response modes to natural and fluent open-domain dialogue modes with deep contextual understanding capabilities. In financial services such as intelligent investment advisory, telephone customer service, and risk confirmation, as well as in medical scenarios such as intelligent consultation, chronic disease management, and mental health counseling, the demand for intelligent assistants capable of complex, continuous dialogue, low latency, and high quality is increasing. However, while existing end-to-end voice dialogue generation solutions have improved in terms of naturalness, they still suffer from high latency in voice generation and low quality in voice synthesis. Therefore, reducing voice generation latency and improving voice synthesis quality have become urgent problems to be solved. Summary of the Invention

[0003] This application provides a method and apparatus for generating dialogue speech based on causal audio coding, so as to reduce speech generation delay and improve speech synthesis quality.

[0004] In a first aspect, this application provides a dialogue speech generation method based on causal audio coding, the method comprising: Based on the audio acquisition module corresponding to the voice dialogue system, short frame sampling is performed on the user's input voice to obtain at least one short frame audio. Each of the short audio frames is streamed to a preset causal audio encoder for encoding to obtain an audio embedding vector corresponding to each of the short audio frames. The dialogue history information is obtained, and the dialogue history information and each audio embedding vector are processed based on the large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. Obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors, and the target timbre feature vector to obtain synthesized speech.

[0005] Secondly, this application also provides a dialogue speech generation apparatus based on causal audio coding, the apparatus comprising: The audio acquisition module is used to perform short-frame sampling of the user's input speech based on the audio acquisition module corresponding to the voice dialogue system, and obtain at least one short-frame audio. An audio encoding module is used to stream each of the short-frame audios to a preset causal audio encoder for encoding, thereby obtaining an audio embedding vector corresponding to each of the short-frame audios. The feature prediction module is used to acquire dialogue history information and process the dialogue history information and each audio embedding vector based on a large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. The speech synthesis module is used to obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, to perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors and the target timbre feature vector to obtain synthesized speech.

[0006] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the dialogue speech generation method based on causal audio coding as described above.

[0007] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the dialogue speech generation method based on causal audio coding as described above.

[0008] This application discloses a method and apparatus for generating dialogue speech based on causal audio coding. Based on an audio acquisition module corresponding to a voice dialogue system, the method performs short-frame sampling of user input speech to obtain at least one short-frame audio. Each short-frame audio is streamed to a preset causal audio encoder for encoding to obtain an audio embedding vector corresponding to each short-frame audio. Dialogue history information is acquired, and the dialogue history information and each audio embedding vector are processed based on a large language model to obtain a predicted text token and a predicted prosodic feature vector corresponding to each audio embedding vector. A target timbre feature vector corresponding to the target speaker is acquired, and based on a speech synthesis model, speech synthesis is performed on each predicted text token, each predicted prosodic feature vector, and the target timbre feature vector to obtain synthesized speech. This application reduces response time and avoids speech generation delay by streaming and encoding short-frame audio without waiting for the user's complete voice input. Secondly, it decouples style representation into timbre feature vector and prosodic feature vector, overcoming the problem that existing single coupled style vectors easily lead to confusion between timbre and prosodic features. In multi-turn dialogue, it can stably maintain the target speaker's timbre, and at the same time, it presets reasonable prosodic based on historical information, thus improving the quality of synthesized speech. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a first schematic flowchart of a dialogue speech generation method based on causal audio coding provided in an embodiment of this application; Figure 2 This is a second schematic flowchart of a dialogue speech generation method based on causal audio coding provided in an embodiment of this application; Figure 3 This is a third schematic flowchart of a dialogue speech generation method based on causal audio coding provided in an embodiment of this application; Figure 4 A schematic block diagram of a dialogue speech generation device based on causal audio coding, provided for embodiments of this application; Figure 5 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] This application provides a method and apparatus for generating dialogue speech based on causal audio coding. The method can be applied to a voice dialogue system or server. By streaming and encoding short audio frames, it eliminates the need to wait for complete user voice input, reducing response time and avoiding speech generation delays. Furthermore, by decoupling style representation into timbre feature vectors and prosodic feature vectors, it overcomes the problem of confusion between timbre and prosodic features caused by existing single-coupled style vectors. This ensures stable maintenance of the target speaker's timbre in multi-turn dialogues, while pre-setting reasonable prosodices based on historical information, improving the quality of synthesized speech. The server can be a standalone server or a server cluster; the voice dialogue system includes: An audio acquisition module is used to sample user input speech in short frames using an audio acquisition device corresponding to the voice dialogue system to obtain at least one short frame audio. The audio acquisition device may be a built-in device of the voice dialogue system or an external device that is communicatively connected to the voice dialogue system. The causal audio encoder module is used to encode the acquired short-frame audio to obtain the audio embedding vector corresponding to each short-frame audio. The large language model processing module is used to process the dialogue history information and each of the audio embedding vectors based on the large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each of the audio embedding vectors. The speech synthesis module is used to synthesize speech from each of the predicted text tokens, each of the predicted prosodic feature vectors, and the target timbre feature vector using the StyleTTS speech synthesis model, thereby obtaining synthesized speech.

[0016] In one embodiment, the voice dialogue system further includes a dual-channel style encoder module for extracting and encoding timbre and prosody features from user input speech to obtain timbre feature vectors and prosody feature vectors.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] Please see Figure 1 , Figure 1This is a schematic flowchart illustrating a dialogue speech generation method based on causal audio coding, provided in an embodiment of this application. This dialogue speech generation method based on causal audio coding can be applied to voice dialogue systems and servers. It reduces response time and avoids speech generation delays by streaming and encoding short audio frames without waiting for complete user voice input. Furthermore, it decouples style representation into timbre feature vectors and prosodic feature vectors, overcoming the problem of confusion between timbre and prosodic features caused by existing single-coupled style vectors. This ensures stable maintenance of the target speaker's timbre in multi-turn dialogues and improves the quality of synthesized speech by pre-setting reasonable prosodices based on historical information.

[0019] like Figure 1 As shown, the dialogue speech generation method based on causal audio coding specifically includes steps S101 to S104.

[0020] S101. Based on the audio acquisition module corresponding to the voice dialogue system, perform short frame sampling on the user's input voice to obtain at least one short frame audio. In one embodiment, the audio acquisition module may consist of audio acquisition devices such as a microphone, which acquires user input speech in real time and splits the continuous speech stream into short audio segments of a preset frame length through frame-level segmentation. For example, a frame length of 20ms / frame is set, and the acquired user input speech is segmented using this frame length to obtain at least one short audio frame. There is no need to wait for complete speech input before processing; short frame sampling can be performed at the preset frame length as soon as user speech input is detected, until the user finishes speaking.

[0021] In one embodiment, whether a user's speech has ended can be identified using sound activity detection or sentence end detection.

[0022] S102. Stream each of the short-frame audios to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short-frame audios. In one embodiment, short-frame audio is streamed to a causal audio encoder, which employs a causal attention mechanism to process the audio incrementally frame by frame and outputs a low-dimensional audio embedding vector for each short-frame audio.

[0023] Among them, streaming transmission refers to the continuous and real-time transmission of short-frame audio samples to a causal encoder for processing. It does not require waiting for all audio samples to be completed and can process short-frame audio samples in real time, thereby improving processing efficiency.

[0024] Further, the step of streaming each of the short-frame audios to a preset causal audio encoder for encoding to obtain an audio embedding vector corresponding to each of the short-frame audios includes: streaming each of the short-frame audios to the historical audio encoder, and when the causal audio encoder receives the short-frame audio, extracting acoustic features from the short-frame audio based on the causal convolutional layer of the causal audio encoder to obtain acoustic features; and encoding the acoustic features based on the causal self-attention encoding layer of the causal audio encoder to obtain an audio embedding vector corresponding to each of the short-frame audios.

[0025] In one embodiment, the causal audio encoder employs a Conformer-Causal architecture, comprising causal convolutional layers and causal self-attention encoding layers. The Conformer-Causal architecture introduces causal convolutions onto the standard Conformer (Convolution-augmented Transformer, a deep learning model that integrates convolutional neural networks and Transformer architectures) model to ensure that the model relies only on current and past context when processing audio sequences, thereby supporting low-latency real-time inference.

[0026] The causal audio encoder can process each short frame of audio it receives and generate the corresponding audio embedding vector without waiting for the complete audio input.

[0027] In a specific embodiment, the received short-frame audio is input into a causal convolutional layer to extract acoustic features (such as Mel spectrum and temporal features). The causal convolutional layer only uses the currently received short-frame audio and all previous short-frame audio information to extract acoustic features from the currently received short-frame audio, without any future dependence.

[0028] The extracted acoustic features are transmitted to a causal self-attention encoding layer for global temporal information integration. Specifically, for the current short frame audio, the causal self-attention only focuses on the acoustic features of the current short frame audio and the short frames audio preceding it, without any future information dependency. Through the causal self-attention encoding layer, the local acoustic features of the current short frame audio are self-attention encoded with historical global information to generate an audio embedding vector corresponding to the current short frame audio that includes historical information dependencies.

[0029] Furthermore, after streaming each of the short-frame audios to a preset causal audio encoder for encoding to obtain the audio embedding vectors corresponding to each of the short-frame audios, the method further includes: caching each of the audio embedding vectors to obtain a continuous audio embedding vector sequence; and transmitting the audio embedding vector sequence to the large language model when a preset condition is met.

[0030] In one embodiment, the audio embedding vectors generated by frame segmentation are stored in a sliding window queue, the queue length of which is dynamically adjusted in chronological order. Each received frame's audio embedding vector is immediately added to the end of the queue. When a preset condition is met, the accumulated embedding vectors in the current queue, i.e., the audio embedding vector sequence, are extracted and transmitted to the streaming input window of the large language model.

[0031] In one embodiment, the preset condition may be the detection of a keyframe, including a speech end frame, which can be detected using a voice activity detection and sentence end detection model. Specifically, a speech pause is detected by voice activity detection, and if the pause time is greater than a preset time threshold, it is determined that a keyframe has been detected. Alternatively, a sentence end detection model can be used to identify sentence end markers in the user's input speech in terms of grammar or semantics, such as interjections. If a sentence end marker is detected, it is determined that a keyframe has been detected.

[0032] In another embodiment, if no keyframe is detected, the transmission of the audio embedding vector sequence is forcibly triggered when the cumulative audio duration corresponding to the cached embedding sequence reaches a preset duration threshold, so as to avoid the user waiting too long.

[0033] S103. Obtain dialogue history information, and process the dialogue history information and each audio embedding vector based on the large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. In one embodiment, dialogue history information is loaded. This dialogue history information includes historical text transcriptions and historical prosodic feature vectors corresponding to previous rounds of dialogue in the current dialogue window. Each round of dialogue text transcription carries a corresponding confidence weight. It is understood that text transcription refers to the text obtained through speech conversion.

[0034] In one embodiment, the received dialogue history information and audio embedding vectors are streamed and decoded using a large language model to predict the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. Streaming and decoding means that the large language model can generate the predicted text token and predicted prosodic feature vector corresponding to the currently received audio embedding vector as soon as it receives the dialogue history information and audio embedding vector, without waiting for the complete audio embedding vector to be input into the large language model.

[0035] In one embodiment, the text composed of predicted text tokens corresponding to each audio embedding vector is the response text generated by the voice dialogue system based on the user's voice input.

[0036] S104. Obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors and the target timbre feature vector to obtain synthesized speech.

[0037] In one embodiment, the target timbre feature vector corresponding to the target speaker can be selected by the user, or matched by the voice dialogue system from a preset timbre database according to the user's preferences.

[0038] In a specific embodiment, a speech synthesis model is used to synthesize speech from each predicted text token, each predicted prosodic feature vector, and the target timbre feature vector to obtain synthesized speech.

[0039] In the above embodiments, by streaming and encoding short-frame audio, there is no need to wait for the user's complete voice input, which reduces the response time and avoids speech generation delay. Secondly, the style representation is decoupled into timbre feature vector and prosodic feature vector, which overcomes the problem that existing single coupled style vectors are prone to confusion between timbre and prosodic features. In multi-turn dialogue, the timbre of the target speaker can be kept unchanged. At the same time, reasonable prosodices are preset according to historical information, which improves the quality of synthesized speech.

[0040] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a dialogue speech generation method based on causal audio coding, provided in an embodiment of this application. This dialogue speech generation method based on causal audio coding can be applied to voice dialogue systems and servers. It improves the efficiency of updating dialogue history information through parallel processing, thereby increasing the efficiency of obtaining dialogue history information during dialogue speech synthesis and thus improving speech synthesis efficiency. Furthermore, by employing different historical information update strategies based on different confidence levels, it avoids the impact of erroneous transcription on historical information, thereby improving the quality of speech synthesis.

[0041] like Figure 2 As shown, after step S102, the dialogue speech generation method based on causal audio coding further includes steps S201 to S204.

[0042] S201. Based on the dual-channel style encoder corresponding to the voice dialogue system, extract and encode the timbre features and prosody features of each audio embedding vector to obtain the current timbre feature vector, the current prosody feature vector and the first confidence level. In one embodiment, the dual-channel style encoder includes two branches: a timbre extraction branch and a prosody extraction branch. The timbre extraction branch extracts timbre feature vectors, and the prosody extraction branch extracts prosodic feature vectors. Each audio embedding vector is streamed to the dual-channel style encoder and simultaneously transmitted to both the timbre extraction and prosody extraction branches. The timbre extraction and prosody extraction branches process the audio embedding vectors in parallel to obtain the current timbre feature vector and its confidence level, and the current prosodic feature vector and its confidence level. The confidence levels corresponding to the current timbre feature vector and the current prosodic feature vector are then weighted and fused to obtain a first confidence level.

[0043] Furthermore, the dual-channel style encoder includes a timbre extraction branch and a prosody extraction branch. The dual-channel style encoder corresponding to the voice dialogue system extracts and encodes timbre and prosody features from each audio embedding vector to obtain a current timbre feature vector, a current prosody feature vector, and a first confidence level. This includes: extracting and encoding timbre features from the streaming audio embedding vector based on the timbre extraction branch to obtain the timbre feature vector and a third confidence level; extracting and encoding prosody features from the streaming audio embedding vector based on the prosody extraction branch to obtain the prosody feature vector and a fourth confidence level; and weighting and fusing the third confidence level and the fourth confidence level to obtain the first confidence level.

[0044] In one embodiment, the timbre extraction branch can use a pre-trained speaker embedding model (such as ECAPA-TDNN, a deep learning-based voiceprint recognition model) to process each audio embedding vector and output the current timbre feature vector corresponding to each audio embedding vector. The current timbre feature vector is a static vector representing the speaker's identity, and is updated only with a small step size every 5 sentences of dialogue or when a speaker switch is detected (to avoid fluctuations), supporting pre-computation caching. Specifically, when the timbre extraction branch receives the first audio embedding vector, it immediately begins timbre extraction. If the matching degree between the extracted timbre and the timbre features of the historical speaker reaches a preset matching degree threshold, the timbre extraction ends, and the timbre feature vector of the historical speaker is used as the current timbre feature vector, reducing unnecessary feature extraction processes. If the extracted timbre has no matching historical speaker timbre features, it continues to receive subsequent audio embedding vectors and performs timbre extraction one by one, updating the previously obtained timbre features with the timbre features of the latest audio embedding vector, until the last audio embedding vector is processed, and the current timbre feature vector is obtained.

[0045] The prosodic extraction branch can employ a CNN (Convolutional Neural Network) + LSTM (Long Short-Term Memory) temporal modeling structure to output a short-term dynamic prosodic feature vector, containing information such as emotion and speech rate. The prosodic extraction branch extracts prosodic features from the received audio embedding vectors and updates the previously obtained prosodic features with the latest audio embedding vector until the last audio embedding vector is processed, generating the final current prosodic feature vector.

[0046] The current timbre feature vector and the current prosody feature vector are each mapped to the text token space dimension of the large language model with different projections.

[0047] In one embodiment, when generating timbre feature vectors and prosody feature vectors, the timbre extraction branch and the prosody extraction branch also output the confidence scores of the timbre feature vectors and prosody feature vectors, respectively, and generate a first confidence score by weighted fusion based on the confidence scores of the timbre feature vectors and prosody feature vectors.

[0048] S202. Stream each of the short-frame audios to the speech recognition model for speech recognition, and obtain the current text segment and second confidence level corresponding to each of the short-frame audios. In one embodiment, each short audio frame is streamed to a speech recognition model, which then generates an N-best (top N best) set of transcribed text candidates for each short audio frame in real time, along with the ASR probability of each candidate text segment. The N-best transcribed text candidate set contains the top N candidate text segments in terms of ASR probability, and the ASR probability of each selected text segment is used as a second confidence level.

[0049] In another embodiment, the speech recognition process of the speech recognition model can also be performed in parallel with the encoding process of the causal audio encoder. Specifically, after obtaining at least one short frame audio through short frame sampling, each short frame audio is streamed to the speech recognition model for speech recognition to obtain the current text segment and second confidence level corresponding to each short frame audio. Step S201 is then executed after the causal audio encoder has finished encoding, thereby improving the efficiency of updating dialogue history information and thus improving the efficiency of generating the next round of responses.

[0050] S203. Based on the first confidence level and the second confidence level, obtain the target confidence level; In one embodiment, the first confidence level and the second confidence level are weighted and fused to generate the target confidence level. The weighting coefficients for the first and second confidence levels can be set according to actual needs.

[0051] In another embodiment, the target confidence level can also be the lowest value between the first confidence level and the second confidence level, i.e., target confidence level = min[first confidence level, second confidence level].

[0052] S204. Update the dialogue history information based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence level corresponding to each of the short audio frames.

[0053] Furthermore, updating the dialogue history information based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence score corresponding to each of the short audio frames includes: matching a target update strategy in a preset update strategy based on the target confidence score; and updating the dialogue history information based on the target update strategy, the previous timbre feature vector, the current prosody feature vector, and the current text segment.

[0054] In one embodiment, historical information is updated by adopting a corresponding update strategy based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence. Update strategies include replacement, score accumulation, filtering, and hierarchical caching.

[0055] In a specific embodiment, when the target confidence level is greater than or equal to the first confidence threshold, the corresponding text segment, timbre feature vector, and prosodic feature vector in the dialogue history information are replaced with the current text segment, the current timbre feature vector, and the current prosodic feature vector, and the target confidence level is synchronously updated into the history information; when the target confidence level is less than the first confidence threshold but greater than or equal to the second confidence threshold, the current text segment, the current timbre feature vector, and the current prosodic feature vector are added to the dialogue history information with a lower weight (which can be set as needed); when the target confidence level is less than the second confidence threshold, the current text segment, the current timbre feature vector, and the current prosodic feature vector can be ignored, that is, the dialogue history information is not updated.

[0056] The first confidence threshold is greater than the second confidence threshold.

[0057] In another embodiment, the above-described process of speech recognition - timbre and prosody feature extraction - target confidence acquisition - dialogue history information update can also be performed in parallel when the synthesized speech is played to the user after speech synthesis.

[0058] In the above embodiments, since dialogue history information affects the synthesis of response speech, parallel processing is used to improve the efficiency of updating dialogue history information, thereby increasing the efficiency of obtaining dialogue history information during dialogue speech synthesis and thus improving speech synthesis efficiency. Secondly, different historical information update strategies are adopted according to different confidence levels to avoid the impact of erroneous transcription on historical information, thereby improving the quality of speech synthesis.

[0059] Please see Figure 3 , Figure 3This is a schematic flowchart illustrating a dialogue speech generation method based on causal audio coding, provided in an embodiment of this application. This dialogue speech generation method based on causal audio coding can be applied to voice dialogue systems and servers. It employs a pre-trained causal audio encoder, a dual-path style encoder, a speech synthesis model, and a large language model, avoiding training from scratch and improving training speed. By obtaining a joint loss function through automatically tuned weight parameters, dynamic balance is achieved, improving training effectiveness and system performance. Style decoupling combined with contrastive consistency constraint training maintains consistency in speaker identity and prosody in multi-speaker and noisy environments, making the final synthesized speech more stable in terms of naturalness and speaker similarity.

[0060] like Figure 3 As shown, the dialogue speech generation method based on causal audio coding specifically includes steps S301 to S303.

[0061] S301. Obtain the pre-trained causal audio encoder, the pre-trained dual-channel style encoder, the pre-trained speech synthesis model, and the pre-trained large language model. In one embodiment, the pre-trained causal audio encoder can adopt a Conformer-Causal architecture, be pre-trained from a large-scale streaming speech database, support frame-by-frame incremental encoding, and rely only on current and historical frame information without future dependencies.

[0062] The pre-trained dual-path style encoder consists of two independent branches, which are obtained through pre-training on a multi-speaker emotional speech dataset. Specifically, the timbre extraction branch can use an ECAPA-TDNN pre-trained speaker embedding model to output a static timbre feature vector; the prosody extraction branch can use a CNN+LSTM temporal model to output a dynamic prosody feature vector.

[0063] The pre-trained speech synthesis model can be obtained by using the StyleTTS2 (text-to-speech model with style diffusion) speech model, which is pre-trained on a noisy speech dataset.

[0064] The pre-trained large language model is obtained by pre-training on a large-scale audio-text alignment dataset.

[0065] S302. The preset text cross-entropy loss function, style regression loss function, style contrast loss function, timbre control loss function and confidence weighted loss function are weighted and fused to obtain the joint loss function; In one embodiment, text cross-entropy loss is used to train a large language model to obtain the optimal predicted text token. The text cross-entropy loss function is:

[0066] in, For large language model parameters, The predicted text token corresponding to the current audio embedding vector. For audio embedding vector sequences, Let T be the historical text token sequence, and T be the total length of the text sequence.

[0067] In one embodiment, style regression loss is used to train a large language model to obtain the optimal predicted prosody, retaining the L1 baseline regression term. The style regression loss function is:

[0068] in, To predict prosodic vectors, This is the true prosodic vector.

[0069] In one embodiment, the style contrast loss function is used to train the prosodic extraction branch in a dual-branch style encoder. By applying InfoNCE contrastive learning to the prosodic vectors within a batch of samples, it ensures that semantically or emotionally similar vectors are grouped together, and that different emotions or different speakers are distinguished, thereby improving the discriminativeness and controllability of style vectors. The style contrast loss function is as follows:

[0070] in, This is the current prosodic feature vector. τ is the prosodic feature vector corresponding to the semantic or emotional positive sample (constructed through label or nearest neighbor retrieval), and τ is the temperature parameter. This represents the prosodic feature vector for all samples.

[0071] In one embodiment, the timbre fidelity control loss is used to train the speech synthesis model to ensure that the synthesized speech retains the timbre of the target speaker. The timbre control loss function is:

[0072] in, For synthesized speech, This is the embedding vector for the synthesized speech. For the target speaker's voice, The embedding vector corresponding to the target speaker's speech.

[0073] In one embodiment, noise and uncertainty in ASR and speaker segmentation are simulated during training. Confidence is introduced to weight context samples, enabling the large language model to assign optimal weights to historical information, thus ignoring low-confidence context. The confidence-weighted loss function is:

[0074] in, Weights for historical transcribed text or style. Historical transcription text or style, This refers to the local loss of historical transcribed text or style.

[0075] In one embodiment, the joint loss function is:

[0076] Among them, parameters The weight coefficients corresponding to each loss function can be automatically tuned using a small-scale validation set. Specifically, a candidate range is set for each weight parameter (e.g., ...). (Discrete values ​​within the range), based on a small-scale validation set, traverse all candidate combinations, calculate the comprehensive validation index (such as convergence speed) for each combination, and select the optimal weight.

[0077] A small-scale validation set covers samples from multiple speakers (different genders, accents, and ages); different noise environments (quiet, street, office); multi-turn dialogue samples (testing cross-turn style consistency); and style transfer samples.

[0078] S303. Obtain the training dataset, and based on the training dataset and the joint loss function, jointly train the pre-trained causal audio encoder, the pre-trained dual-path style encoder, the pre-trained speech synthesis model and the pre-trained large language model to obtain the speech dialogue system.

[0079] In one embodiment, the training dataset includes speech data (with / without noise) from multi-turn dialogues, corresponding text transcriptions, multiple speaker timbre labels, prosodic (emotion / speed) labels, and confidence level annotations.

[0080] In a specific embodiment, streaming speech segments collected from the training dataset are input into a causal audio encoder to generate audio embedding vectors. These embedding vectors are then transmitted to a large language model and a dual-path style encoder. The large language model predicts text tokens and prosodic feature vectors, while the dual-path style encoder extracts timbre and prosodic features. The predicted text tokens and prosodic feature vectors are then transmitted to a speech synthesis model for speech synthesis. The extracted timbre and prosodic features are used to update historical information. A joint loss function is calculated for each iteration, and backpropagation is performed using this joint loss function to synchronously update the parameters of the causal audio encoder, dual-path style encoder, large language model, and speech synthesis model until a preset stopping condition is reached. This stopping condition can be set according to actual needs, such as when the joint loss function value is less than a preset loss threshold or when the maximum number of iterations is reached.

[0081] In another embodiment, the speech synthesis model incorporates noise enhancement and denoising loss during training to make the synthesis more robust under noisy conditions.

[0082] In another embodiment, after training, the large language model is subjected to task-specific distillation and combined with quantization or pruning for deployment on mobile devices. A local style extraction and federated learning periodic update mechanism is adopted to reduce cloud computing and network overhead.

[0083] In the above embodiments, a pre-trained causal audio encoder, a dual-path style encoder, a speech synthesis model, and a large language model are used to avoid training from scratch and improve training speed. The joint loss function is obtained through automatically tuned weight parameters to achieve dynamic balance, improve training effect, and enhance system performance. Style decoupling combined with contrastive consistency constraint training can maintain the consistency of speaker identity and prosody in multi-speaker and noisy environments, making the final synthesized speech more stable in terms of naturalness and speaker similarity.

[0084] Please see Figure 4 , Figure 4 This application provides a schematic block diagram of a dialogue speech generation apparatus based on causal audio coding, which is used to execute the aforementioned dialogue speech generation method based on causal audio coding. The dialogue speech generation apparatus based on causal audio coding can be configured in a voice dialogue system or a server.

[0085] like Figure 4 As shown, the dialogue speech generation device 400 based on causal audio coding includes: The audio acquisition module 401 is used to perform short frame sampling on the user's input voice based on the audio acquisition module corresponding to the voice dialogue system, and obtain at least one short frame audio. The audio encoding module 402 is used to stream each of the short frame audios to a preset causal audio encoder for encoding, thereby obtaining an audio embedding vector corresponding to each of the short frame audios. The feature prediction module 403 is used to acquire dialogue history information and process the dialogue history information and each audio embedding vector based on a large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. The speech synthesis module 404 is used to obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, to perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors and the target timbre feature vector to obtain synthesized speech.

[0086] Furthermore, the audio encoding module 402 includes: An acoustic feature extraction unit is used to stream each of the short frame audios to the historical audio encoder, and when the causal audio encoder receives the short frame audio, to extract acoustic features from the short frame audio based on the causal convolutional layer of the causal audio encoder to obtain acoustic features. The audio embedding vector acquisition unit is used to encode the acoustic features based on the causal self-attention coding layer of the causal audio encoder to obtain the audio embedding vector corresponding to each short frame audio.

[0087] Furthermore, the dialogue speech generation device 400 based on causal audio coding also includes a historical information update module, which includes: The feature extraction unit is used to extract and encode timbre and prosodic features of each audio embedding vector based on the dual-channel style encoder corresponding to the voice dialogue system, and obtain the current timbre feature vector, the current prosodic feature vector and the first confidence level. The speech recognition unit is used to stream each of the short audio frames to the speech recognition model for speech recognition, and to obtain the current text segment and the second confidence level corresponding to each of the short audio frames. The target confidence level acquisition unit is used to obtain a target confidence level based on the first confidence level and the second confidence level; The information update unit is used to update the dialogue history information based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence level corresponding to each of the short audio frames.

[0088] Furthermore, the dual-channel style encoder includes a timbre extraction branch and a prosody extraction branch, and the feature extraction unit includes: The timbre feature extraction subunit is used to extract and encode timbre features from the streaming audio embedding vector based on the timbre extraction branch, and obtain the timbre feature vector and the third confidence level. The prosodic feature extraction subunit is used to extract and encode prosodic features from the streaming audio embedding vector based on the prosodic extraction branch, so as to obtain the prosodic feature vector and the fourth confidence level. The first confidence level acquisition subunit is used to weight and fuse the third confidence level and the fourth confidence level to obtain the first confidence level.

[0089] Furthermore, the information update unit includes: The strategy matching subunit is used to match the target update strategy among preset update strategies based on the target confidence level; The information update subunit is used to update the dialogue history information based on the target update strategy, the previous timbre feature vector, the current prosody feature vector, and the current text segment.

[0090] Furthermore, the dialogue speech generation device 400 based on causal audio coding also includes a system training module for: The system component module acquisition unit is used to acquire the pre-trained causal audio encoder, the pre-trained dual-channel style encoder, the pre-trained speech synthesis model, and the pre-trained large language model. The joint loss function acquisition unit is used to perform weighted fusion of the preset text cross-entropy loss function, style regression loss function, style contrast loss function, timbre control loss function and confidence weighted loss function to obtain the joint loss function; A joint training unit is used to acquire a training dataset and, based on the training dataset and the joint loss function, jointly train the pre-trained causal audio encoder, the pre-trained dual-path style encoder, the pre-trained speech synthesis model, and the pre-trained large language model to obtain the speech dialogue system.

[0091] Furthermore, the dialogue speech generation device 400 based on causal audio coding also includes: A vector caching module is used to cache each of the audio embedding vectors to obtain a continuous sequence of audio embedding vectors; The vector transmission module is used to transmit the audio embedding vector sequence to the large language model when preset conditions are met.

[0092] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0093] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0094] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a voice dialogue system or a server.

[0095] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0096] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any dialogue speech generation method based on causal audio coding.

[0097] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0098] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any dialogue speech generation method based on causal audio coding.

[0099] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0101] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Based on the audio acquisition module corresponding to the voice dialogue system, short frame sampling is performed on the user's input voice to obtain at least one short frame audio. Each of the short audio frames is streamed to a preset causal audio encoder for encoding to obtain an audio embedding vector corresponding to each of the short audio frames. The dialogue history information is obtained, and the dialogue history information and each audio embedding vector are processed based on the large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. Obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors, and the target timbre feature vector to obtain synthesized speech.

[0102] In one embodiment, when the processor streams each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames, it is configured to: Each of the short-frame audio streams is transmitted to the historical audio encoder, and when the causal audio encoder receives the short-frame audio, acoustic features are extracted from the short-frame audio based on the causal convolutional layer of the causal audio encoder to obtain acoustic features; Based on the causal self-attention coding layer of the causal audio encoder, the acoustic features are encoded to obtain the audio embedding vectors corresponding to each short frame audio.

[0103] In one embodiment, after the processor streams each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames, it is further configured to: Based on the dual-channel style encoder corresponding to the voice dialogue system, timbre features and prosodic features are extracted and encoded for each audio embedding vector to obtain the current timbre feature vector, the current prosodic feature vector and the first confidence level. Each of the short audio frames is streamed to a speech recognition model for speech recognition, and the current text segment and second confidence level corresponding to each of the short audio frames are obtained. Based on the first confidence level and the second confidence level, the target confidence level is obtained; The dialogue history information is updated based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence score corresponding to each of the short audio frames.

[0104] In one embodiment, the dual-channel style encoder includes a timbre extraction branch and a prosody extraction branch. When the processor performs timbre and prosody feature extraction and encoding on each audio embedding vector based on the dual-channel style encoder corresponding to the speech dialogue system, to obtain the current timbre feature vector, the current prosody feature vector, and a first confidence level, it is used to: Based on the timbre extraction branch, timbre features are extracted and encoded from the streaming audio embedding vector to obtain the timbre feature vector and the third confidence level. Based on the prosody extraction branch, prosody features are extracted and encoded from the streaming audio embedding vector to obtain the prosody feature vector and the fourth confidence level. The third confidence level and the fourth confidence level are weighted and fused to obtain the first confidence level.

[0105] In one embodiment, when the processor updates the dialogue history information based on the current timbre feature vector, the current prosodic feature vector, the current text segment, and the target confidence score corresponding to each of the short audio frames, it is configured to: Based on the target confidence level, match the target update strategy in the preset update strategy; The dialogue history information is updated based on the target update strategy, the previous timbre feature vector, the current prosody feature vector, and the current text segment.

[0106] In one embodiment, before implementing the audio acquisition module corresponding to the voice dialogue system to perform short-frame sampling of the user's input speech and obtain at least one short-frame audio, the processor is also configured to implement: Acquire pre-trained causal audio encoder, pre-trained dual-channel style encoder, pre-trained speech synthesis model, and pre-trained large language model; The preset text cross-entropy loss function, style regression loss function, style contrast loss function, timbre control loss function, and confidence weighted loss function are weighted and fused to obtain the joint loss function; Obtain a training dataset, and based on the training dataset and the joint loss function, jointly train the pre-trained causal audio encoder, the pre-trained dual-path style encoder, the pre-trained speech synthesis model, and the pre-trained large language model to obtain the speech dialogue system.

[0107] In one embodiment, after the processor streams each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames, it is further configured to: Each of the audio embedding vectors is cached to obtain a continuous sequence of audio embedding vectors; When the preset conditions are met, the audio embedding vector sequence is transmitted to the large language model.

[0108] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the dialogue speech generation methods based on causal audio coding provided in the embodiments of this application.

[0109] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating conversational speech based on causal audio coding, characterized by, include: Based on the audio acquisition module corresponding to the voice dialogue system, short frame sampling is performed on the user's input voice to obtain at least one short frame audio. Each of the short audio frames is streamed to a preset causal audio encoder for encoding to obtain an audio embedding vector corresponding to each of the short audio frames. The dialogue history information is obtained, and the dialogue history information and each audio embedding vector are processed based on the large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. Obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors, and the target timbre feature vector to obtain synthesized speech.

2. The causal-based audio coding method for generating conversational speech according to claim 1, wherein, The step of streaming each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames includes: Each of the short-frame audio streams is transmitted to the historical audio encoder, and when the causal audio encoder receives the short-frame audio, acoustic features are extracted from the short-frame audio based on the causal convolutional layer of the causal audio encoder to obtain acoustic features; Based on the causal self-attention coding layer of the causal audio encoder, the acoustic features are encoded to obtain the audio embedding vectors corresponding to each short frame audio.

3. The causal-based audio coding method for conversational speech generation according to claim 1, wherein, After streaming each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames, the method further includes: Based on the dual-channel style encoder corresponding to the voice dialogue system, timbre features and prosodic features are extracted and encoded for each audio embedding vector to obtain the current timbre feature vector, the current prosodic feature vector and the first confidence level. Each of the short audio frames is streamed to a speech recognition model for speech recognition, and the current text segment and second confidence level corresponding to each of the short audio frames are obtained. Based on the first confidence level and the second confidence level, the target confidence level is obtained; The dialogue history information is updated based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence score corresponding to each of the short audio frames.

4. The causal-based audio coding method of generating conversational speech according to claim 3, characterized in that, The dual-channel style encoder includes a timbre extraction branch and a prosody extraction branch. The dual-channel style encoder, based on the speech dialogue system, extracts and encodes timbre and prosody features from each audio embedding vector to obtain a current timbre feature vector, a current prosody feature vector, and a first confidence level, including: Based on the timbre extraction branch, timbre features are extracted and encoded from the streaming audio embedding vector to obtain the timbre feature vector and the third confidence level. Based on the prosody extraction branch, prosody features are extracted and encoded from the streaming audio embedding vector to obtain the prosody feature vector and the fourth confidence level. The third confidence level and the fourth confidence level are weighted and fused to obtain the first confidence level.

5. The causal-based audio coding method of generating conversational speech according to claim 3, wherein, The step of updating the dialogue history information based on the current timbre feature vector, the current prosody feature vector, the current text segment, and the target confidence score corresponding to each of the short audio frames includes: Based on the target confidence level, match the target update strategy in the preset update strategy; The dialogue history information is updated based on the target update strategy, the previous timbre feature vector, the current prosody feature vector, and the current text segment.

6. The causal-based audio coding method of generating conversational speech according to claim 1, wherein, Before the audio acquisition module based on the voice dialogue system performs short-frame sampling on the user's input voice to obtain at least one short-frame audio, it further includes: Acquire pre-trained causal audio encoder, pre-trained dual-channel style encoder, pre-trained speech synthesis model, and pre-trained large language model; The preset text cross-entropy loss function, style regression loss function, style contrast loss function, timbre control loss function, and confidence weighted loss function are weighted and fused to obtain the joint loss function; Obtain a training dataset, and based on the training dataset and the joint loss function, jointly train the pre-trained causal audio encoder, the pre-trained dual-path style encoder, the pre-trained speech synthesis model, and the pre-trained large language model to obtain the speech dialogue system.

7. The causal audio coding based conversational speech generation method according to any one of claims 1 to 6, characterized by, After streaming each of the short audio frames to a preset causal audio encoder for encoding to obtain the audio embedding vector corresponding to each of the short audio frames, the method further includes: Each of the audio embedding vectors is cached to obtain a continuous sequence of audio embedding vectors; When the preset conditions are met, the audio embedding vector sequence is transmitted to the large language model.

8. An apparatus for generating conversational speech based on causal audio encoding, the apparatus comprising: a causal audio encoder configured to encode an audio signal to generate a causal audio signal; and a speech synthesizer configured to synthesize speech from the causal audio signal. include: The audio acquisition module is used to perform short-frame sampling of the user's input speech based on the audio acquisition module corresponding to the voice dialogue system, and obtain at least one short-frame audio. An audio encoding module is used to stream each of the short-frame audios to a preset causal audio encoder for encoding, thereby obtaining an audio embedding vector corresponding to each of the short-frame audios. The feature prediction module is used to acquire dialogue history information and process the dialogue history information and each audio embedding vector based on a large language model to obtain the predicted text token and predicted prosodic feature vector corresponding to each audio embedding vector. The speech synthesis module is used to obtain the target timbre feature vector corresponding to the target speaker, and based on the speech synthesis model, to perform speech synthesis on each of the predicted text tokens, each of the predicted prosodic feature vectors and the target timbre feature vector to obtain synthesized speech.

9. A computer device, comprising: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the dialogue speech generation method based on causal audio coding as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the dialogue speech generation method based on causal audio coding as described in any one of claims 1 to 7.

Images