Voice generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a voice generation method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that a preset voice enhancement model is used for extracting training voice features of training prompt voice data, filtering the training voice features to obtain selected voice features, and decoding the selected voice features and the training prompt voice data to obtain enhanced voice data; the preset speech generation model is used for: extracting training text features from a training text; generating predicted voice data for the training text features according to the selected voice features; adjusting parameters of a preset voice generation model according to the preset voice data, the predicted voice data, the enhanced voice data and the original voice data to obtain a target voice generation model; and generating target voice data for the target text through the target voice generation model. The method can be applied to business systems needing a large amount of data, such as financial science and technology and health medical treatment, and voice data consistent with a speaking object in style can be stably output.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applied to the fields of fintech and healthcare, and particularly to a speech generation method and apparatus, computer equipment and storage medium. Background Technology

[0002] Text-to-speech systems can generate speech consistent with the target speaker's style based on a small amount of target prompt speech data. However, in practical applications, the target prompt speech data is often affected by environmental noise, channel distortion, and other factors, leading to varying degrees of feature degradation in the synthesized speech data. In related technologies, to overcome the noise impact of the target prompt speech data, a separate speech enhancement module is used to denoise the target prompt speech data before combining it with the denoised target prompt speech data for speech generation. However, setting up a separate speech enhancement module significantly increases the number of system parameters and computational resources. Furthermore, the independent nature of the speech enhancement and speech generation tasks leads to instability in the style of the synthesized speech data. For example, in a financial scenario, a virtual financial assistant with speech generation capabilities interacts with users through this function. If the style of the speech data output by the virtual financial assistant is unstable during the Q&A process, it can negatively impact the user experience. Similarly, in healthcare, a medical assistant with speech generation capabilities interacts with patients through this function. If the style of the medical assistant's output speech data is unstable, it can reduce patients' trust in the medical assistant and decrease its usage rate. Therefore, how to reliably output speech data consistent with the speaking style of the person being spoken to has become a pressing technical problem that needs to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a speech generation method and apparatus, computer device and storage medium, which aim to stably output speech data consistent with the speaking style of the speaker.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech generation method, the method comprising: Acquire training text and training prompt speech data of a reference speaker; wherein, the training prompt speech data is speech data obtained by adding noise to the original speech data; The training prompt speech data is encoded using a preset speech enhancement model to obtain training speech features; The selected speech features are obtained by filtering noise features from the training speech features using the preset speech enhancement model; The selected speech features and the training prompt speech data are decoded using the preset speech enhancement model to obtain enhanced speech data. The training text is encoded using a preset speech generation model to obtain training text features; The training text features are used to generate speech using the preset speech generation model and the selected speech features to obtain predicted speech data. The model parameters of the preset speech generation model are adjusted based on the preset speech data, the predicted speech data, the enhanced speech data, and the original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording the training text; The target text and the target prompt speech data of the target speaker are obtained. The target speech data is obtained by generating speech from the target text using the target speech generation model and the target prompt speech data.

[0005] In some embodiments, the preset speech enhancement model includes: a speech encoder, a feature selector, and a speech decoder; the feature selector includes at least one first expert network, a first gating network, and a second gating network; the selected speech features include: enhanced speech features and generated speech features. The step of filtering noise features from the training speech features using the preset speech enhancement model to obtain selected speech features includes: The first expert network is weighted using the first gating network and the trained speech features to obtain a first weight matrix. The first expert network is weighted using the second gating network and the trained speech features to obtain a second weight matrix; Candidate speech features are obtained by filtering noise features from the trained speech features using each of the first expert networks. The enhanced speech features are obtained by concatenating at least one of the candidate speech features using the first weight matrix. The generated speech features are obtained by concatenating at least one of the candidate speech features using the second weight matrix.

[0006] In some embodiments, the step of assigning weights to the first expert network using the first gating network and the trained speech features to obtain a first weight matrix includes: The importance of the first expert network is measured using the first gating network and the trained speech features to obtain importance measurement data. The importance measurement data is normalized to obtain the first weight value for each of the first expert networks; The first weight values ​​are concatenated to obtain the first weight matrix.

[0007] In some embodiments, the preset speech generation model includes: a text encoder, a variable adapter, a spectrum decoder, and a first vocoder; The step of generating speech from the training text features using the preset speech generation model and the selected speech features to obtain predicted speech data includes: The training text features are used to predict speech features through the variable adapter and the generated speech features to obtain predicted speech features. The predicted spectral features are obtained by using the spectrum decoder and the generated speech features to predict the spectral features of the predicted speech features. The predicted speech data is obtained by outputting the predicted spectral features into speech using the first vocoder.

[0008] In some embodiments, the variable adapter includes at least one second expert network, a third gating network, and a speech feature prediction network; The step of predicting speech features from the training text features using the variable adapter and the generated speech features to obtain predicted speech features includes: The third weight matrix is ​​obtained by assigning weights to each of the second expert networks using the third gating network and the generated speech features. The training text features are predicted by the second expert network to obtain the first speech feature; At least one of the first speech features is concatenated according to the third weight matrix to obtain the second speech feature; The predicted speech features are obtained by predicting speech features of the target text features using the speech feature prediction network and the second speech feature.

[0009] In some embodiments, the spectrum decoder includes: at least one third expert network, a spectrum decoding network, and a fourth gating network; the step of predicting the predicted spectrum features by using the spectrum decoder and the generated speech features to obtain the predicted spectrum features includes: The fourth weight matrix is ​​obtained by assigning weights to each of the third expert networks using the fourth gating network and the generated speech features. The predicted speech features are then subjected to spectral prediction using the third expert network to obtain the first spectral features. At least one of the first spectral features is concatenated according to the fourth weight matrix to obtain the second spectral feature; The predicted spectral features are obtained by normalizing the second spectral features using the spectral decoding network.

[0010] In some embodiments, the speech decoder includes: an attention network, a feedforward network, and a second vocoder; The step of decoding the selected speech features and the training prompt speech data using the preset speech enhancement model to obtain enhanced speech data includes: The training prompt speech data is projected into key features and value features through the attention network; The enhanced speech features are projected into query features using the attention network. The value features are fused based on the query features and the key features to obtain fused features; The fused features are normalized using the feedforward network to obtain normalized features; The normalized features are processed by the second vocoder to generate speech, thereby obtaining the enhanced speech data.

[0011] To achieve the above objectives, a second aspect of this application provides a speech generation apparatus, the apparatus comprising: The data acquisition module is used to acquire training text and training prompt speech data of the reference speaker; wherein, the training prompt speech data is speech data obtained by adding noise to the original speech data; The speech encoding module is used to encode the training prompt speech data through a preset speech enhancement model to obtain training speech features; The noise feature filtering module is used to filter noise features from the training speech features through the preset speech enhancement model to obtain selected speech features. The speech decoding module is used to decode the selected speech features and the training prompt speech data through the preset speech enhancement model to obtain enhanced speech data; The text encoding module is used to encode the training text using a preset speech generation model to obtain training text features; The training speech generation module is used to generate speech from the training text features using the preset speech generation model and the selected speech features, so as to obtain predicted speech data. The model parameter adjustment module adjusts the model parameters of the preset speech generation model based on the preset speech data, the predicted speech data, the enhanced speech data, and the original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording the training text; The target speech generation module is used to acquire target text and target prompt speech data of the target speaker, and generate speech from the target text using the target speech generation model and the target prompt speech data to obtain target speech data.

[0012] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] The speech generation method, apparatus, computer device, and storage medium proposed in this application optimize a preset speech enhancement model and a preset speech generation model collaboratively. Specifically, noise is first added to the original speech data to generate training prompt speech data. The preset speech generation model filters the noise features from the training prompt speech data and outputs selected speech features, generating enhanced speech data based on the selected speech features. Simultaneously, the preset speech generation model extracts training text features from the training text and uses the selected speech features in the speech prediction process of the training text features to output predicted speech data. Then, the predicted speech data, preset speech data, enhanced speech data, and original speech data are jointly used to optimize the preset speech generation model to determine the target speech generation model. Therefore, during the training process of the preset speech generation model, it learns under noisy prompt speech and, in conjunction with the speech enhancement function of the speech enhancement model, generates speech data with a more natural timbre consistent with the reference speaker, thereby training a target speech generation model that can stably output a timbre consistent with the reference speaker. Therefore, in the application of the target speech generation model, the target text is generated by using the target speech generation model and the target prompt speech data to output target speech data with a timbre consistent with the target speaker, stable tone and content, and high naturalness. Attached Figure Description

[0015] Figure 1 This is a flowchart of the speech generation method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S103 in the process; Figure 3 This is a flowchart illustrating the training process of the preset speech enhancement model and the preset speech generation model; Figure 4 This is a flowchart illustrating the feature selector in the speech generation method provided in this application embodiment; Figure 5 yes Figure 2 The flowchart of step S201 in the text; Figure 6This is a flowchart illustrating the specific process of the speech decoder in the speech generation method provided in this application embodiment; Figure 7 yes Figure 1 The flowchart of step S104 in the process; Figure 8 yes Figure 1 The flowchart of step S106 in the process; Figure 9 This is an overall framework diagram of the preset speech generation model in the speech generation method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the variable adapter structure in the speech generation method provided in this application embodiment; Figure 11 yes Figure 8 The flowchart of step S801 in the process; Figure 12 This is a schematic diagram of the spectrum decoder structure in the speech generation method provided in this application embodiment; Figure 13 yes Figure 8 The flowchart of step S802 in the process; Figure 14 This is a schematic diagram of the structure of the speech generation device provided in the embodiments of this application; Figure 15 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0020] Text-to-Speech (TTS) is a technology that converts text into speech. Based on speech synthesis technology, it transforms input text information into natural and fluent speech output. Its core value lies in three aspects: 1. Accessibility: Providing text-to-speech services for visually impaired users, improving information retrieval efficiency; 2. Enhanced Interaction: Enabling voice interaction in intelligent customer service and in-vehicle systems, optimizing user experience; 3. Content Production: Automating the generation of multimedia content such as audiobooks and video dubbing. Modern TTS systems employ a deep neural network architecture, with a typical process including three stages: text preprocessing, acoustic feature extraction, and vocoder synthesis.

[0021] Zero-shot TTS models are a technique in the field of text-to-speech (TTS) synthesis that refers to the ability of a model to generate speech with a specific style, timbre, or language without having seen it before or been specifically trained. The core of this technology lies in its ability to generate speech based on unseen languages ​​or contexts, without requiring specialized training for each language or accent.

[0022] Speech Enhancement (SE) module: This is a signal processing technology that improves speech intelligibility and comfort by filtering, transforming, or reconstructing noisy speech signals. Its core value lies in three aspects: 1. Improved communication quality: Eliminating background noise and echo in scenarios such as VoIP and video conferencing to ensure clear speech transmission; 2. Optimization for smart devices: Providing noise immunity for smart speakers, in-vehicle voice systems, etc., improving voice wake-up and recognition accuracy; 3. Innovation in hearing assistance: Enhancing speech signals in medical devices such as hearing aids and cochlear implants to improve the communication experience for people with hearing impairments.

[0023] Mixture-of-Experts (MoE) is a neural network architecture that uses a gating network to dynamically select a small subset of subnetworks called "experts" for each input, thereby improving model capacity and computational efficiency through sparse activation.

[0024] Self-supervised learning (SSL) is a powerful machine learning paradigm that leverages unlabeled data to extract effective features for various downstream tasks. Before delving into the details of self-supervised learning, we will first analyze its definition, main technical methods, implementation strategies, and its advantages and limitations in practical applications.

[0025] FastSpeech2 is a fast, high-quality end-to-end text-to-speech (TTS) model. It is an improved version of FastSpeech and is particularly suitable for scenarios such as real-time voice broadcasting and intelligent customer service.

[0026] Attention Mechanism: Originating from research on human vision, in cognitive science, due to information processing bottlenecks, humans selectively focus on a portion of all information while ignoring other visible information. This mechanism is commonly referred to as the attention mechanism. Different parts of the human retina possess varying degrees of information processing capacity, or acuity, with only the fovea having the highest acuity.

[0027] Feed-Forward Neural Networks (FNNs) are a basic artificial neural network structure where information flows unidirectionally within the network. They are widely used for tasks such as classification and regression. A feed-forward neural network layer typically consists of multiple neurons arranged hierarchically to form an input layer, hidden layers (which can be one or more), and an output layer. Each neuron in a layer only receives the output from the previous layer as input; there are no feedback connections within or between layers. This unidirectional flow characteristic allows feed-forward neural network layers to handle complex nonlinear problems.

[0028] In the field of artificial intelligence, Text-to-Speech (TTS) systems are synthesizing increasingly natural and expressive speech, primarily due to the use of massive amounts of "field" data (audiobooks, blogs, etc.) during training. Zero-shot TTS systems possess good scalability and speaker transfer capabilities, generating speech consistent with the style of a reference speaker based on a small number of voice prompts. However, during the text-to-speech process, the synthesized speech data exhibits varying degrees of degradation due to the susceptibility of the prompt speech data to environmental noise and channel distortion. Related technologies address the noise problem in the prompt speech data by using a speech enhancement module as an independent front-end to denoise the noisy prompt speech data before inputting it into the text-to-speech model. However, this additional front-end setup increases the number of system parameters and computational overhead, and there is a lack of joint optimization between the speech enhancement module and the text-to-speech model.

[0029] In the fintech field, to improve the comfort of insurance consultation services, voice data from female insurance customer service representatives is often used as prompt voice data to generate voice data consistent with the speaking style of insurance customer service representatives. However, if a separate voice enhancement model needs to be set up to filter the prompt voice data during the voice generation process, it increases the system's computational overhead and also affects the efficiency of responding to insurance consultations.

[0030] In the healthcare field, for patients with visual impairments or mobility difficulties, text-to-speech models are needed to generate audio data from pre-written medical orders. To enhance patient trust in the audio data, the doctor's voice is used as a prompt, generating audio data that matches the doctor's speaking style during diagnosis. If a separate voice enhancement module is used to filter the prompt audio data, a malfunction in this module can affect the timeliness of speech synthesis and add extra computational overhead.

[0031] Based on this, embodiments of this application provide a speech generation method and apparatus, a computer device, and a storage medium. During the training process of a preset speech generation model, a preset speech enhancement model and a preset speech generation model are jointly trained to allow the preset speech generation model to learn noisy speech features and continuously optimize the output speech data. This maintains the stability and consistency of speech generation even under noisy prompts, resulting in a stable and accurate target speech generation model. In practical applications of the target speech generation model, target speech data is automatically generated from the target text using the target speech generation model and target prompt speech data, generating target speech data with the same speech style as the target speaker.

[0032] The speech generation method, apparatus, computer equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the speech generation method in this application is described.

[0033] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] The speech generation method provided in this application relates to the field of artificial intelligence technology and is applied in the fields of fintech and healthcare. The speech generation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech generation method, but is not limited to the above forms.

[0036] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0037] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0038] Figure 1 This is an optional flowchart of the speech generation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0039] Step S101: Obtain training prompt speech data of training text and reference speaker; wherein, training prompt speech data is speech data obtained by adding noise to the original speech data; Step S102: Encode the training prompt speech data using a preset speech enhancement model to obtain training speech features; Step S103: Noise features are filtered from the training speech features using a preset speech enhancement model to obtain selected speech features; Step S104: The selected speech features and training prompt speech data are decoded using a preset speech enhancement model to obtain enhanced speech data; Step S105: Encode the training text using a preset speech generation model to obtain training text features; Step S106: Generate speech from the training text features using a preset speech generation model and selected speech features to obtain predicted speech data; Step S107: Adjust the model parameters of the preset speech generation model based on the preset speech data, predicted speech data, enhanced speech data, and original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording training text. Step S108: Obtain the target text and the target prompt speech data of the target speaker; generate speech from the target text using the target speech generation model and the target prompt speech data to obtain the target speech data.

[0040] Steps S101 to S108 as illustrated in this embodiment involve setting a preset speech enhancement model and a preset speech generation model. Through collaborative learning between these two models, the preset speech generation model can stably and accurately output speech data even with noisy prompt speech data. Specifically, the preset speech enhancement model encodes and filters the training prompt speech data to obtain selected speech features without noise. Then, the preset speech generation model generates predicted speech data using the selected speech features and training text features. Finally, the predicted speech data, preset speech data, enhanced speech data, and original speech data are jointly used to train the preset speech generation model, allowing it to learn under noisy prompt speech data and output speech data stably and accurately. Therefore, by sharing feature space and employing a joint optimization strategy, the preset speech generation model can improve the naturalness and expressiveness of the synthesized speech. After training the preset speech generation model, a target speech generation model is obtained, which can accurately generate target speech data that matches the target speaker.

[0041] In step S101 of some embodiments, training text and training prompt speech data can be extracted from a training database built in the domain in which the speech generation method is applied. Alternatively, training text and training prompt speech data can be extracted in real time from existing publicly available open-source databases. Other methods can also be used to obtain training text and training prompt speech data, and these are not limited to these. For example, if the speech generation method is applied to an insurance customer service scenario, the speech data of insurance customer service personnel during the service process can be collected in advance to build a training database. If the speech generation method is applied to a medical record audio playback scenario, the speech data of communication between various doctors and patients can be stored to form a training database.

[0042] The raw speech data is obtained by recording the speech of the reference speaker. If the raw speech data contains noise, it can be used directly as training prompt speech data. If the raw speech data is clear and free of noise, noise needs to be added to the raw speech data before outputting the training prompt speech data.

[0043] In step S102 of some embodiments, the preset speech enhancement model includes a speech encoder, a feature selector, and a speech decoder. The speech encoder is a pre-trained self-supervised learning model capable of extracting latent features from the training prompt speech data as training speech features. Specifically, the self-supervised learning model includes an input convolutional encoder and L Transformer encoding modules. The training prompt speech data is input to the convolutional encoder for encoding, and then input into the L Transformer encoding modules for encoding. The latent feature output by each Transformer encoding module is defined as... Then, all the latent features are concatenated in hierarchical order to obtain comprehensive multi-scale training speech features, and the training speech features are represented as follows: This embodiment integrates low-level local acoustic information with high-level semantic and prosodic information in its training speech features. It should be noted that the training prompt speech data is noisy audio. A speech encoder consisting of an input convolutional encoder and L Transformer encoding modules is used. The first few Transformer encoding modules extract the shallow local acoustic information of the training prompt speech data, while the later modules extract the deep semantic and prosodic information. Therefore, concatenating the L latent features yields multi-scale training speech features. These training speech features, integrating shallow and deep speech features, can guide the generation of speech data consistent with the speaking style of a reference speaker.

[0044] In step S103 of some embodiments, noise feature filtering is performed by a feature selector in a preset speech enhancement model. This filter filters noise from the trained speech features to generate selected speech features with clearer speech content. It should be noted that the feature selector is a MoE-based feature selector. MoE represents a hybrid expert network, so the feature selector is a selector composed of multiple first expert networks. These first expert networks filter noise features and then fuse them to output selected speech features, providing speech prompt features with higher clarity to guide speech generation.

[0045] In some embodiments, the feature selector includes at least one first expert network, a first gating network, and a second gating network. The selected speech features include enhanced speech features and generated speech features. The first expert network is used to filter noise features from the training speech features. The first gating network is used to assign weight values ​​to each of the first expert networks. The second gating network is also used to assign weight values ​​to the first expert networks. Enhanced speech features are defined as SE features and serve as input to the speech encoder to generate enhanced speech data. Generated speech features are defined as TTS features and are used to participate in speech generation.

[0046] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S205: Step S201: The first expert network is weighted using the first gating network and the trained speech features to obtain the first weight matrix; Step S202: The first expert network is weighted using the second gating network and the trained speech features to obtain the second weight matrix; Step S203: Noise features are filtered from the training speech features through each first expert network to obtain candidate speech features; Step S204: At least one candidate speech feature is concatenated using the first weight matrix to obtain enhanced speech features; Step S205: At least one candidate speech feature is concatenated using the second weight matrix to obtain the generated speech feature.

[0047] In step S201 of some embodiments, the first weight matrix includes at least one first weight value, and each first weight value characterizes the importance of the first expert network to speech enhancement. Specifically, as... Figure 3 and Figure 4 As shown, Figure 3 This is a flowchart illustrating the training process of the preset speech enhancement model and the preset speech generation model. Figure 4 This is a flowchart of the feature selector process. The first gating network is defined as the SE Router, which is specifically a feedforward network. Therefore, the score of each first expert network is calculated through a feedforward network, defined as... The scores are then normalized to obtain the first weight value for each first expert network, defined as follows: This characterizes the importance of the first expert network to the current training speech features.

[0048] In step S202 of some embodiments, the second weight matrix includes at least one second weight value, and each second weight value characterizes the importance of the first expert network to speech generation. Specifically, the second gating network is defined as TTSRouter, which is a feedforward network. Therefore, the score of each first expert network is assigned through a feedforward network and trained speech features, and the score is then normalized to obtain the second weight of each first expert network, defined as... This represents the importance of the first expert network to the training speech features. It should be noted that the first and second weight values ​​are not entirely the same.

[0049] In step S203 of some embodiments, such as Figure 4 As shown, each first expert network is defined as Furthermore, each first expert network shares the input training speech features and performs noise filtering on the training speech features to output candidate speech features.

[0050] It should be noted that each first expert network is implemented as a two-layer feedforward network, capable of projecting high-dimensional training speech features into a low-dimensional embedding space. High-dimensional training speech features often contain redundant information. Projecting them into a low-dimensional space is essentially a compression and selection mechanism, which can remove irrelevant features and retain more contributing information. This not only helps the subsequent first and second gating networks learn more appropriate weight allocations, but also reduces the complexity of subsequent computations and improves speech generation efficiency.

[0051] In step S204 of some embodiments, the first weight matrix includes at least one first weight value. Then, each first weight value and the corresponding candidate speech feature are concatenated. Specifically, the candidate speech features are weighted and summed or weighted and averaged using an adder to output enhanced speech features.

[0052] In step S205 of some embodiments, the second weight matrix includes at least one second weight value. Then, each second weight value and its corresponding candidate speech feature are concatenated. Specifically, the concatenation of candidate speech features is implemented using an adder, and the generated speech feature is output. It should be noted that in the process of enhancing and generating speech features, the first and second weight matrices represent the pre-set important weight values ​​for each first expert network. Therefore, the output enhanced and generated speech features are obtained by concatenating the top K candidate speech features based on their weight values.

[0053] Specifically, in the first gating network and the second gating network, the score and weight value of each first expert network are calculated as shown in formula (1):

[0054] (1) In the formula, The score for the first expert network. For weight values, , For the first The parameters corresponding to the first expert network To train speech features.

[0055] In steps S201 to S205 of this embodiment, candidate speech features are first obtained by filtering noise features from the training speech features through each first expert network. A first gating network assigns a first weight value to each first expert network according to the training speech features, determining a first weight matrix. Then, a second gating network assigns a second weight value to each first expert network according to the training speech features, determining a second weight matrix. Finally, the candidate speech features are concatenated into enhanced speech features according to the first weight matrix, and into generated speech features according to the second weight matrix, outputting the corresponding speech prompt features for speech enhancement and speech generation, respectively.

[0056] Please see Figure 5 In some embodiments, step S201 may include, but is not limited to, steps S501 to S503: Step S501: The importance of the first expert network is measured using the first gating network and the trained speech features to obtain importance measurement data; Step S502: Normalize the importance measurement data to obtain the first weight value of each first expert network. Step S503: Concatenate the first weight values ​​to obtain the first weight matrix.

[0057] In step S501 of some embodiments, as disclosed above, the first gating network is a feedforward network. The feedforward network calculates the score of each first expert network according to the training speech features as importance measurement data, and the importance measurement data characterizes the feature decomposition ability of each first expert network on the training speech features.

[0058] In step S502 of some embodiments, importance measurement data is input into a normalization layer for normalization processing to determine a first weight value, which represents the importance of the first expert network to the currently trained speech features.

[0059] In step S503 of some embodiments, the first weight values ​​are combined into a first weight matrix according to the sorting order of the first expert network.

[0060] In steps S501 to S503 of this embodiment, the importance of each first expert network is calculated according to the training speech features, the first weight value is determined, and the first weight values ​​are concatenated into a first weight matrix to determine the importance of the first expert network and serve as an indicator for concatenating candidate speech features, thereby outputting clearer enhanced speech features.

[0061] It should be noted that the construction process of the second weight matrix is ​​the same as that of the first weight matrix, and will not be repeated here.

[0062] In step S104 of some embodiments, such asFigure 3 As shown, the enhanced speech data is obtained by decoding the training prompt speech data and selected speech features through a speech decoder, and the enhanced speech data is output.

[0063] In step S104 of some embodiments, such as Figure 6 As shown, the speech decoder includes an attention network, a feedforward network, and a second vocoder. By inputting training cue speech data and enhanced speech features into the attention network, attention processing is performed through the attention mechanism. Then, the feedforward network performs nonlinear mapping and normalization. Finally, the normalized features are input into the second vocoder to generate enhanced speech data.

[0064] Please see Figure 7 In some embodiments, step S104 may include, but is not limited to, steps S701 to S705: Step S701: Project the training prompt speech data into key features and value features using an attention network; Step S702: Project the enhanced speech features into query features using an attention network; Step S703: The value features are fused based on the query features and key features to obtain the fused features; Step S704: Normalize the fused features using a feedforward network to obtain normalized features; Step S705: The normalized features are processed by a second vocoder to generate speech data, thereby obtaining enhanced speech data.

[0065] In steps S701 and S702 of some embodiments, the speech decoder is a speech enhancement decoder. In this speech decoder, it is necessary to first fuse the input training prompt speech data with the enhanced speech features based on an attention mechanism, thereby achieving more context-aware speech enhancement under noisy speech prompt conditions. Specifically, as... Figure 6 As shown, in the attention fusion mechanism, the training prompt speech data is projected into key features and value features, where key features are defined as Key and value features are defined as Value. The enhanced speech features are projected into query features, defined as Query.

[0066] In step S703 of some embodiments, a fusion process is performed on the value features. This involves calculating the similarity between the query features and the key features as attention weights, and then using these attention weights to perform a weighted summation of the value features to obtain the fused features. The fused features are represented as follows: (2) In the formula, As a feature of fusion, For value characteristics, For attention weights, Key features, To query features, To query the depth of key features and key features, this is used to scale the dot product to prevent the gradient from being too large or too small.

[0067] The fusion of value features is accomplished by employing an attention mechanism, which involves fusing features from noisy training prompt speech data to output denoised speech features.

[0068] In step S704 of some embodiments, the fused features are discrete, high-level speech features. In order to output continuous and high-quality speech features, a feedforward network is needed to perform nonlinear mapping and layer normalization on the fused features, which is equivalent to nonlinear transformation and enhancement, and outputs normalized features with richer and more stable speech feature representation.

[0069] In step S705 of some embodiments, the second vocoder is used to convert the features into a speech waveform, specifically converting frame-level acoustic features into enhanced speech data efficiently and with high fidelity. It should be noted that the second vocoder is a HiFi-GAN, including a waveform generator and a multi-period discriminator. First, the waveform generator progressively upsamples the low-frame-rate, high-dimensional normalized features to a high-sampling-rate waveform, and then the multi-period discriminator converts the waveform into enhanced speech data.

[0070] In steps S701 to S705 of this embodiment, an attention mechanism is used to project the training prompt speech data into key features and value features, and to project the enhanced speech features into query features. The value features are then fused into a fused feature based on the query and key features. Finally, the fused feature is normalized into a normalized feature, and then converted into enhanced speech data. Therefore, based on the attention mechanism, the fusion of input training prompt speech data and enhanced speech features is achieved, enabling context-aware speech enhancement even with noisy prompt speech, and outputting enhanced speech data.

[0071] As previously disclosed, after generating enhanced speech data, a preset speech enhancement model can be trained using both the enhanced and original speech data to construct a target speech enhancement model with stronger enhancement effects. Furthermore, after using the enhanced speech data to train the preset speech enhancement model, it can also be used as training prompt speech data for the preset speech generation model when data is scarce. It should be noted that the enhanced speech data does not completely replace the training prompt speech data; the preset speech generation model is allowed to learn in noisy environments, enabling it to accurately extract speech features from noisy prompt speech data and output speech data with natural pronunciation and a timbre completely consistent with the style of the reference speaker.

[0072] In some embodiments, the preset speech generation model includes a text encoder, a variable adapter, a spectrum decoder, and a first vocoder. The text encoder is a text-to-phoneme encoder; the variable adapter is a MoE-based adapter and includes at least one second expert network to extract text features; the spectrum decoder converts speech features into spectral features and includes at least one third expert network; the first vocoder converts spectral features into speech waveforms to output predicted speech data. It should be noted that the preset speech generation model is based on the basic architecture of FastSpeech2 and incorporates the MoE variable adapter and MoE spectrum decoder structure to further improve the robustness and controllability of the preset speech generation model in complex scenarios such as multiple speakers and speech data with noisy cues.

[0073] In step S105 of some embodiments, the text encoder encodes the training text, specifically by extracting the training text features. The text encoder is a phoneme encoder used to convert the training text into a dense vector sequence rich in linguistic information that can be processed by a neural network. Therefore, the training text features are a sequence of feature vectors.

[0074] In step S106 of some embodiments, the generated speech features and training text features are converted into predicted speech data by combining a variable adapter, a spectrum decoder and a first vocoder, and the content is the training text, while the speech style and phonemes are consistent with the reference speaking object.

[0075] Please see Figure 8 In some embodiments, step S106 may include, but is not limited to, steps S801 to S803: Step S801: Predict speech features by using a variable adapter and generated speech features to predict the speech features of the training text features; Step S802: Spectral feature prediction is performed on the predicted speech features using a spectrum decoder and generated speech features to obtain the predicted spectral features; Step S803: The predicted spectral features are output as speech by the first vocoder to obtain the predicted speech data.

[0076] In step S801 of some embodiments, the variable adapter is a MoE-based adapter, which integrates a second expert network. This adapter, with prompts for generating speech features, selects the optimal second expert network and converts the training text features into speech features with a reference speaker through the second expert network. Simultaneously, the MoE-based variable adapter significantly enhances the speaker transfer capability in zero-prompt scenarios and the synthesis robustness under noisy conditions.

[0077] In step S802 of some embodiments, the decoder part adopts the MoE-FFT structure to generate high-fidelity Mel spectrum features from the predicted speech features represented in the latent space, thereby obtaining the predicted spectral features.

[0078] In step S803 of some embodiments, the predicted spectral features are used as an acoustic intermediate representation. The predicted spectral features are input into a second vocoder for waveform reconstruction to obtain predicted speech data with high naturalness, consistent timbre with the reference speaker, and robustness to noise.

[0079] In steps S801 to S803 of this embodiment, the training text features are mapped to predicted speech features through a variable adapter and generated speech features. Then, the predicted speech features are converted into predicted spectral features through a spectrum decoder and generated speech features. Finally, the waveform of the predicted spectral features is restored through a first vocoder, outputting predicted speech data with high naturalness and timbre consistent with the reference speaker. Even with noisy prompt speech data, highly accurate speech data can be generated.

[0080] In some embodiments, such as Figure 9 and Figure 10 As shown, Figure 9 This is a diagram illustrating the overall framework of the pre-defined speech generation model. Figure 10 This is a schematic diagram of the variable adapter structure; through Figure 9 It is known that the training text features are input to the variable adapter, and the generated speech features are input to the duration predictor, energy predictor, and pitch predictor, respectively. The duration predictor extracts speech duration features from the generated speech features, the energy predictor extracts speech energy features from the generated speech features, and the pitch predictor extracts speech pitch features from the generated speech features. Then, the extracted speech duration, energy, and pitch features are explicitly modeled to achieve fine-grained control over speech prosody features. Finally, the speech duration, energy, and pitch features are fused with the training text features to output predicted speech features. Figure 10 It is known that the variable adapter includes at least one second expert network, a third gating network, and a speech feature prediction network. The third gating network is used to select the second expert network according to different training text features to output predicted speech features that meet the requirements.

[0081] Please see Figure 11 In some embodiments, step S801 includes, but is not limited to, steps S1101 to S1104: Step S1101: Weights are assigned to each second expert network through the third gating network and the generated speech features to obtain the third weight matrix; Step S1102: Predict speech features from the training text features using a second expert network to obtain the first speech features; Step S1103: At least one first speech feature is concatenated according to the third weight matrix to obtain a second speech feature; Step S1104: Use the speech feature prediction network and the second speech feature to predict the speech features of the target text and obtain the predicted speech features.

[0082] In step S1101 of some embodiments, such as Figure 10 As shown, the second gating network is defined as Gata. The generated speech features are input into the second gating network, and weights are assigned to each second expert network based on the generated speech features to determine a third weight matrix. The third weight matrix represents the selective activation of different second expert networks. Specifically, at least one second expert network is defined as... The third weight matrix includes at least one third weight value, each of which represents the activation state of its corresponding second expert network and also represents the importance of the second expert network output to the predicted speech features. It should be noted that the specific process of the second gating network outputting the third weight matrix is ​​the same as that of the first gating network, and will not be repeated here.

[0083] In step S1102 of some embodiments, such as Figure 10 As shown, the training text features are input into the second expert network. The second expert network predicts speech features based on the training text features to generate the first speech features. The first speech features output by each second expert network are not necessarily the same.

[0084] In step S1103 of some embodiments, as disclosed above, the third weight matrix includes at least one third weight value, and at least one first speech feature is weighted and summed or weighted and averaged according to the third weight value to determine the second speech feature. Therefore, in this embodiment, the third gating network dynamically selects the optimal second expert network based on the input generated speech features, realizing adaptive acoustic modeling under multiple conditions, significantly enhancing the speech generation model's ability to transfer speech objects in zero-cue scenarios and its synthesis robustness under noisy conditions.

[0085] In step S1104 of some embodiments, such as Figure 10 As shown, the speech feature prediction network consists of a two-layer one-dimensional convolutional network with ReLU activation, with each layer followed by a normalization and Dropout layer. Finally, a linear layer outputs the predicted speech features. It should be noted that the speech feature prediction network incorporates three key acoustic features—duration, pitch, and energy—from the second speech feature, injecting speech variation information into the training text features to determine the predicted speech features.

[0086] In steps S1101 to S1104 of this embodiment, a second expert network predicts the first speech feature from the training text features. A third gating network is then used to assign weights to the second expert network to determine a third weight matrix. Based on this third weight matrix, the first speech feature is concatenated into a second speech feature, achieving adaptive learning modeling under multiple conditions to output a more accurate second speech feature. Finally, a speech feature prediction network incorporates three key acoustic features—duration, pitch, and energy—from the second speech feature to output predicted speech features. This enables the generation of predicted speech features consistent with the speaking style of the reference speaker, even under noisy cues.

[0087] In some embodiments, please refer to Figure 12 The spectrum decoder is responsible for converting the predicted speech features modulated by the variable adapter into predicted spectrum features that are perceived by the human ear. The spectrum decoder includes at least one third expert network, a spectrum decoding network, and a fourth gating network; the at least one third expert network is defined as... The fourth gating network assigns weight values ​​to each third expert network based on the generated speech features, so as to dynamically select the output of the third expert network, which can output predictive spectral features with high naturalness and the same timbre as the reference speaker.

[0088] Please see Figure 13 In some embodiments, step S802 may include, but is not limited to, steps S1301 to S1304: Step S1301: Weights are assigned to each third expert network through the fourth gating network and the generated speech features to obtain the fourth weight matrix; Step S1302: Perform spectral prediction on the predicted speech features through a third expert network to obtain the first spectral features; Step S1303: At least one first spectral feature is concatenated according to the fourth weight matrix to obtain a second spectral feature; Step S1304: Use the speech feature prediction network and the second speech feature to predict the speech features of the target text and obtain the predicted speech features.

[0089] In step S1301 of some embodiments, the spectrum decoder of this embodiment adopts a MoE-FFT structure to generate high-fidelity predicted spectrum features from the predicted speech features represented in the latent space. This embodiment also includes a fourth gating network in the spectrum decoder. This fourth gating network selects different third expert networks to participate in spectrum feature prediction, allowing the decoding process to flexibly adjust the spectrum modeling path according to different acoustic contexts or noise categories to generate predicted audio features with high fidelity and consistent timbre with the reference speaker. Specifically, the fourth gating network scores each third expert network for the generated speech features, selects a third expert network to participate in spectrum feature prediction, and outputs a fourth weight matrix. The fourth weight matrix includes at least one fourth weight value, each fourth weight value representing the importance of the third expert network in spectrum feature prediction.

[0090] In step S1302 of some embodiments, the third expert network performs spectral prediction on the predicted speech features to output first spectral features, which can construct first spectral features that are consistent with the timbre of the reference speaker and whose content matches the training text.

[0091] In step S1303 of some embodiments, the first spectral features are spliced ​​according to the fourth weight matrix. Specifically, at least one first spectral feature is weighted and summed or weighted and averaged according to the fourth weight value to output the second spectral feature.

[0092] In step S1304 of some embodiments, such as Figure 12 As shown, the spectrum decoding network is a normalization layer. The second spectrum feature is normalized by the normalization layer to output the predicted spectrum feature, which is used as the acoustic intermediate representation.

[0093] Specifically, both the second and third expert networks use generated speech features as control signal input, and the calculation of the dynamic adjustment of training text features or predicted speech feature mapping process is as follows: (3) in, These are input features. It generates speech features. It is either a second expert network that takes training text features as input, or a third expert network that takes predicted speech features as input. . It is a gated network that takes the generated speech features as input, and can be a third gated network or a fourth gated network. These are the activation weights corresponding to each second or third expert network. A Top-k selection strategy is also used here, retaining the k highest weight values ​​and forcibly setting the remaining Nk weight values ​​to 0.

[0094] Therefore, by using expert networks, conditional expert selection and feature adaptation can be achieved during speech synthesis, resulting in more accurate predicted speech features and predicted spectral features.

[0095] In steps S1301 to S1304 of this embodiment, by adding a third expert network and a fourth gating network to the spectrum decoder, the third expert network outputs a more accurate first spectral feature, and the fourth gating network selects different first spectral features output by the third expert network to participate in the final spectrum prediction. This can adjust the spectrum prediction process according to different acoustic contexts or noise categories, and generate predicted spectral features that are consistent with the timbre of the reference speaking object and have a higher degree of fidelity.

[0096] In step S107 of some embodiments, the model parameters of the preset speech generation model are adjusted simultaneously during the adjustment of model parameters of the preset speech generation model. Specifically, a first loss function is constructed from the enhanced speech data and the original speech data, and the parameters of the preset enhancement model are adjusted according to the first loss function until the first loss function converges, thus completing the parameter adjustment of the preset enhancement model and outputting the target enhancement model. Simultaneously, a second loss function is constructed from the preset speech data and the predicted speech data, and the model parameters of the preset speech generation model are adjusted according to the first and second loss functions until the first and second loss functions converge, thus completing the model parameter adjustment and outputting the target speech generation model.

[0097] In step S108 of some embodiments, after the target speech enhancement model and the target speech generation model are determined, the target speech enhancement model can be used alone for speech enhancement to filter out noise data in the speech data. When the target speech generation model is actually applied, target text and target prompt speech data of the target speaker are acquired. The target prompt speech data is obtained by recording the speech of the target speaker and is used to instruct the target speech generation model to output speech data with the same timbre as the target speaker. It should be noted that the target speech generation model is obtained by training the preset speech generation model in conjunction with the preset speech enhancement model. It can dynamically select the optimal feature combination according to the noise category, acoustic environment, and characteristics of the target speaker in the input target prompt speech data, so as to maintain the stability and consistency of speech generation even under noisy prompt conditions. Therefore, the target speech generation model, in conjunction with the target prompt speech data, generates speech for the target text and outputs the target speech data.

[0098] It should be noted that when the trained target speech generation model is applied to the insurance intelligent customer service scenario, the target prompt speech data of insurance agents is set in advance. The target speech generation model learns the speech style in the target prompt speech data in advance. Then, based on the questions raised by the policyholder, the question-and-answer model automatically outputs the reply text. The target speech generation model can convert the reply text into reply speech data and play it, realizing voice communication between the policyholder and the intelligent customer service. Moreover, the voice content output during the voice communication process is clear and stable, which can improve the policyholder's experience in the process of consulting insurance business.

[0099] It should be noted that the process of the target speech generation model generating target speech data is the same as that of the preset speech generation model generating predicted speech data, and will not be repeated here.

[0100] like Figure 3 As shown, in this embodiment of the application, when training the preset speech enhancement model and the preset speech generation model, noise is first added to the original speech data to obtain training prompt speech data. This training prompt speech data is then input into a speech encoder, which performs speech encoding to output training speech features. Then, a feature selector based on MoE filters the training speech features for noise feature filtering to output selected speech features. It should be noted that when the feature selector outputs the selected speech features, at least one first expert network is fused. Each first expert network shares the input training speech features, but the parameters of each first expert network are independent to ensure feature decomposition capability. Each first expert network is implemented as a two-layer feedforward network, projecting high-dimensional training speech features into a low-dimensional embedding space to output candidate speech features. Simultaneously, a first gating network and a second gating network are set, and a first weight value is assigned to the first expert network according to the training speech features to achieve the construction of the optimal feature combination, determining the generated speech features and enhanced speech features, and defining TTS features and SE features respectively. By inputting the enhanced speech features and training cue speech data together into the speech decoder, which is also a speech enhancement decoder, the input training cue speech data and enhanced speech features are fused based on an attention mechanism to achieve more context-aware speech enhancement under noisy cues, generating enhanced speech data.

[0101] Furthermore, the preset speech generation model consists of a text encoder, a variable adapter, a spectrum decoder, and a first vocoder. The variable adapter includes a third gating network and a second expert network, while the spectrum decoder includes a fourth gating network and a third expert network. Specifically, the variable adapter is used to explicitly model features such as pitch, energy, and duration, enabling fine-grained control over speech prosody features. The second expert network is integrated into the variable adapter, allowing for dynamic selection based on the input generated speech features. This enables adaptive acoustic modeling under multiple conditions, significantly enhancing the preset speech generation model's ability to transfer speech recipients in zero-voice-prompt scenarios and its robustness in speech synthesis under noisy conditions. Therefore, the variable adapter and generated speech features predict predicted speech features from the predicted text features. The spectrum decoder and generated speech features then predict the spectral features of the predicted speech features to output predicted spectral features. Finally, the first vocoder reconstructs the waveform of the predicted spectral features, outputting predicted speech data with high naturalness and timbre consistent with the reference speaker.

[0102] Finally, a first loss function is constructed based on the enhanced speech data and the original speech data, and a second loss function is constructed based on the predicted speech data and the preset speech data. A preset speech generation model is trained using both the first and second loss functions, and a preset speech enhancement model is trained using the first loss function, achieving collaborative optimization between the preset speech generation and enhancement models. After training, it is determined that both the target speech enhancement model and the target speech generation model can be deployed independently, working in an end-to-end joint mode, or performing speech enhancement or speech generation separately according to actual application needs. This offers greater flexibility and lower engineering cost, while also improving the naturalness and noise robustness of the generated speech.

[0103] Please see Figure 14 This application also provides a speech generation apparatus that can implement the above-described speech generation method. The apparatus includes: The data acquisition module 1401 is used to acquire training text and training prompt speech data of the reference speaker; wherein, the training prompt speech data is speech data obtained by adding noise to the original speech data. The speech encoding module 1402 is used to encode the training prompt speech data through a preset speech enhancement model to obtain training speech features; The noise feature filtering module 1403 is used to filter noise features from training speech features using a preset speech enhancement model to obtain selected speech features. The speech decoding module 1404 is used to decode the selected speech features and the training prompt speech data through a preset speech enhancement model to obtain enhanced speech data; The text encoding module 1405 is used to encode the training text using a preset speech generation model to obtain the training text features. The training speech generation module 1406 is used to generate speech from training text features using a preset speech generation model and selected speech features to obtain predicted speech data. The model parameter adjustment module 1407 adjusts the model parameters of the preset speech generation model based on preset speech data, predicted speech data, enhanced speech data, and original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording training text; The target speech generation module 1408 is used to acquire the target text and the target prompt speech data of the target speaker, and to generate speech from the target text using the target speech generation model and the target prompt speech data to obtain the target speech data.

[0104] The specific implementation of this speech generation device is basically the same as the specific embodiment of the speech generation method described above, and will not be repeated here.

[0105] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech generation method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0106] Please see Figure 15 , Figure 15 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 using the speech generation method of the embodiments of this application. The input / output interface 1503 is used to implement information input and output; The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504); The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech generation method.

[0108] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The speech generation method, apparatus, computer device, and storage medium provided in this application embodiment optimize a preset speech enhancement model and a preset speech generation model collaboratively. Specifically, noise is first added to the original speech data to generate training prompt speech data. The preset speech generation model filters the noise features in the training prompt speech data and outputs selected speech features, generating enhanced speech data based on the selected speech features. Simultaneously, training text features are extracted from the training text according to the preset speech generation model, and the selected speech features are used in the speech prediction process to output predicted speech data. The preset speech generation model is optimized by jointly analyzing the predicted speech data, preset speech data, enhanced speech data, and original speech data to determine the target speech generation model. Therefore, during the training process of the preset speech generation model, it learns under noisy prompt speech and, in conjunction with the speech enhancement function of the speech enhancement model, generates speech data with a more natural timbre consistent with the reference speaker, thus training a target speech generation model that can stably output a timbre consistent with the reference speaker. Therefore, in the application of the target speech generation model, the target text is generated by using the target speech generation model and the target prompt speech data to output target speech data with a timbre consistent with the target speaker and with stable and natural tone and content.

[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech generation method, characterized in that, The method includes: Acquire training text and training prompt speech data of a reference speaker; wherein, the training prompt speech data is speech data obtained by adding noise to the original speech data; The training prompt speech data is encoded using a preset speech enhancement model to obtain training speech features; The selected speech features are obtained by filtering noise features from the training speech features using the preset speech enhancement model; The selected speech features and the training prompt speech data are decoded using the preset speech enhancement model to obtain enhanced speech data. The training text is encoded using a preset speech generation model to obtain training text features; The training text features are used to generate speech using the preset speech generation model and the selected speech features to obtain predicted speech data. The model parameters of the preset speech generation model are adjusted based on the preset speech data, the predicted speech data, the enhanced speech data, and the original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording the training text; The target text and the target prompt speech data of the target speaker are obtained. The target speech data is obtained by generating speech from the target text using the target speech generation model and the target prompt speech data.

2. The method according to claim 1, characterized in that, The preset speech enhancement model includes: a speech encoder, a feature selector, and a speech decoder; the feature selector includes at least one first expert network, a first gating network, and a second gating network; the selected speech features include: enhanced speech features and generated speech features. The step of filtering noise features from the training speech features using the preset speech enhancement model to obtain selected speech features includes: The first expert network is weighted using the first gating network and the trained speech features to obtain a first weight matrix. The first expert network is weighted using the second gating network and the trained speech features to obtain a second weight matrix; Candidate speech features are obtained by filtering noise features from the trained speech features using each of the first expert networks. The enhanced speech features are obtained by concatenating at least one of the candidate speech features using the first weight matrix. The generated speech features are obtained by concatenating at least one of the candidate speech features using the second weight matrix.

3. The method according to claim 2, characterized in that, The step of assigning weights to the first expert network using the first gating network and the trained speech features to obtain a first weight matrix includes: The importance of the first expert network is measured using the first gating network and the trained speech features to obtain importance measurement data. The importance measurement data is normalized to obtain the first weight value for each of the first expert networks; The first weight values ​​are concatenated to obtain the first weight matrix.

4. The method according to claim 2, characterized in that, The preset speech generation model includes: a text encoder, a variable adapter, a spectrum decoder, and a first vocoder; The step of generating speech from the training text features using the preset speech generation model and the selected speech features to obtain predicted speech data includes: The training text features are used to predict speech features through the variable adapter and the generated speech features to obtain predicted speech features. The predicted spectral features are obtained by using the spectrum decoder and the generated speech features to predict the spectral features of the predicted speech features. The predicted speech data is obtained by outputting the predicted spectral features into speech using the first vocoder.

5. The method according to claim 4, characterized in that, The variable adapter includes at least one second expert network, a third gating network, and a speech feature prediction network; The step of predicting speech features from the training text features using the variable adapter and the generated speech features to obtain predicted speech features includes: The third weight matrix is ​​obtained by assigning weights to each of the second expert networks using the third gating network and the generated speech features. The training text features are predicted by the second expert network to obtain the first speech feature; At least one of the first speech features is concatenated according to the third weight matrix to obtain the second speech feature; The predicted speech features are obtained by predicting speech features of the target text features using the speech feature prediction network and the second speech feature.

6. The method according to claim 4, characterized in that, The spectrum decoder includes: at least one third expert network, a spectrum decoding network, and a fourth gating network; the step of predicting the predicted spectrum features by using the spectrum decoder and the generated speech features to obtain the predicted spectrum features includes: The fourth weight matrix is ​​obtained by assigning weights to each of the third expert networks using the fourth gating network and the generated speech features. The predicted speech features are then subjected to spectral prediction using the third expert network to obtain the first spectral features. At least one of the first spectral features is concatenated according to the fourth weight matrix to obtain the second spectral feature; The predicted spectral features are obtained by normalizing the second spectral features using the spectral decoding network.

7. The method according to any one of claims 2 to 6, characterized in that, The speech decoder includes: an attention network, a feedforward network, and a second vocoder; The step of decoding the selected speech features and the training prompt speech data using the preset speech enhancement model to obtain enhanced speech data includes: The training prompt speech data is projected into key features and value features through the attention network; The enhanced speech features are projected into query features using the attention network. The value features are fused based on the query features and the key features to obtain fused features; The fused features are normalized using the feedforward network to obtain normalized features; The normalized features are processed by the second vocoder to generate speech, thereby obtaining the enhanced speech data.

8. A speech generation device, characterized in that, The device includes: The data acquisition module is used to acquire training text and training prompt speech data of the reference speaker; wherein, the training prompt speech data is speech data obtained by adding noise to the original speech data; The speech encoding module is used to encode the training prompt speech data through a preset speech enhancement model to obtain training speech features; The noise feature filtering module is used to filter noise features from the training speech features through the preset speech enhancement model to obtain selected speech features. The speech decoding module is used to decode the selected speech features and the training prompt speech data through the preset speech enhancement model to obtain enhanced speech data; The text encoding module is used to encode the training text using a preset speech generation model to obtain training text features; The training speech generation module is used to generate speech from the training text features using the preset speech generation model and the selected speech features, so as to obtain predicted speech data. The model parameter adjustment module adjusts the model parameters of the preset speech generation model based on the preset speech data, the predicted speech data, the enhanced speech data, and the original speech data to obtain the target speech generation model; wherein, the preset speech data is obtained by recording the training text; The target speech generation module is used to acquire target text and target prompt speech data of the target speaker, and generate speech from the target text using the target speech generation model and the target prompt speech data to obtain target speech data.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech generation method according to any one of claims 1 to 7.

Images