A speech synthesis method, device, computer equipment and storage medium

Abstract

The embodiment of the application belongs to the technical field of audio processing, and relates to a speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: calling a configured text-to-speech model, and obtaining first embedding data corresponding to synthesis object identification data in a lookup table of the text-to-speech model, wherein the configured text-to-speech model is embedded with an adapter; performing a feature extraction operation on new object speech data to obtain second embedding data; performing a splicing operation on the first embedding data and the second embedding data to obtain comprehensive embedding data; performing a parameter adjustment operation on the adapter according to the comprehensive embedding data to obtain a fine-tuned text-to-speech model; inputting the comprehensive embedding data and synthesis text data into the fine-tuned text-to-speech model to perform a speech synthesis operation, and obtaining synthesis audio data; and outputting the synthesis audio data to a user terminal. The application greatly reduces the demand for computing resources.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speech synthesis method, apparatus, computer device, and storage medium. Background Technology

[0002] In recent years, thanks to deep learning technology, text-to-speech (TTS) models have made significant progress. These models are able to generate realistic human speech after being trained on high-quality recordings of single or multiple speakers for several hours.

[0003] However, while fine-tuning techniques are widely used to adapt TTS models to new speakers, they face several challenges. Generally, several hours of high-quality speech data are required to fine-tune each speaker. Moreover, this adjustment may compromise the model's speech synthesis quality for already learned speakers. To enable a TTS model to adapt to a new speaker's voice, a large amount of speech data is typically needed for fine-tuning, making it costly to scale the TTS model to a wide range of speakers. Therefore, traditional text-to-speech (TTS) models require a large amount of speech data for fine-tuning to adapt to new speakers, resulting in low efficiency and high costs. Summary of the Invention

[0004] The purpose of this application is to provide a speech synthesis method, apparatus, computer device, and storage medium to solve the problems of low efficiency and high cost of traditional text-to-speech (TTS) models.

[0005] To address the aforementioned technical problems, this application provides a speech synthesis method, employing the following technical solution:

[0006] The system receives a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data;

[0007] The configured text-to-speech model is invoked, and the first embedded data corresponding to the synthesis object identifier data is obtained from the lookup table of the text-to-speech model, wherein the configured text-to-speech model has an embedded adapter.

[0008] Feature extraction is performed on the new object's speech data to obtain the second embedded data;

[0009] The first embedded data and the second embedded data are concatenated to obtain the composite embedded data.

[0010] Based on the comprehensive embedded data, the adapter is adjusted to obtain a fine-tuned text-to-speech model.

[0011] The integrated embedded data and the synthesized text data are input into the fine-tuned text-to-speech model to perform speech synthesis operation, thereby obtaining synthesized audio data.

[0012] The synthesized audio data is output to the user terminal.

[0013] Furthermore, after the step of receiving the speech synthesis request sent by the user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data, the following steps are also included:

[0014] Preprocessing operations are performed on the voice data of the new object.

[0015] Furthermore, the step of performing feature extraction on the new object's speech data to obtain the second embedded data specifically includes the following steps:

[0016] A short-time Fourier transform operation is performed on the speech data of the new object to obtain a reference spectrogram;

[0017] Speech feature data is obtained by performing feature extraction on the reference spectrogram using a convolutional recurrent neural network encoder.

[0018] Obtain the global style token of the new object's speech data;

[0019] The speech feature data and the global style token are input into a multi-head attention layer for weighting to obtain the second embedding data.

[0020] Furthermore, after the step of performing a short-time Fourier transform operation on the new object's speech data to obtain a reference spectrogram, the following step is also included:

[0021] The reference spectrogram is augmented and denoised using an autoencoder network.

[0022] Furthermore, before the step of calling the configured text-to-speech model and retrieving the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model, the following step is also included:

[0023] Read the system database and obtain model training data from the system database;

[0024] Call the initial text-to-speech model, input the model training data into the initial text-to-speech model, and perform model training operation on the initial text-to-speech model according to the model loss function to obtain the trained text-to-speech model;

[0025] The trained text-to-speech model is weighted, and the adapter is embedded into the weighted text-to-speech model to obtain the configured text-to-speech model.

[0026] Furthermore, the model loss function L total Represented as:

[0027] L total =L mel +L pitch +L duration +ω aligner ·L aligner

[0028]

[0029] Among them, y mel,i and Let p represent the target value and the predicted value of the Mel spectrogram at the i-th time step, respectively. i and Let d represent the target pitch value and the predicted pitch value at the i-th time step, respectively; i and represents the target value and predicted value of the duration at the i-th time step, respectively; N represents the total number of time steps, and aligner_loss_function represents the function for calculating the alignment loss.

[0030] To address the aforementioned technical problems, this application also provides a speech synthesis device, which employs the following technical solution:

[0031] The request receiving module is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data;

[0032] The first embedded data acquisition module is used to call the configured text-to-speech model and obtain the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model, wherein the configured text-to-speech model has an embedded adapter.

[0033] The feature extraction module is used to perform feature extraction operations on the new object's speech data to obtain the second embedded data;

[0034] The splicing module is used to splice the first embedded data and the second embedded data to obtain comprehensive embedded data;

[0035] The parameter adjustment module is used to perform parameter adjustment operations on the adapter based on the comprehensive embedded data to obtain a fine-tuned text-to-speech model.

[0036] The speech synthesis module is used to input the integrated embedded data and the synthesized text data into the fine-tuned text-to-speech model to perform speech synthesis operations and obtain synthesized audio data.

[0037] The result output module is used to output the synthesized audio data to the user terminal.

[0038] Furthermore, the feature extraction module includes:

[0039] The short-time Fourier transform submodule is used to perform a short-time Fourier transform operation on the speech data of the new object to obtain a reference spectrogram;

[0040] The feature extraction submodule is used to perform feature extraction operations on the reference spectrogram based on the convolutional recurrent neural network encoder to obtain speech feature data;

[0041] The style token acquisition submodule is used to acquire the global style token of the new object's speech data;

[0042] The weighting submodule is used to input the speech feature data and the global style token into the multi-head attention layer for weighting operations to obtain the second embedded data.

[0043] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0044] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech synthesis method as described above.

[0045] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0046] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method described above.

[0047] This application provides a speech synthesis method, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data; invoking a configured text-to-speech model and obtaining first embedding data corresponding to the synthesis object identification data from a lookup table of the text-to-speech model, wherein the configured text-to-speech model embeds an adapter; performing feature extraction on the new object speech data to obtain second embedding data; concatenating the first embedding data and the second embedding data to obtain comprehensive embedding data; adjusting the parameters of the adapter according to the comprehensive embedding data to obtain a fine-tuned text-to-speech model; inputting the comprehensive embedding data and the synthesis text data into the fine-tuned text-to-speech model for speech synthesis to obtain synthesized audio data; and outputting the synthesized audio data to the user terminal. Compared with existing technologies, this application introduces tiny adapter modules into the base model to ensure that the weights of the base model are not modified. The adapter parameters are adjusted only for the speech data of new speakers. This significantly reduces the demand for computing resources without sacrificing speech naturalness, speaker recognition, and prosodic consistency. Even in the case of scarce data, this application can still maintain good performance. Attached Figure Description

[0048] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0050] Figure 2 This is a flowchart illustrating the implementation of the speech synthesis method provided in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0052] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0057] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0058] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0059] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0060] It should be noted that the speech synthesis method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the speech synthesis device is generally set in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0062] Continue to refer to Figure 2 The diagram shows a flowchart of an embodiment of the speech synthesis method according to this application. The speech synthesis method includes steps S201, S202, S203, S204, S205, S206, and S207.

[0063] In step S201, a speech synthesis request sent by a user terminal is received, wherein the speech synthesis request includes at least new object speech data, synthesized object identification data, and synthesized text data.

[0064] In the embodiments of this application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided in this application. The user terminal may be a mobile terminal such as a mobile phone, smartphone, laptop, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), navigation device, etc., as well as a fixed terminal such as a digital TV, desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not intended to limit this application.

[0065] In step S202, the configured text-to-speech model is invoked, and the first embedded data corresponding to the synthesis object identifier data is obtained from the lookup table of the text-to-speech model. The configured text-to-speech model has an adapter embedded in it.

[0066] In this embodiment, the base model uses FastPitch as the base TTS model. The FastPitch model consists of four parts, including two feedforward transform (FFT) stacks as a phoneme encoder and a Mel decoder, and two convolutional modules as pitch and duration predictors. The encoder processes the input phoneme token x and produces a hidden state h, which is used by the pitch and duration predictors to predict the average pitch p^ and duration d^ for each token, respectively. The decoder receives a length-adjusted hidden representation of the sum of the encoder output h and the pitch p^ to generate a Mel spectrogram sequence y^. To train the pitch predictor, this application uses the true pitch p extracted by PYIN and averages the input tokens. For the duration predictor, this application uses a learnable aligner. The training loss consists of the mean squared error between the predicted and true modalities plus the alignment loss.

[0067] Let L mel For Mel spectrogram loss, L pitch For pitch loss, L duration For duration loss, L aligner For alignment loss, ω aligner The weights are used to align the loss. Then the total loss L... total It can be represented as:

[0068] L total =L mel +L pitch +L duration +ω aligner ·L aligner

[0069]

[0070]

[0071] Among them, y mel,i and Let p represent the target value and the predicted value of the Mel spectrogram at the i-th time step, respectively. i and Let d represent the target pitch value and the predicted pitch value at the i-th time step, respectively; i and represents the target value and predicted value of the duration at the i-th time step, respectively; N represents the total number of time steps, and aligner_loss_function represents the function for calculating the alignment loss.

[0072] In this embodiment, the original layer normalization layers are replaced with CLN adapters in the encoder, pitch and duration predictor, and decoder of the model. Each adapter is a tiny module embedded between the layers of a pre-trained and frozen network. During training, only the parameters of these adapters are updated with gradients, while the rest of the network remains unchanged. The adapter layers typically compress the input data to a low-dimensional space using a downsampling feedforward network, followed by processing through a non-linear activation function and an upsampling feedforward network. To ensure training stability, an initialization method close to the identity mapping is required, thus the adapters contain skip connections. This design ensures that the performance of the original network is not affected in the early stages of training. This application also provides optional dropout, layer normalization, and operations to initialize the final layer to zero to ensure that the module can perform the identity transformation. Placing the adapters after the output of each transformer layer allows them to be more widely applied to different modules.

[0073] In step S203, feature extraction is performed on the new object's speech data to obtain the second embedded data.

[0074] In this embodiment, the feature extraction operation may involve performing a short-time Fourier transform on the new object's speech data to obtain a reference spectrogram; performing feature extraction on the reference spectrogram using a convolutional recurrent neural network encoder to obtain speech feature data; obtaining a global style token for the new object's speech data; and inputting the speech feature data and the global style token into a multi-head attention layer for weighting to obtain second embedding data.

[0075] In step S204, the first embedded data and the second embedded data are spliced ​​together to obtain the comprehensive embedded data.

[0076] In this embodiment, conventional speaker representation methods are implemented by adding a speaker embedding table. A limitation of this approach is its inability to generalize to new speakers not encountered during training. Therefore, this application combines the speaker embedding (SE1) obtained from the lookup table with the speaker embedding (SE2) obtained from a reference spectrogram of a specific speaker and global style tokens. The reference spectrogram is fed into a convolutional recurrent neural network (RNN) encoder, followed by a multi-head attention layer. The attention module outputs weights, summing these style tokens as the speaker representation embedding. The final speaker embedding, SEfinal, is obtained by adding SE1 and SE2.

[0077] In the embodiments of this application, the application can learn tokens without any explicit style or prosodic labels, but can still learn a wide range of acoustic expressiveness, such as speech rate, speaker identity and speaking style.

[0078] In the multi-speaker FastPitch model of this application, the various parts of FastPitch, including the encoder, decoder, pitch prediction, duration prediction, and aligner, are conditionally processed by using the speaker representation as additional input. The input of each component is combined with the speaker representation. This application uses Conditional Layer Normalization (CLN) to condition the model to fit the characteristics of a specific speaker. The conditional network consists of two linear layers that transform the speaker representation into CL.

[0079] In step S205, the adapter is adjusted according to the integrated embedded data to obtain the fine-tuned text-to-speech model.

[0080] In step S206, the integrated embedded data and the synthesized text data are input into the fine-tuned text-to-speech model to perform speech synthesis operation, and synthesized audio data is obtained.

[0081] In step S207, synthesized audio data is output to the user terminal.

[0082] In this embodiment, several small adapter modules are embedded in the original network. The original network weights are kept unchanged, and the adapters are fine-tuned only for the speech data of new speakers. This efficient parameter fine-tuning method can create a new model that shares a large number of parameters with the original model. First, a basic multi-speaker TTS model is pre-trained on a large and diverse TTS dataset. To extend the model to new speakers, several adapters—small modules—are added to the basic model. This application uses a standard adapter, a unified adapter, or BitFit. Next, the pre-trained model is frozen, and the adapters are fine-tuned only on the new speaker data. The outputs of multiple adapters are combined for model inference.

[0083] In this embodiment, a speech synthesis method is provided, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data; invoking a configured text-to-speech model and obtaining first embedded data corresponding to the synthesis object identification data from a lookup table of the text-to-speech model, wherein the configured text-to-speech model embeds an adapter; performing feature extraction on the new object speech data to obtain second embedded data; concatenating the first embedded data and the second embedded data to obtain comprehensive embedded data; adjusting the parameters of the adapter according to the comprehensive embedded data to obtain a fine-tuned text-to-speech model; inputting the comprehensive embedded data and the synthesis text data into the fine-tuned text-to-speech model for speech synthesis to obtain synthesized audio data; and outputting the synthesized audio data to the user terminal. Compared with existing technologies, this application introduces tiny adapter modules into the base model to ensure that the weights of the base model are not modified. The adapter parameters are adjusted only for the speech data of new speakers. This significantly reduces the demand for computing resources without sacrificing speech naturalness, speaker recognition, and prosodic consistency. Even in the case of scarce data, this application can still maintain good performance.

[0084] In some optional implementations of the embodiments of this application, after receiving the speech synthesis request sent by the user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data, the following steps are further included:

[0085] Perform preprocessing operations on the new object's speech data.

[0086] In the embodiments of this application, the preprocessing operation aims to remove irrelevant information and enhance the effective features in the speech signal, thereby improving the accuracy of subsequent analysis and recognition processes.

[0087] In this embodiment, the preprocessing operations include sampling and quantization, denoising, pre-emphasis, framing and windowing, feature extraction, and normalization. Specifically:

[0088] (1) Sampling and quantization:

[0089] Sampling: According to the Nyquist sampling theorem, analog speech signals are converted into digital signals. The sampling frequency is usually chosen to be more than twice the highest frequency of the speech signal to avoid aliasing.

[0090] Quantization: Mapping the analog value of each sampled point to a finite number of values, usually by choosing an appropriate number of bits (such as 16 bits) to represent each sampled value.

[0091] (2) Noise reduction:

[0092] Techniques such as digital filters and noise masking are used to remove noise components from speech signals. Digital filters can filter speech signals according to specific noise characteristics, while noise masking cancels out noise by adjusting the volume and frequency of the speech signal.

[0093] (3) Pre-weighting:

[0094] Pre-emphasis processing of the input digital speech signal aims to emphasize the high-frequency components of the speech, remove the influence of lip radiation, and increase the high-frequency resolution of the speech. This pre-emphasis is typically achieved using a first-order FIR high-pass digital filter.

[0095] (4) Frame division and windowing:

[0096] Because speech signals have short-term stationarity, they can be divided into short segments (called analysis frames) for processing. Frame segmentation is usually achieved using a weighted method with movable, finite-length windows; commonly used window functions include rectangular windows and Hamming windows.

[0097] (5) Feature extraction:

[0098] Parameters representing speech characteristics are extracted from the framed speech signal. Commonly used feature extraction methods include MFCC, LPC, and PLP. Among them, MFCC simulates the human auditory system, mapping the frequency domain to the Mel frequency domain, and then extracts cepstral coefficients that can represent speech characteristics; LPC determines the prediction coefficients by minimizing the prediction error, and these coefficients can represent the spectral characteristics of the speech signal.

[0099] (6) Standardization:

[0100] The extracted feature data is standardized to ensure that all data are within the same scale. This helps to avoid errors and biases caused by different data scales.

[0101] In some optional implementations of the embodiments of this application, the step of performing feature extraction on the new object's speech data to obtain the second embedded data specifically includes the following steps:

[0102] A short-time Fourier transform is performed on the speech data of the new object to obtain a reference spectrogram;

[0103] Speech feature data is obtained by performing feature extraction on the reference spectrogram using a convolutional recurrent neural network encoder.

[0104] Retrieve the global style token for the new object's speech data;

[0105] The speech feature data and global style token are input into the multi-head attention layer for weighting to obtain the second embedding data.

[0106] In the embodiments of this application, the reference spectrogram is a spectral representation of the speaker's speech, typically obtained through short-time Fourier transform (STFT) or other similar methods. It provides the time-frequency characteristics of the speaker's speech.

[0107] In this embodiment, the global style token represents a series of global style features learned from a large amount of data. They can be viewed as a kind of "style vector" used to describe the speaker's speech style or characteristics.

[0108] In this embodiment, the convolutional recurrent neural network encoder first processes a reference spectrogram to extract features. Convolutional layers may be used to capture local features, while RNN layers are used to capture dependencies in the time series. The encoder's output is a high-level representation that includes the speaker's speech features.

[0109] In this embodiment, the multi-head attention layer receives the encoder output and global style tokens as input. The attention mechanism calculates the correlation or weight between each style token and the encoder output. These weights are used to weight and sum the style tokens to generate a weighted style representation.

[0110] In this embodiment, the powerful feature extraction capabilities of deep learning models and the flexibility of attention mechanisms are combined to generate a rich and accurate speaker representation.

[0111] In some optional implementations of the embodiments of this application, after the step of performing a short-time Fourier transform operation on the new object speech data to obtain a reference spectrogram, the following steps are further included:

[0112] The reference spectrogram is augmented and denoised using an autoencoder network.

[0113] In this embodiment, the autoencoder network is a deep learning model that learns a compressed representation of the input data (encoding) and attempts to reconstruct the original data from that representation (decoding) to achieve feature extraction and dimensionality reduction. In the field of signal processing, especially in audio signal processing, autoencoder networks can be used to complete information and denoise reference spectrograms.

[0114] In the embodiments of this application, information completion refers to the process by which an autoencoder network can predict and fill in the missing parts when the reference spectrogram is partially missing due to some reason (such as loss or damage during signal transmission). This process is similar to interpolation or inpainting techniques in image inpainting, but is more complex and sophisticated.

[0115] In this embodiment, denoising refers to the fact that noise is a common problem in audio signal processing. By training an autoencoder network, it can learn to extract features of a clean audio signal from a noisy reference spectrogram. Specifically, the network takes the noisy reference spectrogram as input and attempts to reconstruct a noise-free, original reference spectrogram. In this process, the network automatically ignores or reduces the influence of noise, thereby achieving denoising of the audio signal.

[0116] In this embodiment, an autoencoder network is used for information completion and noise reduction. By inputting the reference spectrogram into the encoder part of the autoencoder, its low-dimensional feature representation is extracted, and then the complete reference spectrogram is reconstructed through the decoder part, so as to reduce the information loss and distortion problems introduced during network transmission.

[0117] In some optional implementations of the embodiments of this application, before the step of calling the configured text-to-speech model and obtaining the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model, the following steps are further included:

[0118] Read the system database and retrieve the model training data from the system database;

[0119] Call the initial text-to-speech model, input the model training data into the initial text-to-speech model, and perform model training operation on the initial text-to-speech model according to the model loss function to obtain the trained text-to-speech model;

[0120] The trained text-to-speech model is weighted, and the adapter is embedded into the weighted text-to-speech model to obtain the configured text-to-speech model.

[0121] In some optional implementations of the embodiments of this application, the model loss function L total Represented as:

[0122] L total =L mel +L pitch +L duration +ω aligner ·L aligner

[0123]

[0124] Among them, y mel,i and Let p represent the target value and the predicted value of the Mel spectrogram at the i-th time step, respectively. i and Let d represent the target pitch value and the predicted pitch value at the i-th time step, respectively; iand represents the target value and predicted value of the duration at the i-th time step, respectively; N represents the total number of time steps, and aligner_loss_function represents the function for calculating the alignment loss.

[0125] In some optional implementations of this application's embodiments, to fine-tune the frozen pre-trained FastPitch model on new speaker adaptation data, this application only adjusts those parameter-efficient or speaker-related modules. Initially, this application introduces these parameter-efficient modules into the pre-trained model. Specifically, this application adds standard adapters to the phoneme encoder, Mel decoder, pitch and duration predictors, and phoneme aligner. Simultaneously, this application experiments with LoRA and Prefix Tuning techniques in the self-attention networks of the encoder and decoder. For any layer with bias, this application applies the BitFit technique. Next, to construct a representation of the speaker's identity, this application extracts the speaker embedding (SE2) from the reference spectrogram and GST. This application combines this embedding with the speaker embedding (SE1) obtained by weighted averaging through a pre-trained speaker embedding lookup table to form the final speaker embedding SEfinal, as shown in the example. Figure 1 As shown, these weights are learned via gradient descent during fine-tuning. Furthermore, this application unfreezes the scaling and biasing linear layers, as previous studies have confirmed the effectiveness of this module in controlling speaker identity. By training only a small number of parameters, this application enables speaker-adaptive optimization of its TTS model in an efficient manner.

[0126] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0127] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0129] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0130] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0131] like Figure 3 As shown, the speech synthesis device 200 of this application embodiment includes:

[0132] The request receiving module 210 is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data;

[0133] The first embedded data acquisition module 220 is used to call the configured text-to-speech model and obtain the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model. The configured text-to-speech model has an adapter embedded in it.

[0134] Feature extraction module 230 is used to perform feature extraction operations on the new object speech data to obtain the second embedded data;

[0135] The splicing module 240 is used to splice the first embedded data and the second embedded data to obtain the comprehensive embedded data;

[0136] The parameter adjustment module 250 is used to perform parameter adjustment operations on the adapter based on the comprehensive embedded data to obtain a fine-tuned text-to-speech model.

[0137] The speech synthesis module 260 is used to input the integrated embedded data and the synthesized text data into the fine-tuned text-to-speech model to perform speech synthesis operations and obtain synthesized audio data.

[0138] The result output module 270 is used to output synthesized audio data to the user terminal.

[0139] In this embodiment, a speech synthesis device 200 is provided, comprising: a request receiving module 210, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesized text data; a first embedded data acquisition module 220, configured to call a configured text-to-speech model and obtain the first embedded data corresponding to the synthesis object identification data from a lookup table of the text-to-speech model, wherein the configured text-to-speech model embeds an adapter; a feature extraction module 230, configured to perform feature extraction on the new object speech data to obtain second embedded data; a splicing module 240, configured to splice the first embedded data and the second embedded data to obtain comprehensive embedded data; a parameter adjustment module 250, configured to perform parameter adjustment on the adapter according to the comprehensive embedded data to obtain a fine-tuned text-to-speech model; a speech synthesis module 260, configured to input the comprehensive embedded data and synthesized text data into the fine-tuned text-to-speech model for speech synthesis to obtain synthesized audio data; and a result output module 270, configured to output the synthesized audio data to the user terminal. Compared with existing technologies, this application introduces tiny adapter modules into the base model to ensure that the weights of the base model are not modified. The adapter parameters are adjusted only for the speech data of new speakers. This significantly reduces the demand for computing resources without sacrificing speech naturalness, speaker recognition, and prosodic consistency. Even in the case of scarce data, this application can still maintain good performance.

[0140] In some optional implementations of the embodiments of this application, the feature extraction module includes:

[0141] The Short Time Fourier Transform (SFT) submodule is used to perform a SFT operation on the new object's speech data to obtain a reference spectrogram.

[0142] The feature extraction submodule is used to perform feature extraction operations on the reference spectrogram based on the convolutional recurrent neural network encoder to obtain speech feature data.

[0143] The Style Token Acquisition submodule is used to acquire the global style token for the voice data of a new object.

[0144] The weighted submodule is used to input speech feature data and global style tokens into the multi-head attention layer for weighting operations to obtain the second embedding data.

[0145] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of this application.

[0146] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only the computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0147] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0148] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may also include both internal storage units and external storage devices of the computer device 300. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for speech synthesis methods. Furthermore, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0149] In some embodiments, the processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 320 is typically used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions stored in the memory 310 or to process data, for example, to execute computer-readable instructions for the speech synthesis method.

[0150] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0151] The computer device provided in this application, by introducing tiny adapter modules into the base model, ensures that the weights of the base model are not modified, and adjusts the adapter parameters only for the speech data of new speakers. This significantly reduces the demand for computing resources without sacrificing speech naturalness, speaker recognition, and prosodic consistency. Even in the case of scarce data, this application can still maintain good performance.

[0152] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the speech synthesis method described above.

[0153] The computer-readable storage medium provided in this application, by introducing tiny adapter modules into the base model, ensures that the weights of the base model are not modified, and adjusts the adapter parameters only for the speech data of new speakers. This significantly reduces the demand for computing resources without sacrificing speech naturalness, speaker recognition, and prosodic consistency. Even in the case of scarce data, this application can still maintain good performance.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0155] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that, Includes the following steps: The system receives a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data; The configured text-to-speech model is invoked, and the first embedded data corresponding to the synthesis object identifier data is obtained from the lookup table of the text-to-speech model, wherein the configured text-to-speech model has an embedded adapter. Feature extraction is performed on the new object's speech data to obtain the second embedded data; The first embedded data and the second embedded data are concatenated to obtain the composite embedded data. Based on the comprehensive embedded data, the adapter is adjusted to obtain a fine-tuned text-to-speech model. The integrated embedded data and the synthesized text data are input into the fine-tuned text-to-speech model to perform speech synthesis operation, thereby obtaining synthesized audio data. The synthesized audio data is output to the user terminal.

2. The speech synthesis method according to claim 1, characterized in that, After receiving the speech synthesis request sent by the user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data, the following steps are further included: Preprocessing operations are performed on the voice data of the new object.

3. The speech synthesis method according to claim 1, characterized in that, The step of performing feature extraction on the new object's speech data to obtain the second embedded data specifically includes the following steps: A short-time Fourier transform operation is performed on the speech data of the new object to obtain a reference spectrogram; Speech feature data is obtained by performing feature extraction on the reference spectrogram using a convolutional recurrent neural network encoder. Obtain the global style token of the new object's speech data; The speech feature data and the global style token are input into a multi-head attention layer for weighting to obtain the second embedding data.

4. The speech synthesis method according to claim 3, characterized in that, After the step of performing a short-time Fourier transform on the new object's speech data to obtain a reference spectrogram, the following step is also included: The reference spectrogram is augmented and denoised using an autoencoder network.

5. The speech synthesis method according to claim 1, characterized in that, Before the step of calling the configured text-to-speech model and retrieving the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model, the following steps are also included: Read the system database and obtain model training data from the system database; Call the initial text-to-speech model, input the model training data into the initial text-to-speech model, and perform model training operation on the initial text-to-speech model according to the model loss function to obtain the trained text-to-speech model; The trained text-to-speech model is weighted, and the adapter is embedded into the weighted text-to-speech model to obtain the configured text-to-speech model.

6. The speech synthesis method according to claim 1, characterized in that, The model loss function L total Represented as: L total L mel +L pitch +L duration +ω aligner ·L aligner Among them, y mel,i and Let p represent the target value and the predicted value of the Mel spectrogram at the i-th time step, respectively. i and Let d represent the target pitch value and the predicted pitch value at the i-th time step, respectively; i and represents the target value and predicted value of the duration at the i-th time step, respectively; N represents the total number of time steps, and aligner_loss_function represents the function for calculating the alignment loss.

7. A speech synthesis device, characterized in that, include: The request receiving module is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least new object speech data, synthesis object identification data, and synthesis text data; The first embedded data acquisition module is used to call the configured text-to-speech model and obtain the first embedded data corresponding to the synthesis object identifier data from the lookup table of the text-to-speech model, wherein the configured text-to-speech model has an embedded adapter. The feature extraction module is used to perform feature extraction operations on the new object's speech data to obtain the second embedded data; The splicing module is used to splice the first embedded data and the second embedded data to obtain comprehensive embedded data; The parameter adjustment module is used to perform parameter adjustment operations on the adapter based on the comprehensive embedded data to obtain a fine-tuned text-to-speech model. The speech synthesis module is used to input the integrated embedded data and the synthesized text data into the fine-tuned text-to-speech model to perform speech synthesis operations and obtain synthesized audio data. The result output module is used to output the synthesized audio data to the user terminal.

8. The speech synthesis device according to claim 7, characterized in that, The feature extraction module includes: The short-time Fourier transform submodule is used to perform a short-time Fourier transform operation on the speech data of the new object to obtain a reference spectrogram; The feature extraction submodule is used to perform feature extraction operations on the reference spectrogram based on the convolutional recurrent neural network encoder to obtain speech feature data; The style token acquisition submodule is used to acquire the global style token of the new object's speech data; The weighting submodule is used to input the speech feature data and the global style token into the multi-head attention layer for weighting operations to obtain the second embedded data.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the speech synthesis method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method as described in any one of claims 1 to 6.

Images