Speech synthesis methods, speech synthesis systems, electronic devices and storage media

Abstract

This application provides a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium, belonging to the field of financial technology. The method includes: adjusting the characters of an acquired sample text sequence according to a character adjustment sub-model of an original speech synthesis model to obtain an initial sample variable sequence; performing speech synthesis on the initial sample variable sequence according to an initial speech prediction sub-model to determine a first predicted speech; obtaining a candidate speech synthesis model based on the first predicted speech, and performing character filtering on the initial sample variable sequence to obtain a target sample variable sequence, inputting this sequence into the candidate speech synthesis model to obtain a second predicted speech; determining a target speech synthesis model based on the second predicted speech; and inputting the target text sequence into the target speech synthesis model for speech synthesis to obtain the target synthesized speech. This application can generate synthesized speech that is natural-sounding and semantically fluent.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of fintech and the socio-economic landscape, people have increasingly higher demands for banking services. In scenarios such as intelligent customer service, multi-turn dialogues, and robotic outbound calls, conveying relevant information to target audiences through natural and semantically accurate speech is one of the most effective and direct methods to improve customer experience and service levels. Currently, deep learning-based speech synthesis systems typically use vocoders for speech synthesis. However, when synthesizing speech from texts with significant semantic variations and variable lengths, the existing speech synthesis methods cannot accurately capture the different contextual relationships within the text, thus affecting the accuracy of the vocoder's speech synthesis and generating unnatural and semantically incoherent synthesized speech. Therefore, improving the accuracy of text-to-speech prediction and generating natural and semantically fluent synthesized speech has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium that can improve the prediction accuracy of text-to-speech and generate synthesized speech that is natural in expression and semantically fluent.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] Obtain a sample text sequence and sample speech of the sample text sequence;

[0006] The sample text sequence is input into a preset original speech synthesis model, which includes a character modulation sub-model and an initial speech prediction sub-model.

[0007] The sample text sequence is adjusted according to the character adjustment sub-model to obtain the initial sample variable sequence.

[0008] The initial sample variable sequence is processed for speech prediction based on the initial speech prediction sub-model to obtain the first predicted speech.

[0009] Based on the first predicted speech and the sample speech, the parameters of the initial speech prediction sub-model are adjusted to obtain a candidate speech synthesis model;

[0010] Based on the first predicted speech and the sample speech, the initial sample variable sequence is filtered by characters to obtain the target sample variable sequence;

[0011] The target sample variable sequence is input into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech;

[0012] The parameters of the candidate speech synthesis model are adjusted based on the second predicted speech and the sample speech to obtain the target speech synthesis model.

[0013] The acquired target text sequence is input into the target speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0014] In some embodiments, the sample text sequence includes initial sample characters, and the step of adjusting the text characters of the sample text sequence according to the character adjustment sub-model to obtain an initial sample variable sequence includes:

[0015] Random characters are extracted from the sample text sequence to obtain the initial variable characters for the sample;

[0016] The initial character of the sample and the initial variable character of the sample are merged to obtain the initial sample variable sequence.

[0017] In some embodiments, the initial sample variable sequence includes sample candidate characters, and the step of adjusting the parameters of the initial speech prediction sub-model based on the first predicted speech and the sample speech to obtain a candidate speech synthesis model includes:

[0018] Spectral loss is calculated based on the first predicted speech and the sample speech to obtain prediction loss data;

[0019] Based on the predicted loss data, partial derivatives are calculated on the candidate characters of the sample to obtain character variable data;

[0020] The character variable data is averaged based on the number of sample text sequences to obtain character metric data.

[0021] A numerical comparison is performed between the preset character measurement threshold and the character measurement data to obtain the measurement comparison result;

[0022] Based on the metric comparison results, the candidate characters in the sample are filtered to obtain the target characters in the sample.

[0023] The parameters of the initial speech prediction sub-model are adjusted based on the initial character of the sample, the initial variable character of the sample, and the target character of the sample to obtain the candidate speech synthesis model.

[0024] In some embodiments, adjusting the parameters of the initial speech prediction sub-model based on the initial sample character, the initial sample variable character, and the target sample character to obtain the candidate speech synthesis model includes:

[0025] The target character in the sample is judged to obtain the target variable character in the sample;

[0026] Based on the initial character of the sample, the initial variable character of the sample, the target character of the sample, and the target variable character of the sample, a pseudo-discovery calculation is performed to obtain character pseudo-discovery data;

[0027] The false discovery data of the characters is compared with the preset false discovery threshold to obtain the false discovery comparison result;

[0028] Based on the false discovery comparison results, the parameters of the initial speech prediction sub-model are adjusted to obtain the candidate speech synthesis model.

[0029] In some embodiments, adjusting the parameters of the initial speech prediction sub-model based on the false discovery comparison results to obtain the candidate speech synthesis model includes:

[0030] If the false discovery comparison result indicates that the false character discovery data is greater than the preset false discovery threshold, the text character adjustment of the sample text sequence according to the character adjustment sub-model is performed again to update the initial sample variable sequence.

[0031] The parameters of the initial speech prediction sub-model are adjusted based on the updated sequence of initial sample variables to obtain the candidate speech synthesis model.

[0032] In some embodiments, the step of filtering the initial sample variable sequence based on the first predicted speech and the sample speech to obtain the target sample variable sequence includes:

[0033] Based on the first predicted speech and the sample speech, the initial sample variable sequence is filtered by characters to obtain a sample filtering sequence, which includes the sample target characters;

[0034] The target characters in the sample are subjected to character recognition to obtain character recognition results;

[0035] If the character recognition result indicates that the currently recognized target character of the sample is the initial variable character of the sample, the currently recognized target character of the sample is removed to obtain the target sample variable sequence.

[0036] In some embodiments, the speech prediction sub-model includes a speech character encoding layer, an attention layer, a linear projection layer, a post-processing layer, and a prediction output layer. The step of performing speech synthesis processing on the initial sample variable sequence based on the initial speech prediction sub-model to obtain the first predicted speech includes:

[0037] The initial sample variable sequence is processed by speech character encoding according to the speech character encoding layer to obtain speech sample encoding features;

[0038] Based on the attention layer, the context features of the speech sample encoding features are extracted to obtain the current step context features of the first sample.

[0039] The current step context features of the first sample and the preset Mel spectrum are concatenated to obtain the context features to be processed;

[0040] The context features to be processed are input into a preset two-layer long short-term memory layer for context feature prediction to obtain the current step context features of the second sample;

[0041] The current step context features of the second sample are linearly projected according to the linear projection layer to obtain the current step projection scalar data.

[0042] The post-processing layer performs spectral update processing on the current step context features of the second sample to update the preset Mel spectrum;

[0043] The first predicted speech is obtained by performing speech synthesis processing on the current step projection scalar data based on the prediction output layer.

[0044] To achieve the above objectives, a second aspect of this application provides a speech synthesis system, the system comprising:

[0045] The text acquisition module is used to acquire a sample text sequence and sample speech of the sample text sequence;

[0046] The model input module is used to input the sample text sequence into a preset original speech synthesis model, which includes a character modulation sub-model and an initial speech prediction sub-model.

[0047] The text character adjustment module is used to adjust the text characters of the sample text sequence according to the character adjustment sub-model to obtain an initial sample variable sequence.

[0048] The first speech prediction module is used to perform speech synthesis processing on the initial sample variable sequence according to the initial speech prediction sub-model to obtain the first predicted speech.

[0049] The first parameter adjustment module is used to adjust the parameters of the initial speech prediction sub-model according to the first predicted speech and the sample speech to obtain a candidate speech synthesis model.

[0050] The character filtering module is used to filter the initial sample variable sequence based on the first predicted speech and the sample speech to obtain the target sample variable sequence;

[0051] The second speech prediction module is used to input the target sample variable sequence into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech.

[0052] The second parameter adjustment module is used to adjust the parameters of the candidate speech synthesis model according to the second predicted speech and the sample speech to obtain the target speech synthesis model.

[0053] A target speech synthesis module is used to input the acquired target text sequence into the target speech synthesis model for speech synthesis processing to obtain target synthesized speech. To achieve the above objective, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any of the first aspects of this application.

[0054] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects of the present application.

[0055] The speech synthesis method, speech synthesis system, electronic device, and storage medium proposed in this application acquire sample text sequences and sample speech of the sample text sequences. The sample text sequences are then input into a preset original speech synthesis model, which includes a character adjustment sub-model and an initial speech prediction sub-model. The character adjustment sub-model is used to adjust the text characters of the sample text sequences to obtain an initial sample variable sequence. The initial speech prediction sub-model is then used to perform speech prediction processing on the initial sample variable sequence to obtain a first predicted speech. The parameters of the initial speech prediction sub-model are then adjusted based on the first predicted speech and the sample speech to obtain a candidate speech synthesis model. Subsequently, the initial sample variable sequence is used to perform character filtering based on the first predicted speech and the sample speech to obtain a target sample variable sequence. This target sample variable sequence is then input into the candidate speech synthesis model for speech synthesis processing to obtain a second predicted speech. The parameters of the candidate speech synthesis model are then adjusted based on the second predicted speech and the sample speech to obtain the target speech synthesis model. This application embodiment trains the initial speech prediction sub-model by combining the character adjustment sub-model, which can accurately capture different contextual relationships in the text. Therefore, when the target text sequence is processed by speech synthesis according to the target speech synthesis model provided in this application embodiment, the prediction accuracy of text to speech can be effectively improved, and synthesized speech with natural expression and fluent semantics can be generated. Attached Figure Description

[0056] Figure 1 This is a first flowchart of the speech synthesis method provided in the embodiments of this application;

[0057] Figure 2 yes Figure 1 A flowchart illustrating the specific method of step S110;

[0058] Figure 3 yes Figure 1 A flowchart illustrating the specific method of step S120;

[0059] Figure 4 yes Figure 1 A flowchart illustrating the specific method of step S150;

[0060] Figure 5 yes Figure 4 A flowchart illustrating the specific method of step S420;

[0061] Figure 6 yes Figure 1 A flowchart illustrating the specific method of step S170;

[0062] Figure 7 yes Figure 6 A flowchart illustrating the specific method of step S610;

[0063] Figure 8 This is a block diagram of the module structure of the speech synthesis system provided in the embodiments of this application;

[0064] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0070] Natural Language Processing (NLP): NLP is an interdisciplinary field combining computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. NLP also involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0071] Text-to-Speech (TTS): TTS is a technology that converts text into speech. TTS generally includes two steps: the first step is text processing, which mainly converts the text into a phoneme sequence and marks the start and end times, frequency changes, and other information of each phoneme; the second step is speech synthesis, which mainly generates speech based on the phoneme sequence (and the marked start and end times, frequency changes, and other information).

[0072] L1 loss function: used to minimize the error, which is the sum of all absolute differences between the true and predicted values.

[0073] With the rapid development of fintech and the socio-economic landscape, people have increasingly higher demands for banking services. In scenarios such as intelligent customer service, multi-turn dialogues, and robotic outbound calls, conveying relevant information to target audiences through natural and semantically accurate speech is one of the most effective and direct methods to improve customer experience and service levels. Currently, deep learning-based speech synthesis systems typically use Mel-spectrum as an intermediate representation for speech synthesis and perform speech synthesis via vocoders. However, when synthesizing speech from texts with significant semantic variations and variable lengths, the existing speech synthesis methods fail to accurately capture the different contextual relationships within the text, thus affecting the accuracy of the vocoder's speech synthesis and generating unnatural and semantically incoherent synthesized speech. Therefore, improving the accuracy of text-to-speech prediction and generating natural and semantically fluent synthesized speech has become an urgent technical problem to be solved.

[0074] Based on this, the speech synthesis method, speech synthesis system, electronic device and storage medium provided in the embodiments of this application can improve the prediction accuracy of text to speech and generate synthesized speech that is natural and semantically fluent.

[0075] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0077] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user voice data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0080] Please refer to Figure 1 , Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. In some embodiments of this application, the speech synthesis method proposed in this application includes, but is not limited to, steps S110 to S190. The following is a detailed description... Figure 1 These nine steps will be explained in detail.

[0081] Step S110: Obtain the sample text sequence and the sample speech of the sample text sequence;

[0082] Step S120: Input the sample text sequence into the preset original speech synthesis model. The original speech synthesis model includes a character modulation sub-model and an initial speech prediction sub-model.

[0083] Step S130: Adjust the text characters of the sample text sequence according to the character adjustment sub-model to obtain the initial sample variable sequence;

[0084] Step S140: Perform speech prediction processing on the initial sample variable sequence according to the initial speech prediction sub-model to obtain the first predicted speech;

[0085] Step S150: Adjust the parameters of the initial speech prediction sub-model based on the first predicted speech and the sample speech to obtain the candidate speech synthesis model;

[0086] Step S160: Filter the initial sample variable sequence according to the first predicted speech and the sample speech to obtain the target sample variable sequence;

[0087] Step S170: Input the target sample variable sequence into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech;

[0088] Step S180: Adjust the parameters of the candidate speech synthesis model based on the second predicted speech and the sample speech to obtain the target speech synthesis model;

[0089] Step S190: Input the acquired target text sequence into the target speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0090] In steps S110 to S190 of some embodiments, a sample text sequence and sample speech of the sample text sequence are acquired. The sample text sequence is then input into a preset original speech synthesis model, which includes a character adjustment sub-model and an initial speech prediction sub-model. Then, the sample text sequence is adjusted according to the character adjustment sub-model to obtain an initial sample variable sequence. The initial sample variable sequence is then processed for speech prediction according to the initial speech prediction sub-model to obtain a first predicted speech. The parameters of the initial speech prediction sub-model are then adjusted based on the first predicted speech and the sample speech to obtain a candidate speech synthesis model. Next, the initial sample variable sequence is filtered for characters according to the first predicted speech and the sample speech to obtain a target sample variable sequence. The target sample variable sequence is then input into the candidate speech synthesis model for speech synthesis processing to obtain a second predicted speech. The parameters of the candidate speech synthesis model are then adjusted based on the second predicted speech and the sample speech to obtain the target speech synthesis model. This application embodiment trains the initial speech prediction sub-model by combining the character adjustment sub-model, which can accurately capture different contextual relationships in the text. Therefore, when the target text sequence is processed by speech synthesis according to the target speech synthesis model provided in this application embodiment, the prediction accuracy of text to speech can be effectively improved, and synthesized speech with natural expression and fluent semantics can be generated.

[0091] In step S110 of some embodiments, in order to generate synthesized speech that is natural and semantically fluent, a training sample set is first obtained. This training sample set includes at least one set of sample data, which includes sample text sequences and sample speech of the sample text sequences. The sample speech is used to represent a reference speech for the sample text sequences during model training. The sample text sequences can be text phoneme sequences obtained through a text phoneme conversion model, such as the Deep Voice3 model or a grapheme to phoneme (G2P) model.

[0092] It should be noted that the storage format of the sample audio in this application can be MP3, CDA, WAV, WMA, RA, MIDI, OGG, APE or AAC, etc., and this application does not limit it.

[0093] It should be noted that the speech synthesis method of this application can also be used to assist applications such as car radio and announcements, car navigation, electronic dictionaries, consumer electronics, smartphones, smart speakers, voice assistants, e-book reading, etc., to generate synthesized speech that is natural and semantically fluent based on the input target text sequence.

[0094] It's important to note that in speech synthesis, since the lengths of the input text sequence and the output audio sequence are often not the same, it's not possible to directly map each character of the input sequence to the target pronunciation. Therefore, an encoder-decoder structure is needed to adjust the model. The encoder extracts and compresses information from the input text sequence into a fixed-length context vector, which serves as the encoded representation of the input text. The decoder then uses this encoded context vector and performs certain transformations to obtain the target output sequence. In Tacotron2, the encoder module includes convolutional layers and BLSTM layers. The characters of the input text sequence are first represented as 512-dimensional text embeddings, then passed through three convolutional layers, each containing 512 5×1 convolutional kernels, followed by batch normalization and ReLU activation. The resulting data is then passed to the BLSTM layer, which contains 512 units (256 units in each direction) to generate encoded features. The encoder's output is passed to an attention module to focus on the positional information of the input sequence. Positional features are calculated using 32 31×1 convolutional kernels and mapped onto a 128-dimensional vector as the representation of attention probabilities. Therefore, Tacotron 2 proposes a recurrent S2S feature prediction structure with an attention mechanism to predict Mel-frequency spectra. This method uses three convolutional layers with 512-dimensional filters at the encoder input to model long-range contextual relationships among the five characters, compensating for the poor capture of long-range dependencies by subsequent RNN structures in practical applications. However, this fixed-length contextual relationship extraction may not accurately capture different contextual relationships throughout the text, lacking flexibility.

[0095] In step S120 of some embodiments, to address the aforementioned problems and improve the flexibility of the Tacotron 2 model in predicting the Mel spectrum, and to optimize the input feature representation of the subsequent vocoder, the original speech synthesis model provided in this application embodiment is an improvement on the Tacotron 2 model. The original speech synthesis model includes a character adjustment sub-model and an initial speech prediction sub-model. The character adjustment sub-model is used to adjust the encoder in the initial speech prediction sub-model, i.e., the speech character encoding layer mentioned in this application. This speech character encoding layer can adaptively select the number of characters used in each encoding, i.e., it does not always fixate on modeling long-range contextual relationships using only 5 characters, but rather enables the trained target speech synthesis model to dynamically capture different contextual relationships in the input text, which is beneficial for improving the reliability of text-to-speech prediction results, thereby improving the speech generation quality of the Tacotron 2-based model.

[0096] In step S130 of some embodiments, the character adjustment sub-model proposed in this application is a technique that can improve the variable selection process when training a deep neural network. It can dynamically adjust the number of characters used to build the model based on the input sample text sequence.

[0097] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the specific method of step S130 provided in the embodiments of this application. In some embodiments of this application, the sample text sequence includes initial sample characters, and step S130 may specifically include, but is not limited to, steps S210 and S220. The following describes the process in conjunction with... Figure 2 These two steps will be explained in detail.

[0098] Step S210: Randomly extract characters from the sample text sequence to obtain the initial variable characters of the sample;

[0099] Step S220: Merge the initial character of the sample and the initial variable character of the sample to obtain the initial sample variable sequence.

[0100] In step S210 of some embodiments, surrogate variables need to be added to the model before model training. Specifically, the initial variable characters of the sample text sequence after random character extraction are the surrogate variables. The features corresponding to these surrogate variables are obtained by random sampling with or without repetition from the character encoding matrix corresponding to the sample text sequence. As a surrogate for the unknown zero variable, the corresponding zero distribution can be obtained.

[0101] In step S220 of some embodiments, after obtaining the initial variable characters of the samples, the training set for the first training becomes the five initial variable characters of the samples obtained by the Tacotron 2 model and the randomly sampled initial variable characters of the samples. The character encoding matrix corresponding to the initial variable sequence obtained after character merging is used as the training object of the model.

[0102] It should be noted that the training data during the first training session will be divided into multiple batches based on the batch size, which represents the number of sample text sequences transmitted to the program in a single session.

[0103] In step S140 of some embodiments, after obtaining an initial sample variable sequence by selecting characters and updating the sample text sequence according to the selected initial variable characters, the initial sample variable sequence is processed for speech prediction according to the initial speech prediction sub-model based on the Tacotron 2 structure to obtain the first predicted speech.

[0104] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the specific method of step S140 provided in the embodiments of this application. In some embodiments of this application, the speech prediction sub-model includes a speech character encoding layer, an attention layer, a linear projection layer, a post-processing layer, and a prediction output layer. Therefore, step S140 may specifically include, but is not limited to, steps S310 to S370. The following describes the process in conjunction with... Figure 3 These seven steps will be explained in detail.

[0105] Step S310: Perform speech character encoding processing on the initial sample variable sequence according to the speech character encoding layer to obtain speech sample encoding features;

[0106] Step S320: Extract context features from the speech sample encoding features based on the attention layer to obtain the current step context features of the first sample;

[0107] Step S330: Concatenate the current step context features of the first sample with the preset Mel spectrum to obtain the context features to be processed;

[0108] Step S340: Input the context features to be processed into the preset two-layer long short-term memory layer to predict the context features and obtain the current step context features of the second sample.

[0109] Step S350: Perform linear projection processing on the current step context features of the second sample according to the linear projection layer to obtain the current step projection scalar data;

[0110] Step S360: Perform spectrum update processing on the current step context features of the second sample according to the post-processing layer to update the preset Mel spectrum;

[0111] Step S370: Perform speech synthesis processing on the current step projection scalar data based on the prediction output layer to obtain the first predicted speech.

[0112] In steps S310 to S370 of some embodiments, the speech character encoding layer used in this embodiment is an encoder based on the Tacotron 2 model, including a character embedding layer and a preprocessing layer connected in sequence. This speech character encoding layer is then connected to an attention layer, a linear projection layer, a post-processing layer, and a prediction output layer. The initial sample variable sequence is used as the input to the character embedding layer, where each character is represented as a one-hot encoded vector. By setting an initial parameter and a trainable embedding matrix, the one-hot vector is mapped to a character embedding vector, meaning each character in the input initial sample variable sequence is represented as a 256-dimensional character embedding vector. During model training, the embedding matrix is ​​trained through backpropagation, similar to other network layer parameters, to obtain an embedding matrix that represents the text of each character set. Then, the character embedding vector undergoes a series of nonlinear transformations through the pre-network, and the nonlinearly transformed character embedding vector is input to the attention layer and then through the linear projection layer to obtain the encoded representation of the initial sample variable sequence, projecting the features into a simplified continuous vector space in conjunction with the context. Subsequently, the post-processing layer performs spectral update processing on the current step context features of the second sample to update the preset Mel spectrum. This preset Mel spectrum is used as a new Mel spectrum template for subsequent iterative training of the model. The first predicted speech represents the predicted speech obtained by the model during its first training after adding the initial variable characters of the sample. This embodiment of the application, by combining the character adjustment sub-model with the initial speech prediction sub-model for model training, can dynamically capture the long-range contextual relationships of the input target text sequence, which is beneficial to improving the reliability of the text-to-Mel spectrum prediction results.

[0113] It should be noted that the prediction output layer in this embodiment includes a decoder and a vocoder. The decoder is a Tacotron 2-based decoder with an attention mechanism, including an attention mechanism, a preprocessing network, a Long Short-Term Memory (LSTM) neural network, and a linear mapping layer. Specifically, the decoder is an autoregressive recurrent neural network, and the attention context vector is obtained through the decoder's attention mechanism. During the decoding process, the decoding result of the previous step is used as the current step's projected scalar data as input, and it passes through a preprocessing network consisting of two fully connected layers, each layer consisting of 256 hidden ReLU units. The output of the preprocessing network is concatenated with the attention context vector calculated by the attention mechanism and input together into a two-layer LSTM for decoding. Each LSTM layer contains 1024 units to obtain the predicted speech coding features. Then, the predicted speech coding features are processed for speech synthesis by the vocoder.

[0114] In step S150 of some embodiments, the first predicted speech is obtained by training by merging the initial sample characters and the randomly sampled initial variable characters. Then, the parameters of the initial speech prediction sub-model are adjusted according to the first predicted speech and the sample speech to improve the ability of the initial speech prediction sub-model to adaptively select the number of characters used in each encoding.

[0115] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the specific method of step S150 provided in the embodiments of this application. In some embodiments of this application, the initial sample variable sequence includes sample candidate characters, then step S150 may specifically include, but is not limited to, steps S410 to S460. The following describes the process in conjunction with... Figure 4 These six steps will be explained in detail.

[0116] Step S410: Calculate the spectral loss based on the first predicted speech and the sample speech to obtain the predicted loss data;

[0117] Step S420: Calculate the partial derivatives of the candidate characters in the sample based on the predicted loss data to obtain the character variable data;

[0118] Step S430: Calculate the mean of the character variable data based on the number of sample text sequences to obtain character metric data;

[0119] Step S440: Compare the preset character measurement threshold and the character measurement data to obtain the measurement comparison result;

[0120] Step S450: Based on the measurement comparison results, the candidate characters in the sample are filtered to obtain the target characters in the sample;

[0121] Step S460: Adjust the parameters of the initial speech prediction sub-model based on the initial sample character, the initial sample variable character, and the sample target character to obtain the candidate speech synthesis model.

[0122] In step S410 of some embodiments, character importance measurement based on the loss function can effectively identify the importance of characters in capturing long-range contextual relationships. For example, in the scenario of an intelligent robot, if the text sequence input to the target object is "I want to cook a dish today, how about making scrambled eggs with tomatoes?", then "I" is not very important in capturing contextual relationships, but "dish" and "scrambled eggs with tomatoes" are relatively more important. Therefore, by measuring the importance of each character, the character encoding matrix can be effectively adjusted in this embodiment. The first predicted speech and the sample speech can be calculated using the L1 loss function, or the L2 loss function, mean squared error loss function, cross-entropy loss function, contrastive loss function, etc., without specific limitations here.

[0123] In steps S420 to S430 of some embodiments, the character metric data of the initial sample variable sequence is used to characterize the importance of all sample initial characters and sample initial variable characters in the initial sample variable sequence. The sample candidate characters include sample initial characters and sample initial variable characters. The importance metric for each sample candidate character in the initial sample variable sequence is calculated as shown in formula (1), and the obtained character variable data is denoted as CV. j ,j is used to represent the character id of the sample candidate character.

[0124]

[0125] Where i represents the index of the sample text sequence, n represents the number of sample text sequences in the training sample set, L represents the loss function used to calculate the spectral loss, and x (i) Let f represent the encoded features of the sample text sequence i, and let f denote the model structure of the initial speech synthesis model based on Tacotron 2 from character encoding to generating the Mel spectrum corresponding to the first predicted speech. Then f(x) (i) ) represents the Mel spectrum corresponding to the first predicted speech, y (i) Let f(x) represent the Mel spectrum of the sample speech corresponding to the sample text sequence i. Therefore, based on the prediction loss data f(x) (i) Calculate the partial derivatives of the candidate characters in the sample to obtain the character variable data. The mean of the character variable data is calculated based on the number of sample text sequences n, resulting in the character metric (CV) data for each candidate character. jRegardless of the number of network layers or structure, the embodiments of this application can conveniently use backpropagation to calculate character metric data (CV) through loss functions and partial derivatives. j .

[0126] In steps S440 to S460 of some embodiments, in order to estimate the False Discovery Rate (FDR) of the current model, it is first necessary to use the reverse elimination method to remove unimportant characters. That is, by removing characters that do not meet the requirements one by one, only important characters that meet the requirements remain. The preset character measurement threshold is a hyperparameter ε pre-set in the character adjustment sub-model. This hyperparameter ε is used to characterize the strength of the variable selection by the character adjustment sub-model, and the hyperparameter ε is between 0 and 1. By comparing the preset character measurement threshold and the character measurement data, when the measurement comparison result indicates that the identified character measurement data is less than the preset character measurement threshold, the sample candidate character corresponding to the character measurement data is removed. In this embodiment, the sample candidate characters whose identified character measurement data is less than the preset character measurement threshold are retained according to the preset hyperparameter ε, and the retained sample candidate characters are used as sample target characters, which can effectively reduce the computational load of the model and improve the efficiency of model training.

[0127] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating the specific method of step S460 provided in an embodiment of this application. In some embodiments of this application, step S460 may specifically include, but is not limited to, steps S510 to S540, as described below. Figure 5 These four steps will be explained in detail.

[0128] Step S510: Perform character judgment on the sample target character to obtain the sample target variable character;

[0129] Step S520: Perform pseudo-discovery calculation based on the initial character of the sample, the initial variable character of the sample, the target character of the sample, and the target variable character of the sample to obtain character pseudo-discovery data;

[0130] Step S530: Compare the false discovery data of the characters with the preset false discovery threshold to obtain the false discovery comparison result;

[0131] Step S540: Adjust the parameters of the initial speech prediction sub-model based on the false discovery comparison results to obtain the candidate speech synthesis model.

[0132] In steps S510 to S540 of some embodiments, in order to determine the recognition status of the model after adding proxy variables, this embodiment of the application uses the error discovery rate (FDR) to determine the training status of the model, and the FDR is used to represent the false character discovery data. Specifically, the number of sample target characters is denoted as r, and r is used to represent the number of character variables retained after passing through the original speech synthesis model network. The sample target characters are judged by the samples to determine the sample target variable characters, and the number of sample target variable characters is denoted as r0, and r0 is used to represent the number of proxy variables among the remaining sample target characters. Then, the number of initial sample characters is denoted as p, where p represents the number of original variables in the initial sample variable sequence, and the number of initial sample variable characters is denoted as q, where q represents the number of added proxy variables. Therefore, the specific calculation of the FDR corresponding to the original speech synthesis model in this embodiment of the application is shown in formula (2).

[0133]

[0134] It should be noted that, in order to better improve the reliability of the Mel spectrum prediction results of the initial speech prediction sub-model of this application, this application proposes a preset pseudo-discovery threshold, which is denoted as cutoff. The pseudo-discovery data of characters is compared with cutoff to obtain the pseudo-discovery comparison result. The pseudo-discovery comparison result is used to determine whether the training results of the current initial speech prediction sub-model meet the required reliability requirements.

[0135] It should be noted that, for example, if the original number of variables p is 5 and the number of added proxy variables q is 3, then when one real sample initial character and one added proxy variable are removed in the first iteration, the number of variables retained by the network is r = 5 + 3 - 1 - 1 = 6, and the number of proxy variables removed is 1, so the number of remaining proxy variables r0 is 2.

[0136] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating the specific method of step S540 provided in an embodiment of this application. In some embodiments of this application, step S540 may specifically include, but is not limited to, steps S610 and S620, as described below. Figure 6 These two steps will be explained in detail.

[0137] Step S610: If the false discovery comparison result indicates that the false discovery data of the character is greater than the preset false discovery threshold, perform text character adjustment on the sample text sequence according to the character adjustment sub-model again to update the initial sample variable sequence.

[0138] Step S620: Adjust the parameters of the initial speech prediction sub-model according to the updated initial sample variable sequence to obtain the candidate speech synthesis model.

[0139] In steps S610 and S620 of some embodiments, if the false discovery comparison result indicates that the false character discovery data is greater than the preset false discovery threshold, it means that the training result of the current initial speech prediction sub-model does not meet the required reliability requirements. Therefore, it is necessary to adjust the text characters of the sample text sequence according to the character adjustment sub-model to update the initial sample variable sequence. Then, speech prediction is performed again based on the updated initial sample variable sequence. If the false discovery comparison result indicates that the false character discovery data is less than or equal to the preset false discovery threshold, it means that the training result of the current initial speech prediction sub-model meets the required reliability requirements. In this case, the training of the model is stopped, and a candidate speech synthesis model is constructed based on the model parameters and model structure of the current initial speech prediction sub-model.

[0140] It's important to note that when the hyperparameter ε is close to 1, the reverse elimination is most "aggressive," resulting in a smaller number of steps required for the cutoff set by FDR. Conversely, when the hyperparameter ε is very close to 0, the FDR determination takes a long time. For example, consider an initial sample variable sequence containing 5 initial sample characters and 3 surrogate variable characters. Assume the FDR cutoff is set to 0.01, and that 3 variables need to be removed to reach this cutoff. Therefore, if ε is very close to 1, it might only require one iteration to remove all 3 surrogate variables. However, if ε is very small, the algorithm might only remove one surrogate variable per iteration, requiring 3 iterations to achieve an FDR less than or equal to the cutoff.

[0141] In step S160 of some embodiments, during the second training phase of the candidate speech synthesis model, it is necessary to remove the remaining proxy variables in order to train the final target synthesized speech model.

[0142] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating the specific method of step S160 provided in an embodiment of this application. In some embodiments of this application, step S160 may specifically include, but is not limited to, steps S710 to S730, as described below. Figure 7 These three steps will be explained in detail.

[0143] Step S710: Based on the first predicted speech and the sample speech, the initial sample variable sequence is filtered to obtain a sample filtering sequence, which includes the sample target characters.

[0144] Step S720: Perform character recognition on the target character in the sample to obtain the character recognition result;

[0145] Step S730: If the character recognition result indicates that the target character of the currently recognized sample is the initial variable character of the sample, remove the target character of the currently recognized sample to obtain the target sample variable sequence.

[0146] In steps S710 to S730 of some embodiments, when FDR is less than or equal to cutoff, the first stage of model training is terminated, and a sample selection sequence consisting of sample target characters is determined based on the first predicted speech and sample speech. Then, character recognition is performed on each sample target character, i.e., it is determined whether there are still proxy variables. If the character recognition result indicates that the currently recognized sample target character is the sample initial variable character, character removal is performed on the currently recognized sample target character to obtain a target sample variable sequence, which does not contain proxy variables.

[0147] In step S170 of some embodiments, the model structure of the candidate speech synthesis model is the same as that of the initial speech prediction sub-model described above, and will not be repeated here.

[0148] It should be noted that, using LJSpeech as the training dataset as an example, starting from the model training, if a batch size of 32 is chosen, then each training iteration involves 32 sample text sequences. For example, if the batch size is 1, the acquired sample text sequence might be "Printing, in the only sense with which we are atpresent concerned…". Specifically, the Tacotron2 speech character encoding layer, due to the convolutional kernel settings, can only see 5 initial sample characters at a time, such as "prInt" and "IngIn". Then, after training the original speech synthesis model constructed using the variable selection and Tacotron2 network as described in this application, the resulting candidate speech synthesis model, before being input into the BiLSTM of the speech character encoding layer, can see different combinations of initial sample characters each time, for example, becoming "prn" and "ngln". After further training on the Tacotron 2-based candidate speech synthesis model, the Mel-spectral output of the second predicted speech is obtained. The second predicted speech in audio form can then be obtained through the vocoder in the candidate speech synthesis model.

[0149] In step S180 of some embodiments, speech loss is calculated based on the second predicted speech and the sample speech. The loss function used for this speech loss calculation can be any of the L1 loss function, L2 loss function, etc., as described above, without specific limitation. Furthermore, the preset training termination condition for parameter adjustment of the model in this embodiment can be that the similarity accuracy of the candidate speech synthesis model is greater than or equal to a preset accuracy threshold. This similarity accuracy is calculated based on the similarity between the sample speech and the second predicted speech. The function used for similarity calculation can be selected according to actual needs, such as cosine similarity calculation, time axis comparison method, etc., without specific limitation. Alternatively, the speech loss value of the candidate speech synthesis model can be less than a preset loss threshold, such as a preset loss threshold of 0.05, without specific limitation.

[0150] In step S190 of some embodiments, in specific applications, the speech synthesis system proposed in this application can be applied to a terminal. This speech synthesis system deploys the target speech synthesis model trained by this application, and the system is used to execute the aforementioned speech synthesis method. When a target object needs to perform speech synthesis on the terminal, the target object can input the target text sequence to be synthesized on the terminal page. Then, the terminal page can display a pop-up box, allowing the target object to generate a speech synthesis service request by touching the synthesized speech button in the pop-up box. The request, containing the target text sequence, is then sent to the port containing the speech synthesis system for speech synthesis processing. Afterwards, the synthesized target speech is received and played back through the terminal's speaker. The speech synthesis method used in this embodiment can effectively improve the accuracy of text-to-speech prediction and allows the target object to hear naturally expressed and semantically fluent synthesized speech.

[0151] For example, in the scenario of intelligent robots in fintech, such as when the input text sequence to the target object is "I see the weather is nice today, let's go for a run," the word "I" is not as important for capturing the context, but "the weather is nice" and "run" are relatively more important. Therefore, when performing speech synthesis, this application can adjust the model parameters according to the different importance of characters to improve the prediction accuracy of the target speech synthesis model. Furthermore, the synthesized speech from the text is more natural and semantically more fluent. At the same time, using the speech synthesis method provided in this application can reduce manual labor costs while ensuring the same service level as human customer service.

[0152] Please refer to Figure 8 , Figure 8This is a schematic diagram of the module structure of a speech synthesis system provided in an embodiment of this application. In some embodiments of this application, the speech synthesis system includes a text acquisition module 810, a model input module 820, a text character adjustment module 830, a first speech prediction module 840, a first parameter adjustment module 850, a character filtering module 860, a second speech prediction module 870, a second parameter adjustment module 880, and a target speech synthesis module 890.

[0153] The text acquisition module 810 is used to acquire sample text sequences and sample speech of the sample text sequences;

[0154] The model input module 820 is used to input the sample text sequence into the preset original speech synthesis model, which includes a character modulation sub-model and an initial speech prediction sub-model.

[0155] The text character adjustment module 830 is used to adjust the text characters of the sample text sequence according to the character adjustment sub-model to obtain the initial sample variable sequence.

[0156] The first speech prediction module 840 is used to perform speech synthesis processing on the initial sample variable sequence according to the initial speech prediction sub-model to obtain the first predicted speech.

[0157] The first parameter adjustment module 850 is used to adjust the parameters of the initial speech prediction sub-model according to the first predicted speech and the sample speech to obtain the candidate speech synthesis model.

[0158] The character filtering module 860 is used to filter the initial sample variable sequence based on the first predicted speech and the sample speech to obtain the target sample variable sequence.

[0159] The second speech prediction module 870 is used to input the target sample variable sequence into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech.

[0160] The second parameter adjustment module 880 is used to adjust the parameters of the candidate speech synthesis model according to the second predicted speech and the sample speech to obtain the target speech synthesis model.

[0161] The target speech synthesis module 890 is used to input the acquired target text sequence into the target speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0162] It should be noted that the speech synthesis system in this application embodiment is used to execute the above-described speech synthesis method, and the speech synthesis system in this application embodiment corresponds to the aforementioned speech synthesis method. For the specific training process, please refer to the aforementioned speech synthesis method, which will not be described in detail here.

[0163] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method described in the embodiments of this application.

[0164] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0165] The following is combined Figure 9 The electronic devices described in the embodiments of this application will be described in detail.

[0166] Please refer to Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0167] The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0168] The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the speech synthesis method of the embodiments of this application.

[0169] The input / output interface 930 is used to implement information input and output;

[0170] The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0171] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0172] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method described in the embodiments of this application.

[0174] This application provides a speech synthesis method, speech synthesis system, electronic device, and storage medium. The method involves acquiring a sample text sequence and sample speech of that sequence, inputting the sample text sequence into a preset original speech synthesis model. This original speech synthesis model includes a character adjustment sub-model and an initial speech prediction sub-model. Then, the character adjustment sub-model adjusts the text characters of the sample text sequence to obtain an initial sample variable sequence. The initial speech prediction sub-model then performs speech prediction processing on the initial sample variable sequence to obtain a first predicted speech. The parameters of the initial speech prediction sub-model are adjusted based on the first predicted speech and the sample speech to obtain a candidate speech synthesis model. Next, the initial sample variable sequence is screened for characters based on the first predicted speech and the sample speech to obtain a target sample variable sequence. This target sample variable sequence is then input into the candidate speech synthesis model for speech synthesis processing to obtain a second predicted speech. Finally, the parameters of the candidate speech synthesis model are adjusted based on the second predicted speech and the sample speech to obtain the target speech synthesis model. This application embodiment trains the initial speech prediction sub-model by combining a character adjustment sub-model, which dynamically captures the long-range contextual relationships of the input target text sequence, improving the reliability of text-to-speech prediction and thus enhancing the accuracy of the vocoder in speech synthesis. The target sample variable sequence is obtained by character-filtering the initial sample variable sequence based on the first predicted speech and sample speech. This target sample variable sequence is then used to further adjust the parameters of the candidate speech synthesis model after character adjustment, generating a target speech synthesis model that produces natural and semantically fluent synthesized speech, and improving the final speech quality generated by the model built on Tacotron 2. Furthermore, the technical solution provided in this application embodiment can be combined with various deep learning Mel-spectrum prediction models with almost no manual intervention and strong generalization capabilities.

[0175] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0177] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0180] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0181] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0183] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain a sample text sequence and sample speech of the sample text sequence, wherein the sample text sequence includes sample initial characters; The sample text sequence is input into a preset original speech synthesis model, which includes a character modulation sub-model and an initial speech prediction sub-model. The initial sample variable sequence is obtained by adjusting the text characters of the sample text sequence according to the character adjustment sub-model. Specifically, this includes: randomly extracting characters from the sample text sequence to obtain initial sample variable characters; merging the initial sample characters and the initial sample variable characters to obtain an initial sample variable sequence, wherein the initial sample variable sequence includes sample candidate characters. The initial sample variable sequence is processed by speech synthesis according to the initial speech prediction sub-model to obtain the first predicted speech; Spectral loss is calculated based on the first predicted speech and the sample speech to obtain prediction loss data; partial derivatives are calculated on the sample candidate characters based on the prediction loss data to obtain character variable data; the mean of the character variable data is calculated based on the number of sample text sequences to obtain character metric data; a preset character metric threshold and the character metric data are numerically compared to obtain a metric comparison result; the sample candidate characters are screened based on the metric comparison result to obtain sample target characters; the parameters of the initial speech prediction sub-model are adjusted based on the initial sample characters, the initial sample variable characters, and the sample target characters to obtain a candidate speech synthesis model; Based on the first predicted speech and the sample speech, the initial sample variable sequence is filtered by characters to obtain the target sample variable sequence; The target sample variable sequence is input into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech; The parameters of the candidate speech synthesis model are adjusted based on the second predicted speech and the sample speech to obtain the target speech synthesis model. The acquired target text sequence is input into the target speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

2. The method according to claim 1, characterized in that, The step of adjusting the parameters of the initial speech prediction sub-model based on the initial sample character, the initial sample variable character, and the sample target character to obtain the candidate speech synthesis model includes: The target character in the sample is judged to obtain the target variable character in the sample; Based on the initial character of the sample, the initial variable character of the sample, the target character of the sample, and the target variable character of the sample, a pseudo-discovery calculation is performed to obtain character pseudo-discovery data; The false discovery data of the characters is compared with the preset false discovery threshold to obtain the false discovery comparison result; Based on the false discovery comparison results, the parameters of the initial speech prediction sub-model are adjusted to obtain the candidate speech synthesis model.

3. The method according to claim 2, characterized in that, The step of adjusting the parameters of the initial speech prediction sub-model based on the false discovery comparison results to obtain the candidate speech synthesis model includes: If the false discovery comparison result indicates that the false character discovery data is greater than the preset false discovery threshold, the text character adjustment of the sample text sequence according to the character adjustment sub-model is performed again to update the initial sample variable sequence. The parameters of the initial speech prediction sub-model are adjusted based on the updated sequence of initial sample variables to obtain the candidate speech synthesis model.

4. The method according to claim 1, characterized in that, The step of filtering the initial sample variable sequence based on the first predicted speech and the sample speech to obtain the target sample variable sequence includes: Based on the first predicted speech and the sample speech, the initial sample variable sequence is filtered by characters to obtain a sample filtering sequence, which includes the sample target characters; The target characters in the sample are subjected to character recognition to obtain character recognition results; If the character recognition result indicates that the currently recognized target character of the sample is the initial variable character of the sample, the currently recognized target character of the sample is removed to obtain the target sample variable sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The speech prediction sub-model includes a speech character encoding layer, an attention layer, a linear projection layer, a post-processing layer, and a prediction output layer. The step of performing speech synthesis processing on the initial sample variable sequence based on the initial speech prediction sub-model to obtain the first predicted speech includes: The initial sample variable sequence is processed by speech character encoding according to the speech character encoding layer to obtain speech sample encoding features; Based on the attention layer, the context features of the speech sample encoding features are extracted to obtain the current step context features of the first sample. The current step context features of the first sample and the preset Mel spectrum are concatenated to obtain the context features to be processed; The context features to be processed are input into a preset two-layer long short-term memory layer for context feature prediction to obtain the current step context features of the second sample; The current step context features of the second sample are linearly projected according to the linear projection layer to obtain the current step projection scalar data. The post-processing layer performs spectral update processing on the current step context features of the second sample to update the preset Mel spectrum; The first predicted speech is obtained by performing speech synthesis processing on the current step projection scalar data based on the prediction output layer.

6. A speech synthesis system, characterized in that, The system includes: The text acquisition module is used to acquire a sample text sequence and sample speech of the sample text sequence, wherein the sample text sequence includes sample initial characters; The model input module is used to input the sample text sequence into a preset original speech synthesis model, which includes a character modulation sub-model and an initial speech prediction sub-model. The text character adjustment module is used to adjust the text characters of the sample text sequence according to the character adjustment sub-model to obtain an initial sample variable sequence. Specifically, it includes: randomly extracting characters from the sample text sequence to obtain initial sample variable characters; merging the initial sample characters and the initial sample variable characters to obtain an initial sample variable sequence, wherein the initial sample variable sequence includes sample candidate characters. The first speech prediction module is used to perform speech synthesis processing on the initial sample variable sequence according to the initial speech prediction sub-model to obtain the first predicted speech. The first parameter adjustment module is used to calculate the spectral loss based on the first predicted speech and the sample speech to obtain prediction loss data; calculate the partial derivative of the sample candidate characters based on the prediction loss data to obtain character variable data; calculate the mean of the character variable data based on the number of sample text sequences to obtain character metric data; compare the preset character metric threshold with the character metric data to obtain a metric comparison result; filter the sample candidate characters based on the metric comparison result to obtain sample target characters; and adjust the parameters of the initial speech prediction sub-model based on the initial sample characters, the initial sample variable characters, and the sample target characters to obtain a candidate speech synthesis model. The character filtering module is used to filter the initial sample variable sequence based on the first predicted speech and the sample speech to obtain the target sample variable sequence; The second speech prediction module is used to input the target sample variable sequence into the candidate speech synthesis model for speech synthesis processing to obtain the second predicted speech. The second parameter adjustment module is used to adjust the parameters of the candidate speech synthesis model according to the second predicted speech and the sample speech to obtain the target speech synthesis model. The target speech synthesis module is used to input the acquired target text sequence into the target speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images