Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, finance and medical treatment, and relates to a speech synthesis method, which comprises the steps of performing text regularization on a to-be-processed text to obtain a regularized text; performing oral rewriting on the regularized text to obtain an oral text; performing text feature discretization on the spoken text to obtain text discrete features; performing voice feature reasoning on the text discrete features to obtain voice discrete features; decoding the voice discrete features to obtain voice spectrum information; and through the trained vocoder, converting the voice frequency spectrum information into audio frequency, and outputting synthetic voice. The invention further provides a speech synthesis device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-processed text can be stored in a block chain. According to the invention, clear and natural voice can be ensured to be generated, the quality and reality of text-to-voice conversion are effectively improved, and the real-time performance of voice synthesis is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology and digital medical technology, and in particular to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of science and technology, the application of text-to-speech (TTS) technology in the financial and medical fields is becoming more and more widespread, providing users with a more convenient and efficient way to interact with information. Especially in scenarios such as financial transaction prompts and medical report interpretation, TTS technology can convert complex text information into speech, helping users quickly obtain key information. Existing TTS technology, especially neural speech synthesis technology based on deep learning, has made significant progress and can generate highly natural speech, greatly improving the user experience.

[0003] However, despite the broad application prospects of these technologies in the financial and medical fields, they still face a series of challenges. First, high-quality training data is the core of neural speech synthesis technology, but in the financial and medical fields, due to the professionalism and sensitivity of the data, it is particularly difficult to obtain and organize a large amount of such data. This not only increases the difficulty of implementing the technology, but also limits its widespread application.

[0004] Secondly, high demand for computing resources is another problem facing current neural speech synthesis technology. In the financial and medical fields, the requirements for real-time performance and accuracy are extremely high, and deep learning models usually require a large amount of computing resources for training and reasoning. This not only increases operating costs, but also places high demands on the performance of hardware equipment.

[0005] In addition, although neural speech synthesis technology has made significant progress in speech naturalness and fluency, there is still a slight gap compared to real human speech. In the fields of finance and medicine, this gap may lead to misunderstanding or omission of information, which in turn affects users' decisions and judgments. In scenarios such as real-time transaction prompts and emergency medical information transmission, users expect to get voice feedback quickly. However, existing technologies are often difficult to meet this demand due to their high computational complexity, resulting in delayed information transmission. Because the inherent mechanism of deep learning models is complex and difficult to understand intuitively, technicians lack clear guidance when optimizing model performance. This not only increases the difficulty of technical implementation, but also limits the application effect of technology in specific scenarios.

[0006] In summary, although deep learning-based neural speech synthesis technology has made significant progress in the fields of finance and medicine, it still needs in-depth research and improvement in areas such as training data acquisition, computing resource optimization, speech naturalness and fluency improvement, real-time performance improvement, interpretability enhancement, and emotional expression and intonation changes to better meet the needs of practical applications. Summary of the invention

[0007] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, computer device and storage medium to solve the technical problems in the related art that speech synthesis relies on a large amount of high-quality training data, has high computing resource requirements, has low naturalness and fluency of speech, and lacks real-time performance and interpretability.

[0008] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solution:

[0009] Obtaining a text to be processed, and performing text regularization on the text to be processed to obtain a regularized text;

[0010] Rewriting the regularized text into a colloquial text to obtain a colloquial text;

[0011] Discretizing the text features of the spoken text to obtain text discrete features;

[0012] Performing speech feature inference on the text discrete features to obtain speech discrete features;

[0013] Decoding the speech discrete features to obtain speech spectrum information;

[0014] The speech spectrum information is converted into audio through the trained vocoder, and synthesized speech is output.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device, which adopts the following technical solution:

[0016] A regularization module, used for obtaining a text to be processed, performing text regularization on the text to be processed, and obtaining a regularized text;

[0017] A rewriting module, used for rewriting the regularized text into colloquial language to obtain a colloquial text;

[0018] A text discretization module, used for discretizing the text features of the spoken text to obtain text discrete features;

[0019] A speech discrete module, used for performing speech feature inference on the text discrete features to obtain speech discrete features;

[0020] A decoding module, used for decoding the discrete features of the speech to obtain speech spectrum information;

[0021] The speech synthesis module is used to convert the speech spectrum information into audio through a trained vocoder and output synthesized speech.

[0022] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0023] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method described above when executing the computer-readable instructions.

[0024] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0025] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the speech synthesis method described above are implemented.

[0026] Compared with the prior art, this application has the following beneficial effects:

[0027] The present application provides a speech synthesis method, which obtains synthesized speech and realizes text-to-speech conversion by sequentially performing text regularization, colloquial rewriting, text feature discretization, speech feature inference, speech decoding and audio conversion on an acquired text to be processed. By regularizing the text, redundant information can be removed, and the consistency of text data and the text quality can be improved; by rewriting the text in a colloquial way, the readability and naturalness of the text can be enhanced, which helps to generate a more natural, fluent and expressive speech; by discretizing the text features and the speech features of the colloquial text, the effectiveness and accuracy of the speech information in the transmission process can be ensured, and the processing depth and precision of the speech data can be improved; by decoding the discrete speech features, more refined spectral information can be obtained, and more refined control of speech details can be achieved, which is conducive to the generation of high-quality speech; by converting the speech spectral information into a continuous audio signal through a vocoder, it is ensured that a clear and natural speech is generated, which effectively improves the quality and realism of the text-to-speech conversion and improves the real-time performance of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0030] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;

[0031] Figure 3 is a flowchart of a specific implementation of the speech synthesis method according to the present application;

[0032] Figure 4 is a structural schematic diagram of an embodiment of a speech synthesis model according to the present application;

[0033] Figure 5 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present application;

[0034] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0039] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0041] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0042] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the speech synthesis device is generally arranged in the server / terminal device.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application, comprising the following steps:

[0045] Step S201, obtaining a text to be processed, performing text regularization on the text to be processed, and obtaining a regularized text.

[0046] Among them, the text to be processed is the text that currently needs to be converted into speech. For example, in the financial field, the text to be processed includes financial transaction reminder text, customer service and communication text, emergency notification and alarm text, financial knowledge explanation text, etc.; in the medical field, the text to be processed includes medical report interpretation text, medical knowledge popularization text, etc.

[0047] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can obtain the text to be processed through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0048] It should be emphasized that in order to further ensure the privacy and security of the text to be processed, the above text to be processed can also be stored in a node of a blockchain.

[0049] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0050] Text regularization is the process of standardizing the text to be processed, including removing stop words, removing HTML tags, normalizing punctuation, removing special characters, unifying uppercase and lowercase letters, text segmentation, part-of-speech tagging, and text format standardization. Text regularization can use pre-configured regular expressions or trained regularization models.

[0051] In some optional implementations, the step of performing text regularization on the text to be processed to obtain the regularized text includes:

[0052] Inputting the text to be processed into a trained regularization model, wherein the regularization model includes a first embedding layer and a plurality of regular decoder layers connected in stacked layers;

[0053] The first embedding layer converts the text to be processed into an embedding vector to obtain an initial text embedding vector;

[0054] The initial text embedding vector is input into multiple regular decoder layers for autoregressive generation in sequence to obtain the optimal regularized text;

[0055] The optimal regularized text is post-processed to obtain the final regularized text.

[0056] In this embodiment, the regularization model is a GPT model, which adopts a basic Transformer decoder-only structure, including a first embedding layer (Embedding) and multiple regular decoder layers connected in layers. The number of regular decoder layers can be set according to actual conditions. For example, 12 regular decoder layers connected in sequence are set. Different regular decoder layers can process text at different scales. For example, regular decoder layers of 1 to 4 layers are used to learn surface features at the phrase level of text, regular decoder layers of 5 to 8 layers are used to learn syntactic features at the syntactic level of text, and regular decoder layers of 9 to 12 layers are used to learn semantic features at the semantic level of text.

[0057] Among them, the training process of the regularization model includes: obtaining a text data set, in which each text in the text data set corresponds to a real regularized text as a label; randomly dividing the text data set into a training set and a test set in a ratio of 8:2; inputting the training set into the pre-trained GPT model, and allowing the model to generate a regularized text sequence word by word according to the context information of the input text through autoregressive generation, and in the generation process, using a beam search algorithm, according to the probability score of the model, select the best as the final predicted regularized text; according to the loss value of the predicted regularized text and the corresponding real regularized text, and using optimization techniques such as gradient accumulation and learning rate scheduling, the model parameters of the GPT model are adjusted until the model converges to obtain a model to be verified; testing the model to be verified through the test set to obtain a test result, and outputting the final model as the regularized model when the test result meets the preset conditions.

[0058] Each regular decoder layer includes a masked multi-head attention mechanism layer and a fully connected layer. During the model training process, the masked multi-head attention mechanism layer uses a multi-head attention mechanism and a masked language modeling method to randomly mask some text fragments, allowing the model to predict the masked text content based on the context, thereby capturing the semantic information and dependencies of the text, thereby improving the model's ability to understand language. Among them, the output of the masked multi-head attention mechanism layer is layer-normalized and then input into the fully connected layer for processing, and the fully connected layer is post-normalized. For example, the initial text embedding vector is input into the regular decoder layer, and the output vector of the model is obtained after processing through the masked multi-head attention mechanism layer, layer normalization, fully connected layer and post-normalization.

[0059] In this embodiment, the trained regularization model is used to regularize the text to be processed, and the probability distribution of all predicted regularized texts is output. A decoding strategy such as beam search or top-k sampling is used to generate an optimal regularized text that conforms to the language rules and semantic coherence according to the given input text. The generated optimal regularized text is post-processed, such as removing repeated words, correcting possible grammatical errors, etc., to improve the quality and readability of the text.

[0060] Regularizing the text to be processed through the regularization model can improve the efficiency and accuracy of regularization, improve the standardization and readability of the generated text, and reduce the probability of grammatical errors and inappropriate expressions in the text, laying the foundation for subsequent tasks.

[0061] Step S202, rewriting the regularized text into a colloquial language to obtain a colloquial text.

[0062] Colloquial rewriting of a text means converting formal, written language into a more natural, everyday, and easy-to-understand spoken form. This process usually involves adjusting sentence structure, choosing more colloquial vocabulary, and adding or reducing details to fit the rhythm and habits of spoken communication.

[0063] In some optional implementations of this embodiment, the step of rewriting the regularized text into colloquial language to obtain the colloquial text includes:

[0064] Inputting the regularized text into a trained spoken language rewriting model, wherein the spoken language rewriting model includes a second embedding layer and a plurality of spoken language decoder layers connected in a stacked manner;

[0065] The regularized text is converted into an embedding vector through the second embedding layer to obtain a regularized text embedding vector;

[0066] The regular text embedding vector is input into multiple spoken language decoder layers for autoregressive generation in sequence to obtain the probability distribution of spoken language text;

[0067] The spoken text is obtained according to the probability distribution of the spoken text.

[0068] In this embodiment, the spoken language rewriting model uses a pre-trained GPT model as a backbone network, including a second embedding layer and multiple spoken language decoder layers connected in stacked layers. The number of spoken language decoder layers can also be set according to actual conditions. For example, 12 layers of stacked spoken language decoder layers can be used, and the output of each spoken language decoder layer and the regular text embedding vector are used as the input of the next spoken language decoder layer.

[0069] In this embodiment, multiple spoken language decoder layers may be hierarchically divided, and different layers may process text features of different scales. Alternatively, different attention heads in each spoken language decoder layer may process text features of different scales.

[0070] The spoken rewriting model is trained using a dataset consisting of a massive corpus of spoken text, which is expected to include original text and regularized text.

[0071] In some optional implementations, each spoken language decoder layer includes a masked multi-head attention mechanism layer and a fully connected layer. Layer normalization is performed after the masked multi-head attention mechanism layer, and post-normalization is performed after the fully connected layer. Finally, the probability distribution of all spoken texts is output through a Linear layer.

[0072] In this embodiment, according to the probability distribution of the spoken text, a greedy search or beam search algorithm is used to sample and generate an optimal spoken text sequence from the probability distribution of the spoken text as the final spoken text for output.

[0073] By rewriting the regularized text into colloquial language through the spoken language rewriting model, the efficiency and accuracy of spoken language rewriting can be improved, the readability and affinity of the text can be enhanced, the comprehensibility of the text can be improved, and the naturalness and fluency of speech synthesis can be improved, thereby expanding the application scenarios.

[0074] Step S203, discretizing the text features of the spoken text to obtain text discrete features.

[0075] In this embodiment, the spoken text is input into the trained text discrete model, and the text is converted into a series of discrete tokenID sequences by analyzing the semantic and grammatical structure of the spoken text and using the vocabulary and encoding rules in the model.

[0076] In some optional implementations of this embodiment, the step of discretizing the text features of the spoken text to obtain text discrete features includes:

[0077] Inputting the spoken text into the trained text discretization model, wherein the text discretization model includes a preprocessing layer, a word segmentation layer, and a text encoding layer;

[0078] Preprocessing the spoken text through a preprocessing layer to obtain a preprocessed spoken text;

[0079] Input the preprocessed spoken text into the word segmentation layer for word segmentation to obtain word segmentation units;

[0080] The word segmentation units are input into the text encoding layer, and the word segmentation units are converted into text discrete features using the vocabulary and encoding rules in the text discrete model.

[0081] Among them, the text discretization model is trained on the basis of the pre-trained tokenizer model. In the process of discretizing text features, the dimension and sparsity of the discrete representation vector are controlled according to the vocabulary size and encoding method of the tokenizer model to optimize the subsequent speech synthesis performance. Specifically, the word segmentation unit is obtained by segmenting the colloquial text; the word segmentation unit is input into the text encoding layer, and each word segmentation unit is mapped to the corresponding tokenID using the predefined vocabulary. For unregistered word segmentation units, the subword segmentation method is used to process the segmentation into characters and / or subwords, and the characters and / or subwords are mapped to the vocabulary; for special symbols in the text such as punctuation, numbers, etc., specific encoding methods such as [PUNCT], [NUM], etc. are used to mark them to retain their semantic information. At the same time, in order to control the dimension and sparsity of the discrete representation vector, the maximum sequence length can be set, for example, the maximum sequence length is 128, and the sequence exceeding the length is truncated, and the sequence within the length is padded. Next, we use the position encoding mechanism in the tokenizer model to add position information to the discrete representation vector to obtain the final text discrete features, where the position information is used to reflect the order and structural relationship of the text. During the encoding process, we use character-level encoding to split uncommon words and new words in the spoken text into single characters for representation, ensuring the coverage and flexibility of the discrete representation vector.

[0082] It should be noted that in the encoding process, through multiple iterative segmentation and encoding, discrete representations of different granularities are obtained, such as character level, word level, subword level, etc., to fully capture the local and global features of the text.

[0083] By converting spoken text into text discrete features, the computational difficulty of the model can be reduced, the demand for computing resources can be reduced, and it will help to better understand the text and generate high-quality speech in the subsequent speech synthesis process.

[0084] Step S204, performing speech feature inference on the text discrete features to obtain speech discrete features.

[0085] The core of speech synthesis is to convert text features into speech features. First, it is necessary to convert text discrete features into corresponding speech discrete features. Speech discrete features refer to the discrete representation of speech features corresponding to the text.

[0086] In this embodiment, discrete feature conversion is performed by a trained speech feature conversion model, which is obtained by training with a corresponding training data set based on a pre-trained GPT model. The speech feature conversion model includes a third embedding layer and a plurality of speech feature decoder layers connected in layers, and the number of speech feature decoder layers can be set according to actual conditions. Each speech feature decoder layer includes a masked multi-head attention mechanism layer and a fully connected layer, and the text discrete features are inferred by the masked multi-head attention mechanism layer and the fully connected layer to obtain speech discrete features.

[0087] In some optional implementations of this embodiment, the step of performing speech feature inference on text discrete features to obtain speech discrete features includes:

[0088] Inputting the text discrete features into the trained speech feature conversion model, wherein the speech feature conversion model includes a third embedding layer and a plurality of speech feature decoder layers connected in a stacked manner;

[0089] The text discrete features are converted into embedding vectors through the third embedding layer to obtain text feature embedding vectors;

[0090] The text feature embedding vector is input into multiple speech feature decoder layers for masked multi-head attention mechanism processing to map the text to speech and obtain speech discrete features.

[0091] In this embodiment, each speech feature decoder layer includes a masked multi-head attention mechanism layer and a fully connected layer, and layer normalization is performed after the masked multi-head attention mechanism layer, and post-normalization is performed after the fully connected layer. Specifically, the text feature embedding vector is input into the first speech feature decoder layer, and the self-attention calculation is performed on the text feature embedding vector by each head of the masked multi-head attention mechanism layer to obtain the output of each head, and the outputs of multiple heads are fused to obtain the text attention feature representation of the speech feature decoder layer, and the text attention feature representation and the text feature embedding vector are used as the input of the next speech feature decoder layer, and the calculation is repeated until all speech feature decoder layers are processed, and the final text attention feature representation is output, and the final text attention feature representation is mapped to the dimensional space of speech features to obtain speech discrete feature output.

[0092] By converting text discrete features into speech discrete features, the focus of speech features can be enhanced, the processing depth and accuracy of speech data can be improved, and the efficiency, fluency and naturalness of subsequent speech synthesis can be improved.

[0093] Step S205: decode the speech discrete features to obtain speech spectrum information.

[0094] Decoding the discrete features of speech into speech spectrum information realizes the conversion of features from discrete to continuous. Among them, the speech spectrum information represents the distribution and intensity of different frequency components in the speech signal, including frequency components, formants, fundamental frequency period, amplitude and energy distribution.

[0095] In this embodiment, the trained speech decoding model is used to decode the discrete features of speech, and the discrete features of speech are mapped to the dimensional space of spectral features. The speech decoding model is trained on the basis of the pre-trained GPT model, including a fourth embedding layer and multiple speech decoder layers connected in layers, the number of speech decoder layers is set according to actual conditions, each speech decoder layer includes a masked multi-head attention mechanism layer and a fully connected layer, layer normalization is performed after the masked multi-head attention mechanism layer, and post-normalization is performed after the fully connected layer.

[0096] Specifically, the speech discrete features are input into a trained speech decoding model, which includes a fourth embedding layer and multiple speech decoder layers connected in a stacked manner; the speech discrete features are embedded in vectors through the fourth embedding layer to obtain speech feature embedding vectors; the speech feature embedding vectors are input into multiple speech decoder layers for speech decoding to obtain speech spectrum information.

[0097] Through the masked multi-head attention mechanism layer of the speech decoder layer, the speech feature embedding vector is weightedly fused using the attention mechanism to obtain the fused speech discrete feature representation, and the fused speech discrete feature representation is layer-normalized to eliminate the differences between different dimensions. The layer-normalized fused speech discrete feature representation is input into the fully connected layer for integration and transformation to obtain the concatenated speech discrete feature representation, and the concatenated speech discrete feature representation is post-normalized to eliminate the dimensional influence between different features. The post-normalized concatenated speech discrete feature representation and the speech feature embedding vector are used as the input of the next speech decoder layer, and the calculation process of the speech decoder layer is repeated until the last speech decoder layer is calculated, and the final speech discrete fusion feature representation is output. The speech discrete fusion feature representation is mapped to the dimensional space of the spectrum information through the linear layer to obtain the speech spectrum information.

[0098] By decoding the discrete features of speech, we can extract high-level spectral features, capture the dependencies of speech features between different time steps, improve the quality and clarity of the spectrum, achieve more refined control over speech details, and thus improve the quality of subsequent synthesized speech.

[0099] Step S206: convert the speech spectrum information into audio through the trained vocoder, and output synthesized speech.

[0100] The speech spectrum information describes the frequency and intensity distribution of the sound in detail. The speech spectrum information is input into the trained vocoder, which converts the speech spectrum information into a continuous audio waveform signal in the time domain, that is, synthesized speech. The converted audio waveform signal is processed, such as volume normalization, removal of silent segments, and fade-in and fade-out, to improve the audio quality. The processed audio waveform signal is encoded into an audio file in the target format, such as WAV, MP3 and other common audio formats, through audio coding technology, and stored in the specified output path or transmitted to the downstream application module to complete the conversion process from text to speech.

[0101] In some optional implementations of this embodiment, the steps of converting speech spectrum information into audio and outputting synthesized speech by the trained vocoder include:

[0102] Inputting speech spectrum information into a vocoder, the vocoder includes an input layer, a feature extraction layer, a convolution layer, a prediction layer, a speech restoration layer and an output layer;

[0103] Preprocess the speech spectrum information through the input layer to obtain a speech input signal;

[0104] Input the speech input signal into the feature extraction layer for feature extraction to obtain speech spectrum features;

[0105] The speech spectrum features are subjected to channel convolution fusion through the convolution layer to obtain convolution fusion spectrum features;

[0106] The convolution fusion spectrum features are input into the prediction layer, and the speech spectrogram is predicted based on the convolution fusion spectrum features;

[0107] The speech time-spectrogram is restored to a speech audio signal through the speech restoration layer;

[0108] The output layer synthesizes the speech waveform of the speech audio signal and outputs the synthesized speech.

[0109] Among them, the vocoder can adopt the VOCOS model. The VOCOS model does not rely on transposed convolution for upsampling, but directly predicts the short-time Fourier transform (STFT) spectrogram through network stacking, and then uses the inverse short-time Fourier transform (iSTFT) to restore the speech.

[0110] In this embodiment, the feature extraction layer can perform speech signal decomposition, feature extraction and encoding on the speech input signal, and the obtained speech spectrum features include frequency band, frequency band signal strength, fundamental frequency, voiced and unvoiced sounds, etc. The convolution layer usually adopts the ConvNeXt convolution module, which is composed of deep convolution and two point-by-point convolutions, and realizes STFT spectrum reconstruction through residual connection, that is, convolution fusion spectrum features are obtained; the prediction layer predicts the speech time spectrum diagram, that is, STFT spectrum, according to the extracted convolution fusion spectrum features; the speech restoration layer is the iSTFT layer, which uses the predicted STFT spectrum to restore the speech signal through iSTFT to obtain the speech audio signal; the output layer generates the final speech waveform for output through inverse transformation or synthesis algorithm according to the characteristic parameters (such as spectrum envelope, fundamental frequency, etc.) and model parameters of the input speech audio signal.

[0111] By converting speech spectrum information into audio and obtaining the final synthesized speech for output, it can ensure the generation of clear and natural speech, effectively improve the quality and realism of text-to-speech conversion, and improve the real-time performance of speech synthesis. The entire speech synthesis process combines deep learning and digital signal processing technology to achieve high-quality text-to-speech synthesis, improve user experience, and expand application scenarios.

[0112] In some optional implementations, the speech synthesis method of the present application is implemented by a deep learning generative speech synthesis model, which includes a regularization model, a spoken language rewriting model, a text discrete model, a speech feature conversion model, a speech decoding model and a vocoder connected in sequence. The text to be processed is input into the generative speech synthesis model, and is processed in sequence by the regularization model, the spoken language rewriting model, the text discrete model, the speech feature conversion model, the speech decoding model and the vocoder to obtain the final synthesized speech. The implementation process is shown in FIG. Figure 3 shown.

[0113] Among them, the regularization model, spoken language rewriting model, speech feature conversion model and speech decoding model are modeled by the pre-trained GPT model. The GPT model structure adopts the basic Transformer decoder-only structure, see Figure 4 As shown in the figure, the text to be processed is converted into regularized text and colloquial text through two conversions of the GPT model. The colloquial text is then converted into text discrete features through the text discrete model. The text discrete features are then converted into speech discrete features through the two-step GPT model. They are then converted into speech spectrum information. Finally, the speech spectrum information is passed through the vocoder to obtain synthesized speech.

[0114] In this embodiment, the regularization model, spoken language rewriting model, speech feature conversion model and speech decoding model can be independent models, or they can share the backbone network, that is, share the GPT model structure. For different tasks, the corresponding fully connected layer is connected after the linear layer of the GPT model structure as the output.

[0115] In order to further optimize the model performance, when training the generative speech synthesis model, multimodal data fusion training is used, that is, the original text, regularized text, spoken text, text discrete features, and speech discrete features are used as training data to train the generative speech synthesis model. During the training process, the regularization model, spoken rewriting model, speech feature conversion model, and speech decoding model are four tasks learned in parallel. This method can reduce the demand for training data and computing resources, accelerate model convergence, and improve training speed.

[0116] By regularizing the text, the present application can remove redundant information and improve the consistency of text data and the text quality; by rewriting the text in a colloquial manner, the readability and naturalness of the text can be enhanced, which helps to generate more natural, fluent and expressive speech; by discretizing the text features and speech features of the colloquial text, the effectiveness and accuracy of the speech information in the transmission process can be ensured, and the processing depth and precision of the speech data can be improved; by decoding the discrete speech features, more refined spectral information can be obtained, and more refined control of the speech details can be achieved, which is conducive to the generation of high-quality speech; by converting the speech spectral information into a continuous audio signal through a vocoder, clear and natural speech can be generated, which effectively improves the quality and realism of text-to-speech conversion and improves the real-time performance of speech synthesis.

[0117] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0118] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0121] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 5 As shown, the speech synthesis device 500 described in this embodiment includes: a regularization module 501, a rewriting module 502, a text discretization module 503, a speech discretization module 504, a decoding module 505 and a speech synthesis module 506. Among them:

[0123] The regularization module 501 is used to obtain a text to be processed, and perform text regularization on the text to be processed to obtain a regularized text;

[0124] The rewriting module 502 is used to rewrite the regularized text into colloquial text to obtain a colloquial text;

[0125] The text discretization module 503 is used to discretize the text features of the spoken text to obtain text discrete features;

[0126] The speech discrete module 504 is used to perform speech feature inference on the text discrete features to obtain speech discrete features;

[0127] The decoding module 505 is used to decode the speech discrete features to obtain speech spectrum information;

[0128] The speech synthesis module 506 is used to convert the speech spectrum information into audio through a trained vocoder and output synthesized speech.

[0129] It should be emphasized that in order to further ensure the privacy and security of the text to be processed, the above text to be processed can also be stored in a node of a blockchain.

[0130] Based on the above-mentioned speech synthesis device 500, by regularizing the text, redundant information can be removed and the consistency of text data and text quality can be improved; by rewriting the text in a colloquial way, the readability and naturalness of the text can be enhanced, which helps to generate more natural, fluent and expressive speech; by discretizing the text features and speech features of the colloquial text, the effectiveness and accuracy of the speech information in the transmission process can be ensured, and the processing depth and precision of the speech data can be improved; by decoding the discrete speech features, more refined spectral information can be obtained, and more refined control of speech details can be achieved, which is conducive to generating high-quality speech; by converting the speech spectral information into a continuous audio signal through a vocoder, clear and natural speech can be generated, which effectively improves the quality and realism of text-to-speech conversion and improves the real-time performance of speech synthesis.

[0131] In some optional implementations of this embodiment, the regularization module 501 is further used to:

[0132] Inputting the to-be-processed text into a trained regularization model, wherein the regularization model comprises a first embedding layer and a plurality of regularization decoder layers connected in stacked layers;

[0133] Performing embedding vector conversion on the text to be processed through the first embedding layer to obtain an initial text embedding vector;

[0134] Inputting the initial text embedding vector into the multiple regularized decoder layers to perform autoregressive generation in sequence to obtain an optimal regularized text;

[0135] The optimal regularized text is post-processed to obtain a final regularized text.

[0136] Regularizing the text to be processed through the regularization model can improve the efficiency and accuracy of regularization, improve the standardization and readability of the generated text, and reduce the probability of grammatical errors and inappropriate expressions in the text, laying the foundation for subsequent tasks.

[0137] In some optional implementations, the rewriting module 502 is further configured to:

[0138] Inputting the regularized text into a trained spoken language rewriting model, wherein the spoken language rewriting model comprises a second embedding layer and a plurality of spoken language decoder layers connected in a stacked manner;

[0139] Performing embedding vector conversion on the regularized text through the second embedding layer to obtain a regularized text embedding vector;

[0140] Inputting the regular text embedding vector into the multiple spoken language decoder layers to perform autoregressive generation in sequence to obtain a probability distribution of spoken language text;

[0141] The spoken text is obtained according to the probability distribution of the spoken text.

[0142] By rewriting the regularized text into colloquial language through the spoken language rewriting model, the efficiency and accuracy of spoken language rewriting can be improved, the readability and affinity of the text can be enhanced, the comprehensibility of the text can be improved, and the naturalness and fluency of speech synthesis can be improved, thereby expanding the application scenarios.

[0143] In some optional implementations, the text discretization module 503 is further used to:

[0144] Inputting the colloquial text into a trained text discrete model, wherein the text discrete model includes a preprocessing layer, a word segmentation layer, and a text encoding layer;

[0145] Preprocessing the spoken text by the preprocessing layer to obtain a preprocessed spoken text;

[0146] Inputting the preprocessed spoken text into the word segmentation layer for word segmentation to obtain word segmentation units;

[0147] The word segmentation unit is input into the text encoding layer, and the word segmentation unit is converted into text discrete features using the vocabulary and encoding rules in the text discrete model.

[0148] By converting spoken text into text discrete features, the computational difficulty of the model can be reduced, the demand for computing resources can be reduced, and it will help to better understand the text and generate high-quality speech in the subsequent speech synthesis process.

[0149] In some optional implementations of this embodiment, the voice discretization module 504 is further used for:

[0150] Inputting the text discrete features into a trained speech feature conversion model, wherein the speech feature conversion model includes a third embedding layer and a plurality of speech feature decoder layers connected in a stacked manner;

[0151] Performing embedding vector conversion on the text discrete features through the third embedding layer to obtain a text feature embedding vector;

[0152] The text feature embedding vector is input into the multiple speech feature decoder layers for masked multi-head attention mechanism processing, the text is mapped to speech, and speech discrete features are obtained.

[0153] By converting text discrete features into speech discrete features, the focus of speech features can be enhanced, the processing depth and accuracy of speech data can be improved, and the efficiency, fluency and naturalness of subsequent speech synthesis can be improved.

[0154] In this embodiment, the decoding module 505 is further used for:

[0155] Inputting the speech discrete features into a trained speech decoding model, wherein the speech decoding model comprises a fourth embedding layer and a plurality of speech decoder layers connected in a stacked manner;

[0156] Performing embedding vector conversion on the speech discrete features through the fourth embedding layer to obtain a speech feature embedding vector;

[0157] The speech feature embedding vector is input into the multiple speech decoder layers for speech decoding to obtain speech spectrum information.

[0158] By decoding the discrete features of speech, we can extract high-level spectral features, capture the dependencies of speech features between different time steps, improve the quality and clarity of the spectrum, achieve more refined control over speech details, and thus improve the quality of subsequent synthesized speech.

[0159] In some optional implementations, the speech synthesis module 506 is further configured to:

[0160] Inputting the speech spectrum information into the vocoder, the vocoder comprising an input layer, a feature extraction layer, a convolution layer, a prediction layer, a speech restoration layer and an output layer;

[0161] Preprocessing the speech spectrum information through the input layer to obtain a speech input signal;

[0162] Inputting the speech input signal into the feature extraction layer to extract features and obtain speech spectrum features;

[0163] Performing channel convolution fusion on the speech spectrum features through the convolution layer to obtain convolution fusion spectrum features;

[0164] Inputting the convolution fusion spectrum features into the prediction layer, and predicting the speech time spectrum graph according to the convolution fusion spectrum features;

[0165] The speech time-spectrogram is restored into a speech audio signal by the speech restoration layer;

[0166] The speech waveform of the speech audio signal is synthesized through the output layer to output synthesized speech.

[0167] By converting speech spectrum information into audio and obtaining the final synthesized speech for output, it can ensure the generation of clear and natural speech, effectively improve the quality and realism of text-to-speech conversion, and improve the real-time performance of speech synthesis. The entire speech synthesis process combines deep learning and digital signal processing technology to achieve high-quality text-to-speech synthesis, improve user experience, and expand application scenarios.

[0168] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0169] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0170] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0171] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the speech synthesis method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0172] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions or process data stored in the memory 61, such as computer-readable instructions for running the speech synthesis method.

[0173] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0174] By regularizing the text, redundant information can be removed and the consistency of text data and text quality can be improved; by rewriting the text in a colloquial way, the readability and naturalness of the text can be enhanced, which helps to generate more natural, fluent and expressive speech; by discretizing the text features and speech features of the colloquial text, the effectiveness and accuracy of speech information in the transmission process can be ensured, and the processing depth and precision of speech data can be improved; by decoding the discrete features of speech, more detailed spectral information can be obtained, and more detailed control of speech details can be achieved, which is conducive to the generation of high-quality speech; the speech spectral information is converted into a continuous audio signal through a vocoder to ensure the generation of clear and natural speech, effectively improving the quality and realism of text-to-speech conversion and improving the real-time performance of speech synthesis.

[0175] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech synthesis method as described above.

[0176] By regularizing the text, redundant information can be removed and the consistency of text data and text quality can be improved; by rewriting the text in a colloquial way, the readability and naturalness of the text can be enhanced, which helps to generate more natural, fluent and expressive speech; by discretizing the text features and speech features of the colloquial text, the effectiveness and accuracy of speech information in the transmission process can be ensured, and the processing depth and precision of speech data can be improved; by decoding the discrete features of speech, more detailed spectral information can be obtained, and more detailed control of speech details can be achieved, which is conducive to the generation of high-quality speech; the speech spectral information is converted into a continuous audio signal through a vocoder to ensure the generation of clear and natural speech, effectively improving the quality and realism of text-to-speech conversion and improving the real-time performance of speech synthesis.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Obtaining a text to be processed, and performing text regularization on the text to be processed to obtain a regularized text; Rewriting the regularized text into a colloquial text to obtain a colloquial text; Discretizing the text features of the spoken text to obtain text discrete features; Performing speech feature inference on the text discrete features to obtain speech discrete features; Decoding the speech discrete features to obtain speech spectrum information; The speech spectrum information is converted into audio through the trained vocoder, and synthesized speech is output.

2. The speech synthesis method according to claim 1, characterized in that: The step of performing text regularization on the text to be processed to obtain the regularized text comprises: Inputting the to-be-processed text into a trained regularization model, wherein the regularization model comprises a first embedding layer and a plurality of regularization decoder layers connected in stacked layers; Performing embedding vector conversion on the text to be processed through the first embedding layer to obtain an initial text embedding vector; Inputting the initial text embedding vector into the multiple regularized decoder layers to perform autoregressive generation in sequence to obtain an optimal regularized text; The optimal regularized text is post-processed to obtain a final regularized text.

3. The speech synthesis method according to claim 1, characterized in that: The step of rewriting the regularized text into a colloquial language to obtain a colloquial text comprises: Inputting the regularized text into a trained spoken language rewriting model, wherein the spoken language rewriting model comprises a second embedding layer and a plurality of spoken language decoder layers connected in a stacked manner; Performing embedding vector conversion on the regularized text through the second embedding layer to obtain a regularized text embedding vector; Inputting the regular text embedding vector into the multiple spoken language decoder layers to perform autoregressive generation in sequence to obtain a probability distribution of spoken language text; The spoken text is obtained according to the probability distribution of the spoken text.

4. The speech synthesis method according to claim 1, characterized in that: The step of discretizing the text features of the spoken text to obtain text discrete features comprises: Inputting the colloquial text into a trained text discrete model, wherein the text discrete model includes a preprocessing layer, a word segmentation layer, and a text encoding layer; Preprocessing the spoken text by the preprocessing layer to obtain a preprocessed spoken text; Inputting the preprocessed spoken text into the word segmentation layer for word segmentation to obtain word segmentation units; The word segmentation unit is input into the text encoding layer, and the word segmentation unit is converted into text discrete features using the vocabulary and encoding rules in the text discrete model.

5. The speech synthesis method according to claim 1, characterized in that: The step of performing speech feature inference on the text discrete features to obtain speech discrete features comprises: Inputting the text discrete features into a trained speech feature conversion model, wherein the speech feature conversion model includes a third embedding layer and a plurality of speech feature decoder layers connected in a stacked manner; Performing embedding vector conversion on the text discrete features through the third embedding layer to obtain a text feature embedding vector; The text feature embedding vector is input into the multiple speech feature decoder layers for masked multi-head attention mechanism processing, the text is mapped to speech, and speech discrete features are obtained.

6. The speech synthesis method according to claim 1, characterized in that: The step of decoding the speech discrete features to obtain speech spectrum information comprises: Inputting the speech discrete features into a trained speech decoding model, wherein the speech decoding model comprises a fourth embedding layer and a plurality of speech decoder layers connected in a stacked manner; Performing embedding vector conversion on the speech discrete features through the fourth embedding layer to obtain a speech feature embedding vector; The speech feature embedding vector is input into the multiple speech decoder layers for speech decoding to obtain speech spectrum information.

7. The speech synthesis method according to claim 1, characterized in that: The steps of converting the speech spectrum information into audio by the trained vocoder and outputting synthesized speech include: Inputting the speech spectrum information into the vocoder, the vocoder comprising an input layer, a feature extraction layer, a convolution layer, a prediction layer, a speech restoration layer and an output layer; Preprocessing the speech spectrum information through the input layer to obtain a speech input signal; Inputting the speech input signal into the feature extraction layer to extract features and obtain speech spectrum features; Performing channel convolution fusion on the speech spectrum features through the convolution layer to obtain convolution fusion spectrum features; Inputting the convolution fusion spectrum features into the prediction layer, and predicting the speech spectrogram according to the convolution fusion spectrum features; The speech time-spectrogram is restored into a speech audio signal through the speech restoration layer; The speech waveform of the speech audio signal is synthesized through the output layer to output synthesized speech.

8. A speech synthesis device, characterized in that: include: A regularization module, used for obtaining a text to be processed, performing text regularization on the text to be processed, and obtaining a regularized text; A rewriting module, used for rewriting the regularized text into colloquial language to obtain a colloquial text; A text discretization module, used for discretizing the text features of the spoken text to obtain text discrete features; A speech discrete module, used for performing speech feature inference on the text discrete features to obtain speech discrete features; A decoding module, used for decoding the discrete features of the speech to obtain speech spectrum information; The speech synthesis module is used to convert the speech spectrum information into audio through a trained vocoder and output synthesized speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images