Speech synthesis method, system and device, storage medium and program product

Abstract

The invention provides a voice synthesis method, system and device, a storage medium and a program product, and relates to the technical field of artificial intelligence and voice processing, and the method comprises the steps: obtaining Mel spectrum data corresponding to to-be-synthesized voice text data; inputting the Mel spectrum data into a neural vocoder based on a selective state space model; and performing long sequence processing on the Mel spectrum data by using the selective state space model based on the neural vocoder to obtain synthesized audio data corresponding to the to-be-synthesized voice text data. According to the invention, the neural vocoder can be constructed based on the state space model for speech synthesis, the high-frequency reconstruction capability is improved, the loss of high-frequency details is avoided, and better synthetic tone quality is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and speech processing technology, and in particular to a speech synthesis method, system, device, storage medium and program product. BACKGROUND

[0002] In the field of modern artificial intelligence and speech processing technology, the importance of text-to-speech (TTS) systems is increasingly prominent. It is widely used in intelligent assistants, barrier-free reading, intelligent customer service and other scenarios, and has become one of the key technologies for human-computer interaction.

[0003] In the TTS system, the vocoder plays a very core role, which undertakes the task of accurately converting intermediate acoustic features into waveform signals. Its performance directly affects the quality and naturalness of the output speech of the TTS system. In related technologies, although the vocoder technology has made some progress, the existing vocoder still has deficiencies in the high-frequency reconstruction capability of audio, which can easily cause the synthesized speech to lose high-frequency details, resulting in unclear and bright sound, affecting the overall sound quality. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis method, system, device, storage medium and program product, which aims to construct a neural vocoder based on a state space model for speech synthesis, improve the high-frequency reconstruction capability to avoid the loss of high-frequency details, and thus obtain better synthesized sound quality.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a speech synthesis method, which comprises:

[0006] Obtaining mel-frequency spectrum data corresponding to the to-be-synthesized speech text data;

[0007] Inputting the mel-frequency spectrum data into a neural vocoder based on a selective state space model; the neural vocoder is constructed based on a selective state space model and a generative adversarial network model;

[0008] Based on the neural vocoder, the mel-frequency spectrum data is processed by the selective state space model to obtain synthesized audio data corresponding to the to-be-synthesized speech text data.

[0009] In some embodiments, the neural vocoder is based on the selective state space model to process the mel-frequency spectrum data to obtain synthesized audio data corresponding to the to-be-synthesized speech text data, comprising:

[0010] capturing long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder employing the selective state space model, to obtain a context-aware feature representation;

[0011] generating synthesized audio data corresponding to the text-to-speech data based on the context-aware feature representation.

[0012] In some embodiments, the neural vocoder comprises a generator comprising a neural network module for a bidirectional sequence modeling task; and

[0013] modeling the mel-spectrogram data based on the neural network module employing the selective state space model for bidirectional sequence modeling, to obtain a bidirectional sequence processing result; the bidirectional sequence processing result is a long-range temporal dependency of the mel-spectrogram data in both directions.

[0014] In some embodiments, the neural network module comprises a forward neural network layer and a backward neural network layer based on the selective state space model;

[0015] modeling the mel-spectrogram data based on the neural network module employing the selective state space model for bidirectional sequence modeling, to obtain a bidirectional sequence processing result, comprises:

[0016] modeling past context based on the forward neural network layer for forward recursive processing of the mel-spectrogram data, to obtain a forward sequence processing result; the forward sequence processing result is a long-range temporal dependency of the mel-spectrogram data in a forward direction, and the bidirectional sequence processing result comprises the forward sequence processing result;

[0017] modeling future context based on the backward neural network layer for inverse recursive processing of the mel-spectrogram data, to obtain an inverse sequence processing result; the inverse sequence processing result is a long-range temporal dependency of the mel-spectrogram data in an inverse direction, and the bidirectional sequence processing result comprises the inverse sequence processing result.

[0018] In some embodiments, the method further comprises:

[0019] fusing the forward sequence processing result and the inverse sequence processing result to obtain a fused long sequence processing result;

[0020] performing linear projection on the fused long sequence processing result to obtain the context-aware feature representation.

[0021] In some embodiments, the generator comprises a decoder module;

[0022] generating, based on the feature representation, synthesized audio data corresponding to the to-be-synthesized speech text data, comprises:

[0023] converting, based on the decoder module, the feature representation into an audio waveform;

[0024] mapping, based on the decoder module, a linear prediction value of the audio waveform to a preset audio amplitude range, to obtain synthesized audio data corresponding to the to-be-synthesized speech text data.

[0025] In some embodiments, before the long-range temporal dependency capturing of the mel-frequency spectrum data by the neural vocoder using the selective state space model, the method further comprises:

[0026] constructing a generator based on the selective state space model;

[0027] training, based on a preset discriminator, the generator by a generative adversarial network, to obtain a neural vocoder based on a generative adversarial network model architecture.

[0028] In some embodiments, after the long sequence processing of the mel-frequency spectrum data by the neural vocoder using the selective state space model to obtain synthesized audio data corresponding to the to-be-synthesized speech text data, the method further comprises:

[0029] obtaining a sound quality verification result of the synthesized audio data;

[0030] updating parameters of a preset discriminator based on the sound quality verification result to obtain an updated discriminator;

[0031] training, based on the updated discriminator, the generator by a generative adversarial network, to obtain an updated generator.

[0032] To achieve the above object, a second aspect of the embodiment of the present application proposes a speech synthesis system, which comprises:

[0033] a data acquisition module configured to acquire mel-frequency spectrum data corresponding to to-be-synthesized speech text data;

[0034] a vocoding module configured to input the mel-frequency spectrum data into a neural vocoder based on a selective state space model, and to perform long sequence processing on the mel-frequency spectrum data by the neural vocoder using the selective state space model, to obtain synthesized audio data corresponding to the to-be-synthesized speech text data.

[0035] To achieve the above object, a third aspect of the embodiments of the present application provides a speech synthesis device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method of the first aspect when executing the computer program.

[0036] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the speech synthesis method of the first aspect when executed by a processor.

[0037] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer program product, comprising a computer program, which implements the speech synthesis method of the first aspect when executed by a processor.

[0038] The speech synthesis method, system, device, computer readable storage medium and computer program product provided by the present application obtain the mel-frequency spectrum data corresponding to the to-be-synthesized speech text data, input the mel-frequency spectrum data into the neural vocoder based on the selective state space model, and obtain the synthesized audio data corresponding to the to-be-synthesized speech text data by using the selective state space model to perform long sequence processing on the mel-frequency spectrum data based on the neural vocoder.

[0039] Compared with the conventional speech synthesis method based on the Griffin-Lim algorithm (Griffin-Lim Algorithm, Griffin-lim), the adaptive weight spectrum interpolation speech transformation and representation method (Speech Transformation and Representation using Adaptive Interpolation of weiGHTed spectrum, STRAIGHT) and the high-quality real-time speech synthesis system based on the vocoder (A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications, WORLD), etc., the embodiments of the present application input the mel-frequency spectrum data corresponding to the to-be-synthesized speech text data into the neural vocoder constructed based on the selective state space model in advance, so as to obtain the synthesized audio data corresponding to the to-be-synthesized speech text data by using the selective state space model to perform long sequence processing on the mel-frequency spectrum data based on the neural vocoder.

[0040] In this way, the embodiments of the present application can utilize the long-sequence processing capabilities of the selective state-space model, which is suitable for tasks requiring long-range dependency modeling, to process long sequences of mel-spectrogram data, thereby avoiding the situation where traditional vocoders ignore long-range dependencies and cause loss of high-frequency details, and thus improving the synthesized sound quality. In other words, the embodiments of the present application can construct a neural vocoder based on the state-space model for speech synthesis, improve the high-frequency reconstruction capabilities and avoid the loss of high-frequency details, thereby obtaining better synthesized sound quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of the steps of the speech synthesis method provided in some embodiments of the present application;

[0042] Figure 2 A flowchart of the steps for constructing a neural vocoder involved in some embodiments of the speech synthesis method provided in the embodiments of the present application;

[0043] Figure 3 A flowchart of steps in other embodiments of the speech synthesis method provided in the embodiment of the present application;

[0044] Figure 4 for Figure 1 Schematic diagram of the detailed process of step S103;

[0045] Figure 5 A schematic diagram of the overall model architecture of a neural vocoder in some embodiments of the speech synthesis method provided in the embodiments of the present application;

[0046] Figure 6 A schematic flow chart of the steps of performing bidirectional sequence modeling on Mel-spectrogram data involved in some embodiments of the speech synthesis method provided in the embodiments of the present application;

[0047] Figure 7 A schematic diagram of the structure of the speech synthesis system provided in an embodiment of the present application;

[0048] Figure 8 This is a schematic diagram of the hardware structure of the speech synthesis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] It should be noted that although the functional modules are divided in the device / system schematic diagram, the logical order is shown in the flowchart, but in some cases, the steps shown or described can be performed in a manner different from the module division in the device / system, or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing the embodiments of the present application only, and is not intended to limit the present application.

[0052] First, the professional technical terms involved in the speech synthesis method provided by the embodiments of the present application are simply explained.

[0053] Neural vocoder.

[0054] Neural vocoder is also called neural voice coder. Wavenet, a kind of deep autoregressive model, marks the emergence of neural vocoder, which uses dilated convolution to directly model the original waveform. Although Wavenet has excellent audio fidelity, its inference is very slow and cannot be practically deployed. In order to solve this problem, efficient autoregressive models (such as WaveRNN) and parallel inference frameworks (such as Parallel WaveNet) are introduced, which provide faster generation with acceptable quality. In addition, the introduction of generative adversarial network model (GAN) into the design of vocoder has brought about a major paradigm shift.

[0055] Mel-spectrogram Generative Adversarial Network (Melgan) proposes a non-autoregressive GAN vocoder that adopts a convolutional architecture with multi-scale discriminators, allowing for fast and efficient waveform generation. High-Fidelity Generative Adversarial Networks (HIFI-GAN) further enhances this approach by introducing a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD), along with a powerful feature matching loss, making it the state-of-the-art (SOTA) and widely used as an acoustic backend module in TTS systems.

[0056] Based on this, Big-scale Variational Generative Adversarial Network (BigVGAN) demonstrates the advantages of GAN-based models over flow-based methods. This research significantly improves the fidelity and robustness of audio by introducing periodic activation functions and anti-aliasing representations into the GAN generator. Additionally, to further improve synthesis efficiency, both the Inverse Short-time Fourier Transform Network (iSTFTNet) and the Fourier-based Neural Vocoder Vocos replace traditional dilated convolutions with Inverse Short-time Fourier Transform (iSTFT) in the GAN architecture for audio waveform reconstruction. These works show that GAN-based training remains one of the most effective solutions for neural vocoders, especially for handling the inherent non-stationarity and high variability characteristics of speech signals.

[0057] State Space Model (SSM).

[0058] SSM is a statistical model used to describe time series data, widely used in machine learning and statistics for handling dynamic systems and time-varying processes. In recent years, SSM has become a powerful sequence modeling paradigm in natural language processing and time series prediction. This type of model captures temporal dependencies through linear state transitions and output mappings, with linear time complexity, strong parallelization capabilities, and robustness to gradient decay for long sequences.

[0059] The SSM can capture the changes of system states over time and the relationship between the observed data and these states. Among them, the new selective state space model architecture Mamba realizes the breakthrough performance of long context tasks through the following innovations: linear time recursion, hardware-aware parallelization, and selective information flow, which reduces the computational overhead to capture long-range dependencies, adopts kernel fusion and memory optimization scanning technology, the inference throughput is 5 times that of similar Transformer models, and through input-related state space parameterization dynamic filtering of irrelevant tokens, the context retention ability is significantly improved.

[0060] Next, the overall idea of the speech synthesis method provided by the embodiments of the present application is described.

[0061] In the field of modern artificial intelligence and speech processing, the importance of text-to-speech (TTS) systems is increasingly prominent, which is widely used in intelligent assistants, barrier-free reading, intelligent customer service and many other scenarios, and has become one of the key technologies of human-computer interaction. The TTS system aims to convert written text into natural and fluent speech output, providing users with convenient and efficient information acquisition and interaction methods. In the TTS system, the vocoder plays a very core role, which undertakes the task of accurately converting intermediate acoustic features into waveform signals, and its performance directly affects the quality and naturalness of the output speech of the TTS system. Ideally, the vocoder should be able to generate highly natural and high-fidelity audio effect speech, and at the same time, in the face of various speakers and complex and diverse data distribution domains, it should exhibit strong and stable synthesis ability to meet the stringent needs of different application scenarios. From the development context of the entire field, the vocoder technology has undergone a great leap from traditional signal processing methods to cutting-edge neural models, and this evolution process reflects the rapid pace of technological innovation and the vigorous vitality of the field.

[0062] Early vocoders are built on the cornerstone of conventional signal processing methods. Classic methods such as Griffin-lim, STRAIGHT and WOLD, try to reconstruct the speech waveform by using carefully designed algorithms from various key acoustic features of the speech signal, such as amplitude, fundamental frequency and spectral envelope. These methods are based on specific speech synthesis assumptions, have good interpretability, and have certain advantages in terms of computational efficiency. However, due to its dependence on fixed assumptions, when faced with complex and variable actual speech scenarios, the generation effect has obvious shortcomings in expressiveness and naturalness, and it is difficult to generate output results comparable to natural human speech. In view of this, with the rise of advanced technologies such as deep learning, the design and development of neural vocoders have gradually become the core focus of the speech synthesis research field, and many researchers and developers hope to use the powerful learning and fitting capabilities of neural networks to break through the performance bottleneck of traditional vocoders.

[0063] At present, although the vocoder technology has made some progress, there are still some key problems that need to be solved in existing vocoders. On the one hand, in terms of high-frequency reconstruction capability of audio, existing vocoders more or less have deficiencies, resulting in loss of high-frequency details in synthesized speech, and the sound is not clear and bright enough, affecting the overall sound quality and naturalness. On the other hand, mainstream vocoders rely too much on convolutional network structure, and the inherent characteristics of convolutional networks make them pay too much attention to local context information when processing speech signals, and seriously ignore long-range dependencies, which makes it difficult to fully utilize global information for accurate synthesis when synthesizing long speech segments or processing complex semantic and prosodic speech, and the coherence and fluency of the speech are greatly compromised. These problems seriously restrict the further promotion and expansion of vocoders in practical applications, and innovative technologies and solutions are needed to solve them.

[0064] In view of this, the embodiments of the present application provide a speech synthesis method, system, device, computer readable storage medium and computer program product, aiming to overcome the shortcomings of the above-mentioned related technologies, based on a state space model to build a neural vocoder for speech synthesis, to improve the high-frequency reconstruction capability and avoid the loss of high-frequency details, so as to obtain better synthesized sound quality.

[0065] Compared with the way of synthesizing speech based on the conventional signal processing methods such as Griffin-lim, STRAIGHT and WOLD of the traditional vocoder, the embodiments of the present application input the mel-spectrum data corresponding to the speech text data to be synthesized into the neural vocoder constructed based on the selective state space model and the generative adversarial network model in advance, and perform long sequence processing on the mel-spectrum data based on the neural vocoder to obtain synthesized audio data corresponding to the speech text data to be synthesized.

[0066] Therefore, the embodiment of the present application can use the selective state space model to process the mel spectrum data in a long sequence by using the long sequence processing capability suitable for long distance dependence modeling tasks, thereby avoiding the loss of high frequency details caused by the traditional vocoder ignoring long range dependence, and optimizing the synthesized sound quality. That is, the embodiment of the present application can construct a neural vocoder based on a state space model to perform speech synthesis, improve the high frequency reconstruction capability to avoid the loss of high frequency details, and thus obtain better synthesized sound quality.

[0067] Next, the speech synthesis method, system, device, computer readable storage medium and computer program product provided by the embodiment of the present application are specifically described by the following embodiments, and first, the speech synthesis method provided by the embodiment of the present application is described in detail.

[0068] It should be noted that in each specific embodiment of the present application, when relevant processing is required according to user information, user behavior data, user historical data and user location information, etc. related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, etc. After obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiment of the present application to function normally is obtained.

[0069] It should be noted that the speech synthesis method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a computer device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like, and can also be a vehicle terminal (for example, a vehicle computing platform) on a vehicle, or a terminal associated with the vehicle, which means that the terminal can communicate with the vehicle based on a network to interact with data. The server end can be a background server terminal device, which can be configured as a physical server, or can be configured as a server cluster or a distributed system formed by a plurality of physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms. The software can be an application, a computer program, and a storage medium carrying the computer program, and the like, which implements the speech synthesis method. It should be understood that, based on different design needs of actual applications, in different feasible embodiments, the terminal, the server end, and the software, and the like, to which the speech synthesis method provided in the embodiments of the present application is applied, can of course also be other forms not listed here, and the speech synthesis method provided in the embodiments of the present application does not specifically limit this.

[0070] In addition, the present application can also be used in a plurality of general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, personal computers (PCs), minicomputers, mainframe computers, vehicles, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0071] For the sake of understanding and description, the speech synthesis method provided in the embodiments of the present application is taken as an example in the following, and each specific embodiment of the present application is described in detail. The implementation of the speech synthesis method provided in the embodiments of the present application by any other form of subject can be referred to the process of the speech synthesis method applied by the terminal device.

[0072] Reference is made to Figure 1 , Figure 1 The speech synthesis method provided by the embodiments of the present application has a step flow diagram in some embodiments. It should be understood that, although Figure 1 and subsequent other step flow diagrams show the execution order of some method steps, the speech synthesis method provided by the embodiments of the present application can of course use an execution order of method steps different from that shown in the figure based on different design needs of actual applications. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the speech synthesis method provided by the embodiments of the present application, and any reasonable change of the order of the method steps shown should be included in the protection scope of the speech synthesis method provided by the embodiments of the present application. Figure 1

[0073] As Figure 1 shown, in some embodiments, the speech synthesis method provided by the embodiments of the present application can include the following steps S101 to S103.

[0074] Step S101: Obtain the mel spectrum data corresponding to the speech text data to be synthesized.

[0075] It should be noted that the speech text data to be synthesized can be text data uploaded by a user and indicated to need speech synthesis through a pre-set human-computer interaction interface by a terminal device.

[0076] When the terminal device performs speech synthesis on the speech text data to be synthesized uploaded by the user in response to a speech synthesis instruction, it first obtains the mel spectrum data corresponding to the speech text data to be synthesized. Among them, the speech synthesis instruction can be triggered by the user when uploading the speech text data to be synthesized.

[0077] In some embodiments, after obtaining the speech text data to be synthesized, the terminal device can use a phoneme conversion tool such as G2P, T5, etc. to first preprocess the text data, and then extract the key parameters of the mel spectrum such as Fourier transform point number n_fft, frame shift hop_length, mel band number n_mels, etc. from the preprocessed text data. Then, the terminal device converts the extracted feature parameters into mel spectrum.

[0078] ​Exemplarily, after obtaining the to-be-synthesized speech text data (such as "Hello, this is an example"), the terminal device first performs processing of the initialization tool and parameters, that is, loading a text coding tool, a pre-trained T5 model (including a word segmenter and a generation model) for subsequent conversion of the text into a phoneme representation, and configuring mel spectrum parameters. The mel spectrum parameters include core parameters of speech processing, such as a sampling rate (22050 Hz), a Fourier transform point number (2048), a frame shift (256), a mel filter number (80), and minimum / maximum frequencies (0 Hz / 8000 Hz), which determine the resolution and frequency range of the mel spectrum.

[0079] Subsequently, the terminal device converts the to-be-synthesized speech text data into a model recognizable digital code (token) through the T5 word segmenter. The token is processed by using the T5 generation model, and a corresponding phoneme sequence (the smallest pronunciation unit of speech, such as an initial and a medial combination in Chinese) is output. The purpose of this step is to convert the "visual symbol" of the text into a "pronunciation symbol" to provide a pronunciation basis for subsequent speech generation.

[0080] Finally, the terminal device converts the phoneme sequence into a mel spectrum (Phonemes to Mel-spectrogram). In this process, the terminal device can first supplement acoustic characteristics, such as duration allocation: assuming a pronunciation duration of each phoneme (such as 0.1 seconds / phoneme), calculating the number of mel spectrum frames corresponding to the total duration (total duration x sampling rate ÷ frame shift), and fundamental frequency (pitch) setting: assigning an average fundamental frequency (such as 100 Hz) to the phoneme sequence to simulate the pitch characteristics of human pronunciation. After that, the terminal device generates a mel spectrum: according to the total frame number and the mel filter number, a mel spectrum matrix (shape [n_mels, total frame number]) that meets the dimension requirement is generated.

[0081] In some embodiments, the terminal device can also apply a professional TTS model (such as Tacotron, GlowTTS) to generate more realistic mel spectrum data corresponding to the to-be-synthesized speech text data by learning the mapping relationship between phonemes and acoustic characteristics.

[0082] Step S102: inputting the mel spectrum data into the neural vocoder based on the selective state space model.

[0083] It should be noted that the terminal device can construct the neural vocoder based on the selective state space model before formally responding to the speech synthesis instruction to perform speech synthesis, and then store the neural vocoder in a local or online database. In this way, when the terminal device performs speech synthesis, the terminal device can directly call the neural vocoder from the local or online database.

[0084] After obtaining the mel-spectrum data, the terminal device can input the mel-spectrum data into the neural vocoder constructed based on the selective state space model in advance, so as to facilitate subsequent long sequence processing of the selective state space model based on the neural vocoder, bidirectional long-range time sequence dependence capture of the mel-spectrum data, and generation of synthesized audio data corresponding to the to-be-synthesized speech text data.

[0085] In some embodiments, the neural vocoder can directly utilize the long sequence processing capability of the selective state space model to capture the bidirectional long-range time sequence dependence of the mel-spectrum data. For example, assuming that the neural vocoder is a speech signal processing method combined with state space model modeling capability, the neural vocoder can utilize the long sequence processing capability of the selective state space model to capture the bidirectional long-range time sequence dependence of the mel-spectrum data corresponding to the to-be-synthesized speech text data when processing the speech signal corresponding to the to-be-synthesized speech text data.

[0086] In other embodiments, the neural vocoder includes a generator. In this case, the neural vocoder can utilize the long sequence processing capability of the selective state space model to capture the bidirectional long-range time sequence dependence of the mel-spectrum data through the generator.

[0087] In some embodiments, the neural vocoder can be a neural vocoder MambaVoc that combines state space model modeling capability with the advantages of adversarial training generation. In this way, the neural vocoder MambaVoc can simultaneously model the past and future context of the mel-spectrum through a selective state space recursive mechanism, achieve bidirectional long-range time sequence dependence capture of the mel-spectrum, and adopt an innovative hybrid architecture, i.e., the architecture of the MambaVoc realized by reconstructing the original Mamba and HiFi-GAN structure.

[0088] Step S103: based on the neural vocoder adopting the selective state space model to process the mel-spectrum data in a long sequence, obtaining synthesized audio data corresponding to the to-be-synthesized speech text data.

[0089] The terminal device generates the synthesized audio data corresponding to the to-be-synthesized speech text data by using the neural vocoder to perform long sequence processing on the mel spectrum data based on the long sequence processing capability of the selective state space model suitable for a long distance dependency modeling task, that is, the neural vocoder simultaneously models the past and future context of the mel spectrum by using the selective state space recursive mechanism to capture bidirectional long-range temporal dependencies, thereby generating the synthesized audio data. In this way, the terminal device can obtain the synthesized audio data output by the neural vocoder and use the synthesized audio data as the synthesized audio data corresponding to the to-be-synthesized speech text data.

[0090] In the embodiments of the present application, a neural vocoder based on a selective state space model is constructed by a terminal device, and then when speech synthesis is needed, mel spectrum data corresponding to to-be-synthesized speech text data is obtained by the terminal device, and the mel spectrum data is input into the pre-constructed neural vocoder, so that the neural vocoder uses the long sequence processing capability of the selective state space model suitable for a long distance dependency modeling task to perform long sequence processing (such as bidirectional long-range temporal dependency capture) on the mel spectrum data, thereby generating synthesized audio data corresponding to the to-be-synthesized speech text data.

[0091] Compared with the way of performing speech synthesis by a conventional vocoder based on a conventional signal processing method, in the embodiments of the present application, the mel spectrum data corresponding to the to-be-synthesized speech text data is input into the neural vocoder constructed based on the selective state space model in advance, and the neural vocoder performs long sequence processing on the mel spectrum data, thereby obtaining the synthesized audio data corresponding to the to-be-synthesized speech text data. In this way, the embodiments of the present application can use the long sequence processing capability of the selective state space model suitable for a long distance dependency modeling task to perform long sequence processing on the mel spectrum data, thereby avoiding the loss of high-frequency details caused by the conventional vocoder ignoring long-range dependencies, and optimizing the synthesized sound quality. That is, the embodiments of the present application can construct a neural vocoder based on a state space model to perform speech synthesis, improve the high-frequency reconstruction capability to avoid the loss of high-frequency details, and thus obtain better synthesized sound quality.

[0092] In some embodiments, the terminal device can construct a neural vocoder MambaVoc that generates high-quality waveforms from input mel spectrums based on a selective state space model and a generative adversarial network model.

[0093] Please refer to Figure 2 , Figure 2 The speech synthesis method provided in the embodiments of the present application involves a step flowchart of constructing a neural vocoder MambaVoc in some embodiments.

[0094] AsFigure 2 As shown, in some embodiments, the speech synthesis method provided by the embodiments of the present application can also construct the neural vocoder MambaVoc for generating high-quality waveforms from input mel-spectra through steps S201 and S202 as shown.

[0095] Step S201: constructing a generator based on a selective state space model.

[0096] When constructing the neural vocoder MambaVoc for generating high-quality waveforms from input mel-spectra, the terminal device can first construct a generator based on a selective state space model.

[0097] In some embodiments, the terminal device can employ a novel selective state space model architecture Mamba to construct the generator based on the selective state space model. In this way, the terminal device can employ a bidirectional Mamba generator architecture to model the past and future context of the mel-spectrum simultaneously through a selective state space recurrent mechanism, thereby achieving bidirectional long-range temporal dependency capture.

[0098] Step S202: performing generative adversarial network training on the generator based on a preset discriminator to obtain a neural vocoder based on a generative adversarial network model architecture.

[0099] It should be noted that the preset discriminator includes a multi-resolution discriminator (MRD) and a multi-period discriminator (MPD).

[0100] After constructing the generator based on the selective state space model, the terminal device can also perform generative adversarial network training on the generator based on the preset discriminators, i.e., the multi-resolution discriminator MRD and the multi-period discriminator MPD, thereby constructing the neural vocoder MambaVoc based on the generative adversarial network model architecture. In this way, the terminal device has constructed the neural vocoder MambaVoc that combines the state space model modeling capability with the advantages of adversarial training generation.

[0101] In this embodiment, considering that the speech signal is similar to the language, and the essence of both is a long-range time sequence signal, so it can fully benefit from the high expressiveness and high efficiency of the Mamba model architecture, therefore, a neural vocoder MambaVoc based on the selective state space model and the generative adversarial network model is constructed by the terminal device to combine the state space model modeling capability and the generation advantage of adversarial training, so that since the core of Mamba is to use the SSM module, it can theoretically realize the linear computational complexity of O(n) (the algorithm efficiency will not "explode" with the increase of the input size, but will grow slowly in proportion to the input size), so it can significantly improve the long sequence processing capability of the neural vocoder MambaVoc, which is suitable for tasks that need to model long-distance dependencies. And because of the linear structure of Mamba, the neural vocoder MambaVoc can also ensure higher sound quality while reasoning quickly during the reasoning phase, thereby improving the processing efficiency of the mel-spectrum data and further improving the overall efficiency of the synthesized speech.

[0102] In some embodiments, since the neural vocoder MambaVoc constructed by the terminal device is based on the generative adversarial network model architecture, the terminal device can also update the neural vocoder MambaVoc based on the discriminator.

[0103] Please refer to Figure 3 , Figure 3 The speech synthesis method provided by the embodiments of the present application has a step flowchart in some other embodiments.

[0104] As Figure 3 shown, in some embodiments, the speech synthesis method provided by the embodiments of the present application, after the step S103 described above: performing long sequence processing on the mel-spectrum data based on the neural vocoder using the selective state space model to obtain synthesized audio data corresponding to the to-be-synthesized speech text data, can further include steps S301 to S303 as shown below.

[0105] Step S301: Obtain the sound quality verification result of the synthesized audio data.

[0106] After the terminal device performs long sequence processing on the mel-spectrum data based on the above-mentioned neural vocoder MambaVoc to obtain synthesized audio data corresponding to the to-be-synthesized speech text data, it can further perform sound quality verification on the "proximity" between the synthesized audio data and the real audio, thereby obtaining the sound quality verification result of the synthesized audio data.

[0107] In some embodiments, the terminal device can evaluate the proximity between the synthesized audio data and the real audio from multiple dimensions such as intelligibility, naturalness, coherence, etc. to obtain the sound quality verification result of the synthesized audio data.

[0108] Exemplarily, the terminal device can select 10 pieces of sample data from the synthesized audio data, each piece of sample data having a duration of 5-10 seconds and covering different speech speeds (normal / fast), different emotions (neutral / happy), and different text types (short sentences / long sentences), and select 10 pieces of matching content (e.g., real-person recordings of the same text) from a real-audio library in the same field as the "real benchmark", and then randomly number a total of 20 pieces of audio (10 synthesized+10 real) to obtain a mixed sample set.

[0109] In addition, the terminal device can also design tools and indicators, that is, determine the audio analysis library (librosa), the speech quality evaluation tool (PESQ, STOI, MOSNet), or the spectrum analysis software (Audacity) as the objective indicator tool, and set up a subjective evaluation table, for example, design the scoring items (1-5 points), including "naturalness" (whether it sounds like a real person speaking), "clarity" (whether the pronunciation is clear and distinguishable), "coherence" (whether the sentence pause is natural), and "no noise" (whether there is current sound / distortion).

[0110] Then, the terminal device can perform quantitative analysis on the mixed sample set by using the designed tools and indicators to verify the objective indicators (for example, compare the basic acoustic characteristics, verify the spectral characteristics, and calculate the standardized sound quality indicators), and calculate the average score of each objective indicator of the 10 pieces of synthesized audio data (for example, the average naturalness score is 3.8, and the average clarity score is 4.2), and compare the calculated score with the score of the real audio (for example, the average naturalness score of the real audio is 4.5, and the difference between the synthesized audio data is ≤1 point, which is considered acceptable) to determine the abnormal cases in the synthesized audio data.

[0111] Finally, the terminal device analyzes the abnormal cases in the synthesized audio data that do not meet the objective indicators, such as checking whether there is high-frequency distortion (such as noise generated when the vocoder is generated) through the spectrum graph, or if the naturalness score is low, observing whether the fundamental frequency curve is abrupt (such as the lack of drop in the emotion sentence tail), or the pause position does not match the real voice (such as not pausing at the comma in a long sentence), to obtain the sound quality verification result of the synthesized audio data and output it. For example, if the comprehensive objective indicators judge that the sound quality of the synthesized audio data does not meet the preset standard (such as "qualified"), the output sound quality verification result is: "high-frequency energy is too high, causing noise, suggest adjusting the mel-spectrum to waveform conversion parameters of the vocoder", or "the fundamental frequency decreases insufficiently at the end of the emotion sentence, the prosody prediction module of the TTS model needs to be optimized", and the like.

[0112] Step S302: updating the parameters of the preset discriminator based on the sound quality verification result to obtain an updated discriminator.

[0113] After obtaining the sound quality verification result of the synthesized audio data, the terminal device can update the discriminator based on the neural vocoder MambaVoc when the sound quality verification result indicates that the sound quality of the synthesized audio data does not reach the preset standard. At this time, the terminal device first updates the parameters of the discriminator based on the sound quality verification result to obtain an updated discriminator.

[0114] For example, when the sound quality verification result indicates that the sound quality of the synthesized audio data does not reach the preset standard, the terminal device can regard the synthesized audio data as a fake sample (denoted as x_fake=G(z), where z is noise) and regard the real audio corresponding to the synthesized audio data as a real sample (denoted as x_real). Then, the discriminator is used to judge the real sample and output a predicted probability D(x_real) (an ideal result is 1, representing "real"). The discriminator is also used to judge the fake sample and output a predicted probability D(x_fake) (an ideal result is 0, representing "fake"). Then, the binary cross-entropy (BCE) is used as a loss function to calculate the loss of the discriminator loss_D:

[0115] loss_D=-E[log(D(x_real))]-E[log(1-D(x_fake))].

[0116] Finally, the terminal device can use the gradient descent method (such as the Adam optimizer) to adjust the parameters of the discriminator according to the gradient of loss_D, so as to reduce loss_D, that is, to enhance the ability of the discriminator to distinguish real and fake samples.

[0117] In some embodiments, if the sound quality verification result of the synthesized audio data obtained by the terminal device indicates that the sound quality of the synthesized audio data reaches the preset standard, the neural vocoder MambaVoc does not need to be updated.

[0118] Step S303: Perform generative adversarial network training on the generator based on the updated discriminator to obtain an updated generator.

[0119] After updating the parameters of the discriminator, the terminal device can perform generative adversarial network training on the generator based on the updated discriminator to obtain an updated generator, and then construct an updated neural vocoder based on the generative adversarial network model architecture. In this case, the terminal device can use the updated neural vocoder as the neural vocoder MambaVoc based on the generative adversarial network model architecture, so as to perform long sequence processing on the mel spectrum data corresponding to the current synthesized speech text data to obtain synthesized audio data, or perform long sequence processing on the mel spectrum data corresponding to new synthesized speech text data in the next round of speech synthesis to obtain synthesized audio data.

[0120] Exemplarily, when the terminal device trains the generator based on the updated discriminator, the terminal device can input random noise into the generator, generate a new fake sample (x_fake' = G(z), at this time the generator parameters have not been updated) by the generator, and provide new feedback by the updated discriminator, that is, the updated discriminator judges the new fake sample x_fake' and outputs the predicted probability D(x_fake'), calculates the generator loss, and the loss function can be:

[0121] loss_G = -E[log(D(x_fake'))].

[0122] Then, the terminal device can use the gradient descent method to adjust the parameters of the generator according to the gradient of loss_G (make loss_G decrease, that is, make the generator generate samples closer to the real data distribution to "trick" the updated discriminator).

[0123] In this way, the terminal device repeatedly generates new fake samples by the generator, calculates the loss by the updated discriminator, and adjusts the parameters of the generator based on the calculated loss. In each round of loop, first, the discriminator is updated according to the judgment result (feedback) of the discriminator on the true and false samples, so that it becomes more "sensitive", and then the new output of the generator is judged by the updated discriminator, and the generator is updated based on the new feedback, so that it becomes "good at cheating". Finally, when the discriminator cannot distinguish between true and false samples (D(x_real) ≈ D(x_fake) ≈ 0.5), the generated samples by the generator are almost consistent with the real data distribution, and the training is completed.

[0124] In this embodiment, the terminal device constructs a generator based on a selective state space model, and trains the generator based on a preset discriminator to obtain a neural vocoder MambaVoc based on a generative adversarial network model architecture. Moreover, the terminal device updates the parameters of the discriminator based on the quality verification result of the synthesized audio data generated by the neural vocoder MambaVoc, and trains the generator based on the updated discriminator to obtain an updated generator. In this way, the discriminator constantly learns the feature distribution of real speech (such as prosody, timbre details), so as to force the generator to generate more human-like speech, thereby further improving the naturalness and authenticity of the synthesized speech. Moreover, the discriminator can also force the generator to learn more comprehensive acoustic features, thereby enhancing the modeling capability of the neural vocoder MambaVoc for complex acoustic features.

[0125] In some embodiments, the terminal device can capture long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder MambaVoc, and perform long-sequence processing on the mel-spectrogram data, so as to generate synthesized audio data corresponding to the to-be-synthesized speech text data.

[0126] Please refer to Figure 4 , Figure 4 For Figure 1 the detailed step flowchart of step S103.

[0127] As Figure 4 shown, in some embodiments, the step S103 of capturing long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder using the selective state space model, and performing long-sequence processing on the mel-spectrogram data to obtain synthesized audio data corresponding to the to-be-synthesized speech text data can include steps S401 and S402 as shown below.

[0128] Step S401: capturing long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder using the selective state space model to obtain context-aware feature representation.

[0129] When the terminal device captures long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder and performs long-sequence processing on the mel-spectrogram data, the terminal device can capture long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder using the selective state space model, so as to obtain context-aware feature representation.

[0130] In some embodiments, the neural vocoder includes a generator, and the terminal device can capture long-range temporal dependencies of the mel-spectrogram data based on the generator of the neural vocoder using the selective state space model.

[0131] The generator of the neural vocoder can include a neural network module (BiMamba Block) for bidirectional sequence modeling task. Based on this, the terminal device can encode temporal dependencies of the past and the future using the new BiMamba block, so as to capture long-range temporal dependencies of the mel-spectrogram data.

[0132] In some embodiments, the step S401 of capturing long-range temporal dependencies of the mel-spectrogram data based on the neural vocoder using the selective state space model can include the following steps:

[0133] capturing bidirectional sequence modeling of the mel-spectrogram data based on the neural network module using the selective state space model to obtain bidirectional sequence processing result; the bidirectional sequence processing result is bidirectional long-range temporal dependencies of the mel-spectrogram data.

[0134] When the terminal device uses a selective state space model based on the neural network module to capture the long-range temporal dependencies of the mel spectrum data, the terminal device can use a structured state space model SSM (i.e., Mamba) as the core component of the neural network module BiMambaBlock, thereby using the selective state space recursive mechanism of the selective state space model based on the neural network module BiMambaBlock to perform bidirectional sequence modeling (forward sequence modeling and reverse sequence modeling) on ​​the mel spectrum data to capture the bidirectional long-range temporal dependencies of the mel spectrum data and obtain a bidirectional sequence processing result. The bidirectional sequence processing result can be the bidirectional long-range temporal dependencies of the mel spectrum data.

[0135] It should be noted that in the neural network module BiMambaBlock, the Mamba layer can learn the parameter matrix based on the following formula The one-dimensional input signal Convert to output And with the help of hidden state Implement state evolution.

[0136]

[0137] y(t)=Ch(t).

[0138] In some embodiments, the Mamba layer can use the zero-order hold method to discretize the above continuous-time system to achieve efficient discrete calculations. For example, assuming the time step is △, the discretized system expression can be:

[0139]

[0140] From this we can get the discrete time recursion equation as follows:

[0141]

[0142] In some embodiments, the Mamba layer can achieve efficient sequence modeling by pre-calculating the global convolution kernel k. The Mamba layer can obtain the following by pre-calculating the global convolution kernel k:

[0143]

[0144] In some embodiments, to enhance model adaptability, the Mamba layer can introduce a selective mechanism to enable parameters B, C and Δ to dynamically respond to input changes, thereby filtering irrelevant information and adjusting the receptive field, i.e., by modifying the model structure or parameters, changing the "perception range" of neurons in a certain layer of the network to the input data.

[0145] Step S402: generating synthesized audio data corresponding to the to-be-synthesized speech text data based on the feature representation.

[0146] After the terminal device captures the long-range temporal dependency of the mel-spectrum data based on the generator of the neural vocoder MambaVoc, thereby obtaining a feature representation with context awareness, the terminal device can further perform coding and decoding processing based on the feature representation, thereby generating synthesized audio data corresponding to the to-be-synthesized speech text data.

[0147] In this embodiment, the terminal device captures the long-range temporal dependency of the mel-spectrum data based on the generator of the neural vocoder MambaVoc using the selective state space model, thereby obtaining a feature representation with context awareness, and then performs coding and decoding processing based on the feature representation, thereby generating synthesized audio data corresponding to the to-be-synthesized speech text data. In this way, this embodiment can use the long sequence processing capability of the selective state space model to perform long sequence processing on the mel-spectrum data through a long distance dependency modeling task, thereby avoiding the loss of high frequency details caused by ignoring long-range dependencies, and thereby optimizing the overall synthesized sound quality.

[0148] Please refer to Figure 5 , Figure 5 The speech synthesis method provided in this embodiment provides a schematic diagram of the overall model architecture of the neural vocoder in some embodiments. As shown in Figure 5 In some embodiments, the terminal device can construct the neural network module BiMambaBlock in the generator by using a bidirectional stacked Mamba layer, thereby performing bidirectional sequence modeling on the mel-spectrum data through the neural network module BiMambaBlock, and thereby capturing the bidirectional long-range temporal dependency of the mel-spectrum data. Among them, the bidirectional stacked Mamba layer includes a forward neural network layer and a reverse neural network layer based on the selective state space model.

[0149] Please refer to Figure 6 , Figure 6 The speech synthesis method provided in this embodiment involves a schematic diagram of the step flow of bidirectional sequence modeling on the mel-spectrum data in some embodiments.

[0150] As Figure 6As shown, in some embodiments, the step of "modeling the mel-spectrogram data in a bidirectional sequence manner by the selective state space model based on the neural network module, to obtain a bidirectional sequence processing result" can include the following steps S601 and S602.

[0151] Step S601: modeling past context of the mel-spectrogram data by forward recursive processing based on the forward neural network layer, to obtain a forward sequence processing result.

[0152] It should be noted that the forward sequence processing result is a long-range temporal dependency relationship of the mel-spectrogram data in the forward direction, i.e., the forward sequence processing result obtained by the forward neural network layer modeling past context of the mel-spectrogram data by forward recursive processing, thereby capturing the long-range temporal dependency relationship of the mel-spectrogram data in the forward direction. And the bidirectional sequence processing result includes the forward sequence processing result.

[0153] When the terminal device models the mel-spectrogram data in a bidirectional sequence manner by the neural network module BiMambaBlock in the generator, it can model past context of the mel-spectrogram data by forward recursive processing based on the forward neural network layer, thereby capturing the long-range temporal dependency relationship of the mel-spectrogram data in the forward direction, to obtain a forward sequence processing result (a long-range temporal dependency relationship of the mel-spectrogram data in the forward direction).

[0154] Step S602: modeling future context of the mel-spectrogram data by reverse recursive processing based on the reverse neural network layer, to obtain a reverse sequence processing result.

[0155] It should be noted that the reverse sequence processing result is a long-range temporal dependency relationship of the mel-spectrogram data in the reverse direction, i.e., the reverse sequence processing result obtained by the reverse neural network layer modeling future context of the mel-spectrogram data by reverse recursive processing, thereby capturing the long-range temporal dependency relationship of the mel-spectrogram data in the reverse direction. And the bidirectional sequence processing result includes the reverse sequence processing result.

[0156] When the terminal device models the mel-spectrogram data in a bidirectional sequence manner by the neural network module BiMambaBlock in the generator, while modeling past context of the mel-spectrogram data by forward recursive processing based on the forward neural network layer, it can also model future context of the mel-spectrogram data by reverse recursive processing based on the reverse neural network layer through a flip operation, thereby capturing the long-range temporal dependency relationship of the mel-spectrogram data in the reverse direction, to obtain a reverse sequence processing result (a long-range temporal dependency relationship of the mel-spectrogram data in the reverse direction).

[0157] In some embodiments, after the terminal device performs bidirectional sequence modeling on the mel-spectrogram data through the forward neural network layer and the reverse neural network layer of the neural network module BiMambaBlock, the outputs of the forward neural network layer and the reverse neural network layer are spliced and projected through a linear layer to form a feature representation with context awareness.

[0158] Based on this, the speech synthesis method provided by the embodiments of the present application can further include the following steps:

[0159] state fusion is performed on the forward sequence processing result and the reverse sequence processing result to obtain a fused long sequence processing result;

[0160] linear projection is performed on the fused long sequence processing result to obtain the feature representation with context awareness.

[0161] The terminal device performs forward long-range temporal dependency capture on the mel-spectrogram data through the forward neural network layer of the neural network module BiMambaBlock in the generator, and takes the processing result (forward long-range temporal dependency) output by the forward neural network layer as a forward sequence processing result. In addition, the terminal device performs reverse long-range temporal dependency capture on the mel-spectrogram data through the reverse neural network layer of the neural network module BiMambaBlock, and takes the processing result (reverse long-range temporal dependency) output by the reverse neural network layer as a reverse sequence processing result. Then, as shown in the following formula, the generator performs state fusion on the forward sequence processing result and the reverse sequence processing result by splicing (cat) the forward sequence processing result and the reverse sequence processing result to obtain a fused long sequence processing result. After that, the generator performs linear projection (linear) on the fused long sequence processing result to obtain a final feature representation with context awareness. Figure 5

[0162] In the embodiments, the neural network module BiMambaBlock of the generator is constructed by bidirectional stacking Mamba layers, the forward neural network layer thereof normally processes a sequence, the reverse neural network layer thereof reversely processes a sequence through a flip operation, and the outputs of the two are spliced and projected through a linear layer to form a final feature representation with context awareness. In this way, the embodiments replace the causal convolution (causal convolution) of Mamba with a standard non-causal one-dimensional convolution (Conv1d) to enable the generator to access the complete sequence of the mel-spectrogram data in the inference stage, so that the generator can fully utilize bidirectional temporal information to capture long-range temporal dependency of the mel-spectrogram data without violating the causality constraint, thereby avoiding loss of high-frequency details and effectively optimizing the synthesized sound quality.​

[0163] In some embodiments, the generator comprises a decoder module. The decoder module can be a decoder module DecoderBlock based on a generative adversarial network.

[0164] As shown in Figure 5 , the terminal device can generate the synthesized audio data corresponding to the to-be-synthesized speech text data based on the feature representation output by the neural network module BiMambaBlock (bidirectional Mamba module) in the decoder module DecoderBlock in the generator.

[0165] Based on this, the above-mentioned step S402: generating synthesized audio data corresponding to the to-be-synthesized speech text data based on the feature representation can comprise the following steps:

[0166] Converting the feature representation into an audio waveform based on the decoder module;

[0167] Mapping linear prediction values of the audio waveform to a preset audio amplitude range based on the decoder module to obtain the synthesized audio data corresponding to the to-be-synthesized speech text data.

[0168] After the terminal device forms the final context-aware feature flag based on the neural network module BiMambaBlock in the generator, it further reconstructs the waveform audio from the feature representation based on the decoder module DecoderBlock in the generator to convert the feature representation into an audio waveform. And, based on the decoder module DecoderBlock, the linear prediction values of the audio waveform are mapped to a preset audio amplitude range (such as [-1, 1]), thereby obtaining the synthesized audio data corresponding to the to-be-synthesized speech text data.

[0169] As shown in Figure 5As shown, the decoder module DecoderBlock in the generator is stacked by a series of upsample blocks UpsampleBlock and residual blocks ResidualBlock, and a periodic activation function Snake1dis introduced in the whole architecture. Among them, the upsample block UpsampleBlock can double the time resolution and gradually reconstruct the fine-grained signal structure. The upsample block UpsampleBlock can include: a ConvTranspose1d layer for time upsampling, a nonlinear activation function Snake1dfor smoothing transition, and a ResidualBlock stacked architecture for local refinement. In addition, the architecture of the residual block ResidualBlock can be as follows: input→Snake1d→DilatedConv1d→Snake1d→Conv1d→Add, the DilatedConv1dcan enable the generator to capture long-range dependencies, and the Snake1dactivation supports high-resolution adaptive frequency modeling. Further, the output of the decoder module DecoderBlock reconstructed from the feature representation is post-processed by the layer: Snake1d→Conv1d→tanh to further optimize the feature distribution and improve harmonic consistency, and finally obtain the synthesized audio data corresponding to the text data to be synthesized. Among them, the tanh activation constrains the output to the range [-1, 1] to comply with the standard waveform amplitude specification and avoid numerical overflow.

[0170] In this embodiment, considering the extrapolation ability of Snake1din the field of time series prediction, the introduction of Snake1dactivation in the decoder module DecoderBlock of the generator can enhance the ability of the generator to generate more natural and periodic waveform patterns, thereby facilitating the generator to capture the inherent harmonic structure of the speech signal when capturing long-range temporal dependencies of the mel-spectrogram data. Moreover, the structure design of the decoder module DecoderBlock can gradually improve the time resolution, and the residual connection can retain low-frequency information to avoid the problem of gradient disappearance, thereby improving the high-frequency reconstruction capability of speech synthesis to obtain better synthesized audio quality.

[0171] Please refer to Figure 7 The embodiment of the present application also provides a speech synthesis system, which can implement the speech synthesis method described above.

[0172] As Figure 7 shown, the speech synthesis system provided by the embodiment of the present application includes a data acquisition module 701 and a vocoding module 702.

[0173] Among them,

[0174] The data acquisition module 701 is configured to acquire mel-frequency spectrum data corresponding to to-be-synthesized speech text data.

[0175] The vocoding module 702 is configured to input the mel-frequency spectrum data into a neural vocoder based on a selective state-space model, and perform long sequence processing on the mel-frequency spectrum data based on the selective state-space model adopted by the neural vocoder to obtain synthesized audio data corresponding to the to-be-synthesized speech text data.

[0176] In some embodiments, the vocoding module 702 is further configured to capture long-range temporal dependencies of the mel-frequency spectrum data based on the selective state-space model adopted by the neural vocoder to obtain a context-aware feature representation, and generate the synthesized audio data corresponding to the to-be-synthesized speech text data based on the feature representation.

[0177] In some embodiments, the neural vocoder includes a generator including a neural network module for a bidirectional sequence modeling task; and the vocoding module 702 is further configured to perform bidirectional sequence modeling on the mel-frequency spectrum data based on the selective state-space model adopted by the neural network module to obtain a bidirectional sequence processing result; the bidirectional sequence processing result is a bidirectional long-range temporal dependency of the mel-frequency spectrum data.

[0178] In some embodiments, the neural network module includes a forward neural network layer and a reverse neural network layer based on the selective state-space model; and the vocoding module 702 is further configured to model past context by performing forward recursive processing on the mel-frequency spectrum data based on the forward neural network layer to obtain a forward sequence processing result; the forward sequence processing result is a forward long-range temporal dependency of the mel-frequency spectrum data, and the bidirectional sequence processing result includes the forward sequence processing result; and model future context by performing reverse recursive processing on the mel-frequency spectrum data based on the reverse neural network layer to obtain a reverse sequence processing result; the reverse sequence processing result is a reverse long-range temporal dependency of the mel-frequency spectrum data, and the bidirectional sequence processing result includes the reverse sequence processing result.

[0179] In some embodiments, the vocoding module 702 is further configured to perform state fusion on the forward sequence processing result and the reverse sequence processing result to obtain a fused long sequence processing result, and perform linear projection on the fused long sequence processing result to obtain the context-aware feature representation.

[0180] In some embodiments, the generator comprises a decoder module; the acoustic coding module 702 is further configured to convert the feature representation into an audio waveform based on the decoder module; and map linear prediction values of the audio waveform to a preset audio amplitude range to obtain the synthesized audio data corresponding to the to-be-synthesized speech text data.

[0181] In some embodiments, the speech synthesis system provided by the embodiments of the present application further comprises:

[0182] a model construction module configured to construct a generator based on a selective state space model; and perform generative adversarial network training on the generator based on a preset discriminator to obtain a neural vocoder based on a generative adversarial network model architecture.

[0183] In some embodiments, the model construction module is further configured to obtain a sound quality verification result of the synthesized audio data; perform parameter updating on the preset discriminator based on the sound quality verification result to obtain an updated discriminator; and perform generative adversarial network training on the generator based on the updated discriminator to obtain an updated generator.

[0184] It should be noted that the specific implementation of the speech synthesis system provided by the embodiments of the present application is basically the same as the specific embodiments of the speech synthesis method described above, and will not be repeated here.

[0185] Please refer to Figure 8 The embodiments of the present application also provide a speech synthesis device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned evaluation method of the visual language model when executing the computer program.

[0186] In some embodiments, the speech synthesis device can be any intelligent terminal such as a vehicle-mounted hardware platform (e.g., a vehicle-mounted computer), a tablet computer, a smart phone, a wearable device, etc.

[0187] As Figure 8 shown, the speech synthesis device provided by the embodiments of the present application can comprise:

[0188] The processor 801 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is configured to execute a related program to implement the technical solutions provided by the embodiments of the present application.

[0189] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 802 and are invoked and executed by the processor 801 to implement the voice synthesis method of the embodiments of the present application.

[0190] The input / output interface 803 is configured to realize information input and output.

[0191] The communication interface 804 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0192] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804) of the device.

[0193] The processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the device.

[0194] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the voice synthesis method.

[0195] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0196] The embodiments of the present application also provide a computer program product, which includes a computer program. The computer program is executed by a processor to implement steps, which are basically the same as the specific embodiments of the voice synthesis method, and thus will not be described herein.

[0197] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0198] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0199] The system embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0200] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system, and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0201] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0202] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0204] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0206] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0207] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtaining Mel spectrum data corresponding to the speech text data to be synthesized; Inputting the mel-spectrogram data into a neural vocoder based on a selective state space model; Based on the neural vocoder, the selective state space model is used to perform long sequence processing on the mel spectrum data to obtain synthesized audio data corresponding to the speech text data to be synthesized.

2. The method according to claim 1, characterized in that The method of performing long sequence processing on the Mel spectrum data using the selective state space model based on the neural vocoder to obtain synthesized audio data corresponding to the speech text data to be synthesized includes: Based on the neural vocoder, the selective state space model is used to capture the long-range temporal dependency of the mel spectrum data to obtain a feature representation with context awareness; Synthesized audio data corresponding to the speech text data to be synthesized is generated based on the feature representation.

3. The method according to claim 2, characterized in that The neural vocoder includes a generator, and the generator includes a neural network module for a bidirectional sequence modeling task; the method of capturing long-range temporal dependencies of the mel spectrum data using the selective state space model based on the neural vocoder includes: Based on the neural network module, the selective state space model is used to perform bidirectional sequence modeling on the mel spectrum data to obtain a bidirectional sequence processing result; the bidirectional sequence processing result is a bidirectional long-range temporal dependency relationship of the mel spectrum data.

4. The method according to claim 3, characterized in that The neural network module includes a forward neural network layer and a reverse neural network layer based on a selective state space model; The performing bidirectional sequence modeling on the mel spectrum data using the selective state space model based on the neural network module to obtain a bidirectional sequence processing result includes: Performing forward recursive processing on the Mel spectrum data based on the feedforward neural network layer to model past context, thereby obtaining a forward sequence processing result; the forward sequence processing result is a forward long-range temporal dependency of the Mel spectrum data, and the bidirectional sequence processing result includes the forward sequence processing result; Based on the reverse neural network layer, the mel spectrum data is subjected to reverse recursive processing to model future context, thereby obtaining a reverse sequence processing result; the reverse sequence processing result is a reverse long-range temporal dependency relationship of the mel spectrum data, and the bidirectional sequence processing result includes the reverse sequence processing result.

5. The method according to claim 4, characterized in that The method further comprises: Performing state fusion on the forward sequence processing result and the reverse sequence processing result to obtain a fused long sequence processing result; Linear projection is performed on the fused long sequence processing result to obtain the context-aware feature representation.

6. The method according to claim 2, characterized in that The generator includes a decoder module; The step of generating synthesized audio data corresponding to the speech text data to be synthesized based on the feature representation includes: converting the feature representation into an audio waveform based on the decoder module; Based on the decoder module, the linear prediction value of the audio waveform is mapped to a preset audio amplitude range to obtain synthesized audio data corresponding to the speech text data to be synthesized.

7. The method according to any one of claims 3 to 6, characterized in that Before capturing the long-range temporal dependency of the mel-spectrogram data using the selective state-space model based on the neural vocoder, the method further includes: Building a generator based on a selective state-space model; The generator is trained with a generative adversarial network based on a preset discriminator to obtain a neural vocoder based on a generative adversarial network model architecture.

8. The method according to any one of claims 3 to 6, characterized in that After performing long sequence processing on the mel-spectrogram data using the selective state space model based on the neural vocoder to obtain synthesized audio data corresponding to the speech text data to be synthesized, the method further includes: Obtaining a sound quality verification result of the synthesized audio data; Based on the sound quality verification result, the parameters of the preset discriminator are updated to obtain an updated discriminator; The generator is trained with a generative adversarial network based on the updated discriminator to obtain an updated generator.

9. A speech synthesis system, characterized in that: The system comprises: A data acquisition module is used to acquire Mel spectrum data corresponding to the speech text data to be synthesized; The vocoding module is configured to input the mel-spectrogram data into a neural vocoder based on a selective state-space model; and to perform long-sequence processing on the mel-spectrogram data using the selective state-space model based on the neural vocoder to obtain synthesized audio data corresponding to the speech text data to be synthesized.

10. A speech synthesis device, characterized in that: The speech synthesis device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 8 is implemented.

Images