A meta-learning based adaptive text-to-speech method and related devices thereof

Abstract

The application belongs to the field of artificial intelligence and relates to a self-adaptive text-to-speech method based on meta-learning, which comprises the following steps: pre-training according to a full data set to obtain an initial value of a preset acoustic model; sampling sound training sample data, performing feature training through the preset acoustic model to generate a mel spectrum, and generating style encoding through a preset style encoder; performing adaptive instance normalization processing on layer normalization of the preset acoustic model to obtain a target acoustic model comprising a target mel spectrum; and finally converting stranger sample data to output target speech data with style encoding. The application also provides a self-adaptive text-to-speech device based on meta-learning, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the data involved in the conversion process can be stored in the blockchain. The application can reduce the complexity of training, realize adaptive learning and conversion of small sample data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an adaptive text-to-speech method based on meta-learning and related equipment. Background Technology

[0002] With the success of neural networks in many applications, Text-to-Speech (TTS) systems based on neural networks have also seen significant improvements in recent years. The fidelity and intelligibility of TTS speech have greatly improved. Applications such as AI voice assistant services and audio navigation systems have been widely developed and deployed. While meeting the demand for high-quality voice generation, customers' needs for personalization are constantly increasing. This requires TTS models to not only generate high-quality speech but also capture the voices of different speakers effectively. However, existing adaptive TTS systems are mainly based on pre-trained models. This involves first training the model from scratch using the voices of multiple speakers, and then fine-tuning it using a small portion of the speaker's speech data. However, this method still requires speakers to provide some speech data, and fine-tuning requires thousands of iterations to converge. Furthermore, if a speaker not present in the training data is encountered, the synthesis effect is relatively poor. Therefore, existing adaptive TTS systems suffer from high training complexity and poor conversion results when dealing with small sample data. Summary of the Invention

[0003] The purpose of this application is to propose an adaptive text-to-speech method and related equipment based on meta-learning. The amount of sound training sample data is small, which can reduce the complexity of training and improve the adaptive learning ability and conversion effect of small sample data.

[0004] To address the aforementioned technical problems, this application provides an adaptive text-to-speech method based on meta-learning, employing the following technical solution:

[0005] A pre-trained acoustic model is obtained by pre-training based on the full dataset of the speakers, wherein the parameters of the pre-trained acoustic model are the preset initial values ​​of the acoustic model.

[0006] Sound training sample data is sampled from the full dataset, and feature training is performed on the sound training sample data using the preset acoustic model to generate a mel spectrum, and style encoding is generated using a preset style encoder.

[0007] Adaptive instance normalization is performed on the layer normalization of the preset acoustic model, and the style code is injected into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries the style code;

[0008] Acquire unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with the style code corresponding to the unfamiliar sample data.

[0009] Furthermore, the steps of generating a mel spectrum by performing feature training based on the sound training sample data using the preset acoustic model, and generating a style code using the preset style encoder, specifically include:

[0010] The sound data from the sound training sample data is input into the preset acoustic model, and a mel spectrum is generated based on the sampling frequency of the sound data.

[0011] The sound data from the sound training sample data is input into the preset style encoder, and the style code is generated based on the sampling frequency and sample precision of the sound data.

[0012] Furthermore, the step of performing adaptive instance normalization processing on the layer normalization of the preset acoustic model, and injecting the style encoding into the preset acoustic model to obtain a target acoustic model including the target mel spectrum specifically includes:

[0013] The first parameter of the style encoding is calculated through the adaptive instance normalization process;

[0014] The second parameter of the mel spectrum is calculated through the adaptive instance normalization process;

[0015] Data matching is performed based on the first parameter of the mel spectrum and the second parameter of the style encoding to output the target mel spectrum with style encoding.

[0016] Furthermore, after the step of performing adaptive instance normalization processing on the layer normalization of the preset acoustic model and injecting the style code into the preset acoustic model, the method further includes the step of:

[0017] Text request sample data is sampled from the full dataset, and the text request sample data is input into the target acoustic model for conversion detection to determine whether to output detection data corresponding to the text request sample data.

[0018] Furthermore, the step of inputting the text request sample data into the target acoustic model for conversion detection and determining whether to output detection data corresponding to the text request sample data includes:

[0019] The target mel spectrum is determined by a preset style discriminator to determine whether it contains the style code.

[0020] A preset phoneme discriminator determines whether the target mel spectrum is aligned with the phonemes corresponding to the input text request sample data.

[0021] To address the aforementioned technical problems, this application also provides an adaptive text-to-speech device based on meta-learning, employing the following technical solution:

[0022] The first training module is used to pre-train the model based on the full dataset of the speakers, and uses the model parameters obtained from the pre-training as the initial values ​​of the preset acoustic model.

[0023] The second training module is used to sample sound training sample data from the full dataset, perform feature training based on the sound training sample data using the preset acoustic model to generate a mel spectrum, and generate style code using a preset style encoder.

[0024] The normalization processing module is used to perform adaptive instance normalization processing on the layer normalization of the preset acoustic model, and inject the style code into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries the style code.

[0025] The conversion module is used to acquire unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with the style code corresponding to the unfamiliar sample data.

[0026] Furthermore, the second training module includes:

[0027] The first generation submodule is used to input the sound data in the sound training sample data into the preset acoustic model and generate a mel spectrum according to the sampling frequency of the sound data;

[0028] The second generation submodule is used to input the sound data in the sound training sample data into the preset style encoder, and generate the style code according to the sampling frequency and sample precision of the sound data.

[0029] Furthermore, the normalization processing module includes:

[0030] The first calculation submodule is used to calculate the first parameter of the style encoding through the adaptive instance normalization process;

[0031] The second calculation submodule is used to calculate the second parameter of the mel spectrum through the adaptive instance normalization process;

[0032] The third calculation submodule is used for the first parameter of the mel spectrum and the second parameter of the style encoding, and outputs the target mel spectrum with style encoding.

[0033] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0034] The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the meta-learning-based adaptive text-to-speech method described in any of the above embodiments.

[0035] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0036] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the meta-learning-based adaptive text-to-speech method described in any of the above embodiments.

[0037] Compared with existing technologies, the embodiments of this application have the following advantages: This application trains a preset acoustic model using extracted sound training sample data to obtain the corresponding mel spectrum and style code. Then, by performing adaptive instance normalization on the layer normalization of the preset acoustic model, the style code is injected into the preset acoustic model, resulting in a target acoustic model including the target mel spectrum, and the target mel spectrum carries the style code. In the acoustic model learning process of this application, not only is the amount of sampled speaker sound training sample data small, but the adaptive instance normalization process can also reduce the complexity of training; moreover, the preset acoustic model performs feature learning based on the sampled sound training sample data and injects the style code into the preset acoustic model. The final target acoustic model, after testing, can generate the corresponding style code and target speech data based on a small amount of unfamiliar sample data when performing text-to-speech conversion. It has strong adaptive learning and conversion capabilities for small sample data, which is more conducive to realizing personalized needs. Attached Figure Description

[0038] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0040] Figure 2 This is a flowchart of an embodiment of an adaptive text-to-speech method based on meta-learning according to this application;

[0041] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202;

[0042] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203;

[0043] Figure 5 This is a flowchart illustrating another specific embodiment of an adaptive text-to-speech method based on meta-learning according to this application;

[0044] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 205;

[0045] Figure 7 This is a schematic diagram of the structure of an embodiment of an adaptive text-to-speech device based on meta-learning according to this application;

[0046] Figure 8 yes Figure 7 A schematic diagram of a specific implementation of the second training module shown;

[0047] Figure 9 yes Figure 7 A schematic diagram of a specific implementation of the normalization processing module shown;

[0048] Figure 10 This is a schematic diagram of another embodiment of an adaptive text-to-speech device based on meta-learning according to this application;

[0049] Figure 11 yes Figure 10 The diagram shows a specific implementation of the judgment module.

[0050] Figure 12 This is a basic structural block diagram of the computer device provided in this embodiment. Detailed Implementation

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0052] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0053] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0055] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0056] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0057] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0058] Server 105 can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] It should be noted that the meta-learning-based adaptive text-to-speech method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the meta-learning-based adaptive text-to-speech method apparatus is generally set in the server / terminal device.

[0060] It should be understood that Figure 1 The number of terminal devices 101, 102, 103, network 104, and server 105 shown is merely illustrative. Depending on implementation needs, any number of terminal devices 101, 102, 103, network 104, and server 105 can be used.

[0061] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of an adaptive text-to-speech method based on meta-learning according to this application. The adaptive text-to-speech method based on meta-learning includes the following steps:

[0062] Step S201: Pre-train the model based on the full dataset of the speakers, and use the pre-trained model parameters as the initial values ​​of the preset acoustic model.

[0063] In this embodiment, an adaptive text-to-speech method based on meta-learning runs on an electronic device (e.g., Figure 1The server / terminal device shown can obtain the full dataset of the aforementioned speaker and perform data transmission via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0064] Specifically, the aforementioned full dataset can be a pre-collected collection of data from multiple speakers. The full dataset contains a large amount of data, potentially tens of thousands of records. Within the full dataset, each speaker's data can include multiple audio segments and corresponding text data, with each audio segment being distinct. In the full dataset, there is a one-to-one correspondence between each speaker's audio data and the speaker, which can be represented by an identifier.

[0065] The aforementioned pre-training of the model based on the full dataset of the speakers refers to the training performed before formal training to provide initial model parameters for the pre-defined acoustic model (text2mel). In deep learning neural networks, the training process is based on gradient descent for parameter optimization, iterating step by step to find the minimum loss function and optimal model weights. Gradient descent requires assigning initial values ​​to each parameter in the model. This pre-training can accelerate the feature learning speed of the subsequent acoustic model and improve its efficiency. Here, text2mel can be a transformer-based acoustic model. Transformers do not require loops but process all words or symbols in the sequence in parallel, while using self-attention to combine context with distant words. By processing all words in parallel and ensuring each word pays attention to other words in the sentence through multiple processing steps, Transformers train quickly and achieve high translation accuracy. Therefore, when text2mel is trained based on the initial values ​​provided by the pre-trained acoustic model, it helps to accelerate the convergence speed of gradient descent during model training, and is more likely to obtain a target acoustic model with low error.

[0066] Step S202: Sample sound training sample data from the full dataset, perform feature training based on the sound training sample data using a preset acoustic model to generate a mel spectrum, and generate style code using a preset style encoder.

[0067] Specifically, in the full dataset, each speaker corresponds to a subset of data. Due to the large amount of data, a subset of speakers can be randomly selected. For example, a subset of 50 speakers can be selected from the full dataset of 5000 speakers for feature training of the acoustic model. The extraction method can also be to extract subsets of speakers ranked first / last in sequence. Other extraction methods are also possible, such as interval sampling.

[0068] More specifically, the aforementioned audio training sample data can be sampled directly from the full dataset, or it can be sampled from the aforementioned subset of data. The audio training sample data can include the speaker's voice data and the corresponding text data, which in this embodiment can also be referred to as the support set (Xs, ts), where Xs represents the speaker's voice data and ts represents the corresponding text data. The acoustic model performs feature training on the audio training sample data to achieve the conversion between text and speech.

[0069] The `text2mel` function acts as a generator, using the frequencies of sounds in the training audio data to produce corresponding Mel spectra. Mel spectra can be mathematically converted to Mel scale based on the frequencies in the audio data. A Mel spectrum is a spectrogram on the Mel scale (a MATLAB function), obtained by multiplying the spectrogram by a Mel filter bank. The MATLAB function generates the signal's spectrum using a short-time Fourier transform. Each filter in the Mel filter bank is a triangular filter, which expands the aforementioned multiplication process.

[0070] The preset style encoder can encode the speaker's voice data to generate a style code corresponding to the voice data. The style code can include the speaker's identity information, pitch, and rhythm.

[0071] In this way, the text2mel generator can be used to generate the mel spectrum corresponding to the sound training sample data, and the style code corresponding to the sound data in the sound training sample data can be generated according to the StyleEncoder.

[0072] Step S203: Perform adaptive instance normalization processing on the layer normalization of the preset acoustic model, and inject style coding into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries style coding.

[0073] Specifically, after the text2mel in the transformer generates the corresponding mel spectrum based on the sound data, the layer normalization of all layers in the text2mel of the transformer structure can be processed using the Adaptive Instance Normalization (AdaIN) method. The style code generated by the StyleEncoder based on the sound data can be injected into the text2mel so that the generated mel spectrum carries the style information of the style code, and finally the target acoustic model with the target mel spectrum is obtained.

[0074] Step S204: Obtain unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with style coding corresponding to the unfamiliar sample data.

[0075] In this embodiment, the aforementioned unfamiliar sample data can be a small amount of text sample data that does not carry sound data, obtained by the target acoustic model in practical applications. After inputting the unfamiliar sample data into the target acoustic model, tests show that when using the target acoustic model for text-to-speech conversion, it can generate corresponding style codes and target speech data based on a small amount of unfamiliar sample data, achieving adaptive learning and conversion of small sample data.

[0076] In this embodiment of the invention, by training text2mel with extracted sound training sample data, the corresponding mel spectrum and style code can be obtained. Then, by performing adaptive instance normalization on the layer normalization of text2mel, the style code is injected into text2mel, resulting in a target acoustic model including the target mel spectrum, and the target mel spectrum carries the style code. In the acoustic model learning process of this application, not only is the amount of sampled speaker sound training sample data small, but the adaptive instance normalization process can also reduce the complexity of training; and text2mel performs feature learning based on the sampled sound training sample data, injecting the style code into text2mel. The final target acoustic model, after testing, can generate the corresponding style code and target speech data based on a small amount of unfamiliar sample data when performing text-to-speech conversion. It has strong adaptive learning and conversion capabilities for small sample data, which is more conducive to realizing personalized needs.

[0077] In some alternative implementations, such as Figure 3 As shown, Figure 3 for Figure 2 A flowchart illustrating a specific embodiment of step 202. Step 202 performed by the aforementioned electronic device specifically includes the following steps:

[0078] Step S2021: Input the sound data from the sound training sample data into the preset acoustic model, and generate the mel spectrum according to the sampling frequency of the sound data.

[0079] Specifically, the sound data from each speaker's training sample data can be input into text2mel in an orderly manner, and then converted according to the relationship between the frequency and mel scale of a segment of sound data. Specifically, the segment of sound data can be divided into multiple segments of sound sources, and the frequencies of each segment of sound sources can be different or the same. Then, based on the frequency of each segment of sound sources, the conversion is performed according to the relationship between frequency and mel scale, thereby obtaining the mel spectrum corresponding to the sound data. The conversion relationship between frequency (f) and mel scale (m) is shown in the following formula (1):

[0080] m = 2595 * log10(1 + f / 700) (1)

[0081] Step S2022: Input the sound data from the sound training sample data into the preset style encoder, and generate style codes based on the sampling frequency and sample precision of the sound data.

[0082] Specifically, audio encoding primarily involves compressing sound information. After digitization, the amount of information in a sound signal is much larger than in analog transmission, and it cannot be transmitted directly like analog television sound; therefore, an additional compression and encoding process is required, which is audio encoding. In this embodiment, sound data can also be input into a StyleEncoder, where the speaker's identity, pitch, and rhythm are encoded based on the sampling frequency and sample precision of the sound data, resulting in a style code corresponding to each sound data point. Each speaker's voice has a different pitch and rhythm. The aforementioned method of encoding using a style encoder can include, but is not limited to, linear predictive coding, sub-band coding, etc.

[0083] In this embodiment of the invention, by inputting sound data into text2mel, the corresponding mel spectrum can be generated according to the above formula (1); and by inputting sound data into StyleEncoder, the speaker's identity, pitch and rhythm of the sound are encoded according to the sampling frequency and sample precision of the sound data, so as to obtain the style code corresponding to each sound data, which is convenient to inject the style code into text2mel when performing adaptive instance normalization processing, and finally obtain the target acoustic model.

[0084] In some alternative implementations, such as Figure 4 As shown, Figure 4 for Figure 2A flowchart illustrating a specific embodiment of step 203. Step 203 performed by the aforementioned electronic device specifically includes the following steps:

[0085] Step S2031: Calculate the first parameter of the style code through adaptive instance normalization.

[0086] Specifically, the aforementioned adaptive instance normalization process may include aligning the mean and variance of the style encoding. Since Adaptive Instance Normalization (AdaIN) has the ability to learn training mapping parameters, a first parameter can be calculated on the input style encoding using AdaIN. This first parameter includes the mean and variance of the style encoding. Alignment processing of the mean and variance is then performed to match the mean and variance of the MEL spectrum.

[0087] Step S2032: Calculate the second parameter of the mel spectrum through adaptive instance normalization.

[0088] Specifically, since the Adaptive Instance Normalization AdaIN has the ability to learn training mapping parameters, the second parameter (variance and mean of the mel spectrum) can be adaptively calculated based on the mel spectrum.

[0089] Step S2033: Perform data matching based on the first parameter of the mel spectrum and the second parameter of the style coding, and output the target mel spectrum with style coding.

[0090] Specifically, by combining the variance and mean of the calculated style code and the variance and mean of the mel spectrum, the result of inputting the style code and mel spectrum into the Adaptive Instance Normalization (AdaIN) can be calculated. This results in the output of the target mel spectrum with the style code, thus obtaining the target acoustic model. In the text2mel learning process of this application, the amount of sampled user voice training data is small, which reduces the computational complexity in the Adaptive Instance Normalization (AdaIN).

[0091] In some alternative implementations, such as Figure 5 As shown, Figure 5 In order to be in Figure 2 A flowchart illustrating a specific embodiment following step 203. After step 203, the aforementioned electronic device can also be used to perform the following steps:

[0092] Step S205: Sample text request sample data from the full dataset, input the text request sample data into the target acoustic model for conversion detection, and determine whether to output detection data corresponding to the text request sample data.

[0093] Specifically, the aforementioned text request sample data can be directly sampled from the full dataset or obtained from a subset of the proposed speakers. The text request sample data is text sample data excluding audio data, and can be used to perform conversion detection on the target acoustic model to determine whether it can achieve text-to-speech functionality. By inputting the text request sample data and the aforementioned style encoding into the target acoustic model, if it has text-to-speech functionality, it will generate a target mel spectrum M' (detection data) with style encoding; if it does not generate a target mel spectrum M' with style encoding or generates it incorrectly, it indicates that the conversion has failed.

[0094] More specifically, to determine whether the target acoustic model generates a target mel spectrum M' corresponding to the text request sample data, at least one discriminator can be set up for detection. The amount of output detection data is consistent with the amount of text request sample data collected; one text request sample corresponds to one output target mel spectrum M'. When there is only one text request sample, one corresponding discriminator can be set up; when there are multiple text request sample data, a corresponding number of discriminators can be set up. By detecting the target acoustic model in advance, errors can be eliminated before practical application, allowing for the improvement and refinement of the target acoustic model, facilitating better practical application.

[0095] In this embodiment, to determine whether the target acoustic model can complete speech conversion, text request sample data is input into the target acoustic model for conversion detection, and multiple discriminators are provided for judgment. In this way, by detecting the target acoustic model in advance, error correction can be performed before practical application, allowing for the improvement and refinement of the target acoustic model, facilitating better practical application.

[0096] In some alternative implementations, such as Figure 6 As shown, Figure 6 for Figure 5 A flowchart illustrating a specific embodiment of step 205. Step 205 performed by the aforementioned electronic device specifically includes the following steps:

[0097] Step S2051: Determine whether the target mel spectrum contains style coding using a preset style discriminator.

[0098] Specifically, the aforementioned style discriminator can be a discriminator used to identify whether the target mel spectrum M' contains style coding. After the target mel spectrum M' is generated, it is further input into the style discriminator for identification.

[0099] Step S2052: Use a preset phoneme discriminator to determine whether the target mel spectrum is aligned with the phonemes corresponding to the input text request sample data.

[0100] Specifically, the aforementioned phoneme discriminator can be a discriminator used to determine whether the target mel spectrum M' is aligned with the phonemes of the input text request sample data. After the target mel spectrum M' is generated, it can be further input into the phoneme discriminator for judgment. When the aforementioned style discriminator determines that the target mel spectrum M' contains style encoding and that the target mel spectrum M' is aligned with the phonemes of the input text request sample data, it indicates that the generated target acoustic model can accurately achieve the text-to-speech function.

[0101] In this embodiment, two discriminators (style discriminator and phoneme discriminator) are provided to determine whether the target mel spectrum M' contains style coding and whether the target mel spectrum M' is aligned with the phonemes of the input text request sample data, thereby detecting the text-to-speech function of the generated target acoustic model. This allows for timely model optimization based on the recognition accuracy of the target acoustic model.

[0102] It should be emphasized that, in order to further ensure the privacy and security of the information involved in the speech conversion process, such as the sound training sample data, mel spectrum, style encoding, target mel spectrum, unfamiliar sample data, and target speech data, the information involved in the speech conversion process can also be stored in a blockchain node.

[0103] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In a distributed computing environment, program modules can reside on local and remote computer storage media, including storage devices.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0105] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0106] Further reference Figure 7As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an adaptive text-to-speech device based on meta-learning. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0107] like Figure 7 As shown, the meta-learning-based adaptive text-to-speech device 700 of this embodiment includes: a first training module 701, a second training module 702, a normalization processing module 703, and a conversion module 704. Wherein:

[0108] The first training module 701 is used to pre-train the model based on the full dataset of the speakers, and uses the model parameters obtained from the pre-training as the initial values ​​of the preset acoustic model.

[0109] The second training module 702 is used to sample sound training sample data from the full dataset, perform feature training based on the sound training sample data using a preset acoustic model to generate a mel spectrum, and generate style code using a preset style encoder.

[0110] The normalization processing module 703 is used to perform adaptive instance normalization processing on the layer normalization of the preset acoustic model, and inject style coding into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries style coding.

[0111] The conversion module 704 is used to acquire unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with style coding corresponding to the unfamiliar sample data.

[0112] Specifically, the aforementioned full dataset can be a pre-collected collection of data from multiple speakers. The full dataset contains a large amount of data, potentially tens of thousands of records. Within the full dataset, each speaker's data can include multiple audio segments and corresponding text data, with each audio segment being distinct. In the full dataset, there is a one-to-one correspondence between each speaker's audio data and the speaker, which can be represented by an identifier.

[0113] The aforementioned model pre-training based on the full dataset of speakers refers to the training performed before formal training to provide initial model parameters for text2mel. The model parameters obtained through pre-training in the first training module 701 can be used as the initial values ​​for text2mel. In deep learning neural networks, the training process is based on gradient descent for parameter optimization, iterating step by step to find the minimum loss function and the optimal model weights. Gradient descent requires assigning an initial value to each parameter in the model. This pre-training can accelerate the feature learning speed of the subsequent acoustic model and improve its efficiency. Here, text2mel can be a transformer-based acoustic model. Thus, when text2mel is trained subsequently based on the initial values ​​provided by pre-training, the convergence speed of gradient descent in the acoustic model training can be accelerated, making it more likely to obtain a target acoustic model with low error.

[0114] Specifically, in the full dataset, each speaker corresponds to a subset of data. Due to the large amount of data, a subset of speakers can be randomly selected. For example, a subset of 50 speakers can be selected from the full dataset of 5000 speakers for feature training of the acoustic model. The extraction method can also be to extract subsets of speakers ranked first / last in sequence. Other extraction methods are also possible, such as interval sampling.

[0115] More specifically, the aforementioned sound training sample data can be sampled directly from the full dataset, or it can be sampled from the aforementioned subset data. The sound training sample data can include the speaker's voice data and the corresponding text data, which in this embodiment can also be referred to as the support set (Xs, ts), where Xs represents the speaker's voice data and ts represents the corresponding text data. The second training module 702 performs feature training on the sound training sample data to achieve the acoustic model's conversion of text to speech.

[0116] `text2mel` can be used as a generator to generate the corresponding MEL spectrum from the frequencies of sounds in the sound training sample data. The MEL spectrum can then be converted to the MEL scale by performing mathematical operations on the frequencies in the sound data. The MEL spectrum is a spectrogram on the MEL scale, obtained by multiplying the spectrogram by a Mel filter bank. The MATLAB function `text2mel` generates the spectrum of the signal using a short-time Fourier transform. Each filter in the Mel filter bank is a triangular filter, which expands the aforementioned dot product process.

[0117] The StyleEncoder can encode the speaker's voice data to generate a style code corresponding to the voice data. The style code can include the speaker's identity information, pitch, and rhythm.

[0118] Specifically, after the text2mel in the aforementioned transformer generates the corresponding mel spectrum based on the sound data, the normalization processing module 703 processes all LayerNormalizations in the text2mel of the transformer structure using the adaptive instance normalization AdaIN method. The style code generated by the StyleEncoder based on the sound data is injected into the text2mel so that the generated mel spectrum carries the style information of the style code, ultimately yielding the target acoustic model with the target mel spectrum.

[0119] After obtaining the target acoustic model, unfamiliar sample data can be acquired. This unfamiliar sample data can be a small amount of text sample data that does not carry sound data, obtained from the actual application of the target acoustic model. After the conversion module 704 inputs the unfamiliar sample data into the target acoustic model, it can finally output the target speech data corresponding to the unfamiliar sample data, and the target speech data also includes the style code corresponding to the unfamiliar sample data.

[0120] In this embodiment of the invention, by training text2mel with extracted sound training sample data, the corresponding mel spectrum and style code can be obtained. Then, by performing adaptive instance normalization on the layer normalization of text2mel, the style code is injected into text2mel, resulting in a target acoustic model including the target mel spectrum, and the target mel spectrum carries the style code. During the acoustic model learning process, not only is the amount of sampled speaker sound training sample data small, but the adaptive instance normalization process can also reduce the complexity of training; furthermore, text2mel performs feature learning based on the sampled sound training sample data and injects the style code into text2mel. The final target acoustic model, after testing, can generate the corresponding style code and target speech data based on a small amount of unfamiliar sample data when performing text-to-speech conversion. It has strong adaptive learning and conversion capabilities for small sample data, which is more conducive to realizing personalized needs.

[0121] See Figure 8 This is a schematic diagram illustrating a specific implementation of the second training module 702. The second training module 702 includes a first generation submodule 7021 and a second generation submodule 7022.

[0122] The first generation submodule 7021 is used to input the sound data from the sound training sample data into a preset acoustic model and generate a mel spectrum based on the sampling frequency of the sound data.

[0123] The second generation submodule 7022 is used to input the sound data in the sound training sample data into the preset style encoder, and generate style codes according to the sampling frequency and sample precision of the sound data.

[0124] In this embodiment of the invention, the first generation submodule 7021 inputs sound data into text2mel and generates the corresponding mel spectrum according to the above formula (1); and the second generation submodule 7022 inputs sound data into StyleEncoder and encodes the speaker's identity, pitch and rhythm of the sound according to the sampling frequency and sample precision of the sound data, so as to obtain the style code corresponding to each sound data, which is convenient to inject the style code into text2mel when performing adaptive instance normalization processing, and finally obtain the target acoustic model.

[0125] See Figure 9 This is a schematic diagram of a specific implementation of the normalization processing module. The normalization processing module 703 includes a first calculation submodule 7031, a second calculation submodule 7032, and a third calculation submodule 7033.

[0126] The first computation submodule 7031 is used to process the first parameter of the computation style encoding through adaptive instance normalization.

[0127] The second calculation submodule 7032 is used to calculate the second parameter of the mel spectrum through adaptive instance normalization processing.

[0128] The third calculation submodule 7033 is used to perform data matching based on the first parameter of the mel spectrum and the second parameter of the style coding, and outputs the target mel spectrum with style coding.

[0129] In this embodiment, because Adaptive Instance Normalization (AdaIN) has the ability to learn and train mapping parameters, the first calculation submodule 7031 calculates the mean and variance of the style encoding based on the input style encoding using AdaIN. The second calculation submodule 7032 adaptively calculates the variance and mean of the mel spectrum based on the mel spectrum. Then, the third calculation submodule 7033 aligns the mean and variance of the style encoding with the mean and variance of the mel spectrum, outputting the target mel spectrum with the style encoding, thus obtaining the target acoustic model. In the text2mel learning process, the amount of user voice training sample data is small, which reduces the computational complexity in AdaIN.

[0130] In some optional implementations of this embodiment, refer to Figure 10 As shown, the above-mentioned device 700 further includes: a judgment module 705, used to sample text request sample data from the full dataset, input the text request sample data into the target acoustic model for conversion detection, and determine whether to output detection data corresponding to the text request sample data.

[0131] In this embodiment, to determine whether the target acoustic model can complete speech conversion, the judgment module 705 inputs text request sample data into the target acoustic model for conversion detection and provides multiple discriminators for judgment. In this way, by detecting the target acoustic model in advance, error correction can be performed before practical application, allowing for the improvement and refinement of the target acoustic model, facilitating better practical application.

[0132] See Figure 11 This is a structural diagram of a specific embodiment of the judgment module. The judgment module 705 includes a first judgment submodule 7051 and a second judgment submodule 7052.

[0133] The first judgment submodule 7051 is used to determine whether the target mel spectrum contains style coding through a preset style discriminator.

[0134] The second judgment submodule 7052 is used to determine whether the target mel spectrum is aligned with the phonemes corresponding to the input text request sample data through a preset phoneme discriminator.

[0135] In this embodiment, the text-to-speech function of the generated target acoustic model is detected by using the stylediscriminator provided by the first judgment submodule 7051 to determine whether the target mel spectrum M' contains style coding, and by using the phonemediscriminator provided by the second judgment submodule 7052 to determine whether the target mel spectrum M' is aligned with the phonemes of the input text request sample data. This allows for timely model optimization based on the recognition accuracy of the target acoustic model.

[0136] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 12 , Figure 12 This is a basic structural block diagram of the computer device in this embodiment.

[0137] Computer device 120 includes a memory 121, a processor 122, and a network interface 123 that are interconnected via a system bus. It should be noted that only computer device 120 with components 121-123 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0138] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0139] The memory 121 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 121 may be an internal storage unit of the computer device 120, such as the hard disk or memory of the computer device 120. In other embodiments, the memory 121 may also be an external storage device of the computer device 120, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 120. Of course, the memory 121 may include both the internal storage unit and the external storage device of the computer device 120. In this embodiment, the memory 121 is typically used to store the operating system and various application software installed on the computer device 120, such as computer-readable instructions based on a meta-learning adaptive text-to-speech method. In addition, the memory 121 can also be used to temporarily store various types of data that have been output or will be output.

[0140] In some embodiments, processor 122 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 122 is typically used to control the overall operation of computer device 120. In this embodiment, processor 122 is used to execute computer-readable instructions stored in memory 121 or to process data, such as executing computer-readable instructions for a meta-learning-based adaptive text-to-speech method.

[0141] The network interface 123 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 120 and other electronic devices.

[0142] This embodiment trains text2mel using extracted audio training sample data to obtain the corresponding mel spectrum and style code. Then, adaptive instance normalization is applied to the text2mel layer normalization process, injecting the style code into text2mel to obtain a target acoustic model including the target mel spectrum with style code. During the acoustic model learning process, the amount of audio training sample data from the speaker is small, and adaptive instance normalization reduces the complexity of training. Furthermore, text2mel performs feature learning based on the sampled audio training sample data and injects the style code into text2mel. The resulting target acoustic model, after testing, can generate corresponding style codes and target speech data from a small amount of unfamiliar sample data when performing text-to-speech conversion. It exhibits strong adaptive learning and conversion capabilities with small sample data, making it more conducive to achieving personalized needs.

[0143] This application also provides another implementation, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the meta-learning-based adaptive text-to-speech method described above.

[0144] This embodiment trains text2mel using extracted audio training sample data to obtain the corresponding mel spectrum and style code. Then, adaptive instance normalization is applied to the text2mel layer normalization process, injecting the style code into text2mel to obtain a target acoustic model including the target mel spectrum with style code. During the acoustic model learning process, the amount of audio training sample data from the speaker is small, and adaptive instance normalization reduces the complexity of training. Furthermore, text2mel performs feature learning based on the sampled audio training sample data and injects the style code into text2mel. The resulting target acoustic model, after testing, can generate corresponding style codes and target speech data from a small amount of unfamiliar sample data when performing text-to-speech conversion. It exhibits strong adaptive learning and conversion capabilities with small sample data, making it more conducive to achieving personalized needs.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the meta-learning-based adaptive text-to-speech method of the various embodiments of this application.

[0146] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An adaptive text-to-speech method based on meta-learning, characterized in that, Includes the following steps: The model is pre-trained based on the full dataset of the speakers, and the model parameters obtained from the pre-training are used as the initial values ​​of the preset acoustic model, wherein the preset acoustic model is a transformer-based acoustic model. Sound training sample data is sampled from the full dataset. The preset acoustic model is used to train features based on the sound training sample data to generate a mel spectrum. A style code is generated using a preset style encoder. The preset style encoder encodes the sound data of the speaker to generate a style code corresponding to the sound data. The style code includes the speaker's identity information, pitch, and rhythm. Adaptive instance normalization is performed on the layer normalization of the preset acoustic model, and the style code is injected into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries the style code; Acquire unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with the style code corresponding to the unfamiliar sample data; The steps of performing adaptive instance normalization processing on the layer normalization of the preset acoustic model, and injecting the style encoding into the preset acoustic model to obtain a target acoustic model including the target mel spectrum specifically include: The first parameter of the style encoding is calculated through the adaptive instance normalization process; The second parameter of the mel spectrum is calculated through the adaptive instance normalization process; Data matching is performed based on the first parameter of the mel spectrum and the second parameter of the style encoding to output the target mel spectrum with style encoding.

2. The adaptive text-to-speech method based on meta-learning according to claim 1, characterized in that, The steps of generating a mel spectrum by performing feature training based on the sound training sample data using the preset acoustic model, and generating a style code using the preset style encoder, specifically include: The sound data from the sound training sample data is input into the preset acoustic model, and a mel spectrum is generated based on the sampling frequency of the sound data. The sound data from the sound training sample data is input into the preset style encoder, and the style code is generated based on the sampling frequency and sample precision of the sound data.

3. The adaptive text-to-speech method based on meta-learning according to claim 1, characterized in that, After the steps of performing adaptive instance normalization processing on the layer normalization of the preset acoustic model and injecting the style code into the preset acoustic model, the method further includes the following steps: Text request sample data is sampled from the full dataset, and the text request sample data is input into the target acoustic model for conversion detection to determine whether to output detection data corresponding to the text request sample data.

4. The adaptive text-to-speech method based on meta-learning according to claim 3, characterized in that, The step of inputting the text request sample data into the target acoustic model for conversion detection and determining whether to output detection data corresponding to the text request sample data includes: The target mel spectrum is determined by a preset style discriminator to determine whether it contains the style code. A preset phoneme discriminator determines whether the target mel spectrum is aligned with the phonemes corresponding to the input text request sample data.

5. An adaptive text-to-speech device based on meta-learning, characterized in that, include: The first training module is used to pre-train the model based on the full dataset of the speakers, and uses the model parameters obtained from the pre-training as the initial values ​​of the preset acoustic model, wherein the preset acoustic model is a transformer-based acoustic model. The second training module is used to sample sound training sample data from the full dataset, perform feature training on the sound training sample data using the preset acoustic model to generate a mel spectrum, and generate style code using a preset style encoder. The preset style encoder encodes the sound data of the speaker to generate a style code corresponding to the sound data. The style code includes the speaker's identity information, pitch, and rhythm. The normalization processing module is used to perform adaptive instance normalization processing on the layer normalization of the preset acoustic model, and inject the style code into the preset acoustic model to obtain a target acoustic model including the target mel spectrum, wherein the target mel spectrum carries the style code. A conversion module is used to acquire unfamiliar sample data and input the unfamiliar sample data into the target acoustic model to output target speech data with the style code corresponding to the unfamiliar sample data; The normalization processing module includes: The first calculation submodule is used to calculate the first parameter of the style encoding through the adaptive instance normalization process; The second calculation submodule is used to calculate the second parameter of the mel spectrum through the adaptive instance normalization process; The third calculation submodule is used for the first parameter of the mel spectrum and the second parameter of the style encoding, and outputs the target mel spectrum with style encoding.

6. The adaptive text-to-speech device based on meta-learning according to claim 5, characterized in that, The second training module includes: The first generation submodule is used to input the sound data in the sound training sample data into the preset acoustic model and generate a mel spectrum according to the sampling frequency of the sound data; The second generation submodule is used to input the sound data in the sound training sample data into the preset style encoder, and generate the style code according to the sampling frequency and sample precision of the sound data.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the meta-learning-based adaptive text-to-speech method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the meta-learning-based adaptive text-to-speech method as described in any one of claims 1 to 4.

Images