A speech synthesis method and device, electronic equipment and storage medium

Abstract

This application provides a speech synthesis method, apparatus, electronic device, and storage medium, belonging to the field of artificial intelligence technology. The method includes acquiring a first speech source, inputting the first speech source into a first backbone network, and pre-training the first backbone network to obtain a reference model; the first speech source includes single-person speech content; constructing a second backbone network based on the first model parameters of the reference model; acquiring a second speech source, inputting the second speech source into the second backbone network, and performing low-resource target training on the second backbone network to obtain an adaptation model; the second speech source includes multi-person speech content; inputting target text material into the adaptation model, and performing speech synthesis through the adaptation model to obtain the target speech spectrum. This application can eliminate the pre-processing steps of model construction, simplify the processing flow, and improve processing efficiency; and achieve high-quality speech synthesis with a small amount of target corpus and label information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method and apparatus, electronic device and storage medium. Background Technology

[0002] Text-to-Speech (TTS) is a generative task that converts a sequence of phonemes from text into acoustic features. From traditional audio concatenation methods to parametric synthesis methods and the currently mainstream neural network speech synthesis methods, there has been a great deal of research on speech synthesis.

[0003] The research methods in related technical solutions typically require large amounts of text and corpus pairs, especially neural network-based speech synthesis methods. Data set size is crucial for improving speech synthesis quality. However, annotating and recording paired speech and text phonemes is extremely time-consuming and labor-intensive. Achieving high-quality speech synthesis with a limited amount of target corpus and annotations remains a highly challenging problem. Summary of the Invention

[0004] The main objective of this application is to provide a speech synthesis method, apparatus, electronic device, and storage medium, which aims to improve the efficiency and accuracy of audio synthesis.

[0005] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0006] A first speech sample is acquired, and the first speech sample is input into a first backbone network. The first backbone network is pre-trained to obtain a reference model. The first speech sample includes single-person speech content.

[0007] Construct a second backbone network based on the first model parameters of the reference model;

[0008] The second speech material is acquired and input into the second backbone network. The second backbone network is then trained with low-resource targets to obtain an adapted model. The second speech material includes speech content from multiple speakers.

[0009] The target text material is input into the adaptation model, and speech synthesis is performed through the adaptation model to obtain the target speech spectrum.

[0010] In some embodiments, the step of obtaining the second speech material, inputting the second speech material into the second backbone network, and performing low-resource target training on the second backbone network to obtain the adapted model includes:

[0011] The second audio material is input into the second backbone network;

[0012] The second backbone network is used to identify the speaker in the second speech material, and speaker codes are generated based on the identification results;

[0013] The second speech material is used to extract features through the second backbone network to obtain phoneme embedding features;

[0014] The speech spectrum material is obtained by combining the speaker encoding and the phoneme embedding features; in the speech spectrum material, each speech spectrum segment carries a speaker tag;

[0015] Based on the second model parameters of the second backbone network obtained from the output speech spectrum material, the adaptation model is determined based on the second model parameters.

[0016] In some embodiments, after the step of inputting the target text material into the adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum, the method includes:

[0017] The actual speech spectrum was collected;

[0018] The first loss value between the real speech spectrum and the target speech spectrum is calculated using the mean square loss function.

[0019] The low-resource target training process of the second backbone network is constrained based on the first loss value.

[0020] In some embodiments, after this step, the method includes:

[0021] The second speech material is vectorized to obtain the first tag embedding vector;

[0022] Obtain the text sequence at any stage of the low-resource target training process of the second backbone network, and perform vectorization processing on the text sequence to obtain the second label embedding vector;

[0023] The second loss value between the first label embedding vector and the second label embedding vector is calculated using the mean square loss function;

[0024] The encoding process of the low-resource target training process of the second backbone network is constrained based on the second loss value.

[0025] In some embodiments, after the step of obtaining the second speech material, inputting the target text material into the adaptation model, and performing speech synthesis through the adaptation model to obtain the target speech spectrum, the method includes:

[0026] The third loss value is determined based on the product of the second loss value and the harmonic parameter;

[0027] Based on the first loss value and the third loss value, determine the common constraint conditions;

[0028] The parameters of the second backbone network during the low-resource target training process are adjusted according to the common constraints to obtain the adapted model.

[0029] In some embodiments, the step of encoding speech spectrum material by combining the speaker coding and the phoneme embedding features includes:

[0030] A phoneme sequence was extracted from the second speech material;

[0031] The position of the phoneme vector in the phoneme sequence is encoded to obtain the position code;

[0032] The phoneme vectors are marked with their positions based on the position encoding and the phoneme embedding features. The marked phoneme embedding features are then encoded with the speaker encoding to obtain the speech spectrum material.

[0033] In some embodiments, the step of inputting the target text material into the adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum includes:

[0034] An audio attribute prediction model is obtained by training based on the first speech material and / or the second speech material; the audio attribute prediction model includes an energy prediction model, a duration prediction model, and a pitch information prediction model.

[0035] The target text material is input into the adaptation model and encoded to obtain intermediate audio material.

[0036] The intermediate-state audio material is input into the energy prediction model to predict the energy information of the speech spectrum.

[0037] The intermediate audio material is input into the duration prediction model to predict the duration information of the speech spectrum;

[0038] The intermediate audio material is input into the pitch information prediction model to predict the pitch information of the speech spectrum.

[0039] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0040] The pre-training unit is used to acquire the first speech material, input the first speech material into the first backbone network, and pre-train the first backbone network to obtain a reference model; the first speech material includes single-person speaking speech content;

[0041] A model building unit is used to build a second backbone network based on the first model parameters of the reference model;

[0042] The low-resource target training unit is used to acquire the second speech material, input the second speech material into the second backbone network, and perform low-resource target training on the second backbone network to obtain the adaptation model; the second speech material includes speech content of multiple people speaking;

[0043] The speech spectrum synthesis unit is used to input the target text material into the adaptation model, and perform speech synthesis through the adaptation model to obtain the target speech spectrum.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the method described in the first aspect above.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the method described in the first aspect.

[0046] This application proposes a speech synthesis method, apparatus, electronic device, and storage medium. The method first trains the backbone network to obtain a reference model through a pre-training process, and then performs low-resource target training based on the reference model to obtain an adapted model. By splitting the model training process into a pre-training stage and a low-resource target stage, and based on the reference model, a large amount of pre-processing steps for model building can be eliminated, simplifying the processing flow and improving processing efficiency. Furthermore, the pre-training process focuses on recognizing single-person speech content, while the low-resource target training process focuses on recognizing multi-person speech content. After obtaining the target adapted model, high-quality speech synthesis can be performed based on this adapted model using a small amount of target corpus and label information. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the steps of a speech synthesis method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the backbone network structure in an embodiment of this application;

[0049] Figure 3This is a schematic diagram of the pre-training phase and the low-resource target training phase in the embodiments of this application;

[0050] Figure 4 This is a flowchart illustrating the steps of training a low-resource target using the second backbone network in an embodiment of this application.

[0051] Figure 5 This is a flowchart illustrating the steps for calculating the first loss value in an embodiment of this application.

[0052] Figure 6 This is a flowchart illustrating the steps for calculating the second loss value in an embodiment of this application;

[0053] Figure 7 This is a flowchart illustrating the steps for determining common constraints in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the structure of a speech synthesis device according to an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of a speech synthesis electronic device according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0063] Text-to-Speech (TTS) is a part of human-computer dialogue, aiming to enable machines to converse with humans. TTS involves artificially generating human voices from text; in other words, it's about generating human pronunciation from given text. Sound is a continuous analog signal, while the synthesis process uses computers to simulate it with digital signals. This requires digital signal processing (DSP) of analog signal information.

[0064] Based on this, embodiments of this application provide a speech synthesis method and apparatus, electronic device, and storage medium. In the speech synthesis of this solution, speaker embedding representations are used to improve the model's coverage of data content learning; that is, by using paired corpora from different speakers to cover more one-to-many mapping relationships between phonemes and synthesized speech. In addition to speaker embedding, the solution also proposes to enhance the speech synthesis quality of low-resource speakers on the adaptation model by using label embedding representations of the reference model.

[0065] This application provides a speech synthesis method, apparatus, electronic device, and storage medium, which are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0066] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. Furthermore, the speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of a speech synthesis method, but is not limited to the above forms.

[0069] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0071] Figure 1 This is an optional flowchart of a speech synthesis method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0072] Step S101: Obtain the first speech material, input the first speech material into the first backbone network, and pre-train the first backbone network to obtain the reference model;

[0073] In this embodiment, the first speech material mainly includes the speech content of a single person speaking. Before constructing the pre-training training set, the collected first speech material undergoes corresponding audio processing. This processing primarily involves extracting phonemes from the speech material and describing it using phoneme sequences. Before inputting the data into the model network, the phonemes in the sequence are vectorized to obtain phoneme vectors (sequences). During training, the first backbone network in this embodiment mainly synthesizes the speech spectrum. The training data input to this backbone network—that is, the text phonemes obtained after phoneme extraction and vectorization in the preceding steps—are encoded and decoded to finally obtain the predicted speech spectrum output. The final trained network model serves as the reference model.

[0074] Specifically, in the embodiments, there is a sufficient amount of text phonemes and speech spectrum data in the pre-training stage. For example, the amount of material data used in the pre-training stage in the embodiments may include 30,000 speech pairs, all of which are Chinese datasets created by manual annotation.

[0075] It should be noted that the backbone network model in this embodiment can be a transformer model, an LSTM model, or a GPT-2 model, etc.; for example Figure 2 As shown, taking the Transformer model as an example, the Transformer model includes an encoder and a decoder. During the pre-training phase, the main process of this backbone network model includes: first, acquiring text phonemes from the training set and inputting them into the Transformer model; second, encoding the phoneme content using the encoder to obtain hidden feature vectors; third, decoding these hidden feature vectors using the decoder, multiplying them by other reference feature vectors, such as speaker features and location features, to obtain a descriptive segment of the speech spectrum; and finally, decoding this descriptive segment in the last layer to obtain the complete speech spectrum content, which is then output. In addition, in this embodiment, the similarity between the sample video description segment and the reference video description segment is calculated using the loss function of the transformer model. The loss function of the transformer model is optimized based on the similarity. The model loss of the loss function is backpropagated, and the model parameters are continuously adjusted until the similarity is greater than or equal to the similarity threshold. The optimization of the transformer model is then stopped, and a backbone network model that meets the requirements is obtained, which is the reference model in this embodiment.

[0076] Step S102: Construct a second backbone network based on the first model parameters of the reference model;

[0077] In an embodiment, such as Figure 3 As shown, the second backbone network is identical to the first backbone network in the embodiment. The difference lies in the training phase they participate in: the first backbone network is the backbone network used in the pre-training phase, while the second backbone network is the backbone network used in the low-resource-target training phase. Since the structure of the backbone networks is exactly the same, the structure of the second backbone network will not be described in detail.

[0078] Specifically, in the embodiment, before the low-resource target training stage, the embodiment first initializes the model parameters of the backbone network according to the model parameters of the reference model when initializing the backbone network, based on the reference model obtained in the pre-training stage.

[0079] Step S103: Obtain the second speech material, input the second speech material into the second backbone network, and perform low-resource target training on the second backbone network to obtain the adaptation model;

[0080] The second audio material includes speech content from multiple people. Unlike the first audio material obtained in step S101, which primarily contains speech content from a single person, the second audio material mainly contains speech content from multiple people conversing (speaking). Specifically, in this embodiment, the purpose of low-resource target training is to enable the model to perform speech recognition and feature extraction from the speech content from multiple people speaking, and to output the speech spectrum through corresponding encoding and decoding processes. The output speech spectrum is a single-tone attribute (single person) speech spectrum. In this embodiment, the network model obtained after the backbone network completes low-resource target training is the adaptation model.

[0081] Specifically, in this embodiment, step S103 mainly involves guiding the low-resource target training process based on the pre-trained reference model. During the low-resource target training process, the same backbone network as in the training phase is first used. The parameters of the reference model obtained in the pre-training phase are used to initialize the network parameters during backbone network initialization. The input data is also a text phoneme sequence, and the output is the speech spectrum of the text corresponding to different speakers. When the speakers are different, the speech activates the speaker encoding module in the backbone network, and the adaptation model can use different encodings for representation. The difference between the two phases is that in the pre-training phase, there is only a single speaker, and the corresponding speaker encoding remains constant.

[0082] Step S104: Input the target text material into the adaptation model, and perform speech synthesis through the adaptation model to obtain the target speech spectrum;

[0083] The target text material refers to the speech material file that needs to be synthesized (or audio synthesized). This speech material file can be the speech content of a single person or the speech content of a human dialogue. Specifically, in this embodiment, based on the adaptation model obtained after the low-resource target training phase, the acquired target text material is input into this adaptation model. The encoder in the adaptation model, such as an encoder composed of 4 layers of feedforward transformer, encodes the text to obtain a phoneme sequence vector. Other attribute feature information, such as energy, duration, and pitch, is extracted from the obtained phoneme sequence vector. This other attribute feature information is used as corresponding feature labels. During the decoding of the phoneme sequence vector to obtain the target speech spectrum, the target speech spectrum is labeled using feature labels. Finally, the adaptation model outputs the target speech spectrum with the corresponding feature labels.

[0084] like Figure 4 The diagram shown is a flowchart illustrating the steps of low-resource target training of the second backbone network in the technical solution of this application. In some feasible embodiments, step S103, which involves acquiring the second speech material, inputting the second speech material into the second backbone network, and performing low-resource target training on the second backbone network to obtain the adaptation model, may further include steps S1031-S1035 during implementation.

[0085] S1031, input the second voice material into the second backbone network;

[0086] Specifically, in the low-resource target training process of the embodiment, after obtaining the second speech material, necessary preprocessing is performed on the content of the speech material. For example, the audio format file in the speech material is transcoded to obtain the corresponding text format content. In some feasible ways, the embodiment can directly obtain the text format material content. After obtaining the text format (training) material content, the embodiment preprocesses the text content to obtain the corresponding phoneme sequence, and inputs it into the second backbone network.

[0087] Exemplarily, the phone in the phoneme sequence in the embodiment is the smallest speech unit divided according to the natural attributes of speech. Analyzed based on the pronunciation actions in a syllable, one action constitutes one phone. And the phoneme sequence refers to a sequence composed of several phones. In the process of converting text content into a phoneme sequence, not only is the text converted into phonetic notations, but also since the pronunciation duration and pauses of each phone in speech are different; after converting the text into phonemes in the embodiment, certain prosody information is usually added to help the acoustic model improve the naturalness of the synthesized speech. The added prosody information can be divided into four prosody levels: phoneme (L0), word (L1), breath break (L3), and sentence (L4).

[0088] S1032. Identify the speaker in the second speech material through the second backbone network, and generate a speaker code according to the identification result;

[0089] Specifically in the embodiment, in step S1032, first perform voiceprint recognition on the material content of the voice format of the second speech material, train a voiceprint matching model according to the audio format material in the voice format material, obtain the voiceprint matching model, and then perform voiceprint matching on the audio format material through this voiceprint matching model to determine the speaker, and assign a corresponding speaker label to the material. Among them, the speaker label can also be in the form of a speaker code to label the material. The voiceprint matching (recognition) model adopted in step S1032 of the embodiment can adopt non-parametric models, parametric models, or machine learning models such as artificial neural networks and support vector machines.

[0090] S1033. Extract features from the second speech material through the second backbone network to obtain phoneme embedding features;

[0091] Among them, the phoneme embedding features are the characteristic content of the speech material represented by vectors. Exemplarily, in the embodiment, for the phoneme sequence "ni3 hao3" corresponding to the material text "你好", first vectorize this phoneme sequence to obtain a phoneme vector sequence, and then, in the embodiment, extract features through the embedding layer in the second backbone network to obtain phoneme embedding features; specifically in the embodiment, the embedding layers set in the second backbone network and the first backbone network can be mapped according to the total number of (phoneme vector) words input and the dimension of the feature vector to be represented.

[0092] S1034. Combine the speaker code and the phoneme embedding features for encoding to obtain a speech spectrum material; in the speech spectrum material, each segment of speech spectrum carries a speaker label;

[0093] Specifically, in this embodiment, the speaker encoding obtained in step S1032 and the phoneme embedding features obtained in step S1033 are input into the encoder of the second backbone network for fusion output to obtain the corresponding speech spectrum material. Each speech spectrum in the obtained speech spectrum material carries a speaker label. It should be noted that the first or second speech material in this embodiment also integrates speech spectrum material for training. The speech spectrum material is predicted by the second backbone network during the low-resource target training process. The loss value between the prediction result and the actual result is calculated using a loss function, and the model parameters of the second backbone network are tuned based on this loss value to finally obtain the trained adaptation model.

[0094] S1035, based on the second model parameters of the second backbone network after obtaining the speech spectrum material from the output, determine the adaptation model based on the second model parameters;

[0095] Specifically, in the embodiment, the speech spectrum obtained by the prediction of the second backbone network in step S1034 and the speech spectrum in the training set are used to calculate the loss value through the loss function, and the model parameters of the second backbone network are optimized through the loss value so that the prediction output of the second backbone network is closer to the actual speech spectrum, and the speech spectrum synthesized by the final adaptation model is more in line with the speech spectrum of real people.

[0096] like Figure 5 The diagram shows the flowchart of the steps for calculating the first loss value in the embodiment. In some feasible embodiments, the training process of the model can be a process of continuously converging the loss value by constraining it, thereby optimizing the model parameters. Therefore, in a certain round of training, after obtaining the target speech spectrum through the model, the method obtains the second speech material, inputs the second speech material into the second backbone network, and performs low-resource target training on the second backbone network to obtain the adapted model. Step S1035 may include steps S10351-S10353 in the specific embodiment:

[0097] S10351, the actual speech spectrum was acquired;

[0098] Specifically, in this embodiment, unlike the speaker's speech spectrum integrated in the training dataset (first speech material and second speech material), the embodiment can also obtain the real speech spectrum by collecting the real speech material of the target object and extracting the speech spectrum of this speech material.

[0099] S10352, calculate the first loss value between the real speech spectrum and the target speech spectrum using the mean square loss function;

[0100] The first loss value is the difference between the target speech spectrum synthesized by the adaptation model and the real speech spectrum. This loss value includes, but is not limited to, any one of the regression evaluation metrics, such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Average Error (MAE). Specifically, in the low-resource target training process of the embodiment, in addition to making the generated target speech spectrum as close as possible to the real speech spectrum, taking MSE error as an example, the MSE loss function between the generated spectrum and the real spectrum is denoted as L. hard And satisfy the following calculation formula:

[0101]

[0102] in, It is the Mel spectrum generated by the adaptation model. It is the Mel spectrum corresponding to the real waveform file, and N represents the total number of samples. Here, the samples refer to the real speech spectrum or the target speech spectrum. That is to say, the real speech spectrum and the target speech spectrum should be the same in number.

[0103] S10353, constrain the low-resource target training process of the second backbone network according to the first loss value;

[0104] Specifically, in the embodiment, during the training of the second backbone network, L is minimized. hard The value of L is used to continuously fine-tune the model parameters of the second backbone network as a constraint. In the embodiment, L hard The loss calculated using a neural network is used to optimize the network layer parameters through gradient backpropagation, ultimately resulting in L... hard It continues to decrease until it converges.

[0105] like Figure 6 The diagram shown is a flowchart illustrating the steps for calculating the second loss value in an embodiment. In some feasible embodiments, step S1035, which involves acquiring the second speech material, inputting the second speech material into the second backbone network, and training the second backbone network with low-resource targets to obtain the adapted model, may further include steps S10354-S10357 in specific implementations.

[0106] S10354, The second speech material is vectorized to obtain the first tag embedding vector;

[0107] The first tag embedding vector refers to the tag embedding vector obtained by serializing the phonemes of all text-format (phoneme) material content in the second speech material, extracting features such as keywords through natural language processing, and vectorizing the features such as keywords.

[0108] S10355: Obtain the text sequence at any stage of the low-resource target training process of the second backbone network, and perform vectorization processing on the text sequence to obtain the second label embedding vector.

[0109] Specifically, in the embodiment, the text phoneme sequence of the second speech material is input into the two backbone networks to obtain the intermediate text sequence obtained in the model during the low-resource target training process of the second backbone network, for example, after speaker encoding and before decoding output. The obtained intermediate text sequence is also vectorized and labeled to obtain the second label embedding vector.

[0110] S10356, Calculate the second loss value between the first label embedding vector and the second label embedding vector using the mean square loss function;

[0111] Specifically, in this embodiment, combining the first label embedding vector and the second label embedding vector obtained in steps S10354 and S10355, the MSE loss function between the two label embedding representations can also be calculated and denoted as L. ref This is the second loss value, which is calculated using the following formula:

[0112]

[0113] in, This is the Mel spectrum output by the reference model. L ref The loss is also optimized by backpropagating the gradient to improve the network parameters, thereby making L... ref As small as possible.

[0114] S10357, constrain the encoding process of the low-resource target training process of the second backbone network according to the second loss value;

[0115] Specifically, in the embodiment, the loss value L is minimized. ref As another constraint in the low-resource target training process of the second backbone network, it constrains the model encoding and synthesis processes, so that the target speech spectrum obtained by the synthesized output of the adapted model after training is more in line with the real human voice.

[0116] like Figure 7 The diagram shown is a flowchart illustrating the steps for determining common constraints in an embodiment; in some feasible embodiments, the method proceeds after obtaining the loss value L.hard and loss value L ref Subsequently, the two loss values ​​can be used as common constraints to optimize the training process of the second backbone network. Furthermore, in this embodiment, the step S1035, which involves acquiring the second speech material, inputting the second speech material into the second backbone network, and performing low-resource target training on the second backbone network to obtain the adapted model, may further include steps S10358-S103510:

[0117] S10358, Determine the third loss value based on the product of the second loss value and the harmonic parameter;

[0118] Specifically, in the embodiment, the second loss value L in the low-resource target training process is obtained based on steps S10354-S10356. ref Then, based on the harmonic parameter α determined in the embodiment and the second loss value L ref The product αL ref The third loss value is an intermediate loss value, where the harmonic parameter is a hyperparameter. In the embodiment, it can be determined through multiple experiments that when the harmonic parameter α is set to 0.5, the final synthesized speech effect of the adaptation model can achieve the best results.

[0119] S10359, Determine the common constraint conditions based on the first loss value and the third loss value;

[0120] Specifically, in the embodiment, the adapted model obtained from the final low-resource target training process can be based on the loss value L. hard and loss value L ref The common constraints impose a constraint on the model's training process. The mathematical expression of this common constraint is as follows:

[0121] Loss = L hard +αL ref

[0122] Here, Loss represents the loss value calculated under common constraints.

[0123] S103510, the parameters of the second backbone network in the low-resource target training process are adjusted according to the common constraints to obtain the adapted model;

[0124] For example, in this embodiment, the MSE loss function between the generated spectrum and the real spectrum is first calculated, denoted as L. hard Minimize MSE loss L hafdAdditionally, the text sequence is simultaneously input into the second backbone network to obtain label embedding representations. That is, while keeping the reference model parameters unchanged, the spectrum of the text sequence to be generated is performed, using the input before the decoder module as the label embedding representation. Similarly, the text sequence also obtains label embedding representations during the training of the adaptation model. The MSE loss function between the two label embedding representations can be denoted as L. ref This constrains the content encoding integrity of the trained adaptation model. The final training of the low-resource target adaptation model is performed by L. hard +αL ref Common constraints.

[0125] In some feasible embodiments, the step S1034 of encoding speech spectrum material by combining speaker coding and phoneme embedding features may include steps S10341-S10343:

[0126] S10341, Extract the phoneme sequence from the second speech material;

[0127] In this embodiment, a phoneme sequence is first obtained through the aforementioned phoneme extraction and serialization process; the specific process is similar to the aforementioned steps and will not be described in detail here.

[0128] S10342, the position of the phoneme vector in the phoneme sequence is encoded to obtain the position code;

[0129] In this embodiment, positional encoding encodes the position of phonemes within a phoneme sequence. For example, when transcoding speech material to text material, the resulting text content is "Xiaoming loves to eat apples, Xiaohong bought a new Apple phone." First, this is converted into a corresponding phoneme sequence, then mapped into a space where each phoneme is represented by a vector. Positional encoding can simply use the phoneme positions 1, 2, 3, 4, 5, 6 for positional encoding, and then add this to the vector representation. Therefore, the phoneme sequence feature representations corresponding to the first "apple" and the second "apple" in the text content are not the same variables.

[0130] More specifically, the specific position encoding method used in this embodiment is as follows: if the current position is an odd number, such as 1, 3, 5, or 7, the encoding is obtained using the following calculation formula:

[0131]

[0132] For even-numbered positions, the following calculation formula is used:

[0133]

[0134] Where t is the phoneme vector at the corresponding position, d is the dimension of the vector, and k is the integer part of the corresponding position divided by 2.

[0135] S10343, Based on the position coding and phoneme embedding features, the position of the phoneme vector is marked, and the position-marked phoneme embedding features are encoded with the speaker coding to obtain the speech spectrum material;

[0136] The positional encoding obtained through step S10342 is used to mark the position of the phoneme vectors in the phoneme sequence. Combined with the speaker encoding and the embedding features of the phoneme sequence, the phoneme sequence features are input into the encoder to encode the phoneme sequence features.

[0137] In some feasible embodiments, the method includes steps S104, which involves inputting the target text material into an adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum. In practice, this step may include steps S1041-S1045:

[0138] S1041, an audio attribute prediction model is obtained by training based on the first speech material and / or the second speech material;

[0139] The audio attribute prediction models include energy prediction models, duration prediction models, and pitch information prediction models. For example... Figure 2 As shown, in this specific embodiment, after transcoding the speech material in the training dataset, the text input to the second backbone network is first preprocessed to obtain the corresponding phoneme sequence. Then, combined with position encoding and speaker encoding, it is input into the encoder, which consists of four feedforward transformer layers. The encoded vector is input into three different predictors to predict energy, duration, and pitch, respectively. The training process of the predictors is designed to process the energy extraction, pitch extraction, and duration annotation information of the speech audio. During the prediction process, the corresponding energy, duration, and pitch information are directly predicted and fused with the encoder output before being input into the decoder. Here, the decoder uses two two-dimensional convolutional layers, each with 256 3*3 convolutional kernels. It can be understood that the adapted model obtained after completing the low-resource target training in this embodiment may include sub-models for energy prediction, duration prediction, and pitch information prediction.

[0140] S1042, Input the target text material into the adaptation model, encode it to obtain intermediate audio material;

[0141] Specifically, in the implementation example, the adaptation model is applied to the synthesis scenario of the target speech spectrum. First, the target text material is processed in the necessary preprocessing to obtain the corresponding phoneme sequence. Similar to the training stage, the embedding features of the phoneme sequence are obtained, and combined with position encoding and speaker encoding, they are input into the encoder to encode the intermediate audio material.

[0142] S1043, Input the intermediate audio material into the energy prediction model to predict the energy information of the speech spectrum;

[0143] S1044, Input the intermediate audio material into the duration prediction model to predict the duration information of the speech spectrum;

[0144] S1045, input the intermediate audio material into the pitch information prediction model to predict the pitch information of the speech spectrum;

[0145] It should be noted that the prediction processes for the energy, duration, and pitch of the speech spectrum in this embodiment can be performed in parallel, without a logical sequential order. Specifically, in this embodiment, the three predictors use the same network structure: a four-layer CNN network plus two-layer LSTM network layers. However, the predictors have different functions, namely, predicting energy, duration, and pitch information separately. The training process and the backbone network are jointly trained together; each predictor corresponds to a loss function, and the losses are weighted and added to the total loss function. Joint training is achieved by optimizing the total loss. In the training set, duration, energy, and pitch information can be extracted from audio samples separately. During the prediction process, there are no real labels; the prediction capabilities learned during training are used to output the corresponding energy, duration, and pitch.

[0146] Refer to the instruction manual appendix Figure 2 and attached Figure 3 The implementation process of the embodiments of the technical solution of this application is described in full as follows:

[0147] The implementation provides a multi-speaker target domain adaptation speech synthesis method; in speech synthesis, speaker embedding representation is used to improve the model's coverage of data content learning, that is, to cover more one-to-many mapping relationships between phonemes and synthesized speech through paired corpora of different speakers; in addition to speaker embedding, the scheme also proposes a reference model's label embedding representation to enhance the speech synthesis quality of low-resource speakers on the adaptation model.

[0148] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-mentioned speech synthesis method. The apparatus includes:

[0149] The pre-training unit is used to acquire the first speech material, input the first speech material into the first backbone network, and pre-train the first backbone network to obtain a reference model; the first speech material includes single-person speech content;

[0150] The model building unit is used to construct a second backbone network based on the first model parameters of the reference model.

[0151] The low-resource target training unit is used to acquire the second speech material, input the second speech material into the second backbone network, and perform low-resource target training on the second backbone network to obtain the adaptation model; the second speech material includes speech content of multiple people speaking;

[0152] The speech spectrum synthesis unit is used to input the target text material into the adaptation model, and then perform speech synthesis through the adaptation model to obtain the target speech spectrum.

[0153] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0154] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0155] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0156] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 according to an embodiment of this application for speech synthesis.

[0157] The input / output interface 903 is used to implement information input and output;

[0158] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0159] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0160] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0161] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described speech synthesis method.

[0162] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0163] The speech synthesis method provided in this application involves obtaining a reference model from the backbone network of the synthesized speech spectrum through a pre-training process, and then performing low-resource target training based on the reference model to obtain an adapted model. By splitting the model training process into a pre-training stage and a low-resource target stage, and based on the reference model, a large amount of pre-processing steps for model building can be eliminated, simplifying the processing flow and improving processing efficiency. Furthermore, the pre-training process focuses on recognizing single-person speech content, while the low-resource target training process focuses on recognizing multi-person speech content. After obtaining the target adapted model, high-quality speech synthesis can be performed based on this adapted model using a small amount of target corpus and label information.

[0164] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0165] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0168] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0169] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: A first speech sample is acquired, and the first speech sample is input into a first backbone network. The first backbone network is pre-trained to obtain a reference model. The first speech sample includes single-person speech content. Construct a second backbone network based on the first model parameters of the reference model; The second speech material is acquired and input into the second backbone network. The second backbone network is then trained with low-resource targets to obtain an adapted model. The second speech material includes speech content from multiple speakers. The target text material is input into the adaptation model, and speech synthesis is performed through the adaptation model to obtain the target speech spectrum; Collect real speech material of the target object, and extract the speech spectrum from the real speech material to obtain the real speech spectrum; The first loss value between the real speech spectrum and the target speech spectrum is calculated using the mean square loss function. The training process for low-resource targets in the second backbone network is constrained based on the first loss value; The step of acquiring the second speech material, inputting the second speech material into the second backbone network, and performing low-resource target training on the second backbone network to obtain the adaptation model includes: The second audio material is input into the second backbone network; The second backbone network is used to perform voiceprint recognition on the speaker in the second speech material, and a speaker code is generated based on the recognition result; wherein, the speaker code feature is a speaker label used to distinguish different speakers; The second speech material is vectorized through the second backbone network to obtain a phoneme vector sequence. Feature extraction is performed on the phoneme vector sequence to obtain phoneme embedding features. The speech spectrum material is obtained by combining the speaker encoding and the phoneme embedding features; the speech spectrum in the speech spectrum material carries a speaker tag. The second model parameters of the second backbone network after outputting the speech spectrum material are obtained, and the adaptation model is determined based on the second model parameters.

2. The speech synthesis method according to claim 1, characterized in that, After the step of inputting the target text material into the adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum, the method includes: The second speech material is vectorized to obtain the first tag embedding vector; Obtain the text sequence at any stage of the low-resource target training process of the second backbone network, and perform vectorization processing on the text sequence to obtain the second label embedding vector; The second loss value between the first label embedding vector and the second label embedding vector is calculated using the mean square loss function; The encoding process of the low-resource target training process of the second backbone network is constrained based on the second loss value.

3. The speech synthesis method according to claim 2, characterized in that, After the step of inputting the target text material into the adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum, the method includes: The third loss value is determined based on the product of the second loss value and the harmonic parameter; Based on the first loss value and the third loss value, determine the common constraint conditions; The parameters of the second backbone network during the low-resource target training process are adjusted according to the common constraints to obtain the adapted model.

4. The speech synthesis method according to claim 1, characterized in that, The step of encoding the speech spectrum material by combining the speaker encoding and the phoneme embedding features includes: A phoneme sequence was extracted from the second speech material; The position of the phoneme vector in the phoneme sequence is encoded to obtain the position code; The phoneme vectors are marked with their positions based on the position encoding and the phoneme embedding features. The marked phoneme embedding features are then encoded with the speaker encoding to obtain the speech spectrum material.

5. A speech synthesis method according to any one of claims 1-4, characterized in that, The step of inputting the target text material into the adaptation model and performing speech synthesis through the adaptation model to obtain the target speech spectrum includes: An audio attribute prediction model is obtained by training based on the first speech material and / or the second speech material; the audio attribute prediction model includes an energy prediction model, a duration prediction model, and a pitch information prediction model. The target text material is input into the adaptation model and encoded to obtain intermediate audio material. The intermediate-state audio material is input into the energy prediction model to predict the energy information of the speech spectrum. The intermediate audio material is input into the duration prediction model to predict the duration information of the speech spectrum; The intermediate audio material is input into the pitch information prediction model to predict the pitch information of the speech spectrum.

6. A speech synthesis device, characterized in that, The device includes: The pre-training unit is used to acquire the first speech material, input the first speech material into the first backbone network, and pre-train the first backbone network to obtain a reference model; the first speech material includes single-person speaking speech content; A model building unit is used to build a second backbone network based on the first model parameters of the reference model; The low-resource target training unit is used to acquire the second speech material, input the second speech material into the second backbone network, and perform low-resource target training on the second backbone network to obtain the adaptation model; the second speech material includes speech content of multiple people speaking; The speech spectrum synthesis unit is used to input the target text material into the adaptation model, and perform speech synthesis through the adaptation model to obtain the target speech spectrum; Collect real speech material of the target object, and extract the speech spectrum from the real speech material to obtain the real speech spectrum; The first loss value between the real speech spectrum and the target speech spectrum is calculated using the mean square loss function. The training process for low-resource targets in the second backbone network is constrained based on the first loss value; The step of acquiring the second speech material, inputting the second speech material into the second backbone network, and performing low-resource target training on the second backbone network to obtain the adaptation model includes: The second audio material is input into the second backbone network; The second backbone network is used to perform voiceprint recognition on the speaker in the second speech material, and a speaker code is generated based on the recognition result; wherein, the speaker code feature is a speaker label used to distinguish different speakers; The second speech material is vectorized through the second backbone network to obtain a phoneme vector sequence. Feature extraction is performed on the phoneme vector sequence to obtain phoneme embedding features. The speech spectrum material is obtained by combining the speaker encoding and the phoneme embedding features; the speech spectrum in the speech spectrum material carries a speaker tag. The second model parameters of the second backbone network after outputting the speech spectrum material are obtained, and the adaptation model is determined based on the second model parameters.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 5.

8. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 5.

Images