Speech synthesis method, apparatus, device, and storage medium

Abstract

The present disclosure provides a speech synthesis method, device, equipment and storage medium. The method comprises the following steps: preprocessing a to-be-processed text to obtain a phoneme sequence corresponding to the to-be-processed text; inputting the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence; and inputting the target speech frame into a second acoustic model to generate a target speech point. The method can solve the problem that the duration of traditional speech synthesis cannot be controlled, and can realize the synthesis effect of different people and various types of languages while ensuring high-quality sound quality.

Description

Technical Field

[0001] This disclosure relates to the field of speech synthesis, and in particular to a speech synthesis method, apparatus, device, and storage medium. Background Technology

[0002] Text-to-Speech (TTS) is a technology that transforms computer-generated or externally input text information into fluent, understandable human speech. As TTS technology advances in most scenarios, the demand for cross-language communication is growing rapidly.

[0003] Currently, most cross-language methods are trained by having speakers record their voices in different languages, but this approach has limitations. For example, it requires speakers to have a high level of fluency in each language, which increases the difficulty of recording and raises the time, money, and manpower costs. Secondly, new speakers cannot be used for cross-language transfer if they only have monolingual data. Summary of the Invention

[0004] This disclosure provides a speech synthesis method, apparatus, device, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0005] According to a first aspect of this disclosure, a speech synthesis method is provided, characterized in that the method comprises:

[0006] The text to be processed is preprocessed to obtain a phoneme sequence corresponding to the text to be processed.

[0007] The phoneme sequence is input into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence, wherein the first acoustic model is a model with an attention mechanism;

[0008] The target speech frame is input into the second acoustic model to generate target speech points.

[0009] In one possible implementation, the step of inputting the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence includes:

[0010] The phoneme sequence is input into the acoustic coding layer within the first acoustic model to obtain a first coding feature corresponding to the phoneme sequence.

[0011] The first encoded feature is input into the speaker-removed network layer to obtain the first encoded feature after removing speaker information and language information, which is used as the second encoded feature.

[0012] The second coding feature is processed according to the predicted phoneme duration or the preset input duration to determine the speech frame number information of each feature in the second coding feature, and the second coding feature with the speech frame number information is used as the target coding feature.

[0013] In the decoding layer within the first acoustic model, target speaker information and target language information are added. The target encoded features are input into the decoding layer to generate target speech frames using the target speaker information and the target language information. The decoding layer is equipped with an attention mechanism.

[0014] In one possible implementation, before processing the encoded features according to the predicted text duration, the method further includes:

[0015] The phoneme sequence is input into the duration model to generate a predicted phoneme duration corresponding to the phoneme sequence.

[0016] In one possible implementation, the phoneme sequence is input into a duration model to generate a predicted phoneme duration corresponding to the phoneme sequence, including:

[0017] The phoneme sequence is input into the duration coding layer in the duration model to obtain the duration coding features corresponding to the phoneme sequence;

[0018] The duration encoding features are input into the duration predictor within the duration model to generate predicted phoneme durations corresponding to each phoneme in the phoneme sequence.

[0019] In one possible implementation, before inputting the phoneme sequence into the first acoustic model, the method further includes:

[0020] The training sample set is obtained by processing the sample audio. The training sample set includes phoneme sample data and its corresponding phoneme duration data, speaker sample data and language sample data.

[0021] The first acoustic model and the second acoustic model are trained based on the phoneme sample data, and the first acoustic model is corrected using standard speech frames to update the model parameters of the first acoustic model, and the second acoustic model is corrected using the sample audio to update the model parameters of the second acoustic model, wherein the standard speech frame is information obtained by extracting the MEL spectrum from the sample audio.

[0022] In one possible implementation, the phoneme sample data is used to train the first acoustic model, and standard speech frames are used for correction to update the model parameters of the first acoustic model, including:

[0023] The phoneme sample data is input into the acoustic coding layer within the first acoustic model to obtain the first coded sample feature corresponding to the phoneme sample data.

[0024] The first encoded sample feature is input into the speaker removal network layer. By increasing the loss function value output by the speaker removal network layer, the parameter information in the speaker removal network layer is adjusted to obtain the first encoded sample feature of the speaker removal sample data and language sample data, which is used as the second encoded sample feature.

[0025] The second encoded sample features are processed according to a preset input duration to determine the speech frame number information of each feature in the second encoded sample features, and the second encoded sample features with the speech frame number information are used as target encoded sample features.

[0026] Speaker sample data and language sample data are added to the decoding layer within the first acoustic model. The target encoded features are input into the decoding layer to generate target sample speech frames using the speaker sample data and the language sample data. The decoding layer is equipped with an attention mechanism.

[0027] The target sample speech frame is corrected using standard speech frames to update the model parameters of the first acoustic model.

[0028] According to a second aspect of this disclosure, a speech synthesis apparatus is provided, characterized in that the apparatus comprises:

[0029] The phoneme acquisition module is used to preprocess the text to be processed to obtain a phoneme sequence corresponding to the text to be processed.

[0030] A speech frame generation module is used to input the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence, wherein the first acoustic model is a model with an attention mechanism;

[0031] The speech point generation module is used to input the target speech frame into the second acoustic model to generate target speech points.

[0032] In one possible implementation, the voice frame generation module is specifically used for:

[0033] The phoneme sequence is input into the acoustic coding layer within the first acoustic model to obtain a first coding feature corresponding to the phoneme sequence.

[0034] The first encoded feature is input into the speaker-removed network layer to obtain the first encoded feature after removing speaker information and language information, which is used as the second encoded feature.

[0035] The second coding feature is processed according to the predicted phoneme duration or the preset input duration to determine the speech frame number information of each feature in the second coding feature, and the second coding feature with the speech frame number information is used as the target coding feature.

[0036] In the decoding layer within the first acoustic model, target speaker information and target language information are added. The target encoded features are input into the decoding layer to generate target speech frames using the target speaker information and the target language information. The decoding layer is equipped with an attention mechanism.

[0037] In one possible implementation, it further includes: a phoneme duration prediction module, used for:

[0038] Before processing the encoded features based on the predicted text duration, the phoneme sequence is input into the duration model to generate the predicted phoneme duration corresponding to the phoneme sequence.

[0039] In one possible implementation, the phoneme duration prediction module is specifically used for:

[0040] The phoneme sequence is input into the duration coding layer in the duration model to obtain the duration coding features corresponding to the phoneme sequence;

[0041] The duration encoding features are input into the duration predictor within the duration model to generate predicted phoneme durations corresponding to each phoneme in the phoneme sequence.

[0042] In one possible implementation, it further includes: a model training module, used to train the phoneme sequence before inputting it into the first acoustic model.

[0043] The training sample set is obtained by processing the sample audio. The training sample set includes phoneme sample data and its corresponding phoneme duration data, speaker sample data and language sample data.

[0044] The first acoustic model and the second acoustic model are trained based on the phoneme sample data, and the first acoustic model is corrected using standard speech frames to update the model parameters of the first acoustic model, and the second acoustic model is corrected using the sample audio to update the model parameters of the second acoustic model, wherein the standard speech frame is information obtained by extracting the MEL spectrum from the sample audio.

[0045] In one possible implementation, the model training module is specifically used for:

[0046] The phoneme sample data is input into the acoustic coding layer within the first acoustic model to obtain the first coded sample feature corresponding to the phoneme sample data.

[0047] The first encoded sample feature is input into the speaker removal network layer. By increasing the loss function value output by the speaker removal network layer, the parameter information in the speaker removal network layer is adjusted to obtain the first encoded sample feature of the speaker removal sample data and language sample data, which is used as the second encoded sample feature.

[0048] The second encoded sample features are processed according to a preset input duration to determine the speech frame number information of each feature in the second encoded sample features, and the second encoded sample features with the speech frame number information are used as target encoded sample features.

[0049] Speaker sample data and language sample data are added to the decoding layer within the first acoustic model. The target encoded features are input into the decoding layer to generate target sample speech frames using the speaker sample data and the language sample data. The decoding layer is equipped with an attention mechanism.

[0050] The target sample speech frame is corrected using standard speech frames to update the model parameters of the first acoustic model.

[0051] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0052] At least one processor; and

[0053] A memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0055] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0056] The speech synthesis method, apparatus, device, and storage medium disclosed herein preprocess the text to be processed to obtain a phoneme sequence corresponding to the text; input the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence; input the target speech frame into a second acoustic model to generate target speech points. This can solve the problem of uncontrollable duration in traditional speech synthesis and achieve the synthesis effect of multiple types of languages ​​from different people while ensuring high-quality sound.

[0057] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0058] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0059] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0060] Figure 1A A schematic diagram illustrating the implementation flow of a speech synthesis method provided in Embodiment 1 of this disclosure is shown;

[0061] Figure 1B A schematic diagram of the model training process in a speech synthesis method provided in Embodiment 1 of this disclosure is shown;

[0062] Figure 2A A schematic diagram illustrating the implementation flow of a speech synthesis method provided in Embodiment 1 of this disclosure is shown;

[0063] Figure 2B A simplified flowchart of a speech synthesis method provided in Embodiment 1 of this disclosure is shown;

[0064] Figure 3 A schematic diagram of the structure of a speech synthesis device provided in Embodiment 3 of this disclosure is shown;

[0065] Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0066] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0067] Example 1

[0068] Figure 1A This is a flowchart illustrating a speech synthesis method provided in Embodiment 1 of this disclosure. This method can be executed by the speech synthesis device provided in this embodiment, which can be implemented using software and / or hardware. Specifically, the method includes:

[0069] S110. Preprocess the text to be processed to obtain the phoneme sequence corresponding to the text to be processed.

[0070] The text to be processed refers to the text that needs to be synthesized into speech. The phoneme sequence refers to a series of phonemes arranged in the order of the text to be processed, where a phoneme is the smallest unit of articulation.

[0071] Specifically, since the purpose of this embodiment is to synthesize speech information from the text to be processed, this embodiment needs to first convert the text to be processed into a phoneme sequence. For example, the text to be processed can be preprocessed by any application software or applet that can achieve this purpose in the existing technology, or the text to be processed can be converted into a phoneme sequence by engineers writing corresponding code programs on the front end. This embodiment does not limit the specific process of its preprocessing.

[0072] S120. Input the phoneme sequence into the first acoustic model to obtain the target speech frame corresponding to the phoneme sequence.

[0073] The first acoustic model is a model with an attention mechanism. This first acoustic model can be a neural network model with an attention mechanism, implemented at the decoding layer. This attention mechanism allows the first acoustic model to learn the speech patterns of real people, rather than simply outputting monotonous, mechanical language. After learning, the speech synthesized by the first acoustic model can achieve a smoother, more realistic, and rhythmic effect. The target speech frame refers to the number of speech frames corresponding to the phoneme sequence needed for speech synthesis, as well as the duration of each frame. Each frame is approximately 10ms-20ms long.

[0074] Specifically, in this embodiment, the acoustic model for synthesizing the final speech information can be divided into a first acoustic model and a second acoustic model. The first acoustic model may include neural network layers such as an acoustic coding layer, a speaker removal network layer, and a decoding layer, which are used to generate target speech frames corresponding to phoneme sequences.

[0075] S130. Input the target speech frame into the second acoustic model to generate target speech points.

[0076] The target speech point refers to the speech point that corresponds to the phoneme sequence and can be directly stored in the computer, i.e., the final synthesized speech information. The second acoustic model can be a neural network model containing a neural vocoder, used to synthesize the input target speech frame into target speech points.

[0077] The second acoustic model in this embodiment can be trained independently, modeling directly on speech points and generating speech points through speech frames. This eliminates the traditional vocoder approach and enables the sound to achieve a high fidelity effect.

[0078] The disclosed speech synthesis method preprocesses the text to be processed to obtain a phoneme sequence corresponding to the text; inputs the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence; inputs the target speech frame into a second acoustic model to generate target speech points. This method can solve the problem of uncontrollable duration in traditional speech synthesis and achieve the synthesis effect of multiple types of languages ​​from different people while ensuring high-quality sound.

[0079] In this embodiment of the disclosure, before inputting the phoneme sequence into the first acoustic model, the method further includes: acquiring sample audio and processing it to obtain a training sample set, wherein the training sample set includes phoneme sample data and corresponding phoneme duration data, speaker sample data, and language sample data; training the first acoustic model and the second acoustic model based on the phoneme sample data, and correcting the first acoustic model using standard speech frames to update the model parameters of the first acoustic model, and correcting the second acoustic model using sample audio to update the model parameters of the second acoustic model, wherein the standard speech frame is information obtained by extracting the sample audio using MEL spectrum.

[0080] The sample audio can be original audio files containing multiple languages ​​and different speakers, used to train the first and second acoustic models. In the sample audio, each identical speaker uses only one language. The training sample set can be a sample set directly used for training, including phoneme sample data and its corresponding phoneme duration data, speaker sample data, and language sample data. Phoneme sample data refers to the phoneme sequence corresponding to the original sample audio; phoneme duration data refers to the duration data labeled for each phoneme in the phoneme sample data; speaker sample data can be labeled speaker information, such as a person's name, and data containing that speaker's timbre; language sample data refers to labeled language data, such as Chinese, English, French, etc. The standard speech frame refers to the relatively accurate speech frame parameter information about the sample audio obtained by processing the sample audio. For example, the standard speech frame can be the information obtained by extracting the MEL spectrum from the sample audio.

[0081] Specifically, after obtaining high-quality sample audio, this embodiment can decode the sample audio to obtain a training sample set, which can then be manually labeled and corrected; alternatively, the sample audio can be directly manually labeled to obtain a training sample set, which contains all information related to the sample audio. This embodiment then uses the training sample set to train the first acoustic model and the second acoustic model, continuously correcting the model parameters of the first and second acoustic models to obtain a mature acoustic model.

[0082] In this embodiment, a first acoustic model is trained using phoneme sample data and corrected using standard speech frames to update the model parameters of the first acoustic model. This includes: inputting phoneme sample data into the acoustic coding layer of the first acoustic model to obtain first coded sample features corresponding to the phoneme sample data; inputting the first coded sample features into a speaker removal network layer, adjusting the parameter information within the speaker removal network layer by increasing the loss function value output by the speaker removal network layer to obtain first coded sample features that remove speaker sample data and language sample data, which are then used as second coded sample features; processing the second coded sample features according to a preset input duration to determine the speech frame number information for each feature in the second coded sample features, and using the second coded sample features with the speech frame number information as target coded sample features; adding speaker sample data and language sample data to the decoding layer within the first acoustic model, inputting the target coded features into the decoding layer to generate target sample speech frames using speaker sample data and language sample data, wherein the decoding layer has an attention mechanism; and correcting the target sample speech frames using standard speech frames to update the model parameters of the first acoustic model.

[0083] The first encoded sample feature refers to the abstract encoded features of phoneme sample data extracted through the acoustic coding layer. The second encoded sample feature refers to the abstract encoded features that do not contain speaker sample data or language sample data. The target encoded sample feature refers to the abstract encoded feature labeled with speech frame number information. The target sample speech frame can be speech frame data corresponding to the phoneme sample data generated by adding the required speaker sample data and language sample data to the target encoded sample features. The preset input duration is any duration information set according to actual needs. The coding layer can encode some kind of data; for example, the acoustic coding layer encodes phoneme sequences and learns abstract knowledge of phonemes.

[0084] Specifically, in this embodiment, phoneme sample data is input into the acoustic coding layer within the first acoustic model to obtain the first coded sample features corresponding to the phoneme sample data. Since this embodiment aims to enable the first acoustic model to generate speech effects from different speakers and languages ​​using the same phoneme sample data, it is necessary to remove the original speaker and language information contained in the phoneme sample data during the early training phase. Specifically, this embodiment sets a speaker removal network layer after the acoustic coding layer. This speaker removal network layer includes a predictor, used to output the first coded sample features that do not contain speaker and language sample data, and to output a loss function related to the speaker removal network layer. Furthermore, since removing the speaker network layer in the early training phase cannot achieve the effect of removing speaker and language sample data, this embodiment needs to continuously adjust the parameter information of the speaker removal network layer by continuously increasing the loss function value to obtain a first coded sample feature unrelated to the speaker, i.e., the second coded sample feature. Because speaker and language sample data are strongly correlated, this embodiment implicitly removes language sample data by removing speaker sample data.

[0085] Secondly, this embodiment can process the second encoded sample features according to a preset input duration to determine the speech frame count information of each feature in the second encoded sample features. This embodiment can control the speech frame count information of each feature in the second encoded sample features to be the same by setting the same preset input duration, or it can control the speech frame count information of each feature in the second encoded sample features to be different by setting different preset input durations. Finally, this embodiment uses the second encoded sample features with speech frame count information as target encoded sample features and inputs them into the decoding layer. Since this embodiment has removed speaker sample data and language sample data in the early stage, speaker sample data and language sample data need to be added to the decoding layer to generate target sample speech frames using speaker sample data and language sample data. Only then can the generation effect of the target sample speech frames be verified by standard speech frames, and the model parameters of the first acoustic model are adjusted by continuous correction until the target sample speech frames generated by the first acoustic model are almost the same as the standard speech frames.

[0086] Finally, after training the first acoustic model, this embodiment inputs the target sample speech frame into the second acoustic model to train it. Specifically, the second acoustic model outputs target speech points by inputting the target sample speech frame. Since the sample audio is composed of speech points, if the second acoustic model learns well, the generated target speech points will be very similar to the sample audio. Therefore, this embodiment uses sample audio to verify the generation effect of the target speech points. By continuously correcting, the model parameters of the second acoustic model are adjusted until a mature second acoustic model is obtained, and the training ends. A detailed training process diagram can be seen as follows. Figure 1B As shown.

[0087] This embodiment uses an end-to-end model to train the first and second acoustic models, and introduces a speaker removal network layer, which can achieve good cross-language transfer effect. Furthermore, the duration of the generated speech can be manually set during the generation process to control the generated speech, thus solving the problem of uncontrollable speech duration in the prior art.

[0088] Example 2

[0089] Figure 2A This is a flowchart of a speech synthesis method provided in Embodiment 2 of this disclosure. Based on the above embodiments, this embodiment further includes inputting a phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence. The method comprises: inputting the phoneme sequence into an acoustic coding layer within the first acoustic model to obtain a first coding feature corresponding to the phoneme sequence; inputting the first coding feature into a speaker removal network layer to obtain a first coding feature with speaker and language information removed, which serves as a second coding feature; processing the second coding feature according to a predicted phoneme duration or a preset input duration to determine the speech frame number information for each feature in the second coding feature, and using the second coding feature with the speech frame number information as a target coding feature; adding target speaker information and target language information to a decoding layer within the first acoustic model, inputting the target coding feature into the decoding layer, and generating a target speech frame using the target speaker information and the target language information. The decoding layer includes an attention mechanism. Specifically, this method includes:

[0090] S210. Preprocess the text to be processed to obtain the phoneme sequence corresponding to the text to be processed.

[0091] S220. Input the phoneme sequence into the acoustic coding layer within the first acoustic model to obtain the first coding feature corresponding to the phoneme sequence.

[0092] The first coding feature refers to the abstract coding feature of the phoneme sequence data extracted through the acoustic coding layer.

[0093] S230. Input the first coding feature into the speaker-removed network layer to obtain the first coding feature after removing speaker information and language information, which is used as the second coding feature.

[0094] The second coding feature refers to the abstract coding feature that does not contain speaker information or language information.

[0095] Since the first acoustic model contains speaker information and language information during the training process, after the phoneme sequence is input into the acoustic encoding layer in this embodiment, the obtained first encoded feature may inevitably carry speaker information and language information. Therefore, in this embodiment, the first encoded feature is input into the speaker removal network layer to obtain a first encoded feature that definitely does not contain speaker information and language information, which is used as the second encoded feature.

[0096] S240. Process the second encoded feature according to the predicted phoneme duration or the preset input duration, determine the speech frame number information of each feature in the second encoded feature, and use the second encoded feature with the speech frame number information as the target encoded feature.

[0097] Among them, the target encoded feature refers to an abstract encoded feature marked with speech frame number information. The predicted phoneme duration is the duration information of each phoneme obtained by processing the phoneme sequence.

[0098] In this embodiment, the duration of the target encoded feature can be controlled by using the predicted phoneme duration predicted by the duration model, or by the preset input duration input manually. Whether processing the second encoded feature according to the predicted phoneme duration or the preset input duration, the purpose of this embodiment is to make the second encoded feature carry data with duration labels. For example, the two characters "today" each occupy 5 speech frames and 4 speech frames in the speech.

[0099] In the embodiment of the present disclosure, before processing the encoded feature according to the predicted text duration, it further includes: inputting the phoneme sequence into the duration model to generate a predicted phoneme duration corresponding to the phoneme sequence. <0000​​​​​​​​​​Specifically, the duration model in this embodiment may include a duration coding layer and a duration predictor. The phoneme sequence is input into the duration coding layer to obtain duration coding features corresponding to the phoneme sequence. The duration coding features are then input into the duration predictor to generate predicted phoneme durations corresponding to each phoneme in the phoneme sequence.

[0104] It should be noted that the duration model in this embodiment can also be a model trained based on phoneme sample data in the training sample set of this application before use. For example, the phoneme sample data can be input into the duration encoding layer of the duration model to obtain duration encoding sample features corresponding to the phoneme sample data. Then, the duration encoding sample features can be input into the duration predictor to obtain the duration corresponding to each phoneme in the phoneme sample data. Finally, the model parameters of the duration model can be adjusted by correcting the phoneme duration data in the training sample set.

[0105] S250. Add target speaker information and target language information to the decoding layer in the first acoustic model, input the target encoding features into the decoding layer, and generate a target speech frame using the target speaker information and the target language information.

[0106] The decoding layer incorporates an attention mechanism.

[0107] The target speech frame refers to the speech frame data corresponding to the phoneme sequence generated by adding the required speaker and language information to the target encoded features. The target speaker information can be information with arbitrary vocal characteristics set according to requirements, and the target language information can be information with arbitrary language characteristics set according to requirements.

[0108] Specifically, this embodiment incorporates target speaker embedding and target language information embedding during the decoding stage to generate corresponding target speech frames. This means it can generate target speech frames that utilize both the target speaker information and the target language information. A simplified process diagram can be found in the provided image. Figure 2B Embedding refers to a technique for multidimensionalizing information. For example, when embedding language information, embedding networks are used to expand information such as Chinese or English into 128-dimensional numbers.

[0109] S260. Input the target speech frame into the second acoustic model to generate the target speech points.

[0110] This embodiment effectively solves the problem of the inability to finely control the duration of traditional speech synthesis, and can achieve high-quality, near-human sound quality while maintaining controllable duration. Furthermore, by introducing speaker information and language information, this embodiment enables the model to synthesize speech containing other language information even when the speaker speaks only a single language during training.

[0111] Example 3

[0112] Figure 3 This is a schematic diagram of a speech synthesis method apparatus provided in an embodiment of the present disclosure. The apparatus specifically includes:

[0113] The phoneme acquisition module 310 is used to preprocess the text to be processed to obtain a phoneme sequence corresponding to the text to be processed.

[0114] The speech frame generation module 320 is used to input the phoneme sequence into the first acoustic model to obtain the target speech frame corresponding to the phoneme sequence, wherein the first acoustic model is a model with an attention mechanism;

[0115] The speech point generation module 330 is used to input the target speech frame into the second acoustic model to generate target speech points.

[0116] In one embodiment, the speech frame generation module 320 is specifically used for: inputting a phoneme sequence into the acoustic coding layer within a first acoustic model to obtain a first coding feature corresponding to the phoneme sequence; inputting the first coding feature into a speaker removal network layer to obtain a first coding feature with speaker information and language information removed, which is used as a second coding feature; processing the second coding feature according to the predicted phoneme duration or a preset input duration to determine the speech frame number information of each feature in the second coding feature, and using the second coding feature with the speech frame number information as the target coding feature; adding target speaker information and target language information to the decoding layer within the first acoustic model, inputting the target coding feature into the decoding layer, and generating a target speech frame using the target speaker information and the target language information, wherein the decoding layer is equipped with an attention mechanism.

[0117] In one possible implementation, the system further includes a phoneme duration prediction module, which is used to input the phoneme sequence into the duration model before processing the encoding features according to the predicted text duration, and generate a predicted phoneme duration corresponding to the phoneme sequence.

[0118] In one possible implementation, the phoneme duration prediction module is specifically used to: input the phoneme sequence into the duration coding layer within the duration model to obtain duration coding features corresponding to the phoneme sequence; input the duration coding features into the duration predictor within the duration model to generate predicted phoneme durations corresponding to each phoneme in the phoneme sequence.

[0119] In one possible implementation, the system further includes: a model training module, configured to acquire sample audio and process it to obtain a training sample set before inputting the phoneme sequence into the first acoustic model, wherein the training sample set includes phoneme sample data and corresponding phoneme duration data, speaker sample data, and language sample data; train the first acoustic model and the second acoustic model based on the phoneme sample data, and correct the first acoustic model using standard speech frames to update the model parameters of the first acoustic model, and correct the second acoustic model using sample audio to update the model parameters of the second acoustic model, wherein the standard speech frame is information obtained by extracting the MEL spectrum from the sample audio.

[0120] In one embodiment, the model training module is specifically used for: inputting phoneme sample data into the acoustic coding layer of the first acoustic model to obtain first coded sample features corresponding to the phoneme sample data; inputting the first coded sample features into the speaker removal network layer, adjusting the parameter information in the speaker removal network layer by increasing the loss function value output by the speaker removal network layer, so as to obtain the first coded sample features of speaker sample data and language sample data removed, which are used as second coded sample features; processing the second coded sample features according to a preset input duration, determining the speech frame number information of each feature in the second coded sample features, and using the second coded sample features with speech frame number information as target coded sample features; adding speaker sample data and language sample data to the decoding layer in the first acoustic model, inputting the target coded features into the decoding layer, generating target sample speech frames using speaker sample data and language sample data, wherein the decoding layer is equipped with an attention mechanism; and correcting the target sample speech frames using standard speech frames to update the model parameters of the first acoustic model.

[0121] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0122] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0124] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as speech synthesis methods. For example, in some embodiments, the speech synthesis method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the speech synthesis method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the speech synthesis method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0131] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0132] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0134] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: The text to be processed is preprocessed to obtain a phoneme sequence corresponding to the text to be processed. The phoneme sequence is input into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence, wherein the first acoustic model is a model with an attention mechanism; The target speech frame is input into the second acoustic model to generate target speech points; The step of inputting the phoneme sequence into the first acoustic model to obtain the target speech frame corresponding to the phoneme sequence includes: The phoneme sequence is input into the acoustic coding layer within the first acoustic model to obtain a first coding feature corresponding to the phoneme sequence. The first encoded feature is input into the speaker-removed network layer to obtain the first encoded feature after removing speaker information and language information, which is used as the second encoded feature. The second coding feature is processed according to the predicted phoneme duration or the preset input duration to determine the speech frame number information of each feature in the second coding feature, and the second coding feature with the speech frame number information is used as the target coding feature. In the decoding layer within the first acoustic model, target speaker information and target language information are added. The target encoded features are input into the decoding layer to generate target speech frames using the target speaker information and the target language information. The decoding layer is equipped with an attention mechanism.

2. The method according to claim 1, characterized in that, Before processing the encoded features based on the predicted text duration, the process also includes: The phoneme sequence is input into the duration model to generate a predicted phoneme duration corresponding to the phoneme sequence.

3. The method according to claim 2, characterized in that, Inputting the phoneme sequence into a duration model to generate predicted phoneme durations corresponding to the phoneme sequence includes: The phoneme sequence is input into the duration coding layer in the duration model to obtain the duration coding features corresponding to the phoneme sequence; The duration encoding features are input into the duration predictor within the duration model to generate predicted phoneme durations corresponding to each phoneme in the phoneme sequence.

4. The method according to claim 3, characterized in that, Before inputting the phoneme sequence into the first acoustic model, the following is also included: The training sample set is obtained by processing the sample audio. The training sample set includes phoneme sample data and its corresponding phoneme duration data, speaker sample data and language sample data. The first acoustic model and the second acoustic model are trained based on the phoneme sample data, and the first acoustic model is corrected using standard speech frames to update the model parameters of the first acoustic model, and the second acoustic model is corrected using the sample audio to update the model parameters of the second acoustic model, wherein the standard speech frame is information obtained by extracting the MEL spectrum from the sample audio.

5. The method according to claim 4, characterized in that, The first acoustic model is trained using the phoneme sample data and corrected using standard speech frames to update the model parameters of the first acoustic model, including: The phoneme sample data is input into the acoustic coding layer within the first acoustic model to obtain the first coded sample feature corresponding to the phoneme sample data. The first encoded sample feature is input into the speaker removal network layer. By increasing the loss function value output by the speaker removal network layer, the parameter information in the speaker removal network layer is adjusted to obtain the first encoded sample feature of the speaker removal sample data and language sample data, which is used as the second encoded sample feature. The second encoded sample features are processed according to a preset input duration to determine the speech frame number information of each feature in the second encoded sample features, and the second encoded sample features with the speech frame number information are used as target encoded sample features. Speaker sample data and language sample data are added to the decoding layer within the first acoustic model. The target encoded features are input into the decoding layer to generate target sample speech frames using the speaker sample data and the language sample data. The decoding layer is equipped with an attention mechanism. The target sample speech frame is corrected using standard speech frames to update the model parameters of the first acoustic model.

6. A speech synthesis device, characterized in that, The device includes: The phoneme acquisition module is used to preprocess the text to be processed to obtain a phoneme sequence corresponding to the text to be processed. A speech frame generation module is used to input the phoneme sequence into a first acoustic model to obtain a target speech frame corresponding to the phoneme sequence, wherein the first acoustic model is a model with an attention mechanism; The speech point generation module is used to input the target speech frame into the second acoustic model to generate target speech points; The speech frame generation module is specifically used for: The phoneme sequence is input into the acoustic coding layer within the first acoustic model to obtain a first coding feature corresponding to the phoneme sequence. The first encoded feature is input into the speaker-removed network layer to obtain the first encoded feature after removing speaker information and language information, which is used as the second encoded feature. The second coding feature is processed according to the predicted phoneme duration or the preset input duration to determine the speech frame number information of each feature in the second coding feature, and the second coding feature with the speech frame number information is used as the target coding feature. In the decoding layer within the first acoustic model, target speaker information and target language information are added. The target encoded features are input into the decoding layer to generate target speech frames using the target speaker information and the target language information. The decoding layer is equipped with an attention mechanism.

7. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images