Speech synthesis model establishment method, application method, device and storage medium

Abstract

The present invention discloses a method for establishing a speech synthesis model, an application method, a device, and a storage medium. The method for establishing the speech synthesis model includes: obtaining an initial character string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer; inputting the initial character string into the character layer to perform model training on the character layer, and obtaining character latent variables output by the character layer; inputting the initial phonemes and character latent variables into the phoneme layer to perform model training on the phoneme layer, and obtaining phoneme latent variables output by the phoneme layer; and inputting the initial acoustic features and phoneme latent variables into the acoustic feature layer to perform model training on the acoustic feature layer, thereby obtaining a speech synthesis model. The method improves the naturalness and synthesis speed of speech synthesized by the speech synthesis model.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a method for establishing a speech synthesis model, an application method, a device and a storage medium. Background Art

[0002] Currently, many end-to-end architectures for English speech synthesis systems achieve excellent sound quality, nearly resembling human pronunciation. However, in non-alphabetic languages such as Chinese and Japanese, end-to-end speech synthesis using text as input is difficult. This is due to the diversity of text and tones. For example, Mandarin Chinese has five tones, including the light tone. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present invention provide a method for establishing a speech synthesis model, an application method, a device and a storage medium, aiming to improve the naturalness and synthesis speed of synthesized speech.

[0005] To achieve the above-mentioned purpose, the first aspect of an embodiment of the present invention proposes a method for establishing a speech synthesis model, comprising: obtaining an initial character string, an initial phoneme, an initial acoustic feature and a basic training model, wherein the basic training model includes a character layer, a phoneme layer and an acoustic feature layer; inputting the initial character string into the character layer so that the character layer undergoes model training, and obtaining the character latent variables output by the character layer; inputting the initial phoneme and the character latent variables into the phoneme layer so that the phoneme layer undergoes model training, and obtaining the phoneme latent variables output by the phoneme layer; inputting the initial acoustic feature and the phoneme latent variables into the acoustic feature layer so that the acoustic feature layer undergoes model training, thereby obtaining a speech synthesis model.

[0006] In some embodiments, the character layer includes a first encoder, a first decoder, and a first converter; inputting the initial character string into the character layer to enable the character layer to perform model training and obtaining the character latent variables output by the character layer includes: inputting the initial character string into the first encoder to obtain an intermediate character string; inputting the intermediate character string into the first decoder to obtain a final character string; obtaining a string loss value based on the initial character string and the final character string, and adjusting the character layer based on the string loss value; inputting the intermediate character string into the first converter to obtain the character latent variables.

[0007] In some embodiments, the phoneme layer includes a second encoder, a second decoder and a second converter; the inputting the initial phoneme and the character latent variable into the phoneme layer so that the phoneme layer can perform the model training and obtaining the phoneme latent variable output by the phoneme layer includes: inputting the initial phoneme into the second encoder to obtain an intermediate phoneme; inputting the intermediate phoneme into the second decoder to obtain a final phoneme; obtaining a phoneme loss value based on the initial phoneme and the final phoneme, and adjusting the phoneme layer based on the phoneme loss value; inputting the character latent variable into the second encoder to perform supervised training on the phoneme layer; inputting the intermediate phoneme into the second converter to obtain a phoneme latent variable.

[0008] In some embodiments, the acoustic feature layer includes a third encoder and a third decoder; inputting the initial acoustic features and the phoneme latent variables into the acoustic feature layer so that the acoustic feature layer performs the model training includes: inputting the initial acoustic features into the third encoder to obtain intermediate acoustic features; inputting the intermediate acoustic features into the third decoder to obtain final acoustic features; obtaining acoustic feature loss values based on the initial acoustic features and the final acoustic features, and adjusting the acoustic feature layer based on the acoustic feature loss values; inputting the phoneme latent variables into the third encoder to perform supervised training on the acoustic feature layer.

[0009] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present invention proposes an application method of a speech synthesis model, wherein the speech synthesis model is established by the establishment method of any one of the embodiments of the first aspect above, and the application method includes: obtaining a target character string or a target phoneme; inputting the target character string or the target phoneme into the speech synthesis model to obtain a speech spectrum.

[0010] In some embodiments, it includes at least one of the following: when the target character string is input into the speech synthesis model to obtain a speech spectrum, the inputting the target character string into the speech synthesis model to obtain a speech spectrum includes: inputting the target character string into the speech synthesis model, and obtaining a speech spectrum via the character layer, the phoneme layer and the acoustic feature layer; when the target phoneme is input into the speech synthesis model to obtain a speech spectrum, the inputting the target phoneme into the speech synthesis model to obtain a speech spectrum includes: inputting the target phoneme into the speech synthesis model, and obtaining the speech spectrum via the phoneme layer and the acoustic feature layer.

[0011] In some embodiments, the character layer is provided with a first converter connected to the phoneme layer, and the phoneme layer is provided with a second converter connected to the acoustic feature layer; when the target character string is input into the speech synthesis model to obtain a speech spectrum, the step of inputting the target character string into the speech synthesis model and obtaining the speech spectrum via the character layer, the phoneme layer and the acoustic feature layer includes: inputting the target character string into the character layer to obtain a character sequence; inputting the character sequence into the phoneme layer so that the phoneme layer outputs a phoneme sequence, and aligning the character sequence and the phoneme sequence through the first converter; inputting the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and aligning the phoneme sequence and the acoustic feature sequence through the second converter; and synthesizing the speech spectrum according to the character sequence, the phoneme sequence and the acoustic feature sequence.

[0012] In some embodiments, the phoneme layer is provided with a second converter connected to the acoustic feature layer; when the target phoneme is input into the speech synthesis model to obtain a speech spectrum, the target phoneme is input into the speech synthesis model, and the speech spectrum is obtained via the phoneme layer and the acoustic feature layer, including: inputting the target phoneme into the phoneme layer to obtain a phoneme sequence; inputting the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and aligning the phoneme sequence and the acoustic feature sequence through the second converter; and synthesizing the speech spectrum according to the phoneme sequence and the acoustic feature sequence.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor runs the computer program, the method for establishing a speech synthesis model as described in any one of the first aspects above and / or the method for applying a speech synthesis model as described in any one of the second aspects above is executed.

[0014] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present invention proposes a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for establishing a speech synthesis model as described in any one of the first aspects above and / or the method for applying a speech synthesis model as described in any one of the second aspects above.

[0015] The beneficial effects of the embodiments of the present invention include: establishing a speech synthesis model through a method for establishing a speech synthesis model, the method comprising obtaining an initial string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model comprises a character layer, a phoneme layer, and an acoustic feature layer; inputting the initial string into the character layer to train the model, and obtaining character latent variables output by the character layer; inputting the initial phonemes and character latent variables into the phoneme layer to train the model, and obtaining phoneme latent variables output by the phoneme layer; inputting the initial acoustic features and phoneme latent variables into the acoustic feature layer to train the model, thereby obtaining a speech synthesis model. The speech synthesis model obtains a target string or target phoneme; and the target string or target phoneme is input into the speech synthesis model to obtain a speech spectrum. This improves the naturalness of speech synthesized by the speech synthesis model and increases the synthesis speed of speech synthesized by the speech synthesis model.

[0016] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0018] Figure 1 Schematic diagram of a system architecture platform for a method for establishing a speech synthesis model and / or an application method of a speech synthesis model provided in an embodiment of the present invention;

[0019] Figure 2 is a flowchart of a method for establishing a speech synthesis model provided by one embodiment of the present invention;

[0020] Figure 3 is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention;

[0021] Figure 4 is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention;

[0022] Figure 5 is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention;

[0023] Figure 6 is a flow chart of a method for applying a speech synthesis model provided by one embodiment of the present invention;

[0024] Figure 7 is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention;

[0025] Figure 8 is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention;

[0026] Figure 9 is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention;

[0027] Figure 10 This is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0030] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0031] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0032] First, some nouns designed in this invention are analyzed:

[0033] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0034] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0035] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0036] Speech synthesis, also known as text-to-speech (TTS) technology, can convert any text into standard, fluent speech in real time, effectively giving a machine an artificial mouth. It involves multiple disciplines, including acoustics, linguistics, digital signal processing, and computer science, and is a cutting-edge technology in the field of Chinese information processing. Its primary challenge is converting text into audible sound—in other words, enabling machines to speak like humans. This "enabling machines to speak like humans" is fundamentally different from traditional audio playback devices (systems). Traditional audio playback devices (systems), such as tape recorders, achieve "machine speech" by pre-recording and then playing back sound. This approach has significant limitations in terms of content, storage, transmission, convenience, and timeliness. Computer-based speech synthesis, on the other hand, can convert any text into highly natural speech at any time, truly enabling machines to "speak like humans."

[0037] Phoneme: A phoneme is the smallest unit of speech defined by the natural properties of speech. From an acoustic perspective, a phoneme is the smallest unit of speech defined by sound quality. From a physiological perspective, a single pronunciation action forms a phoneme. For example, the word "ma" consists of two pronunciation actions: "m" and "a," making them two phonemes. Sounds produced by the same pronunciation action are the same phoneme, while sounds produced by different pronunciation actions are different phonemes. For example, in "ma-mi," the two "m"s have the same pronunciation action and are the same phoneme, while "a" and "i" have different pronunciation actions and are different phonemes.

[0038] Acoustic characteristics: These are physical quantities that represent the acoustic characteristics of speech. They are also a general term for the acoustic manifestations of various sound elements. Examples include the energy concentration area, formant frequency, formant intensity, and bandwidth that represent timbre, as well as duration, fundamental frequency, and average speech power that represent prosodic characteristics of speech.

[0039] MAE loss function (Mean Square Error): It is the most commonly used error in regression loss function. It is the mean of the sum of the squares of the differences between the predicted value f(x) and the target value y.

[0040] The Attention Mechanism originates from the study of human vision. In cognitive science, due to information processing bottlenecks, humans selectively focus on a portion of all information while ignoring other visible information. By introducing the attention mechanism, information at every position in the source language is preserved. During the decoding process, when generating each target language word, the attention mechanism directly selects relevant information from the source language as auxiliary information. This approach effectively solves the two problems mentioned above. First, it eliminates the need to transmit all source language information through the encoding vector; information at all positions in the source language can be directly accessed at each decoding step. Second, source language information can be directly transmitted to each step in the decoding process, shortening the distance information is transmitted.

[0041] Sequence-to-Sequence Model: Sequence-to-Sequence is an encoder-decoder neural network structure composed of a bidirectional RNN, which can meet the situation where the input and output sequence lengths are different and realize the conversion between one sequence to another.

[0042] Teacher-Forcing: A network training method that is crucial for developing deep learning language models for machine translation, text summarization, image captioning, and many other applications. Instead of using the output of the previous state as the input to the next state, it directly uses the corresponding previous item of the standard answer (groundtruth) in the training data as the input to the next state.

[0043] In existing English speech synthesis systems, many end-to-end architectures achieve excellent sound quality, nearly resembling human pronunciation. However, in non-alphabetic languages such as Chinese and Japanese, end-to-end speech synthesis using text as input is difficult. This is due to the diversity of text and tones. For example, Mandarin Chinese has five tones, including the light tone.

[0044] Based on this, embodiments of the present invention provide a method for establishing a speech synthesis model, an application method, a device, and a storage medium. The method for establishing a speech synthesis model includes obtaining an initial string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer; inputting the initial string into the character layer to train the model, and obtaining character latent variables output by the character layer; inputting the initial phonemes and character latent variables into the phoneme layer to train the model, and obtaining phoneme latent variables output by the phoneme layer; and inputting the initial acoustic features and phoneme latent variables into the acoustic feature layer to train the model, thereby obtaining a speech synthesis model. The speech synthesis model obtains a target string or target phoneme; and inputs the target string or target phoneme into the speech synthesis model to obtain a speech spectrum. This method improves the naturalness of speech synthesized by the speech synthesis model and increases the synthesis speed of speech synthesized by the speech synthesis model.

[0045] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0046] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0047] The method for establishing a speech synthesis model and / or the method for applying a speech synthesis model according to the embodiment of the present invention relate to the field of artificial intelligence and digital medical technology. The method for establishing a speech synthesis model and / or the method for applying a speech synthesis model provided in the embodiment of the present invention can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model, etc., but is not limited to the above forms.

[0048] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0049] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0050] like Figure 1 As shown, Figure 1 It is a schematic diagram of a system architecture platform for a method for establishing a speech synthesis model and / or an application method of a speech synthesis model provided in an embodiment of the present invention.

[0051] The system architecture platform 100 of the embodiment of the present invention includes one or more processors 110 and a memory 120. Figure 1 In the figure, a processor 110 and a memory 120 are taken as an example.

[0052] The processor 110 and the memory 120 may be connected via a bus or other means. Figure 1 The bus connection is taken as an example.

[0053] The memory 120 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 120 may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 120 may optionally include a memory 120 remotely located relative to the processor 110, and these remote memories may be connected to the system architecture platform 100 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the system architecture platform 100, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0055] like Figure 2 As shown, Figure 2 It is a flowchart of a method for establishing a speech synthesis model provided by an embodiment of the present invention. The method for establishing a speech synthesis model in this embodiment of the present invention includes but is not limited to step S200, step S210, step S220 and step S230.

[0056] Step S200: obtaining an initial character string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer;

[0057] Step S210: inputting the initial character string into the character layer to perform model training on the character layer, and obtaining the character latent variables output by the character layer;

[0058] Step S220, inputting the initial phoneme and character latent variables into the phoneme layer to perform model training on the phoneme layer, and obtaining the phoneme latent variables output by the phoneme layer;

[0059] Step S230: Input the initial acoustic features and phoneme latent variables into the acoustic feature layer so that the acoustic feature layer performs model training to obtain a speech synthesis model.

[0060] In this embodiment, a method for establishing a speech synthesis model includes obtaining an initial string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer; inputting the initial string into the character layer to train the model, and obtaining character latent variables output by the character layer; inputting the initial phonemes and character latent variables into the phoneme layer to train the model, and obtaining phoneme latent variables output by the phoneme layer; and inputting the initial acoustic features and phoneme latent variables into the acoustic feature layer to train the model, thereby obtaining a speech synthesis model. The basic training model consists of three different layers, each of which inputs different variables and outputs corresponding variables, thereby improving the overall synthesis speed and work efficiency of the model. Furthermore, the use of dedicated layers to input and output corresponding variables improves the accuracy of the model. An initial character string is input into the character layer of a basic training model to train the character layer, and character latent variables output by the character layer are simultaneously obtained. The character latent variables output by the character layer and the initial phonemes are input into the phoneme layer of the basic training model to train the phoneme layer, and the phoneme latent variables output by the phoneme layer are simultaneously obtained. The phoneme latent variables output by the phoneme layer and the initial acoustic features are input into the acoustic feature layer of the basic training model to train the acoustic feature layer, thereby obtaining a speech synthesis model. The initial character string, initial phonemes, and initial acoustic features are extracted by preprocessing text and audio data in a preset training dataset.

[0061] like Figure 3 As shown, Figure 3 It is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention. The method for establishing a speech synthesis model in this embodiment of the present invention includes but is not limited to step S300, step S310, step S320 and step S330.

[0062] Step S300, inputting the initial character string into the first encoder to obtain an intermediate character string;

[0063] Step S310, inputting the intermediate character string into the first decoder to obtain the final character string;

[0064] Step S320, obtaining a character string loss value according to the initial character string and the final character string, and adjusting the character layer according to the character string loss value;

[0065] Step S330: Input the intermediate character string into the first converter to obtain character latent variables.

[0066] In an embodiment of the present invention, the character layer includes a first encoder, a first decoder and a first converter. The initial string is input into the character layer, that is, the initial string is input into the first encoder to obtain an intermediate string, and the intermediate string is input into the first decoder to obtain a final string, thereby realizing self-encoding and self-decoding of the string, completing string-to-string reconstruction, and calculating the loss between the initial string and the final string through MAE loss to obtain a string loss value, and adjusting the character layer according to the string loss value, performing model training in a self-supervised manner, and adopting an autoregressive method to improve the model training speed and model prediction accuracy of the character layer, accelerate the convergence speed of the model, input the intermediate string into the first converter to obtain a character latent variable, and the first converter is a sequence-to-sequence model based on the attention mechanism, so that the model of the character layer converges faster.

[0067] In addition, the character layer is trained using the teacher-forced mode, that is, the output at each moment is integrated with the real results of the previous moment for model training, which improves the convergence speed of the model.

[0068] like Figure 4 As shown, Figure 4 It is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention. The method for establishing a speech synthesis model in this embodiment of the present invention includes but is not limited to steps S400, 410, S420, S430 and S440.

[0069] Step S400, inputting the initial phoneme into the second encoder to obtain the intermediate phoneme;

[0070] Step S410, inputting the intermediate phoneme into the second decoder to obtain the final phoneme;

[0071] Step S420, obtaining a phoneme loss value according to the initial phoneme and the final phoneme, and adjusting the phoneme layer according to the phoneme loss value;

[0072] Step S430, inputting the character latent variables into the second encoder to perform supervised training on the phoneme layer;

[0073] Step S440: Input the intermediate phoneme into the second converter to obtain the phoneme latent variable.

[0074] In an embodiment of the present invention, the phoneme layer includes a second encoder, a second decoder, and a second converter. The initial phoneme is input into the phoneme layer, that is, the initial phoneme is input into the second encoder to obtain an intermediate phoneme. The intermediate phoneme is input into the second decoder to obtain a final phoneme, thereby realizing self-encoding and self-decoding of the phoneme and completing phoneme-to-phoneme reconstruction. The loss between the initial phoneme and the final phoneme is calculated by MAE loss to obtain a phoneme loss value. The phoneme layer is adjusted according to the phoneme loss value, and the model training is performed in a self-supervised manner. The autoregressive method is adopted to improve the model training speed and model prediction accuracy of the phoneme layer, accelerate the convergence speed of the model, and use the character latent variable output by the character layer as input to the second encoder for supervised training of the phoneme layer, improve the accuracy and precision of the phoneme layer model training, and achieve alignment between the character layer and the phoneme layer. The intermediate phoneme is input into the second converter to obtain the phoneme latent variable. The second converter is a sequence-to-sequence model based on the attention mechanism, which makes the model of the phoneme layer converge faster.

[0075] In addition, the phoneme layer is trained using a teacher-forced mode, that is, the output at each moment is integrated with the real results of the previous moment for model training, thereby improving the convergence speed of the model.

[0076] like Figure 5 As shown, Figure 5 It is a flowchart of a method for establishing a speech synthesis model provided by another embodiment of the present invention. The method for establishing a speech synthesis model in this embodiment of the present invention includes but is not limited to step S500, step S510, step S520 and step S530.

[0077] Step S500: inputting the initial acoustic features into a third encoder to obtain intermediate acoustic features;

[0078] Step S510: inputting the intermediate acoustic features into a third decoder to obtain final acoustic features;

[0079] Step S520: obtaining an acoustic feature loss value according to the initial acoustic feature and the final acoustic feature, and adjusting the acoustic feature layer according to the acoustic feature loss value;

[0080] Step S530: Input the phoneme latent variable into the third encoder to perform supervised training on the acoustic feature layer.

[0081] In an embodiment of the present invention, the acoustic feature layer includes a third encoder and a third decoder. The initial acoustic feature is input into the acoustic feature layer, that is, the initial acoustic feature is input into the third encoder to obtain an intermediate acoustic feature. The intermediate acoustic feature is input into the third decoder to obtain a final acoustic feature, thereby realizing self-encoding and self-decoding of the acoustic feature, completing the reconstruction of the acoustic feature into the acoustic feature, and calculating the loss between the initial acoustic feature and the final acoustic feature through the MAE loss to obtain the acoustic feature loss value, and adjusting the acoustic feature layer according to the acoustic feature loss value. The model is trained by a self-supervised method, and an autoregressive method is adopted to improve the model training speed and model prediction accuracy of the acoustic feature layer, accelerate the convergence speed of the model, and use the phoneme latent variable output by the phoneme layer as input and input it into the third encoder to perform supervised training on the acoustic feature layer, thereby improving the accuracy and precision of the acoustic feature layer model training and realizing alignment of the phoneme layer and the acoustic feature layer.

[0082] In addition, the acoustic feature layer is trained using a teacher-forced mode, that is, the output at each moment is integrated with the real results of the previous moment for model training, thereby improving the convergence speed of the model.

[0083] like Figure 6 As shown, Figure 6 This is a flowchart of a method for applying a speech synthesis model provided by an embodiment of the present invention. The method for applying a speech synthesis model according to the embodiment of the present invention includes but is not limited to step S600 and step S610.

[0084] Step S600, obtaining a target character string or a target phoneme;

[0085] Step S610: input the target character string or target phoneme into the speech synthesis model to obtain a speech spectrum.

[0086] In an embodiment of the present invention, a method for applying a speech synthesis model includes obtaining a target string or target phoneme, inputting the target string or target phoneme into the speech synthesis model, and obtaining a speech spectrum. The speech synthesis model is obtained by the method for establishing a speech synthesis model according to the above embodiment. After model training, the speech synthesis model supports input of strings or target phonemes. The input of the target string minimizes the effects of polyphones and different tones, thereby improving the naturalness and speed of synthesized speech.

[0087] like Figure 7 and Figure 8 As shown, Figure 7 This is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention. The method for applying a speech synthesis model in this embodiment of the present invention includes but is not limited to step S700. Figure 8This is a flowchart of a method for applying a speech synthesis model provided by another embodiment of the present invention. The method for applying a speech synthesis model in this embodiment of the present invention includes but is not limited to step S800.

[0088] Step S700, when inputting the target character string into the speech synthesis model to obtain the speech spectrum, inputting the target character string into the speech synthesis model to obtain the speech spectrum, includes: inputting the target character string into the speech synthesis model, and obtaining the speech spectrum through the character layer, the phoneme layer and the acoustic feature layer.

[0089] Step S800, when inputting the target phoneme into the speech synthesis model to obtain the speech spectrum, inputting the target phoneme into the speech synthesis model to obtain the speech spectrum, includes: inputting the target phoneme into the speech synthesis model, and obtaining the speech spectrum through the phoneme layer and the acoustic feature layer.

[0090] In an embodiment of the present invention, when a target character string is input into a speech synthesis model to obtain a speech spectrum, the target character string is input into the speech synthesis model to obtain a speech spectrum, including: inputting the target character string into the speech synthesis model to obtain a speech spectrum via the character layer, the phoneme layer and the acoustic feature layer. The input of the target character string can avoid the influence of polyphones and different tones to the greatest extent. The intermediate phoneme layer guides the character string, reduces the intermediate information semantic gap, and improves the naturalness and synthesis speed of the synthesized speech; when the target phoneme is input into the speech synthesis model to obtain a speech spectrum, the target phoneme is input into the speech synthesis model to obtain a speech spectrum, including: inputting the target phoneme into the speech synthesis model to obtain a speech spectrum via the phoneme layer and the acoustic feature layer.

[0091] like Figure 9 As shown, Figure 9 It is a flowchart of an application method of a speech synthesis model provided by another embodiment of the present invention. The application method of the speech synthesis model of the embodiment of the present invention includes but is not limited to step S900, step S910, step S920 and step S930.

[0092] Step S900, inputting the target character string into the character layer to obtain a character sequence;

[0093] Step S910: inputting a character sequence into a phoneme layer so that the phoneme layer outputs a phoneme sequence, and aligning the character sequence and the phoneme sequence through a first converter;

[0094] Step S920: input the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and align the phoneme sequence and the acoustic feature sequence through a second converter;

[0095] Step S930: synthesize a speech spectrum according to the character sequence, the phoneme sequence and the acoustic feature sequence.

[0096] In an embodiment of the present invention, the character layer is provided with a first converter connected to the phoneme layer, and the phoneme layer is provided with a second converter connected to the acoustic feature layer. The target character string is input into the speech synthesis model, that is, the target character string is input into the character layer to obtain a character sequence, the character sequence is input into the phoneme layer so that the phoneme layer outputs a phoneme sequence, and the character sequence and the phoneme sequence are aligned through the first conversion device, the phoneme sequence is input into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and the phoneme sequence and the acoustic feature sequence are aligned through the second converter. The speech spectrum is synthesized based on the character sequence, the phoneme sequence and the acoustic feature sequence, thereby improving the naturalness and synthesis speed of the synthesized speech. The input of the target character string can minimize the influence of polyphones and different tones, and the intermediate phoneme layer guides the character sequence, reduces the intermediate information semantic gap, and improves the naturalness and synthesis speed of the synthesized speech. The first converter is a sequence-to-sequence model based on the attention mechanism, which completes the alignment of character sequence to phoneme sequence, making the character layer model converge faster. The second converter is a sequence-to-sequence model based on the attention mechanism, which completes the alignment of phoneme sequence to acoustic feature sequence, making the phoneme layer model converge faster.

[0097] like Figure 10 As shown, Figure 10 This is a flowchart of an application method of a speech synthesis model provided by another embodiment of the present invention. The application method of the speech synthesis model of the embodiment of the present invention includes but is not limited to step S1000, step S1010 and step 1020.

[0098] Step S1000: input the target phoneme into the phoneme layer to obtain a phoneme sequence;

[0099] Step S1010: inputting the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and aligning the phoneme sequence and the acoustic feature sequence through a second converter;

[0100] Step S1020: synthesize the speech spectrum according to the phoneme sequence and the acoustic feature sequence.

[0101] In this embodiment of the present invention, a second converter is provided at the phoneme layer, connected to the acoustic feature layer. A target phoneme is input into the phoneme layer to generate a phoneme sequence. This phoneme sequence is then input into the acoustic feature layer, causing the acoustic feature layer to output an acoustic feature sequence. The second converter aligns the phoneme sequence and acoustic feature sequence, and synthesizes a speech spectrum based on the phoneme sequence and acoustic feature sequence, improving the naturalness and speed of the synthesized speech. The second converter is a sequence-to-sequence model based on an attention mechanism, which aligns the phoneme sequence with the acoustic feature sequence, enabling faster model convergence at the phoneme layer.

[0102] In addition, an embodiment of the present invention further provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0103] The processor and the memory may be connected via a bus or other means.

[0104] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] It should be noted that the electronic device in this embodiment can be applied as the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model as in the above-mentioned embodiment. The electronic device in this embodiment has the same inventive concept as the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model as in the above-mentioned embodiment. Therefore, these embodiments have the same implementation principles and technical effects, which will not be described in detail here.

[0106] The non-transient software program and instructions required to implement the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model as described above are stored in the memory. When executed by the processor, the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model as described above are executed, for example, Figure 2 Method steps S200 to S230, Figure 3 Steps S300 to S330 of the method, Figure 4 Steps S400 to S440 of the method, Figure 5 Method steps S500 to S530, Figure 6 Method steps S600 to S610, Figure 7 Method step S700, Figure 8 Method step S800, Figure 9 Steps S900 to S930 of the method, Figure 10 Method steps S1000 to S1020.

[0107] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one location or distributed across multiple network units. Some or all of these modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0108] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned electronic device embodiment, so that the above-mentioned processor can execute the method for establishing a speech synthesis model and / or the method for applying a speech synthesis model as described in the above-mentioned embodiment, for example, executing the above-described Figure 2 Method steps S200 to S230, Figure 3 Steps S300 to S330 of the method, Figure 4 Steps S400 to S440 of the method, Figure 5 Method steps S500 to S530, Figure 6 Method steps S600 to S610, Figure 7 Method step S700, Figure 8 Method step S800, Figure 9 Steps S900 to S930 of the method, Figure 10 Method steps S1000 to S1020.

[0109] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0110] It will be understood by those skilled in the art that Figure 2-10 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0112] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0113] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0114] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0115] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store programs.

[0119] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for establishing a speech synthesis model, characterized in that: include: Obtaining an initial character string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer; the character layer includes a first encoder, a first decoder, and a first converter; the phoneme layer includes a second encoder, a second decoder, and a second converter; and the acoustic feature layer includes a third encoder and a third decoder; Inputting the initial character string into the first encoder to obtain an intermediate character string; Inputting the intermediate character string into the first decoder to obtain a final character string; Obtaining a character string loss value according to the initial character string and the final character string, and adjusting the character layer according to the character string loss value; Inputting the intermediate character string into the first converter to obtain character hidden variables; Inputting the initial phoneme into the second encoder to obtain an intermediate phoneme; inputting the intermediate phoneme into the second decoder to obtain a final phoneme; Obtaining a phoneme loss value according to the initial phoneme and the final phoneme, and adjusting the phoneme layer according to the phoneme loss value; Inputting the character latent variables into the second encoder to perform supervised training on the phoneme layer; Inputting the intermediate phoneme into the second converter to obtain a phoneme latent variable; Inputting the initial acoustic features into the third encoder to obtain intermediate acoustic features; Inputting the intermediate acoustic features into the third decoder to obtain final acoustic features; Obtaining an acoustic feature loss value according to the initial acoustic feature and the final acoustic feature, and adjusting the acoustic feature layer according to the acoustic feature loss value; The phoneme latent variables are input into the third encoder to perform supervised training on the acoustic feature layer to obtain a speech synthesis model.

2. A method for applying a speech synthesis model, characterized in that: The application method comprises: Get the target string or target phoneme; Inputting the target character string or the target phoneme into a speech synthesis model to obtain a speech spectrum; The method for establishing the speech synthesis model includes: Obtaining an initial character string, initial phonemes, initial acoustic features, and a basic training model, wherein the basic training model includes a character layer, a phoneme layer, and an acoustic feature layer; the character layer includes a first encoder, a first decoder, and a first converter; the phoneme layer includes a second encoder, a second decoder, and a second converter; and the acoustic feature layer includes a third encoder and a third decoder; Inputting the initial character string into the first encoder to obtain an intermediate character string; Inputting the intermediate character string into the first decoder to obtain a final character string; Obtaining a character string loss value according to the initial character string and the final character string, and adjusting the character layer according to the character string loss value; Inputting the intermediate character string into the first converter to obtain character hidden variables; Inputting the initial phoneme into the second encoder to obtain an intermediate phoneme; inputting the intermediate phoneme into the second decoder to obtain a final phoneme; Obtaining a phoneme loss value according to the initial phoneme and the final phoneme, and adjusting the phoneme layer according to the phoneme loss value; Inputting the character latent variables into the second encoder to perform supervised training on the phoneme layer; Inputting the intermediate phoneme into the second converter to obtain a phoneme latent variable; Inputting the initial acoustic features into the third encoder to obtain intermediate acoustic features; Inputting the intermediate acoustic features into the third decoder to obtain final acoustic features; Obtaining an acoustic feature loss value according to the initial acoustic feature and the final acoustic feature, and adjusting the acoustic feature layer according to the acoustic feature loss value; The phoneme latent variables are input into the third encoder to perform supervised training on the acoustic feature layer to obtain the speech synthesis model.

3. The application method of the speech synthesis model according to claim 2, characterized in that: Include at least one of the following: In the case where the target character string is input into the speech synthesis model to obtain a speech spectrum, the step of inputting the target character string into the speech synthesis model to obtain a speech spectrum includes: inputting the target character string into the speech synthesis model, and obtaining a speech spectrum via the character layer, the phoneme layer, and the acoustic feature layer; In the case where the target phoneme is input into the speech synthesis model to obtain a speech spectrum, the step of inputting the target phoneme into the speech synthesis model to obtain a speech spectrum includes: inputting the target phoneme into the speech synthesis model, and obtaining the speech spectrum via the phoneme layer and the acoustic feature layer.

4. The application method of the speech synthesis model according to claim 3, characterized in that: include: The character layer is provided with a first converter connected to the phoneme layer, and the phoneme layer is provided with a second converter connected to the acoustic feature layer; In the case where the target character string is input into the speech synthesis model to obtain a speech spectrum, the step of inputting the target character string into the speech synthesis model and obtaining the speech spectrum via the character layer, the phoneme layer, and the acoustic feature layer includes: Inputting the target character string into the character layer to obtain a character sequence; Inputting the character sequence into the phoneme layer so that the phoneme layer outputs a phoneme sequence, and aligning the character sequence and the phoneme sequence through the first converter; Inputting the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and aligning the phoneme sequence and the acoustic feature sequence through the second converter; The speech spectrum is synthesized according to the character sequence, the phoneme sequence and the acoustic feature sequence.

5. The application method of the speech synthesis model according to claim 3, characterized in that: include: The phoneme layer is provided with a second converter connected to the acoustic feature layer; In a case where the target phoneme is input into the speech synthesis model to obtain a speech spectrum, inputting the target phoneme into the speech synthesis model and obtaining the speech spectrum via the phoneme layer and the acoustic feature layer includes: Inputting the target phoneme into the phoneme layer to obtain a phoneme sequence; Inputting the phoneme sequence into the acoustic feature layer so that the acoustic feature layer outputs an acoustic feature sequence, and aligning the phoneme sequence and the acoustic feature sequence through the second converter; The speech spectrum is synthesized according to the phoneme sequence and the acoustic feature sequence.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for establishing a speech synthesis model according to claim 1 and / or the method for applying a speech synthesis model according to any one of claims 2 to 5 when executing the computer program.

7. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the method for establishing a speech synthesis model as described in claim 1 and / or the method for applying a speech synthesis model as described in any one of claims 2 to 5.

Images