Speech synthesis method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of speech synthesis, is applied to a text-to-speech scene, and relates to a speech synthesis method, device and equipment and a storage medium thereof, during actual speech synthesis, a pre-trained speech synthesis agent is adopted for speech synthesis, and it is ensured that the speech synthesis efficiency is improved during actual text-to-speech synthesis. According to the method, a more accurate character-acoustic sequence alignment effect can be obtained, introduction of noise can be avoided to a great extent, the audio quality of synthesized speech is ensured, speech synthesis can be quickly realized by adopting a sequence alignment component based on a differentiable aligner, and the speech synthesis method is applied to the financial customer service industry, so that the speech synthesis efficiency is improved. And a better and better tone quality service function can be provided for customers.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, which is applied to text-to-speech scenarios, and in particular to a speech synthesis method, apparatus, device and storage medium thereof. Background Art

[0002] With the rapid development of the Internet, especially the rapid development of the self-media industry and the financial industry, traditional speech synthesis is increasingly moving towards AI intelligent speech synthesis, and speech synthesis technology is often required, that is, outputting a specified speech with the pronunciation of another person object.

[0003] There are several mainstream approaches to existing speech synthesis tasks. The first approach uses instance standardization to decouple and extract the content and timbre information of the input speech information. The second approach splits the speech into multiple features such as content, rhythm, pitch, timbre, etc., and then uses supervised technology and random sampling methods to extract the corresponding features. However, the first instance standardization scheme mentioned above cannot achieve true feature decoupling, and the synthesis effect on speech synthesis tasks is not good; the second scheme mentioned above is prone to cause a large amount of partial duplication in feature selection, resulting in excessive encoding and decoding, and is prone to introducing more noise, making it impossible to guarantee the quality of speech synthesis. Therefore, the existing technology cannot quickly realize speech synthesis in text-to-speech synthesis, and is also prone to introducing noise, resulting in poor quality of synthesized speech, affecting customer conversation experience. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium thereof, so as to solve the problem that the prior art cannot quickly realize speech synthesis in text-to-speech synthesis and easily introduces noise, resulting in poor quality of synthesized speech and affecting customer conversation experience.

[0005] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solutions:

[0006] A speech synthesis method comprises the following steps:

[0007] Get the speech text to be synthesized;

[0008] Serializing the speech text to be synthesized to obtain a speech text sequence to be synthesized;

[0009] Inputting the speech text sequence to be synthesized into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent;

[0010] The acoustic signal is sent to a preset speech sound source to output a synthesized speech.

[0011] Furthermore, the speech synthesis agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component, and a streaming decoding processing component. The step of inputting the speech text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent specifically includes:

[0012] Inputting the speech text sequence to be synthesized into the sequence alignment component, and outputting the speech text sequence aligned with the time phase of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the speech to be synthesized;

[0013] Mapping the speech text sequence that is time-aligned with the target Mel spectrum through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey a Gaussian distribution;

[0014] Inputting the series of high-dimensional acoustic feature representations that obey Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized;

[0015] The acoustic features of the speech to be synthesized are input into the streaming decoding processing component, and the acoustic features are decoded to obtain the target acoustic signal.

[0016] Further, the acoustic feature generation component includes a Mel acoustic feature extraction subcomponent and a first decoding subcomponent, and the step of inputting the series of high-dimensional acoustic feature representations obeying Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized specifically includes:

[0017] Performing acoustic feature extraction on the series of high-dimensional acoustic feature representations that obey Gaussian distribution through the Mel acoustic feature extraction subcomponent to obtain an acoustic feature extraction result;

[0018] Inputting the acoustic feature extraction result into the first decoding subcomponent for decoding to obtain the acoustic features of the speech to be synthesized;

[0019] The streaming decoding component includes a sampling subcomponent and a second decoding subcomponent. The step of inputting the acoustic features of the speech to be synthesized into the streaming decoding component, performing acoustic feature decoding, and obtaining a target acoustic signal specifically includes:

[0020] Inputting the acoustic features of the speech to be synthesized into the sampling subcomponent for Gaussian sampling to obtain the acoustic features after sampling;

[0021] The sampled acoustic features are decoded by the second decoding subcomponent to obtain a target acoustic signal.

[0022] Furthermore, before executing the step of inputting the to-be-synthesized speech text sequence into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent, the method further includes:

[0023] Obtain batches of reference voices of target voice users;

[0024] Performing noise reduction training on the acoustic feature generation component through the batch reference speech to obtain an acoustic feature generation component after noise reduction training; and,

[0025] Obtaining all acoustic features generated by the acoustic feature generation component after the noise reduction training;

[0026] All acoustic features generated by the acoustic feature generation component that has completed the noise reduction training are used to perform noise reduction training on the streaming decoding component to obtain a streaming decoding component that has completed the noise reduction training.

[0027] Furthermore, the acoustic feature generation component further includes a first encoding subcomponent, and the step of performing noise reduction training on the acoustic feature generation component through the batch reference speech to obtain the acoustic feature generation component after the noise reduction training specifically includes:

[0028] Step 501, inputting the batch of reference speech into the Mel acoustic feature extraction subcomponent to obtain the acoustic feature extraction results in the Mel spectrum form corresponding to all reference speech respectively;

[0029] Step 502, performing a first noise addition process on the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech sounds, to obtain acoustic feature extraction results after noise addition;

[0030] Step 503, using the first encoding subcomponent to perform feature encoding on the acoustic feature extraction result after the noise is added, to obtain an acoustic feature encoding result;

[0031] Step 504: input the acoustic feature encoding result into the first decoding subcomponent for decoding to obtain the acoustic features corresponding to all the reference speech respectively;

[0032] Step 505, mapping the acoustic features corresponding to all the reference speech sounds respectively through a preset second linear mapping layer to obtain high-dimensional acoustic feature representations corresponding to all the reference speech sounds respectively;

[0033] Step 506, using a maximum likelihood estimation method, calculating the loss value of the high-dimensional acoustic feature representations corresponding to all the reference speech sounds compared to the acoustic feature extraction result in the Mel spectrum form;

[0034] Step 507: if the loss value exceeds a preset first loss threshold, adjust the encoding and decoding parameters of the acoustic feature generation component and / or adjust the mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss value does not exceed the first loss threshold, and the training of the acoustic feature generation component is completed;

[0035] Step 508: If the loss value does not exceed the first loss threshold, a trained acoustic feature generation component is obtained.

[0036] Furthermore, after executing the step of acquiring the batch reference voices of the target voice users, the method further includes:

[0037] Step 601, performing text serialization processing on the batch of reference speech to obtain serialized texts corresponding to the batch of reference speech respectively;

[0038] Step 602, obtaining the time domain features corresponding to the batch of reference speech respectively from the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech respectively, as the time domain features extracted by Mel spectra;

[0039] Step 603, deploying the time domain feature as reference knowledge of the sequence alignment component to the sequence alignment component, to obtain a sequence alignment component with reference knowledge deployed;

[0040] Step 604: input the serialized texts corresponding to the batch of reference speech into the sequence alignment component after the reference knowledge deployment, and output the speech text sequence aligned with the time of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the batch of reference speech;

[0041] Step 605, mapping the speech text sequence that is time-aligned with the target Mel spectrum through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey Gaussian distribution;

[0042] Step 606, extracting time domain features corresponding to the batch of reference speech respectively from the series of high-dimensional acoustic feature representations obeying Gaussian distribution as the time domain features adjusted by the sequence alignment component;

[0043] Step 607, obtaining the similarity between the time domain features adjusted by the sequence alignment component and the time domain features extracted from the Mel spectrum by comparison calculation;

[0044] Step 608: if the similarity does not exceed the preset similarity threshold, the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer are adjusted, and steps 604 to 607 are re-executed until the similarity exceeds the preset similarity threshold, and the training of the sequence alignment component is completed;

[0045] Step 609: If the similarity exceeds a preset similarity threshold, a trained sequence alignment component is obtained.

[0046] Further, the streaming decoding component further includes a second encoding subcomponent, and the step of performing noise reduction training on the streaming decoding component using all acoustic features generated by the acoustic feature generation component after the noise reduction training to obtain the streaming decoding component after the noise reduction training specifically includes:

[0047] Step 701, performing a second noise addition process on all acoustic features generated by the acoustic feature generation component after the noise reduction training, to obtain all acoustic features after noise addition;

[0048] Step 702, using the second encoding subcomponent to perform feature encoding on all the acoustic features after the noise is added, to obtain an acoustic feature encoding result;

[0049] Step 703: input the acoustic feature encoding result into the second decoding subcomponent for decoding to obtain acoustic signals corresponding to all reference speech sounds;

[0050] Step 704, using a maximum likelihood estimation method, calculating the loss values ​​of the acoustic signals corresponding to all the reference speech sounds compared to all the acoustic features generated by the acoustic feature generation component after the noise reduction training;

[0051] Step 705: if the loss value exceeds a preset second loss threshold, the encoding and decoding parameters of the streaming decoding component are adjusted, and steps 701 to 704 are re-executed until the loss value does not exceed the second loss threshold, and the training of the streaming decoding component is completed;

[0052] Step 706: If the loss value does not exceed the second loss threshold, a trained streaming decoding component is obtained.

[0053] In order to solve the above technical problems, the embodiment of the present application also provides a speech synthesis device, which adopts the following technical solution:

[0054] A speech synthesis device, comprising:

[0055] A text acquisition module is used to acquire the speech text to be synthesized;

[0056] A serialization processing module, used for performing serialization processing on the speech text to be synthesized to obtain a speech text sequence to be synthesized;

[0057] A speech synthesis module, used for inputting the speech text sequence to be synthesized into a pre-trained speech synthesis agent, and obtaining an acoustic signal synthesized by the speech synthesis agent;

[0058] The speech output module is used to send the acoustic signal to a preset speech sound source and output a synthesized speech.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0060] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech synthesis method when executing the computer-readable instructions.

[0061] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0062] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method as described above.

[0063] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0064] The speech synthesis method described in the embodiment of the present application uses a pre-trained speech synthesis agent to perform speech synthesis when performing actual speech synthesis, which ensures that when the actual text-to-speech synthesis is performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent, thereby ensuring the audio quality of the synthesized speech. Moreover, a sequence alignment component based on a differentiable aligner is used to quickly realize speech synthesis. The speech synthesis method is applied to the financial customer service industry, which can provide customers with higher-quality and good sound quality service functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0067] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;

[0068] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0069] Figure 4 is a flowchart of a specific embodiment of training the speech synthesis agent in the speech synthesis method described in the present application;

[0070] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 is shown;

[0071] Figure 6 is a flowchart of a specific embodiment of training the sequence alignment component in the speech synthesis method described in the present application;

[0072] Figure 7 yes Figure 4 A flowchart of a specific embodiment of step 404 is shown;

[0073] Figure 8 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present application;

[0074] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0076] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0077] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0078] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0079] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0080] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0081] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0082] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the speech synthesis device is generally arranged in the terminal device.

[0083] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0084] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application. The speech synthesis method comprises the following steps:

[0085] Step 201, obtaining the speech text to be synthesized.

[0086] In this embodiment, the speech text to be synthesized refers to a piece of text content to be converted into speech, for example: in the field of financial technology, the intelligent customer service has pre-set text content when answering customer questions. The speech synthesis method provided in this application can convert the text content into synthesized speech. The speech synthesis method can also be used in teaching auxiliary scenarios such as intelligent recitation, intelligent storytelling, and intelligent reading, such as audio books.

[0087] Step 202, serialize the speech text to be synthesized to obtain a speech text sequence to be synthesized.

[0088] Specifically, the speech text to be synthesized is serialized to obtain a speech text sequence to be synthesized, that is, the characters are serialized and sorted according to the order of the characters in the speech text to be synthesized, so as to obtain a speech text sequence to be synthesized, which is convenient for subsequent speech synthesis according to the sequence.

[0089] Step 203: input the speech text sequence to be synthesized into a pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent.

[0090] In this embodiment, the speech synthesis agent is, for example, a speech synthesis component or a speech synthesis model.

[0091] In this embodiment, the speech synthesis agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component and a streaming decoding processing component. The sequence alignment component based on the differentiable aligner, for example, the alignment module Differentiable Aligner of EFfixTTS2, EFfixTTS2, that is, EFTS2 is an efficient and excellent speech synthesis model structure, and its alignment module Differentiable Aligner is a fully differentiable internal alignment module. Compared with some other non-differentiable or external alignment modules, such as MAS search in VITSS and MFA in FastSpeech, the alignment module of EFTS2 is accurate and computationally efficient.

[0092] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0093] Step 301, inputting the speech text sequence to be synthesized into the sequence alignment component, and outputting the speech text sequence aligned with the time phase of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the speech to be synthesized;

[0094] Specifically, in the acoustic Mel spectrum, the horizontal axis is the time axis, which is used to represent the change of the sound frequency in the time period, and the vertical axis is the frequency axis, which is used to characterize the change of the frequency value of the sound frequency; it can also be understood that the acoustic Mel spectrum contains time domain features and frequency domain features. The target Mel spectrum, for example, is: the speech text to be synthesized is output as Zhang San's speech. At this time, the acoustic Mel spectrum corresponding to Zhang San's speech is the target Mel spectrum.

[0095] The sequence alignment component is used to align the speech text sequence to be synthesized, so that the subsequently synthesized speech is closer to the real target object sound, more humane, fluent and natural.

[0096] Step 302, mapping the speech text sequence that is time-aligned with the target Mel spectrum through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey a Gaussian distribution;

[0097] Specifically, by mapping the speech text sequence that is time-aligned with the target Mel spectrum through a preset first linear mapping layer, a series of high-dimensional acoustic feature representations that obey Gaussian distribution are obtained, which facilitates the subsequent processing steps to obtain the acoustic Mel spectrum.

[0098] Step 303, inputting the series of high-dimensional acoustic feature representations that obey Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized;

[0099] Specifically, the acoustic feature generation component includes a Mel spectrum acoustic feature extraction subcomponent and a first decoding subcomponent, wherein the Mel spectrum acoustic feature extraction subcomponent is used to extract Mel spectrum acoustic features from the high-dimensional feature representation, and the first decoding subcomponent is used to decode the extracted acoustic features to obtain the acoustic features of the speech to be synthesized.

[0100] In this embodiment, the step of inputting the series of high-dimensional acoustic feature representations that obey Gaussian distribution into the acoustic feature generation component to generate the acoustic features of the speech to be synthesized specifically includes: performing acoustic feature extraction on the series of high-dimensional acoustic feature representations that obey Gaussian distribution through the Mel acoustic feature extraction subcomponent to obtain acoustic feature extraction results; inputting the acoustic feature extraction results into the first decoding subcomponent to perform decoding processing to obtain the acoustic features of the speech to be synthesized.

[0101] Step 304: input the acoustic features of the speech to be synthesized into the streaming decoding processing component, perform acoustic feature decoding, and obtain a target acoustic signal.

[0102] Specifically, the streaming decoding component includes a sampling subcomponent and a second decoding subcomponent. The sampling subcomponent is used to sample and process the acoustic features of the speech to be synthesized to obtain more comprehensive acoustic features, and the second decoding subcomponent is used to decode the acoustic features into audio signals.

[0103] In this embodiment, the step of inputting the acoustic features of the speech to be synthesized into the streaming decoding component, performing acoustic feature decoding, and obtaining the target acoustic signal specifically includes: inputting the acoustic features of the speech to be synthesized into the sampling subcomponent for Gaussian sampling to obtain the sampled acoustic features; performing acoustic feature decoding on the sampled acoustic features by the second decoding subcomponent to obtain the target acoustic signal. Specifically, the Gaussian sampling refers to the selection of acoustic feature points during sampling that obeys Gaussian distribution.

[0104] In this embodiment, before executing the step of inputting the speech text sequence to be synthesized into the pre-trained speech synthesis agent and obtaining the acoustic signal synthesized by the speech synthesis agent, the method further includes a method of training the speech synthesis agent. By training the speech synthesis agent, the speech synthesized by the speech synthesis agent is made more natural and closer to the pronunciation of the target object, thereby providing customers with better audio service quality.

[0105] Continue to refer Figure 4 , Figure 4 is a flowchart of a specific embodiment of training the speech synthesis agent in the speech synthesis method of the present application, comprising:

[0106] Step 401, obtaining batch reference voices of target voice users;

[0107] Step 402, performing noise reduction training on the acoustic feature generation component through the batch reference speech to obtain an acoustic feature generation component after noise reduction training; and

[0108] Step 403, obtaining all acoustic features generated by the acoustic feature generation component after the noise reduction training;

[0109] In this embodiment, the acoustic feature generation component also includes a first encoding subcomponent. Specifically, the acoustic feature generation component includes a Mel acoustic feature extraction subcomponent, a first encoding subcomponent and a first decoding subcomponent. Before actual application, the acoustic feature generation component is first subjected to noise reduction training, so that when the subsequent actual speech synthesis is performed, the interference of noise can be eliminated as much as possible, so that the audio quality of the synthesized speech is better.

[0110] Continue to refer Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of step 402 shown includes:

[0111] Step 501, inputting the batch of reference speech into the Mel acoustic feature extraction subcomponent to obtain the acoustic feature extraction results in the Mel spectrum form corresponding to all reference speech respectively;

[0112] Step 502, performing a first noise addition process on the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech sounds, to obtain acoustic feature extraction results after noise addition;

[0113] Step 503, using the first encoding subcomponent to perform feature encoding on the acoustic feature extraction result after the noise is added, to obtain an acoustic feature encoding result;

[0114] Step 504: input the acoustic feature encoding result into the first decoding subcomponent for decoding to obtain the acoustic features corresponding to all the reference speech respectively;

[0115] Step 505, mapping the acoustic features corresponding to all the reference speech sounds respectively through a preset second linear mapping layer to obtain high-dimensional acoustic feature representations corresponding to all the reference speech sounds respectively;

[0116] Through linear mapping layer mapping, the acoustic features corresponding to all reference speech are converted into high-dimensional acoustic feature representations, which is convenient for comparison with the acoustic feature extraction results in the form of Mel spectrum.

[0117] Step 506, using a maximum likelihood estimation method, calculating the loss value of the high-dimensional acoustic feature representations corresponding to all the reference speech sounds compared to the acoustic feature extraction result in the Mel spectrum form;

[0118] Step 507: if the loss value exceeds a preset first loss threshold, adjust the encoding and decoding parameters of the acoustic feature generation component and / or adjust the mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss value does not exceed the first loss threshold, and the training of the acoustic feature generation component is completed;

[0119] Step 508: If the loss value does not exceed the first loss threshold, a trained acoustic feature generation component is obtained.

[0120] In essence, the acoustic feature generation component is subjected to noise reduction learning training, by first performing a first noise addition process on the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech sounds, to obtain the acoustic feature extraction results after noise addition, and then performing encoding and decoding processes, and finally calculating the loss value between the decoding processing result and the acoustic feature extraction result before noise addition, adjusting the acoustic feature generation component according to the loss value, and finally obtaining a trained acoustic feature generation component, that is, the trained acoustic feature generation component learns the automatic recognition and exclusion of noise when performing acoustic feature generation, thereby ensuring that the introduction of noise is reduced during actual speech synthesis.

[0121] In this embodiment, after executing the step of acquiring batch reference speech of the target speech user, the speech synthesis method further includes a sequence alignment component training method.

[0122] Continue to refer Figure 6 , Figure 6 is a flowchart of a specific embodiment of training the sequence alignment component in the speech synthesis method of the present application, including:

[0123] Step 601, performing text serialization processing on the batch of reference speech to obtain serialized texts corresponding to the batch of reference speech respectively;

[0124] Step 602, obtaining the time domain features corresponding to the batch of reference speech respectively from the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech respectively, as the time domain features extracted by Mel spectra;

[0125] Step 603, deploying the time domain feature as reference knowledge of the sequence alignment component to the sequence alignment component, to obtain a sequence alignment component with reference knowledge deployed;

[0126] Step 604: input the serialized texts corresponding to the batch of reference speech into the sequence alignment component after the reference knowledge deployment, and output the speech text sequence aligned with the time of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the batch of reference speech;

[0127] Step 605, mapping the speech text sequence that is time-aligned with the target Mel spectrum through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey Gaussian distribution;

[0128] Step 606, extracting time domain features corresponding to the batch of reference speech respectively from the series of high-dimensional acoustic feature representations obeying Gaussian distribution as the time domain features adjusted by the sequence alignment component;

[0129] Step 607, obtaining the similarity between the time domain features adjusted by the sequence alignment component and the time domain features extracted from the Mel spectrum by comparison calculation;

[0130] Step 608: if the similarity does not exceed the preset similarity threshold, the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer are adjusted, and steps 604 to 607 are re-executed until the similarity exceeds the preset similarity threshold, and the training of the sequence alignment component is completed;

[0131] Step 609: If the similarity exceeds a preset similarity threshold, a trained sequence alignment component is obtained.

[0132] By training the sequence alignment component, when actually performing speech synthesis, the alignment effect can be ensured when the speech text sequence to be synthesized is input into the sequence alignment component and the speech text sequence aligned with the time of the target mel spectrum is output.

[0133] Step 404: Perform noise reduction training on the streaming decoding component using all acoustic features generated by the acoustic feature generation component that has completed noise reduction training, to obtain a streaming decoding component that has completed noise reduction training.

[0134] In this embodiment, the streaming decoding component further includes a second encoding subcomponent, that is, the streaming decoding component includes a sampling subcomponent, a second encoding subcomponent and a second decoding subcomponent. By performing noise reduction training on the streaming decoding component, the interference of noise can be eliminated as much as possible during the subsequent actual speech synthesis, so that the audio quality of the synthesized speech is better.

[0135] Continue to refer Figure 7 , Figure 7 yes Figure 4 The flowchart of a specific embodiment of step 404 shown includes:

[0136] Step 701, performing a second noise addition process on all acoustic features generated by the acoustic feature generation component after the noise reduction training, to obtain all acoustic features after noise addition;

[0137] Step 702, using the second encoding subcomponent to perform feature encoding on all the acoustic features after the noise is added, to obtain an acoustic feature encoding result;

[0138] Step 703: input the acoustic feature encoding result into the second decoding subcomponent for decoding to obtain acoustic signals corresponding to all reference speech sounds;

[0139] Step 704, using a maximum likelihood estimation method, calculating the loss values ​​of the acoustic signals corresponding to all the reference speech sounds compared to all the acoustic features generated by the acoustic feature generation component after the noise reduction training;

[0140] Step 705: if the loss value exceeds a preset second loss threshold, the encoding and decoding parameters of the streaming decoding component are adjusted, and steps 701 to 704 are re-executed until the loss value does not exceed the second loss threshold, and the training of the streaming decoding component is completed;

[0141] Step 706: If the loss value does not exceed the second loss threshold, a trained streaming decoding component is obtained.

[0142] In essence, the streaming decoding component is subjected to noise reduction learning training, by first performing a second noise addition process on all acoustic features generated by the acoustic feature generation component after the noise reduction training, obtaining all acoustic features after noise addition, and then performing encoding and decoding processes. Finally, the loss values ​​of the acoustic signals corresponding to all the reference speech signals compared with all acoustic features generated by the acoustic feature generation component after the noise reduction training are calculated, and the streaming decoding component is adjusted according to the loss values. Finally, a trained streaming decoding component is obtained, so that the trained streaming decoding component has learned the automatic recognition and elimination of noise before the actual audio signal decoding, thereby ensuring that the introduction of noise is reduced during the actual speech synthesis.

[0143] Step 204: sending the acoustic signal to a preset speech sound source to output a synthesized speech.

[0144] In this embodiment, the preset voice sound source refers to a voice output component or device, such as a speaker, a sound system, etc. Specifically, the step of sending the acoustic signal to the preset voice sound source and outputting the synthesized voice includes: decoding the acoustic signal, and outputting the decoding result through the voice output component to obtain the synthesized voice.

[0145] In this embodiment, before speech synthesis, the speech synthesis intelligent agent is trained in combination with the batch speech data of the target object. Specifically, alignment training is performed on the sequence alignment component, noise reduction learning training is performed on the acoustic feature generation component, and noise reduction learning training is performed on the streaming decoding component. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent, thereby ensuring the audio quality of the synthesized speech.

[0146] This application trains a speech synthesis agent in combination with batch speech data of the target object before performing actual speech synthesis. Specifically, alignment training is performed on the sequence alignment component, noise reduction learning training is performed on the acoustic feature generation component, and noise reduction learning training is performed on the streaming decoding component. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent, thereby ensuring the audio quality of the synthesized speech. Moreover, the use of a sequence alignment component based on a differentiable aligner can quickly realize speech synthesis. This speech synthesis method is applied to the financial customer service industry, and can provide customers with higher-quality and good sound quality service functions.

[0147] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0148] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0149] In the embodiment of the present application, before performing actual speech synthesis, the speech synthesis agent is first trained in combination with the batch speech data of the target object. Specifically, the sequence alignment component is aligned and trained, the acoustic feature generation component is denoised and trained, and the streaming decoding component is denoised and trained. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent, thereby ensuring the audio quality of the synthesized speech. Moreover, the use of a sequence alignment component based on a differentiable aligner can quickly realize speech synthesis. This speech synthesis method is applied to the financial customer service industry, and can provide customers with better quality and good sound quality service functions.

[0150] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0151] like Figure 8 As shown, the speech synthesis device 800 described in this embodiment includes: a text acquisition module 801, a serialization processing module 802, a speech synthesis module 803 and a speech output module 804. Among them:

[0152] The text acquisition module 801 is used to acquire the speech text to be synthesized;

[0153] A serialization processing module 802 is used to perform serialization processing on the speech text to be synthesized to obtain a speech text sequence to be synthesized;

[0154] The speech synthesis module 803 is used to input the speech text sequence to be synthesized into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent;

[0155] The speech output module 804 is used to send the acoustic signal to a preset speech sound source and output a synthesized speech.

[0156] This application trains a speech synthesis agent in combination with batch speech data of the target object before performing actual speech synthesis. Specifically, alignment training is performed on the sequence alignment component, noise reduction learning training is performed on the acoustic feature generation component, and noise reduction learning training is performed on the streaming decoding component. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent, thereby ensuring the audio quality of the synthesized speech. Moreover, the use of a sequence alignment component based on a differentiable aligner can quickly realize speech synthesis. This speech synthesis method is applied to the financial customer service industry, and can provide customers with higher-quality and good sound quality service functions.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0159] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0160] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 having components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0161] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0162] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of a speech synthesis method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.

[0163] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions or process data stored in the memory 9a, such as the computer-readable instructions for running the speech synthesis method.

[0164] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0165] The computer device proposed in this embodiment belongs to the field of speech synthesis technology and is applied to text-to-speech scenarios. This application trains the speech synthesis agent in combination with the batch speech data of the target object before performing actual speech synthesis. Specifically, the sequence alignment component is trained, the acoustic feature generation component is trained for noise reduction learning, and the streaming decoding component is trained for noise reduction learning. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent to ensure the audio quality of the synthesized speech. Moreover, the sequence alignment component based on the differentiable aligner is used to quickly realize speech synthesis. The speech synthesis method is applied to the financial customer service industry, which can provide customers with better quality and good sound quality service functions.

[0166] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech synthesis method as described above.

[0167] The computer-readable storage medium proposed in this embodiment belongs to the field of speech synthesis technology and is applied to text-to-speech scenarios. This application trains the speech synthesis agent in combination with the batch speech data of the target object before performing actual speech synthesis. Specifically, alignment training is performed on the sequence alignment component, noise reduction learning training is performed on the acoustic feature generation component, and noise reduction learning training is performed on the streaming decoding component. This ensures that when the text-to-speech synthesis is actually performed, a more accurate text-acoustic sequence alignment effect can be obtained, and the introduction of noise can be avoided to a great extent to ensure the audio quality of the synthesized speech. Moreover, the use of a sequence alignment component based on a differentiable aligner can quickly realize speech synthesis. The speech synthesis method is applied to the financial customer service industry, which can provide customers with higher-quality and good sound quality service functions.

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0169] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Get the speech text to be synthesized; Serializing the speech text to be synthesized to obtain a speech text sequence to be synthesized; Inputting the speech text sequence to be synthesized into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent; The acoustic signal is sent to a preset speech sound source to output a synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that: The speech synthesis agent includes a sequence alignment component based on a differentiable aligner, an acoustic feature generation component and a streaming decoding processing component. The step of inputting the speech text sequence to be synthesized into the pre-trained speech synthesis agent to obtain the acoustic signal synthesized by the speech synthesis agent specifically includes: Inputting the speech text sequence to be synthesized into the sequence alignment component, and outputting the speech text sequence aligned with the time phase of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the speech to be synthesized; Mapping the speech text sequence that is time-aligned with the target Mel spectrum through a preset first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey a Gaussian distribution; Inputting the series of high-dimensional acoustic feature representations that obey Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized; The acoustic features of the speech to be synthesized are input into the streaming decoding processing component, and the acoustic features are decoded to obtain the target acoustic signal.

3. The speech synthesis method according to claim 2, characterized in that: The acoustic feature generation component includes a Mel acoustic feature extraction subcomponent and a first decoding subcomponent. The step of inputting the series of high-dimensional acoustic feature representations obeying Gaussian distribution into the acoustic feature generation component to generate acoustic features of the speech to be synthesized specifically includes: Performing acoustic feature extraction on the series of high-dimensional acoustic feature representations that obey Gaussian distribution through the Mel acoustic feature extraction subcomponent to obtain an acoustic feature extraction result; Inputting the acoustic feature extraction result into the first decoding subcomponent for decoding to obtain the acoustic features of the speech to be synthesized; The streaming decoding component includes a sampling subcomponent and a second decoding subcomponent. The step of inputting the acoustic features of the speech to be synthesized into the streaming decoding component, performing acoustic feature decoding, and obtaining a target acoustic signal specifically includes: Inputting the acoustic features of the speech to be synthesized into the sampling subcomponent for Gaussian sampling to obtain the acoustic features after sampling; The sampled acoustic features are decoded by the second decoding subcomponent to obtain a target acoustic signal.

4. The speech synthesis method according to claim 3, characterized in that: Before executing the step of inputting the to-be-synthesized speech text sequence into a pre-trained speech synthesis agent to obtain an acoustic signal synthesized by the speech synthesis agent, the method further comprises: Obtain batches of reference voices of target voice users; Performing noise reduction training on the acoustic feature generation component through the batch reference speech to obtain an acoustic feature generation component after noise reduction training; and, Obtaining all acoustic features generated by the acoustic feature generation component after the noise reduction training; All acoustic features generated by the acoustic feature generation component that has completed the noise reduction training are used to perform noise reduction training on the streaming decoding component to obtain a streaming decoding component that has completed the noise reduction training.

5. The speech synthesis method according to claim 4, characterized in that: The acoustic feature generation component further includes a first encoding subcomponent, and the step of performing noise reduction training on the acoustic feature generation component through the batch reference speech to obtain the acoustic feature generation component after the noise reduction training specifically includes: Step 501, inputting the batch of reference speech into the Mel acoustic feature extraction subcomponent to obtain the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech respectively; Step 502, performing a first noise addition process on the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech sounds, to obtain acoustic feature extraction results after noise addition; Step 503, using the first encoding subcomponent to perform feature encoding on the acoustic feature extraction result after the noise is added, to obtain an acoustic feature encoding result; Step 504: input the acoustic feature encoding result into the first decoding subcomponent for decoding to obtain the acoustic features corresponding to all the reference speech respectively; Step 505, mapping the acoustic features corresponding to all the reference speech sounds respectively through a preset second linear mapping layer to obtain high-dimensional acoustic feature representations corresponding to all the reference speech sounds respectively; Step 506, using a maximum likelihood estimation method, calculating the loss value of the high-dimensional acoustic feature representations corresponding to all the reference speech sounds compared to the acoustic feature extraction result in the Mel spectrum form; Step 507: if the loss value exceeds a preset first loss threshold, adjust the encoding and decoding parameters of the acoustic feature generation component and / or adjust the mapping parameters of the second linear mapping layer, and re-execute steps 501 to 506 until the loss value does not exceed the first loss threshold, and the training of the acoustic feature generation component is completed; Step 508: If the loss value does not exceed the first loss threshold, a trained acoustic feature generation component is obtained.

6. The speech synthesis method according to claim 5, characterized in that: After executing the step of acquiring batch reference voices of target voice users, the method further includes: Step 601, performing text serialization processing on the batch of reference speech to obtain serialized texts corresponding to the batch of reference speech respectively; Step 602, obtaining the time domain features corresponding to the batch of reference speech respectively from the acoustic feature extraction results in the form of Mel spectra corresponding to all reference speech respectively, as the time domain features extracted by Mel spectra; Step 603, deploying the time domain feature as reference knowledge of the sequence alignment component to the sequence alignment component, to obtain a sequence alignment component with reference knowledge deployed; Step 604: inputting the serialized texts corresponding to the batch of reference speech into the sequence alignment component after the reference knowledge deployment, and outputting the speech text sequence aligned with the time of the target Mel spectrum, wherein the target Mel spectrum refers to the Mel spectrum corresponding to the batch of reference speech; Step 605, mapping the speech text sequence that is time-aligned with the target Mel spectrum through the first linear mapping layer to obtain a series of high-dimensional acoustic feature representations that obey Gaussian distribution; Step 606, extracting time domain features corresponding to the batch of reference speech respectively from the series of high-dimensional acoustic feature representations obeying Gaussian distribution as the time domain features adjusted by the sequence alignment component; Step 607, obtaining the similarity between the time domain features adjusted by the sequence alignment component and the time domain features extracted from the Mel spectrum by comparison calculation; Step 608: if the similarity does not exceed the preset similarity threshold, the alignment parameters of the sequence alignment component and / or the mapping parameters of the first linear mapping layer are adjusted, and steps 604 to 607 are re-executed until the similarity exceeds the preset similarity threshold, and the training of the sequence alignment component is completed; Step 609: If the similarity exceeds a preset similarity threshold, a trained sequence alignment component is obtained.

7. The speech synthesis method according to claim 4, characterized in that: The streaming decoding component further includes a second encoding subcomponent, and the step of performing noise reduction training on the streaming decoding component using all acoustic features generated by the acoustic feature generation component after the noise reduction training to obtain the streaming decoding component after the noise reduction training specifically includes: Step 701, performing a second noise addition process on all acoustic features generated by the acoustic feature generation component after the noise reduction training, to obtain all acoustic features after noise addition; Step 702, using the second encoding subcomponent to perform feature encoding on all the acoustic features after the noise is added, to obtain an acoustic feature encoding result; Step 703: input the acoustic feature encoding result into the second decoding subcomponent for decoding to obtain acoustic signals corresponding to all reference speech sounds; Step 704, using a maximum likelihood estimation method, calculating the loss values ​​of the acoustic signals corresponding to all the reference speech sounds compared to all the acoustic features generated by the acoustic feature generation component after the noise reduction training; Step 705: if the loss value exceeds a preset second loss threshold, the encoding and decoding parameters of the streaming decoding component are adjusted, and steps 701 to 704 are re-executed until the loss value does not exceed the second loss threshold, and the training of the streaming decoding component is completed; Step 706: If the loss value does not exceed the second loss threshold, a trained streaming decoding component is obtained.

8. A speech synthesis device, characterized in that: include: A text acquisition module is used to acquire the speech text to be synthesized; A serialization processing module, used for performing serialization processing on the speech text to be synthesized to obtain a speech text sequence to be synthesized; A speech synthesis module, used for inputting the speech text sequence to be synthesized into a pre-trained speech synthesis agent, and obtaining an acoustic signal synthesized by the speech synthesis agent; The speech output module is used to send the acoustic signal to a preset speech sound source and output a synthesized speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images