A speech synthesis method, device and storage medium

Abstract

The present application discloses a method for speech synthesis, which includes the following steps: obtaining a target text, and inputting the target text into a target spectrum generated by an acoustic model; using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform; performing pop detection on the speech waveform; if pops are detected in the speech waveform, repeating the step of using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform until there are no pops in the speech waveform; and outputting the speech corresponding to the speech waveform without pops. By applying the technical solution provided by the present application, there can be no pops in the final output speech, so that the final speech is more stable and reliable, and the quality of the synthesized speech is improved. The present application also discloses a speech synthesis device, equipment and storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a speech synthesis method, device and storage medium. Background Art

[0002] Speech synthesis technology is a technology that converts text information into speech information. It involves multiple disciplines such as acoustics, linguistics, and computer science. It is a cutting-edge technology in the field of Chinese information processing. Speech synthesis technology can be used to obtain synthesized speech, which can be played in application scenarios such as user terminals and robots. The quality of synthesized speech directly affects the user's auditory experience.

[0003] Therefore, how to perform speech synthesis so that the final synthesized speech is more stable and reliable and improve the quality of the synthesized speech is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] The purpose of the present application is to provide a speech synthesis method, device and storage medium, so as to make the synthesized speech more stable and reliable and improve the quality of the synthesized speech.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] A speech synthesis method, comprising:

[0007] Acquire a target text, and input the target text into a target spectrum generated by an acoustic model;

[0008] Inferring the target spectrum using a neural network vocoder to obtain a predicted speech waveform;

[0009] Performing pop detection on the speech waveform;

[0010] If it is detected that there is a pop in the speech waveform, repeatedly performing the step of using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform until there is no pop in the speech waveform;

[0011] Output the speech corresponding to the speech waveform without popping sounds.

[0012] In a specific implementation of the present application, the step of performing pop detection on the speech waveform includes:

[0013] The speech waveform is subjected to a time domain popping detection according to an absolute value of the amplitude of the speech waveform and a relative amplitude change rate in the time domain.

[0014] In a specific implementation of the present application, the step of performing time domain pop detection on the speech waveform according to the absolute value of the amplitude and the relative amplitude change rate of the speech waveform in the time domain includes:

[0015] Determine whether there is a pop in the time domain of the speech waveform in the time domain based on the absolute value of the amplitude of each sampling point of the speech waveform in the time domain, and / or the absolute value of the amplitude difference between every two adjacent sampling points, and / or the absolute value of the amplitude difference between every two sampling points separated by one or more sampling points.

[0016] In a specific implementation of the present application, after performing the time domain pop detection on the speech waveform according to the absolute value of the amplitude and the relative amplitude change rate of the speech waveform in the time domain, the method further includes:

[0017] If there is no popping sound in the speech waveform in the time domain, the popping sound detection in the frequency domain is performed on the speech waveform according to the energy of each sub-band frequency interval of the speech waveform in the frequency domain.

[0018] In a specific implementation of the present application, the step of performing frequency domain pop detection on the speech waveform according to the energy of each sub-band frequency interval of the speech waveform in the frequency domain includes:

[0019] Determine whether the absolute value of the energy difference between each two adjacent sub-band frequency intervals of the speech waveform in the frequency domain is greater than a preset energy difference threshold;

[0020] If yes, it is determined that the speech waveform has a pop in the frequency domain;

[0021] If not, it is determined that there is no popping sound in the speech waveform in the frequency domain.

[0022] In a specific implementation of the present application, before outputting the speech waveform without popping sound, the method further includes:

[0023] Performing silence detection on the speech corresponding to the speech waveform without popping sound;

[0024] If it is detected that there is target silence in the speech whose silence duration is greater than a preset duration threshold, the target silence in the speech is processed.

[0025] In a specific implementation of the present application, processing the target silence in the speech includes:

[0026] performing truncation processing on the target silence in the speech,

[0027] Or, the target silence in the speech is replaced by silence whose silence duration is less than the preset duration threshold.

[0028] In a specific implementation of the present application, before the neural network vocoder is used to infer the target spectrum to obtain the predicted speech waveform, the method further includes:

[0029] Performing amplitude detection on the target spectrum;

[0030] The detected amplitude exceeding the preset amplitude range is adjusted to be within the amplitude range.

[0031] A speech synthesis device, comprising:

[0032] Memory for storing computer programs;

[0033] A processor is used to implement the steps of any of the above-mentioned speech synthesis methods when executing the computer program.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned speech synthesis methods are implemented.

[0035] By applying the technical solution provided in the embodiment of the present application, after the target spectrum is inferred by the neural network vocoder to obtain the predicted speech waveform, the speech corresponding to the speech waveform is not directly output, but the speech waveform is first subjected to pop detection, and the corresponding speech is output when there is no pop. In the case of pop, the target spectrum is re-inferred by the neural network vocoder to obtain the predicted speech waveform, and the pop detection is performed on the speech waveform obtained again, until the predicted speech waveform obtained by inferring the target spectrum by the neural network vocoder is not detected, and the speech corresponding to the speech waveform without pop is output, so that there is no pop in the final output speech, so that the final speech is more stable and reliable, and the quality of the synthesized speech is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a flowchart of a speech synthesis method in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of the structure of a speech synthesis device in an embodiment of the present application;

[0039] Figure 3 This is a structural diagram of a speech synthesis device in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The core of this application is to provide a speech synthesis method, which can be implemented by running in the background of a computer or by cloud processing to achieve higher processing efficiency and faster running speed.

[0041] In any scenario where there is a need to convert text information into voice information, the technical solution provided by the embodiment of the present application can be applied to perform speech synthesis, for example, a scenario where a popular science article is to be played to a user in voice.

[0042] In an embodiment of the present application, a target text is obtained, and the target text is input into a target spectrum generated by an acoustic model; the target spectrum is inferred using a neural network vocoder to obtain a predicted speech waveform; a pop detection is performed on the speech waveform; if a pop is detected in the speech waveform, the step of inferring the target spectrum using a neural network vocoder to obtain a predicted speech waveform is repeated until there is no pop in the speech waveform; and the speech corresponding to the speech waveform without the pop is output. After the target spectrum is inferred using a neural network vocoder to obtain a predicted speech waveform, the speech corresponding to the speech waveform is not directly output, but a pop detection is first performed on the speech waveform, and the corresponding speech is output only when there is no pop. In the case of the presence of popping sounds, the target spectrum is re-inferred using a neural network vocoder to obtain a predicted speech waveform, and popping sound detection is performed on the speech waveform obtained again. This is not until the target spectrum is inferred using the neural network vocoder and no popping sound is detected in the predicted speech waveform. Only then will the speech corresponding to the speech waveform without popping sounds be output. This ensures that there are no popping sounds in the final output speech, making the final speech more stable and reliable, thereby improving the quality of the synthesized speech.

[0043] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0044] See also Figure 1 As shown in FIG. 1 , a flow chart of an implementation of a speech synthesis method provided in an embodiment of the present application is provided, and the method may include the following steps:

[0045] S110: Acquire a target text, and input the target text into a target spectrum generated by an acoustic model;

[0046] In the embodiment of the present application, the target text may be text information currently to be subjected to speech synthesis, may be a piece of text information specified by a user, or may be a piece of text information automatically generated by a computer.

[0047] When there is a need for speech synthesis, a target spectrum can be generated based on the target text. Specifically, the target text can be mapped based on a preset mapping relationship to obtain a target spectrum mapped by the target text. The target text can also be input into a pre-acoustic model, and the target spectrum corresponding to the target text can be generated using the acoustic model.

[0048] The spectrum in the embodiment of the present application is the frequency domain amplitude spectrum of the sound waveform, and does not include phase information. The specific form of the spectrum may be a Melton spectrum.

[0049] After obtaining the target spectrum generated based on the target text, you can proceed to the subsequent steps.

[0050] S120: using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform;

[0051] In an embodiment of the present application, a network model can be constructed using a deep neural network, and a neural network vocoder can be obtained by using training data to train the model. The obtained neural network vocoder has the ability to associate the spectrum with the speech. Specifically, the neural network vocoder can be a neural network vocoder based on a generative adversarial network (GAN), such as melgan (a non-autoregressive feedforward convolutional architecture), hifigan (a GAN-based model that can effectively generate high-fidelity speech); it can also be a neural network vocoder based on a recurrent neural network (RNN), such as wavernn (a deep architecture for speech synthesis).

[0052] The training data may include a plurality of information pairs collected in advance, each of which may include a spectrum generated based on text and a corresponding speech waveform, or may include a spectrum converted from a speech waveform and the speech waveform. For example, an audio novel is split into a plurality of audio segments, the spectrum of each audio segment is extracted, and the speech waveform and spectrum corresponding to each audio segment are used as training data to train a neural network vocoder, and the trained neural network vocoder is used to predict the speech waveform corresponding to the target spectrum.

[0053] In the process of model training for the neural network vocoder, the speech information can be divided into multiple frequency bands by means of multiple sub-bands, and then trained and inferred separately, and finally the sub-bands are merged. In addition, in the process of model training for the neural network vocoder, the amplitude of the speech waveform in the training data can be normalized first, such as making its maximum amplitude 0 decibels, and then uniformly reducing the set value, such as reducing it by 6 decibels. In this way, the amplitude of the predicted speech waveform output by the neural network vocoder under normal circumstances during the use phase is below -6 decibels. On the one hand, it can avoid the overload phenomenon of the predicted speech waveform, and on the other hand, it can ensure that there is sufficient sound range for energy detection.

[0054] After the neural network vocoder is trained, the neural network vocoder can be used to infer the target spectrum to obtain the predicted speech waveform.

[0055] S130: Performing pop detection on the speech waveform to determine whether there is pop in the speech waveform; if yes, re-performing S120; if no, proceeding to S140;

[0056] The target spectrum generated based on the target text is obtained, and the target spectrum is inferred using a neural network vocoder. After the predicted speech waveform is obtained, the speech waveform can be further subjected to pop detection.

[0057] A popping sound refers to a sound that is not only suddenly too loud in volume, but also has no clear or understandable information in the sound content.

[0058] If a neural network vocoder is used to infer the target spectrum and there are popping sounds in the predicted speech waveform, if the speech corresponding to the speech waveform is directly output, it will affect the user's sound experience.

[0059] Therefore, the present application does not directly output the speech corresponding to the speech waveform, but performs a pop detection on the speech waveform. If a pop is detected in the speech waveform, the step of using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform is repeated, such as Figure 1 shown.

[0060] It is understandable that the neural network vocoder is essentially based on the principle of probability statistics. Its characteristic is that even if the input is the same each time, the predicted speech waveform inferred by the neural network vocoder will have different values ​​at each sampling point, but the speech content sounds the same, or the predicted speech waveform inferred each time is similar. That is, the neural network vocoder has the working characteristic that the predicted speech waveform is different but does not affect the speech content. On this basis, when it is detected that there is a pop in the speech waveform predicted by the neural network vocoder, the step of using the neural network vocoder to infer the target spectrum and obtain the predicted speech waveform is repeated, which will make the predicted speech waveforms obtained twice before and after similar, but there are still differences. The re-obtained predicted speech waveform can continue to be detected for pops.

[0061] After the pop detection is performed on the speech waveform, if no pop is detected in the speech waveform, the operation of step S140 may be performed.

[0062] S140: Outputting the speech corresponding to the speech waveform without popping sound.

[0063] After using a neural network vocoder to infer the target spectrum, obtain a predicted speech waveform, and perform pop detection on the speech waveform, if no pop is detected in the speech waveform, the speech waveform without pops can be encapsulated into a corresponding audio file using a preset encapsulation format, and the audio file can be decoded into speech for playback in application scenarios such as user terminals and robots.

[0064] By using the method provided in the embodiment of the present application, after the target spectrum is inferred by the neural network vocoder to obtain the predicted speech waveform, the speech corresponding to the speech waveform is not directly output, but the speech waveform is first subjected to pop detection, and the corresponding speech is output when there is no pop. In the case of pop, the target spectrum is re-inferred by the neural network vocoder to obtain the predicted speech waveform, and the pop detection is performed on the speech waveform obtained again, until the predicted speech waveform obtained by inferring the target spectrum by the neural network vocoder is not detected, and the speech corresponding to the speech waveform without pop is output, so that there is no pop in the final output speech, and the final speech is more stable and reliable, which improves the quality of the synthesized speech.

[0065] In one embodiment of the present application, before using the neural network vocoder to infer the target spectrum to obtain the predicted speech waveform, the method may further include the following steps:

[0066] Step 1: Detect the amplitude of the target spectrum;

[0067] Step 2: Adjust the detected amplitude that exceeds the preset amplitude range to within the amplitude range.

[0068] For ease of description, the above two steps are combined for illustration.

[0069] In the embodiment of the present application, after obtaining the target spectrum generated based on the target text, the target spectrum can be further subjected to amplitude detection to determine whether there is an amplitude in the target spectrum that exceeds a preset amplitude range. The amplitude range can be set and adjusted according to actual conditions, and the embodiment of the present application does not limit this.

[0070] After the target spectrum is amplitude-detected, the detected amplitude exceeding the preset amplitude range can be adjusted to within the amplitude range. Specifically, the amplitude exceeding the upper limit of the amplitude range can be replaced by the upper limit, and the amplitude less than the lower limit of the amplitude range can be replaced by the lower limit.

[0071] After adjusting the amplitude of the detected target spectrum that exceeds the preset amplitude range to within the amplitude range, the target spectrum is input into the neural network vocoder, and the neural network vocoder is used to infer the target spectrum to obtain the predicted speech waveform. This can effectively avoid the problem of inaccurate inference caused by the amplitude of the target spectrum input to the neural network vocoder being too large or too small.

[0072] In one embodiment of the present application, performing pop detection on a speech waveform may include the following steps:

[0073] The speech waveform is subjected to a time domain popping detection according to an absolute value of the amplitude of the speech waveform and a relative amplitude change rate in the time domain.

[0074] In an embodiment of the present application, after using a neural network vocoder to infer the target spectrum and obtain a predicted speech waveform, the speech waveform can first be subjected to a time domain pop detection based on the absolute value of the amplitude and the relative rate of change of the amplitude of the speech waveform in the time domain to determine whether there is a pop in the time domain.

[0075] The manifestation of popping sound in the time domain is high energy and sudden changes in energy in time. The volume of the voice can be reflected by measuring the absolute value of the amplitude, and the fluctuation of the volume can be reflected by the relative rate of change of the amplitude. In the embodiment of the present application, it is possible to determine whether there is popping sound in the voice waveform in the time domain by detecting the amplitude of the voice waveform in the time domain state and the change of the amplitude in the time dimension.

[0076] In a specific embodiment of the present application, whether there is a pop in the time domain of the speech waveform in the time domain is determined based on the absolute value of the amplitude of each sampling point of the speech waveform in the time domain, and / or the absolute value of the amplitude difference between every two adjacent sampling points, and / or the absolute value of the amplitude difference between every two sampling points separated by one or more sampling points.

[0077] Specifically, for each sampling point of the speech waveform in the time domain, it can be determined whether the absolute value of the amplitude of the sampling point is greater than a preset first threshold value. If it is greater than the first threshold value, it can be determined that there is popping in the speech waveform in the time domain. If it is less than or equal to the first threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the adjacent subsequent sampling point is greater than a preset second threshold value. If it is greater than the second threshold value, it can be determined that there is popping in the speech waveform in the time domain. If it is less than or equal to the second threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the second subsequent sampling point is greater than a preset third threshold value, and the second subsequent sampling point is separated from the sampling point by one sampling point. If it is greater than the third threshold value, it can be determined that there is popping in the speech waveform in the time domain. If it is less than or equal to the third threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the third subsequent sampling point is greater than a preset fourth threshold value, and the third subsequent sampling point is separated from the sampling point by two sampling values. If it is greater than the fourth threshold value, it can be determined that there is popping in the speech waveform in the time domain. If it is less than or equal to the fourth threshold value, it can be determined that there is no popping in the speech waveform in the time domain.

[0078] The above process can be expressed by the following time domain detection formula:

[0079] ψ=Count:{||Y n ||>α};

[0080] Δ1=Count:{||Y n+1 -Y n ||>β};

[0081] Δ2=Count:{||Y n+2 -Y n ||>γ};

[0082] Δ3=Count:{||Y n+3 -Y n ||>δ};

[0083] Among them, ||Y n|| represents the absolute value of the amplitude of the nth sampling point of the speech waveform in the time domain. Among the four formulas from top to bottom, the first formula is used to calculate the number of sampling points of the speech waveform in the time domain whose absolute value is greater than the first threshold α, the second formula is used to calculate the number of adjacent sampling points of the speech waveform in the time domain whose absolute value of the amplitude difference is greater than the second threshold β, the third formula is used to calculate the number of sampling points of the speech waveform in the time domain whose absolute value of the amplitude difference is greater than the third threshold γ, and the fourth formula is used to calculate the number of sampling points of the speech waveform in the time domain whose absolute value of the amplitude difference is greater than the third threshold δ. The second formula to the third formula reflect the energy change rate. As long as there is a sampling value greater than the corresponding threshold, it can be determined that there is a pop in the speech waveform in the time domain. It can also be determined that there is a pop in the speech waveform in the time domain only when the number of sampling points greater than the corresponding threshold reaches a certain number.

[0084] In one embodiment of the present application, after performing pop detection on the speech waveform in the time domain, the method may include the following steps:

[0085] If there is no popping sound in the speech waveform in the time domain, the popping sound detection in the frequency domain is performed on the speech waveform according to the energy of each sub-band frequency interval of the speech waveform in the frequency domain.

[0086] After performing pop detection on the speech waveform in the time domain, if it is determined that there is no pop in the speech waveform in the time domain, the speech waveform can be further detected in the frequency domain based on the energy of each sub-band frequency interval of the speech waveform in the frequency domain to improve the detection accuracy of the pop.

[0087] The performance of popping in the frequency domain is that the energy in some sub-band frequency intervals is small, but the energy in other sub-band frequency intervals is extremely high. Based on this, the energy changes in each frequency sub-band can be analyzed in the frequency domain to detect popping in the speech waveform. That is, the potential popping can be further screened through the sub-band energy changes in the frequency domain.

[0088] In a specific implementation of the present application, it is determined whether the absolute value of the difference in energy between each two adjacent sub-band frequency intervals of the speech waveform in the frequency domain is greater than a preset energy difference threshold; if so, it is determined that the speech waveform has popping sounds in the frequency domain; if not, it is determined that the speech waveform does not have popping sounds in the frequency domain.

[0089] Specifically, the energy of the speech waveform in each sub-band frequency interval in the frequency domain may be calculated first. For example, the calculation may be performed based on the following formula:

[0090]

[0091] Among them, freq represents any sub-band, such as the frequency range of 100 Hz to 1000 Hz is a sub-band, F freq is the spectral amplitude within subband freq.

[0092] After calculating the energy of the speech waveform in each sub-band frequency interval in the frequency domain, it can be further determined whether the absolute value of the difference in energy between two adjacent sub-band frequency intervals is greater than a preset energy difference threshold. If it is greater than the energy difference threshold, it can be determined that there is popping in the speech waveform in the frequency domain. If the absolute value of the difference in energy between two adjacent sub-band frequency intervals is less than or equal to the energy difference threshold, it can be determined that there is no popping in the speech waveform in the frequency domain. Specifically, the following frequency domain detection formula can be used for determination.

[0093] Δ4=Count:{||H band+1 -H band ||>ε};

[0094] This formula is used to calculate the number of times the absolute value of the difference in energy between two adjacent sub-band frequency intervals of the speech waveform in the frequency domain is greater than the energy difference threshold ε, for example, whether the absolute value of the difference between the sub-band energy of 100 Hz to 1000 Hz and the energy in the sub-band frequency interval of 1000 Hz to 2000 Hz is greater than the energy difference threshold ε.

[0095] In one embodiment of the present application, when no popping sound is detected in the speech waveform, before outputting the speech corresponding to the speech waveform without popping sound, the method further includes:

[0096] Performing silence detection on the speech corresponding to the speech waveform without popping sound;

[0097] If it is detected that there is target silence in the speech whose silence duration is greater than a preset duration threshold, the target silence in the speech is processed.

[0098] If there is silence in the speech corresponding to the speech waveform without popping sound, there will be a long period of silence in the hearing sense during playback, which affects the user's auditory experience. Therefore, in the embodiment of the present application, when the speech waveform is subjected to popping sound detection and no popping sound is detected in the speech waveform, the speech corresponding to the speech waveform can be further subjected to silence detection. If silence is detected in the speech, it can be further determined whether the silence duration is greater than a preset duration threshold. If it is less than or equal to, it can be considered that the silence duration is tolerable. If it is greater, the silence can be processed.

[0099] Specifically, the voice pair can be detected by voice activity detection (VAD) technology to obtain silence without speaking. If the silence duration is greater than the duration threshold, the silence can be truncated or replaced with a preset silence of a shorter duration. That is, the target silence in the voice is processed, including: truncating the target silence in the voice, or replacing the target silence in the voice with silence whose duration is less than the preset duration threshold.

[0100] The silence in the speech corresponding to the speech waveform without popping sound is processed before outputting, which can effectively avoid long periods of silence in the final listening experience.

[0101] Of course, in actual applications, whether to perform the mute processing can be determined based on the switch state of the mute processing, and the mute processing operation can also be not performed when background music is to be added to the voice.

[0102] Through the technical solution of the embodiments of the present application, the stability of the sound quality of the synthesized speech can be improved, the harsh listening problem caused by explosive sounds can be avoided, and at the same time, long periods of silent speech can be reduced. Ultimately, the synthesized speech output to the user is more reliable and of higher quality, so that users who listen to the synthesized speech can obtain a better sound auditory experience.

[0103] The speech synthesis method provided by the present application is described below in a specific application embodiment. When a novel needs to be read aloud in the form of speech, the text of the novel is input into the acoustic model to generate a spectrum, and then the spectrum is input into a neural network vocoder based on a generative adversarial network to predict the speech waveform. For the speech waveform predicted by the neural network vocoder, determine whether the absolute value of the amplitude of each sampling point in the time domain is greater than a preset first threshold value. If it is greater than the first threshold value, it can be determined that there is a pop in the speech waveform in the time domain. If it is less than or equal to the first threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the adjacent subsequent sampling point is greater than a preset second threshold value. If it is greater than the second threshold value, it can be determined that there is a pop in the speech waveform in the time domain. If it is less than or equal to the second threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the second sampling point thereafter is greater than a preset third threshold value, and the second sampling point thereafter is separated from the sampling point by one sampling point. If it is greater than the third threshold value, it can be determined that there is a pop in the speech waveform in the time domain. If it is less than or equal to the third threshold value, it can be further determined whether the absolute value of the amplitude difference between the sampling point and the third sampling point thereafter is greater than a preset fourth threshold value, and the third sampling point thereafter is separated from the sampling point by two sampling values. If it is greater than the fourth threshold value, it can be determined that there is a pop in the speech waveform in the time domain. If it is less than or equal to the fourth threshold value, it can be determined that there is no pop in the speech waveform in the time domain.

[0104] If the speech waveform has popping sounds in the time domain, the neural network vocoder predicts the speech waveform based on the spectrum again, and detects whether the speech waveform has popping sounds in the time domain again in the above manner. If the speech waveform does not have popping sounds in the time domain, the speech waveform is transformed in time-frequency mode, the energy of the speech waveform in each sub-band frequency interval in the frequency domain is calculated, and it is determined whether the absolute value of the difference in energy between each two adjacent sub-band frequency intervals is a preset fifth threshold. If it is greater than the fifth threshold, it can be determined that the speech waveform has popping sounds in the frequency domain. If the absolute value of the difference in energy between each two adjacent sub-band frequency intervals is less than or equal to the fifth threshold, it can be determined that the speech waveform does not have popping sounds in the frequency domain.

[0105] If the speech waveform has popping sounds in the frequency domain, the neural network vocoder predicts the speech waveform based on the spectrum again, and detects whether the speech waveform has popping sounds in the time domain again in the above manner. If the speech waveform does not have popping sounds in the frequency domain, the speech waveform is packaged into an audio file in MP3 format, and then the audio file is decoded into PCM (Pulse Code Modulation) audio data for playback, thus realizing the reading of a novel in the form of speech.

[0106] Corresponding to the above method embodiment, the embodiment of the present application further provides a speech synthesis device. The speech synthesis device described below and the speech synthesis device method described above can be referenced to each other.

[0107] See also Figure 2 As shown, the device may include the following modules:

[0108] A target spectrum acquisition module 210 is used to acquire a target text and input the target text into a target spectrum generated by an acoustic model;

[0109] A speech waveform prediction module 220 is used to infer the target spectrum using a neural network vocoder to obtain a predicted speech waveform;

[0110] The pop detection module 230 is used to detect pops in the speech waveform; if the pop is detected in the speech waveform, the workflow of the speech waveform prediction module 220 is triggered until the pop does not exist in the speech model; if the pop is not detected in the speech waveform, the workflow of the synthetic speech output module 240 is triggered;

[0111] The synthesized speech output module 240 is used to output speech corresponding to a speech waveform without popping sounds.

[0112] When the device provided by the embodiment of the present application is used, after the target spectrum is inferred by the neural network vocoder to obtain the predicted speech waveform, the speech corresponding to the speech waveform is not directly output, but the speech waveform is first subjected to pop detection, and the corresponding speech is output when there is no pop. When there is pop, the target spectrum is re-inferred by the neural network vocoder to obtain the predicted speech waveform, and the pop detection is performed on the speech waveform obtained again, until the predicted speech waveform obtained by inferring the target spectrum by the neural network vocoder is not detected, and the speech corresponding to the speech waveform without pop is output, so that there is no pop in the final output speech, and the final speech is more stable and reliable, which improves the quality of the synthesized speech.

[0113] In a specific implementation of the present application, the pop detection module 230 is used to:

[0114] The speech waveform is subjected to a time domain popping detection according to an absolute value of the amplitude of the speech waveform and a relative amplitude change rate in the time domain.

[0115] In a specific implementation of the present application, the pop detection module 230 is used to:

[0116] Determine whether there is a pop in the time domain of the speech waveform in the time domain based on the absolute value of the amplitude of each sampling point of the speech waveform in the time domain, and / or the absolute value of the amplitude difference between every two adjacent sampling points, and / or the absolute value of the amplitude difference between every two sampling points separated by one or more sampling points.

[0117] In a specific implementation of the present application, the pop detection module 230 is further used to:

[0118] After performing pop detection on the speech waveform in the time domain according to the absolute value of the amplitude and the relative rate of change of the amplitude in the time domain, if there is no pop detection on the speech waveform in the time domain, performing pop detection on the speech waveform in the frequency domain according to the energy in each sub-band frequency interval of the speech waveform in the frequency domain.

[0119] In a specific implementation of the present application, the pop detection module 230 is used to:

[0120] Determine whether the absolute value of the difference in energy between each two adjacent sub-band frequency intervals of the speech waveform in the frequency domain is greater than a preset energy difference threshold; if so, determine that the speech waveform has popping in the frequency domain; if not, determine that the speech waveform does not have popping in the frequency domain.

[0121] In a specific implementation of the present application, a mute processing module is also included, which is used to:

[0122] Before outputting the speech corresponding to the speech waveform without popping sounds, performing silence detection on the speech corresponding to the speech waveform without popping sounds;

[0123] If it is detected that there is target silence in the speech whose silence duration is greater than a preset duration threshold, the target silence in the speech is processed.

[0124] In a specific implementation of the present application, the mute processing module is used to:

[0125] performing truncation processing on the target silence in the speech,

[0126] Or, the target silence in the speech is replaced by silence whose silence duration is less than the preset duration threshold.

[0127] In a specific implementation of the present application, an amplitude adjustment module is also included, which is used to:

[0128] Before using the neural network vocoder to infer the target spectrum and obtain the predicted speech waveform, the target spectrum is amplitude-detected;

[0129] The detected amplitude exceeding the preset amplitude range is adjusted to be within the amplitude range.

[0130] Corresponding to the above method embodiment, the present application embodiment further provides a speech synthesis device, including:

[0131] Memory for storing computer programs;

[0132] A processor is used to implement the steps of the above-mentioned speech synthesis method when executing a computer program.

[0133] like Figure 3 , which is a schematic diagram of the structure of a speech synthesis device, the speech synthesis device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0134] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0135] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the speech synthesis method.

[0136] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0137] Acquire a target text, and input the target text into a target spectrum generated by an acoustic model;

[0138] Inferring the target spectrum using a neural network vocoder to obtain a predicted speech waveform;

[0139] Performing pop detection on the speech waveform;

[0140] If it is detected that there is a pop in the speech waveform, repeatedly performing the step of using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform until there is no pop in the speech waveform;

[0141] Output the speech corresponding to the speech waveform without popping sounds.

[0142] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function (such as a spectrum function and a pop detection function), etc.; the data storage area may store data created during use, such as spectrum data, voice data, etc.

[0143] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0144] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0145] Of course, it should be noted that Figure 3 The structure shown does not constitute a limitation on the speech synthesis device in the embodiment of the present application. In actual applications, the speech synthesis device may include Figure 3 More or fewer components than shown, or combinations of certain components.

[0146] Corresponding to the above method embodiment, the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method are implemented.

[0147] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0148] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0149] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0150] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a target text, and input the target text into a target spectrum generated by an acoustic model; Inferring the target spectrum using a neural network vocoder to obtain a predicted speech waveform; Performing pop detection on the speech waveform; If it is detected that there is a pop in the speech waveform, repeatedly performing the step of using a neural network vocoder to infer the target spectrum to obtain a predicted speech waveform until there is no pop in the speech waveform; Output the speech corresponding to the speech waveform without popping sounds.

2. The speech synthesis method according to claim 1, characterized in that: Performing pop detection on the speech waveform includes: The speech waveform is subjected to a time domain popping detection according to an absolute value of the amplitude of the speech waveform and a relative amplitude change rate in the time domain.

3. The speech synthesis method according to claim 2, characterized in that: The step of performing time domain pop detection on the speech waveform according to the absolute value of the amplitude and the relative amplitude change rate of the speech waveform in the time domain comprises: Determine whether there is a pop in the time domain of the speech waveform in the time domain based on the absolute value of the amplitude of each sampling point of the speech waveform in the time domain, and / or the absolute value of the amplitude difference between every two adjacent sampling points, and / or the absolute value of the amplitude difference between every two sampling points separated by one or more sampling points.

4. The speech synthesis method according to claim 2, characterized in that: After performing the time domain pop detection on the speech waveform according to the absolute value of the amplitude and the relative amplitude change rate of the speech waveform in the time domain, the method further includes: If there is no popping sound in the speech waveform in the time domain, the popping sound detection in the frequency domain is performed on the speech waveform according to the energy of each sub-band frequency interval of the speech waveform in the frequency domain.

5. The speech synthesis method according to claim 4, characterized in that: The performing frequency domain pop detection on the speech waveform according to the energy of each sub-band frequency interval of the speech waveform in the frequency domain comprises: Determine whether the absolute value of the energy difference between each two adjacent sub-band frequency intervals of the speech waveform in the frequency domain is greater than a preset energy difference threshold; If yes, it is determined that the speech waveform has a pop in the frequency domain; If not, it is determined that there is no popping sound in the speech waveform in the frequency domain.

6. The speech synthesis method according to claim 1, characterized in that: Before the outputting of the speech corresponding to the speech waveform without popping sound, the method further comprises: Performing silence detection on the speech corresponding to the speech waveform without popping sound; If it is detected that there is target silence in the speech whose silence duration is greater than a preset duration threshold, the target silence in the speech is processed.

7. The speech synthesis method according to claim 6, characterized in that: Processing the target silence in the speech includes: performing truncation processing on the target silence in the speech, Or, the target silence in the speech is replaced by silence whose silence duration is less than the preset duration threshold.

8. The speech synthesis method according to any one of claims 1 to 7, characterized in that: Before the target spectrum is inferred by the neural network vocoder to obtain the predicted speech waveform, the method further includes: Performing amplitude detection on the target spectrum; The detected amplitude exceeding the preset amplitude range is adjusted to be within the amplitude range.

9. A speech synthesis device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the speech synthesis method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 8 are implemented.

Images