Voice signal synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a voice signal synthesis method and device, electronic equipment and a storage medium, and relates to the technical field of voice synthesis, the method predicts the amplitude spectrum and phase spectrum of a voice signal at a frame level, and avoids the periodic noise problem caused by an up-sampling structure with large calculation amount and transposition convolution. Moreover, according to the method, the frame-level fundamental frequency is obtained from the input voice signal Mel spectrum, the voiced sound frames and the non-voiced sound frames are determined, and the reference amplitude spectrum is generated in an energy information distribution mode, so that extra amplitude spectrum information can be provided for predicting the amplitude spectrum and the phase spectrum of the voice signal frame by frame, the prediction difficulty is reduced, and the prediction efficiency is improved. Therefore, the synthesis efficiency and quality of the voice signal are improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech signal synthesis method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous maturity of artificial intelligence technology, speech synthesis technology has developed rapidly. Most speech synthesis systems first use acoustic models to model text information into low-resolution representations, such as mel spectrograms, and then use vocoders to restore the original waveform signal from the mel spectrogram. Therefore, the performance of the vocoder will significantly affect the quality and speed of synthesized speech. Most traditional vocoders are designed based on the source filter theory of human speech generation. The excitation signal is generated through the excitation parameters of the source, and the final waveform is synthesized through the filtering process. However, traditional vocoders often ignore complex nonlinear changes and generate waveforms of poor quality.

[0003] Thanks to the rapid development of neural network technology, more and more waveform synthesis models based on neural networks have been proposed by scholars. Among them, the waveform generation model based on autoregressive neural network (WaveNet, etc.) and the waveform generation model based on flow neural network (WaveGlow, etc.) have achieved high-quality waveform generation, but due to their very low waveform synthesis efficiency, their use scenarios are still limited.

[0004] The generative adversarial network vocoder, which uses a fully convolutional network as the backbone, has gradually become one of the mainstream vocoders used in academia and industry due to its good synthesis quality and high synthesis efficiency. The generative adversarial network vocoder uses transposed convolution and multi-receptive field fusion modules to directly map the mel spectrogram to the speech waveform, and minimizes the gap between the synthesized waveform and the real waveform through multi-period discriminators and multi-scale discriminators. However, since it needs to upsample from the mel feature to the audio sampling rate (such as 24kHz), the amount of calculation is large and it cannot be directly deployed in products with weak computing power. In addition, the transposed convolution used in upsampling will cause frequency domain aliasing and periodic noise, and the time-frequency information of the mel spectrum is not well utilized. The inverse short-time Fourier transform network (iSTFTNet) vocoder improves the upsampling layer of the generative part of the generative adversarial network vocoder to make the model faster and lighter while ensuring audio quality. However, transposed convolution is still required for upsampling, which is computationally intensive and will cause frequency domain aliasing and periodic noise. Summary of the invention

[0005] The present invention provides a speech signal synthesis method, device, electronic equipment and storage medium to solve the defects existing in the related art.

[0006] The present invention provides a speech signal synthesis method, comprising: Get the Mel spectrum of speech signal; Extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining the voiced frames and the unvoiced frames in the Mel spectrum of the speech signal; Determine the energy information of each frame in the Mel spectrum of the speech signal, and distribute the energy information of the non-voiced frame, and distribute the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; Based on the Mel spectrum of the speech signal and the reference amplitude spectrum, the amplitude spectrum and the phase spectrum of the speech signal are predicted frame by frame, and based on the amplitude spectrum and the phase spectrum, the speech signal is synthesized.

[0007] According to a speech signal synthesis method provided by the present invention, the extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal and determining the voiced sound frame and the non-voiced sound frame in the Mel spectrum of the speech signal includes: Extracting features from the speech signal mel spectrum to obtain a first feature vector and a second feature vector; Activating the first feature vector to obtain an unvoiced and voiced sound label of each frame in the speech signal Mel spectrum; the unvoiced and voiced sound label is used to indicate whether each frame in the speech signal Mel spectrum is a voiced sound frame; An exponential operation is performed on the second eigenvector to obtain the fundamental frequency of each frame in the Mel-spectrogram of the speech signal.

[0008] According to a speech signal synthesis method provided by the present invention, the determining of energy information of each frame in the mel spectrum of the speech signal includes: Converting the speech signal Mel spectrum into a speech signal amplitude spectrum; Calculating the square of the amplitude of each frequency point in the amplitude spectrum of the speech signal, and determining the energy information of each frame in the Mel spectrum of the speech signal based on the sum of the square of the amplitude of each frequency point in each frame in the amplitude spectrum of the speech signal; or, The square of the amplitude of each frequency point in the Mel spectrum of the speech signal is calculated, and the energy information of each frame in the Mel spectrum of the speech signal is determined based on the sum of the squares of the amplitudes of each frequency point in each frame in the Mel spectrum of the speech signal.

[0009] According to a speech signal synthesis method provided by the present invention, the energy information of the voiced sound frame is distributed based on the fundamental frequency of the voiced sound frame, including: Determine the dimension of the reference amplitude spectrum based on the number of Fourier transform points, and determine the frequency interval of adjacent dimensions of the reference amplitude spectrum based on the sampling rate of the speech signal; Based on the energy information of the voiced speech frame, the energy of each harmonic position of the voiced speech frame is determined, and based on the fundamental frequency of the voiced speech frame, the frequency interval and the relationship between the frequency of each harmonic position of the voiced speech frame and the fundamental frequency, the dimension of each harmonic position of the voiced speech frame on the reference amplitude spectrum is determined.

[0010] According to a speech signal synthesis method provided by the present invention, the amplitude spectrum and phase spectrum of the speech signal are predicted frame by frame based on the Mel spectrum of the speech signal and the reference amplitude spectrum, including: Based on the generator network module, a first hidden vector of the Mel spectrum of the speech signal and a second hidden vector of the reference amplitude spectrum are respectively extracted, the first hidden vector and the second hidden vector are added and then dimensionality reduction processing is performed on the added results to obtain a first dimensional vector and a second dimensional vector, residual processing is performed on the first dimensional vector to obtain a third dimensional vector, and the third dimensional vector is spliced ​​with the first dimensional vector, and the amplitude spectrum and the phase spectrum are obtained based on the spliced ​​results; The dimension of the first-dimensional vector is greater than the dimension of the second-dimensional vector.

[0011] According to a speech signal synthesis method provided by the present invention, the step of obtaining a speech signal mel spectrum comprises: Applying a fundamental frequency extraction module in a neural network vocoder to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining voiced frames and non-voiced frames in the Mel spectrum of the speech signal; Applying the reference amplitude spectrum generation module in the neural network vocoder, determining the energy information of each frame in the mel spectrum of the speech signal, and distributing the energy information of the non-voiced frame, distributing the energy information of the voiced frame based on the fundamental frequency of the voiced frame, and obtaining a reference amplitude spectrum; Applying the generator network module in the neural network vocoder, based on the speech signal Mel spectrum and the reference amplitude spectrum, predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, and synthesizing the speech signal based on the amplitude spectrum and the phase spectrum; The generator network module is obtained by jointly training the discriminator based on the sample audio signal by applying the fundamental frequency extraction module and the reference amplitude spectrum generation module.

[0012] According to a speech signal synthesis method provided by the present invention, the training step of the generator network module includes: Inputting the sample Mel spectrum of the sample audio signal into an initial neural network vocoder to obtain a synthetic audio signal of the sample audio signal output by the initial neural network vocoder; The synthesized audio signal is converted into a frequency domain using a plurality of different short-time Fourier transform parameters to obtain a synthesized amplitude spectrum of the synthesized audio signal, and the synthesized amplitude spectrum is converted into a synthesized Mel spectrum using different Mel transform matrices; Calculating a mel reconstruction loss based on the synthesized mel spectrum and the sample mel spectrum, and calculating a generative adversarial loss and a feature matching loss of an initial discriminator based on the synthesized audio signal and the sample audio signal; Based on the Mel reconstruction loss, the feature matching loss and the generative adversarial loss, the initial neural network vocoder and the initial discriminator are iteratively trained to obtain the neural network vocoder and the generator network module therein.

[0013] The present invention also provides a speech signal synthesis device, comprising: A mel spectrum acquisition unit, used for acquiring a mel spectrum of a speech signal; A fundamental frequency extraction unit, used to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determine the voiced sound frame and the unvoiced sound frame in the Mel spectrum of the speech signal; A reference amplitude spectrum generating unit, configured to determine energy information of each frame in the mel spectrum of the speech signal, and to distribute energy information of the non-voiced frames, and to distribute energy information of the voiced frames based on the fundamental frequency of the voiced frames, so as to obtain a reference amplitude spectrum; The speech synthesis unit is used to predict the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the Mel spectrum of the speech signal and the reference amplitude spectrum, and synthesize the speech signal based on the amplitude spectrum and the phase spectrum.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech signal synthesis method as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech signal synthesis method as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech signal synthesis method as described above is implemented.

[0017] The speech signal synthesis method, device, electronic device and storage medium provided by the present invention predict the amplitude spectrum and phase spectrum of the speech signal at the frame level, avoiding the periodic noise problem caused by the upsampling structure with large computational complexity and the transposed convolution. Moreover, the method obtains the fundamental frequency at the frame level from the Mel spectrum of the input speech signal, determines the voiced frame and the non-voiced frame, and generates a reference amplitude spectrum by allocating energy information, which can provide additional amplitude spectrum information for predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, reduce the difficulty of prediction, and thus improve the synthesis efficiency and quality of the speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the speech signal synthesis method provided by the present invention.

[0020] Figure 2 It is a structural schematic diagram of a generator network module in the speech signal synthesis method provided by the present invention.

[0021] Figure 3 It is a schematic diagram for comparing a true amplitude spectrum of a sample audio signal and a sample reference amplitude spectrum in the speech signal synthesis method provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the speech signal synthesis device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Existing adversarial neural network vocoders can be divided into two types. One, such as the iSTFTNet vocoder, directly uses the Mel map to synthesize speech signals through an upsampling structure; the other, such as the Vocos vocoder, directly predicts the amplitude spectrum and phase spectrum from the Mel map through a convolutional structure, and synthesizes speech signals through an inverse short-time Fourier transform.

[0026] However, the upsampling structure is generally a transposed convolution, which requires a large amount of calculation and cannot be directly deployed in products with weak computing power. It will also cause frequency domain aliasing and periodic noise, and it does not make good use of the time-frequency information of the Mel spectrum. The amplitude spectrum and phase spectrum are directly predicted from the Mel map through the convolution structure, which requires a high amount of information in the Mel map. When the dimension of the Mel map is low, the quality of the synthesized speech signal will be significantly reduced, and there may be fundamental frequency jitter.

[0027] Based on this, an embodiment of the present invention provides a method for synthesizing a speech signal.

[0028] Figure 1 FIG. 1 is a flow chart of a speech signal synthesis method provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S1, obtain the speech signal Mel spectrum; S2, extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining the voiced frames and the unvoiced frames in the Mel spectrum of the speech signal; S3, determining the energy information of each frame in the Mel spectrum of the speech signal, and distributing the energy information of the non-voiced frame, and distributing the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; S4, predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the Mel spectrum of the speech signal and the reference amplitude spectrum, and synthesizing the speech signal based on the amplitude spectrum and the phase spectrum.

[0029] Specifically, the speech signal synthesis method provided in the embodiment of the present invention is executed by a speech signal synthesis device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0030] First, step S1 is performed to obtain the Mel-spectrogram of the speech signal. The Mel-spectrogram of the speech signal can be obtained by using the text representation of the speech signal to be synthesized through an acoustic model, or by converting the amplitude spectrum extracted from the speech signal obtained from other speech tasks into the Mel-scale, which is not specifically limited here.

[0031] Then, step S2 is performed to extract the fundamental frequency (F0) of each frame in the speech signal Mel spectrum. The fundamental frequency is the lowest frequency of each frame in the speech signal Mel spectrum in the frequency domain, also known as the fundamental frequency or pitch frequency. The fundamental frequency can be detected by using Librosa's yin function or other fundamental frequency detection algorithms (such as CREPE, Praat, etc.), or it can be extracted by using a trained fundamental frequency prediction module.

[0032] In addition, each frame in the speech signal Mel spectrum can be classified according to the pronunciation mode of each frame in the speech signal Mel spectrum, and the voiced frames and non-voiced frames in the speech signal Mel spectrum can be determined, and each frame can be marked to obtain the unvoiced and voiced labels of each frame. It can be understood that the unvoiced and voiced labels can indicate whether each frame in the speech signal Mel spectrum is a voiced frame. If the i-th frame in the speech signal Mel spectrum is a voiced frame, the unvoiced and voiced label of the i-th frame can be 1. If the i-th frame in the speech signal Mel spectrum is a non-voiced frame, that is, it is not a voiced frame, the unvoiced and voiced label of the i-th frame can be 0.

[0033] Here, the trained fundamental frequency prediction module can also be used to output the unvoiced and voiced sound labels of each frame in the speech signal Mel spectrum, and then determine the voiced sound frames and non-voiced sound frames in the speech signal Mel spectrum. At this time, the fundamental frequency prediction module can simultaneously complete the task of extracting the fundamental frequency of each frame in the speech signal Mel spectrum and the task of determining the unvoiced and voiced sound labels of each frame in the speech signal Mel spectrum. By inputting the speech signal Mel spectrum into the fundamental frequency prediction model, the fundamental frequency prediction model can simultaneously output the fundamental frequency and unvoiced and voiced sound labels of each frame in the speech signal Mel spectrum.

[0034] Then, step S3 is executed to determine the energy information of each frame in the Mel spectrum of the speech signal. Generally speaking, the Mel spectrum is a compressed representation of the spectrum, which is mapped to the Mel scale to simulate the human ear's perception of pitch. Therefore, the amplitude of the Mel spectrum does not directly reflect the energy information of the speech signal and does not directly correspond to the energy information of the speech signal.

[0035] Based on this, in an embodiment of the present invention, the speech signal Mel spectrum can be directly converted into a speech signal spectrum, and then the energy information of each frame in the speech signal Mel spectrum can be calculated according to the segment corresponding to each frame in the speech signal spectrum.

[0036] Since the voiced frames and the non-voiced frames in the speech signal Mel spectrum have been determined in step S2, the energy information of each frame in the speech signal Mel spectrum can be allocated according to the different categories of each frame in the speech signal Mel spectrum to obtain a reference amplitude spectrum. It can be understood that the energy information allocation for each frame is to determine the energy of each frame in each dimension of the reference amplitude spectrum.

[0037] Assume that the number of Fourier transform points is 1024 during signal processing. For 100 frames of speech signal Mel spectrum fbk, the length of the reference amplitude spectrum ref_mag to be generated is 100 frames, and the dimension is 1024 / 2+1=513.

[0038] For non-voiced frames, it is assumed that the energy is evenly distributed in each dimension of the reference amplitude spectrum, and the energy information of the non-voiced frames can be directly distributed. Usually, non-voiced frames can include unvoiced frames and silent frames. Unvoiced frames can be represented by Gaussian white noise, and the energy information of silent frames is 0. If the jth frame is an unvoiced frame, its energy information is energy[j]=5, then its energy in each dimension of the reference amplitude spectrum is ref_mag[j][:]=5 / 513.

[0039] For the voiced sound frame, it is considered that its energy is mainly concentrated on the fundamental frequency and its harmonics, and then the energy of each harmonic can be directly determined according to the number of harmonics, and the position of the fundamental frequency and the position of the harmonics can be determined by using the fundamental frequency of the voiced sound frame to obtain the reference amplitude spectrum. The positions of the fundamental frequency and the harmonics on the reference amplitude spectrum can be the dimensions of the fundamental frequency and the harmonics on the reference amplitude spectrum.

[0040] Here, the reference amplitude spectrum is also called the excitation amplitude spectrum, which can reflect the distribution of inter-frame and intra-frame energy in the Mel spectrum of the speech signal, and can assist in perceiving the change from the Mel spectrum of the speech signal to the true amplitude spectrum.

[0041] Finally, step S4 is executed to predict the amplitude spectrum and phase spectrum of the speech signal frame by frame using the speech signal Mel spectrum and the reference amplitude spectrum. This process can be performed by inputting the speech signal Mel spectrum and the reference amplitude spectrum into a trained generator network module, and the generator network module predicts the amplitude spectrum and phase spectrum of the speech signal frame by frame.

[0042] Furthermore, the amplitude spectrum and the phase spectrum can be synthesized into a complex spectrum, and the complex spectrum can be converted into a speech signal by inverse short-time Fourier transform. A possible implementation is that the number of Fourier transform points of the inverse short-time Fourier transform is 1024, and the frame shift is the same as the frame shift of the input speech signal Mel spectrum.

[0043] The speech signal synthesis method provided in the embodiment of the present invention first obtains the speech signal Mel spectrum; then extracts the fundamental frequency of each frame in the speech signal Mel spectrum, and determines the voiced frame and the non-voiced frame in the speech signal Mel spectrum; thereafter determines the energy information of each frame in the speech signal Mel spectrum, and distributes the energy information of the non-voiced frame, and distributes the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; finally, based on the speech signal Mel spectrum and the reference amplitude spectrum, predicts the amplitude spectrum and phase spectrum of the speech signal frame by frame, and synthesizes the speech signal based on the amplitude spectrum and the phase spectrum. The method predicts the amplitude spectrum and phase spectrum of the speech signal at the frame level, avoiding the periodic noise problem caused by the upsampling structure with large computational amount and the transposed convolution. Moreover, the method obtains the fundamental frequency at the frame level from the input speech signal Mel spectrum, determines the voiced frame and the non-voiced frame, and generates the reference amplitude spectrum by allocating energy information, which can provide additional amplitude spectrum information for predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, reduce the difficulty of prediction, and thus improve the synthesis efficiency and quality of the speech signal.

[0044] On the basis of the above embodiment, the step of extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal and determining the voiced frames and the unvoiced frames in the Mel spectrum of the speech signal includes: Extracting features from the speech signal mel spectrum to obtain a first feature vector and a second feature vector; Activating the first feature vector to obtain an unvoiced and voiced sound label of each frame in the speech signal Mel spectrum; the unvoiced and voiced sound label is used to indicate whether each frame in the speech signal Mel spectrum is a voiced sound frame; An exponential operation is performed on the second eigenvector to obtain the fundamental frequency of each frame in the Mel-spectrogram of the speech signal.

[0045] Specifically, the above step S2 can be implemented by a trained fundamental frequency extraction module. The feature extraction layer in the fundamental frequency extraction module can be used to extract the Mel spectrum of the speech signal to obtain the first feature vector and the second feature vector. The feature extraction layer may include a series of convolutional layers and activation layers. If the dimension of the Mel spectrum fbk of the speech signal is 80, an output out with a frame number of 100 and a dimension of 2 can be obtained after processing by the feature extraction layer. The output includes a first-dimensional output out[:,0] and a second-dimensional output out[:,1]. The first-dimensional output is the first feature vector, and the second-dimensional output is the second feature vector.

[0046] By using the activation network in the baseband extraction module, such as the Sigmoid network, the first feature vector can be activated and constrained to be between (0,1) to obtain a vector p of length 100, which is used as a criterion for judging voiced frames and non-voiced frames. For example, when p[i]>0.5 (i<100), the i-th frame is considered to represent a voiced sound, and when p[i]<=0.5, the i-th frame is considered not to represent a voiced sound, that is, it may be an unvoiced sound or silence. Thus, a Boolean vector uv of length 100 for unvoiced and voiced sound labels can be obtained, whose Boolean values ​​are the unvoiced and voiced sound labels of each frame in the speech signal Mel spectrum, indicating whether each frame is a voiced sound frame.

[0047] By using the exponential operation unit in the fundamental frequency extraction module, the second eigenvector can be exponentially operated by a natural constant, and can be multiplied by the unvoiced and voiced labels of each frame to obtain the fundamental frequency f0 of each frame in the speech signal Mel spectrum. When the i-th frame is a voiced frame, f0[i] represents the fundamental frequency of the i-th frame, in Hz; when the i-th frame is an unvoiced frame, f0[i]=0.

[0048] It is understandable that the fundamental frequency extraction module can be trained by using sample mel spectrograms carrying fundamental frequency labels and unvoiced and voiced sound labels, or can be jointly trained with a module that performs downstream tasks, which is not specifically limited here.

[0049] In the embodiment of the present invention, by extracting the features of the speech signal mel spectrum, the unvoiced and voiced sound labels and the fundamental frequency of each frame can be synchronously acquired, which can reduce the synthesis difficulty of the speech signal and improve the synthesis efficiency.

[0050] On the basis of the above embodiment, determining the energy information of each frame in the mel spectrum of the speech signal includes: Converting the speech signal Mel spectrum into a speech signal amplitude spectrum; Calculating the square of the amplitude of each frequency point in the amplitude spectrum of the speech signal, and determining the energy information of each frame in the Mel spectrum of the speech signal based on the sum of the square of the amplitude of each frequency point in each frame in the amplitude spectrum of the speech signal; or, The square of the amplitude of each frequency point in the Mel spectrum of the speech signal is calculated, and the energy information of each frame in the Mel spectrum of the speech signal is determined based on the sum of the squares of the amplitudes of each frequency point in each frame in the Mel spectrum of the speech signal.

[0051] Specifically, to calculate the energy information of each frame in the speech signal Mel spectrum, the speech signal Mel spectrum can be converted into the speech signal amplitude spectrum, and the amplitude of each frequency point of the speech signal amplitude spectrum can be squared. Then, by summing the squares of the amplitudes of all frequency points, the energy information of each frame can be obtained. This process can obtain very accurate energy information of each frame.

[0052] But in fact, it is not necessary to obtain very accurate energy information for each frame, as long as it can describe the energy changes between frames. Therefore, in this method, the amplitude of each frequency point of the speech signal Mel spectrum can be directly squared, and then the square of the amplitude of all frequency points in each frame is summed to obtain the energy information of each frame. Here, energy is a vector with a length of 100, and energy[i] represents the energy information of the i-th frame.

[0053] It is worth noting that if the Mel spectrum of the input speech signal is a logarithmic Mel spectrum, it needs to be restored to the Mel spectrum before performing the energy acquisition operation.

[0054] The embodiment of the present invention provides two methods for determining the energy information of each frame in the mel spectrum of the speech signal, so that the user can select an appropriate method to determine the energy information according to the needs, thereby providing convenience for the user.

[0055] Based on the above embodiment, the allocating energy information of the voiced sound frame based on the fundamental frequency of the voiced sound frame includes: Determine the dimension of the reference amplitude spectrum based on the number of Fourier transform points, and determine the frequency interval of adjacent dimensions of the reference amplitude spectrum based on the sampling rate of the speech signal; Based on the energy information of the voiced speech frame, the energy of each harmonic position of the voiced speech frame is determined, and based on the fundamental frequency of the voiced speech frame, the frequency interval and the relationship between the frequency of each harmonic position of the voiced speech frame and the fundamental frequency, the dimension of each harmonic position of the voiced speech frame on the reference amplitude spectrum is determined.

[0056] Specifically, if the number of Fourier transform points during signal processing is 1024, for 100 frames of speech signal Mel spectrum fbk, the dimension of the reference amplitude spectrum ref_mag that needs to be generated is 1024 / 2+1=513.

[0057] Assume that the i-th frame is a voiced frame, and assume that the sampling rate of the speech signal to be synthesized is 24000 Hz. The frequency interval of the adjacent dimensions of the reference amplitude spectrum is: 24000 / (513-1)=20 Hz. For the fundamental frequency f0 obtained from the fundamental frequency extraction module, if the fundamental frequency of the i-th frame is f0[i]=203 Hz, it should be located at the round(203 / 20)=10th dimension of the reference amplitude spectrum, where round represents the rounding operation. If the energy information energy[i] of the i-th frame is 50, and the number of harmonics n_frames of the reference amplitude spectrum to be generated is 4, then the energy of each harmonic of the i-th frame is energy[i] / (n_frames+1)=10. For the harmonic position, its frequency is usually an integer multiple of the fundamental frequency, and here it is taken as 2-5 times. The dimensions on the reference amplitude spectrum are round(203*2 / 20)=20, round(203*3 / 20)=30, round(203*4 / 20)=41, round(203*5 / 20)=51. The energy of the above dimensions of the reference amplitude spectrum is 10, that is, ref_mag[i]

[10] =ref_mag[i]

[20] =ref_mag[i]

[30] =ref_mag[i]

[41] =ref_mag[i]

[51] =10, and the energy of other dimensions is 0.

[0058] In the embodiment of the present invention, the energy information of the voiced sound frame is distributed through signal processing, which can simplify the difficulty of energy distribution.

[0059] On the basis of the above embodiment, the step of predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the speech signal Mel spectrum and the reference amplitude spectrum includes: Based on the generator network module, the first hidden vector of the speech signal Mel spectrum and the second hidden vector of the reference amplitude spectrum are extracted respectively, the first hidden vector and the second hidden vector are added and the added results are respectively subjected to dimensionality reduction processing to obtain a first dimensional vector and a second dimensional vector, the first dimensional vector is subjected to residual processing to obtain a third dimensional vector, and the third dimensional vector is spliced ​​with the first dimensional vector, and the amplitude spectrum and the phase spectrum are obtained based on the spliced ​​results.

[0060] Specifically, in the embodiment of the present invention, the amplitude spectrum and phase spectrum of the speech signal can be predicted frame by frame through the generator network module. The structure of the generator network module is as follows: Figure 2 As shown, the generator network module may include a first convolution module, a second convolution module, an addition module, a third convolution module, a fourth convolution module, a residual module, a fifth convolution module, a splicing module, a sixth convolution module and an inverse short-time Fourier transform unit.

[0061] The first convolution module is used to receive the reference amplitude spectrum, perform convolution calculation on the reference amplitude spectrum, change the feature dimension at the same time, and extract the second hidden vector of the reference amplitude spectrum. The first convolution module may include a convolution layer and a normalization layer.

[0062] The second convolution module is used to receive the speech signal Mel spectrum, perform convolution calculation on the Mel spectrum, and change the feature dimension to extract the first hidden vector of the speech signal Mel spectrum. The first convolution module may include a convolution layer and a normalization layer.

[0063] The addition module is connected to the first convolution module and the second convolution module respectively, and is also connected to the third convolution module and the fourth convolution module respectively. The addition module can perform an addition operation on the first hidden vector and the second hidden vector.

[0064] The addition results output by the addition unit are respectively input into the third convolution module and the fourth convolution module. The third convolution module is used to reduce the dimension of the addition result and extract some features to obtain a first dimensional vector, and the fourth convolution module is used to reduce the dimension of the addition result and extract some features to obtain a second dimensional vector. The third convolution module and the fourth convolution module can both include a convolution layer, a normalization layer and an activation layer.

[0065] Here, the dimension of the first dimension vector is greater than the dimension of the second dimension vector, and a possible dimension allocation ratio of the first dimension vector to the second dimension vector is 3: 1. For example, the result of the addition is a 512-dimensional hidden vector, the first dimension vector obtained after processing by the third convolution module becomes a 384-dimensional hidden vector, and the second dimension vector obtained after processing by the fourth convolution module becomes a 128-dimensional hidden vector.

[0066] This approach allows the generator network module to automatically divide the input into two parts according to the difficulty of modeling. The part that is more difficult to model continues to perform the subsequent residual processing, while the part that is easier to model can be processed through a simple network. One conceivable distribution method is that the speech signal Mel spectrum contains more amplitude spectrum information, but loses the phase spectrum information. The reference amplitude spectrum changes relatively continuously between frames and within frames, and recovery is relatively simple, while the phase spectrum changes more dramatically and has a phase wrapping problem, so it is difficult to recover. Therefore, the generator network module can simply process the amplitude information that is easier to recover through the third convolution module, and the phase information that is difficult to recover can be processed through a deeper residual module.

[0067] The first dimension vector obtained by the third convolution module passes through the residual module and the fifth convolution module in sequence. The residual module includes multiple residual convolution modules with jump links, such as ResBlock modules or ConvNext modules. In some possible implementations, each convolution layer (Convolutional layer) in the residual convolution module is composed of several convolution units, and the parameters of each convolution unit are optimized by the back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolution layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0068] The fifth convolution module may include a convolution layer, a normalization layer, and an activation function layer, for further processing the hidden features to obtain a third dimensional vector.

[0069] The third dimensional vector output by the fifth convolution module and the second dimensional vector output by the fourth convolution module are input to the concatenation unit together. The concatenation unit is used to concatenate the second dimensional vector and the third dimensional vector in the dimensional direction to obtain a higher dimensional hidden layer vector.

[0070] The splicing result output by the splicing unit is processed by the sixth convolution module, which includes a convolution layer and a normalization layer, and is used to fuse the splicing result and change the output dimension. One possible implementation is that the output dimension is 1026.

[0071] The output of the sixth convolution module is divided into two parts in the dimension direction. The first part has a dimension of 513, representing the logarithmic amplitude spectrum of the speech signal. The amplitude spectrum can be obtained by performing exponential operation on the logarithmic amplitude spectrum. The second part also has a dimension of 513, representing the phase spectrum of the speech signal.

[0072] In the embodiment of the present invention, the phase spectrum and amplitude spectrum of the speech signal are predicted by the generator network module, and the structure is simple and easy to implement. Moreover, the generator network module introduces a branch of the residual module during processing, which can accurately restore the phase information that is difficult to restore, thereby improving the accuracy of the synthesized speech signal.

[0073] Based on the above embodiment, the step of obtaining the speech signal mel spectrum includes: Applying a fundamental frequency extraction module in a neural network vocoder to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining voiced frames and non-voiced frames in the Mel spectrum of the speech signal; Applying the reference amplitude spectrum generation module in the neural network vocoder, determining the energy information of each frame in the mel spectrum of the speech signal, and distributing the energy information of the non-voiced frame, distributing the energy information of the voiced frame based on the fundamental frequency of the voiced frame, and obtaining a reference amplitude spectrum; Applying the generator network module in the neural network vocoder, based on the speech signal Mel spectrum and the reference amplitude spectrum, predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, and synthesizing the speech signal based on the amplitude spectrum and the phase spectrum; The generator network module is obtained by jointly training the discriminator based on the sample audio signal by applying the fundamental frequency extraction module and the reference amplitude spectrum generation module.

[0074] Specifically, in the embodiment of the present invention, after obtaining the mel spectrum of the speech signal, a neural network type vocoder can also be applied to implement the process of the above steps S2-S4, and the neural network type vocoder can include a fundamental frequency extraction module, a reference amplitude spectrum generation module and a generator network module. The process of step S2 is implemented by the fundamental frequency extraction module, the process of step S3 is implemented by the reference amplitude spectrum generation module, and the process of step S4 is implemented by the generator network module.

[0075] It is understandable that the generator network module can be obtained by using the sample audio signal, combining the baseband extraction module and the reference amplitude spectrum generation module to jointly train the discriminator. During the training process, the structural parameters of the baseband extraction module can also be fine-tuned to achieve joint training of the baseband extraction module, the generator network module and the discriminator, thereby obtaining an applicable baseband extraction module and a generator network module, and constructing an applicable neural network type vocoder by the baseband extraction module, the reference amplitude spectrum generation module and the generator network module.

[0076] During the training process, a Generative Adversarial Network (GAN) can be constructed, which includes a generator and a discriminator. The generator can include a baseband extraction network module, an excitation amplitude spectrum generation module, and a generator network module to be trained. The discriminator can include a multi-period discriminator (MPD) and a multi-resolution discriminator (MRD). Both MPD and MRD take the sample audio signal and the synthesized audio signal obtained by the generator network module as input.

[0077] MPD contains five sub-modules with different periods, which can respectively identify the components with periods of 2, 3, 5, 7 and 11 in the sample audio signal and the synthesized audio signal. Each sub-module includes five layers of two-dimensional convolution.

[0078] MRD contains three submodules with different short-time Fourier transform parameters, which respectively discriminate the amplitude spectrum of the sample audio signal and the amplitude spectrum of the synthetic audio signal under different short-time Fourier transform network (stft) parameters. Each submodule includes 5 layers of two-dimensional convolution.

[0079] In the embodiment of the present invention, the synthesis of speech signals can be achieved through a neural network vocoder, which can simplify the synthesis process and improve the synthesis efficiency.

[0080] Based on the above embodiment, the training steps of the generator network module include: Inputting the sample Mel spectrum of the sample audio signal into an initial neural network vocoder to obtain a synthetic audio signal of the sample audio signal output by the initial neural network vocoder; The synthesized audio signal is converted into a frequency domain using a plurality of different short-time Fourier transform parameters to obtain a synthesized amplitude spectrum of the synthesized audio signal, and the synthesized amplitude spectrum is converted into a synthesized Mel spectrum using different Mel transform matrices; Calculating a mel reconstruction loss based on the synthesized mel spectrum and the sample mel spectrum, and calculating a generative adversarial loss and a feature matching loss of an initial discriminator based on the synthesized audio signal and the sample audio signal; Based on the Mel reconstruction loss, the feature matching loss and the generative adversarial loss, the initial neural network vocoder and the initial discriminator are iteratively trained to obtain the neural network vocoder and the generator network module therein.

[0081] Specifically, in an embodiment of the present invention, in the process of training the generator network module, an initial neural network type vocoder can be constructed first, and the initial neural network type vocoder is the generator in the GAN.

[0082] First, a sample Mel spectrum with original audio features is extracted from a sample audio signal. The original audio signal may be a speech signal of different people, and the sample Mel spectrum may be a logarithmic Mel spectrum.

[0083] Then, the sample Mel spectrum is input into an initial neural network type vocoder, and the sample Mel spectrum is processed by the initial neural network type vocoder to obtain a synthetic audio signal of the sample audio signal corresponding to the sample Mel spectrum.

[0084] In the initial neural network type vocoder, the fundamental frequency extraction module can extract the fundamental frequency of each frame in the sample Mel spectrum, and determine the voiced frames and non-voiced frames in the sample Mel spectrum; the reference amplitude spectrum generation module can determine the energy information of each frame in the sample Mel spectrum, and distribute the energy information of the non-voiced frames, and distribute the energy information of the voiced frames based on the fundamental frequency of the voiced frames to obtain the sample reference amplitude spectrum; the generator network module to be trained can predict the amplitude spectrum and phase spectrum of the synthesized speech signal frame by frame based on the sample Mel spectrum and the sample reference amplitude spectrum, and obtain the synthesized speech signal based on the amplitude spectrum and the phase spectrum.

[0085] like Figure 3 As shown in the figure, the left side is the true amplitude spectrum of the sample audio signal, and the right side is the sample reference amplitude spectrum. It can be seen that the sample reference amplitude spectrum can reflect the distribution of energy between frames and within frames, and can help the generator network module better learn the changes from Mel spectrum to true amplitude spectrum.

[0086] The synthesized audio signal is converted to the frequency domain using a variety of short-time Fourier transform parameters to obtain a synthesized amplitude spectrum of the synthesized audio signal, and the synthesized amplitude spectrum is converted into a synthesized Mel spectrum through different Mel transformation matrices. Both the synthesized audio signal and the sample audio signal can obtain multiple different synthesized Mel spectra.

[0087] The loss function used in the embodiment of the present invention may include Mel reconstruction loss, generative adversarial loss and feature matching loss, wherein the Mel reconstruction loss refers to the difference between the Mel spectrum obtained from the predicted waveform processing and the Mel spectrum obtained from the original waveform processing. The difference between each synthesized Mel spectrum and the sample Mel spectrum can be calculated by synthesizing the Mel spectrum and the sample Mel spectrum, and then the difference is summed as the final Mel reconstruction loss.

[0088] The generative adversarial loss refers to the difference between the synthesized audio signal and the sample audio signal, which can be determined by the least squares method.

[0089] Feature matching loss refers to the degree of difference between the features extracted by the initial discriminator between the synthetic audio signal and the sample audio signal. The sample audio signal and the synthetic audio signal are input to the initial discriminator, so that the initial discriminator extracts synthetic features from the synthetic audio signal and sample features from the sample audio signal, respectively, and uses the synthetic features and sample features to discriminate the sample audio signal and the synthetic audio signal to obtain a discrimination result. The feature matching loss can be calculated using the synthetic features and sample features.

[0090] Finally, the loss function is obtained by weighted summing of the Mel reconstruction loss, generative adversarial loss and feature matching loss. The loss function is continuously optimized using the gradient descent method according to the discrimination result, and the initial neural network vocoder and the initial discriminator, namely, GAN, are iteratively updated to train the GAN until the loss function reaches the expected goal, such as the loss function converges, and the structural parameters of the trained neural network vocoder are obtained.

[0091] In this embodiment, the structural parameters of the trained neural network vocoder can be used to update the structural parameters of the neural network vocoder of the final application, or the trained neural network vocoder can be directly used as the neural network vocoder of the final application.

[0092] like Figure 4 As shown, based on the above embodiment, an embodiment of the present invention provides a speech signal synthesis device, including: A mel spectrum acquisition unit 41, used to acquire a mel spectrum of a speech signal; A fundamental frequency extraction unit 42 is used to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determine the voiced sound frame and the unvoiced sound frame in the Mel spectrum of the speech signal; A reference amplitude spectrum generating unit 43 is used to determine the energy information of each frame in the mel spectrum of the speech signal, and to distribute the energy information of the non-voiced frame, and to distribute the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; The speech synthesis unit 44 is used to predict the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the Mel spectrum of the speech signal and the reference amplitude spectrum, and synthesize the speech signal based on the amplitude spectrum and the phase spectrum.

[0093] On the basis of the above embodiment, in the speech signal synthesis device provided in the embodiment of the present invention, the fundamental frequency extraction unit is specifically used for: Extracting features from the speech signal mel spectrum to obtain a first feature vector and a second feature vector; Activating the first feature vector to obtain an unvoiced and voiced sound label of each frame in the speech signal Mel spectrum; the unvoiced and voiced sound label is used to indicate whether each frame in the speech signal Mel spectrum is a voiced sound frame; An exponential operation is performed on the second eigenvector to obtain the fundamental frequency of each frame in the Mel-spectrogram of the speech signal.

[0094] On the basis of the above-mentioned embodiment, in the speech signal synthesis device provided in the embodiment of the present invention, the reference amplitude spectrum generating unit is specifically used for: Converting the speech signal Mel spectrum into a speech signal amplitude spectrum; Calculating the square of the amplitude of each frequency point in the amplitude spectrum of the speech signal, and determining the energy information of each frame in the Mel spectrum of the speech signal based on the sum of the square of the amplitude of each frequency point in each frame in the amplitude spectrum of the speech signal; or, The square of the amplitude of each frequency point in the Mel spectrum of the speech signal is calculated, and the energy information of each frame in the Mel spectrum of the speech signal is determined based on the sum of the squares of the amplitudes of each frequency point in each frame in the Mel spectrum of the speech signal.

[0095] On the basis of the above-mentioned embodiment, in the speech signal synthesis device provided in the embodiment of the present invention, the reference amplitude spectrum generating unit is specifically used for: Determine the dimension of the reference amplitude spectrum based on the number of Fourier transform points, and determine the frequency interval of adjacent dimensions of the reference amplitude spectrum based on the sampling rate of the speech signal; Based on the energy information of the voiced speech frame, the energy of each harmonic position of the voiced speech frame is determined, and based on the fundamental frequency of the voiced speech frame, the frequency interval and the relationship between the frequency of each harmonic position of the voiced speech frame and the fundamental frequency, the dimension of each harmonic position of the voiced speech frame on the reference amplitude spectrum is determined.

[0096] On the basis of the above-mentioned embodiment, in the speech signal synthesis device provided in the embodiment of the present invention, the speech synthesis unit is specifically used for: Based on the generator network module, a first hidden vector of the Mel spectrum of the speech signal and a second hidden vector of the reference amplitude spectrum are respectively extracted, the first hidden vector and the second hidden vector are added and then dimensionality reduction processing is performed on the added results to obtain a first dimensional vector and a second dimensional vector, residual processing is performed on the first dimensional vector to obtain a third dimensional vector, and the third dimensional vector is spliced ​​with the first dimensional vector, and the amplitude spectrum and the phase spectrum are obtained based on the spliced ​​results; The dimension of the first-dimensional vector is greater than the dimension of the second-dimensional vector.

[0097] On the basis of the above embodiment, the speech signal synthesis device provided in the embodiment of the present invention further includes a neural network type vocoder unit, which is used to: Applying a fundamental frequency extraction module in a neural network vocoder to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining voiced frames and non-voiced frames in the Mel spectrum of the speech signal; Applying the reference amplitude spectrum generation module in the neural network vocoder, determining the energy information of each frame in the mel spectrum of the speech signal, and distributing the energy information of the non-voiced frame, distributing the energy information of the voiced frame based on the fundamental frequency of the voiced frame, and obtaining a reference amplitude spectrum; Applying the generator network module in the neural network vocoder, based on the speech signal Mel spectrum and the reference amplitude spectrum, predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, and synthesizing the speech signal based on the amplitude spectrum and the phase spectrum; The generator network module is obtained by jointly training the discriminator based on the sample audio signal by applying the fundamental frequency extraction module and the reference amplitude spectrum generation module.

[0098] On the basis of the above embodiment, the speech signal synthesis device provided in the embodiment of the present invention further includes a training module for: Inputting the sample Mel spectrum of the sample audio signal into an initial neural network vocoder to obtain a synthetic audio signal of the sample audio signal output by the initial neural network vocoder; The synthesized audio signal is converted into a frequency domain using a plurality of different short-time Fourier transform parameters to obtain a synthesized amplitude spectrum of the synthesized audio signal, and the synthesized amplitude spectrum is converted into a synthesized Mel spectrum using different Mel transform matrices; Calculating a mel reconstruction loss based on the synthesized mel spectrum and the sample mel spectrum, and calculating a generative adversarial loss and a feature matching loss of an initial discriminator based on the synthesized audio signal and the sample audio signal; Based on the Mel reconstruction loss, the feature matching loss and the generative adversarial loss, the initial neural network vocoder and the initial discriminator are iteratively trained to obtain the neural network vocoder and the generator network module therein.

[0099] Specifically, the functions of each module in the speech signal synthesis device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above method embodiment, and the effects achieved are also consistent. Please refer to the above embodiment for details, which will not be repeated in the embodiment of the present invention.

[0100] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the speech signal synthesis method provided in the above embodiments.

[0101] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech signal synthesis method provided in the above embodiments.

[0103] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the speech signal synthesis method provided in the above embodiments.

[0104] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech signal synthesis method, characterized in that: include: Get the Mel spectrum of speech signal; Extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining the voiced frames and the unvoiced frames in the Mel spectrum of the speech signal; Determine the energy information of each frame in the Mel spectrum of the speech signal, and distribute the energy information of the non-voiced frame, and distribute the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; Based on the Mel spectrum of the speech signal and the reference amplitude spectrum, the amplitude spectrum and the phase spectrum of the speech signal are predicted frame by frame, and based on the amplitude spectrum and the phase spectrum, the speech signal is synthesized.

2. The speech signal synthesis method according to claim 1, characterized in that: The extracting the fundamental frequency of each frame in the Mel spectrum of the speech signal and determining the voiced frames and the non-voiced frames in the Mel spectrum of the speech signal comprises: Extracting features from the speech signal mel spectrum to obtain a first feature vector and a second feature vector; Activating the first feature vector to obtain an unvoiced and voiced sound label of each frame in the speech signal Mel spectrum; the unvoiced and voiced sound label is used to indicate whether each frame in the speech signal Mel spectrum is a voiced sound frame; An exponential operation is performed on the second eigenvector to obtain the fundamental frequency of each frame in the Mel-spectrogram of the speech signal.

3. The speech signal synthesis method according to claim 1, characterized in that: The determining of energy information of each frame in the Mel-spectrogram of the speech signal includes: Converting the speech signal Mel spectrum into a speech signal amplitude spectrum; Calculating the square of the amplitude of each frequency point in the amplitude spectrum of the speech signal, and determining the energy information of each frame in the Mel spectrum of the speech signal based on the sum of the square of the amplitude of each frequency point in each frame in the amplitude spectrum of the speech signal; or, The square of the amplitude of each frequency point in the Mel spectrum of the speech signal is calculated, and the energy information of each frame in the Mel spectrum of the speech signal is determined based on the sum of the squares of the amplitudes of each frequency point in each frame in the Mel spectrum of the speech signal.

4. The speech signal synthesis method according to claim 1, characterized in that: The allocating energy information of the voiced sound frame based on the fundamental frequency of the voiced sound frame includes: Determine the dimension of the reference amplitude spectrum based on the number of Fourier transform points, and determine the frequency interval of adjacent dimensions of the reference amplitude spectrum based on the sampling rate of the speech signal; Based on the energy information of the voiced speech frame, the energy of each harmonic position of the voiced speech frame is determined, and based on the fundamental frequency of the voiced speech frame, the frequency interval and the relationship between the frequency of each harmonic position of the voiced speech frame and the fundamental frequency, the dimension of each harmonic position of the voiced speech frame on the reference amplitude spectrum is determined.

5. The speech signal synthesis method according to claim 1, characterized in that: The step of predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the speech signal Mel spectrum and the reference amplitude spectrum includes: Based on the generator network module, a first hidden vector of the Mel spectrum of the speech signal and a second hidden vector of the reference amplitude spectrum are respectively extracted, the first hidden vector and the second hidden vector are added and then dimensionality reduction processing is performed on the added results to obtain a first dimensional vector and a second dimensional vector, residual processing is performed on the first dimensional vector to obtain a third dimensional vector, and the third dimensional vector is spliced ​​with the first dimensional vector, and the amplitude spectrum and the phase spectrum are obtained based on the spliced ​​results; The dimension of the first-dimensional vector is greater than the dimension of the second-dimensional vector.

6. The speech signal synthesis method according to any one of claims 1 to 5, characterized in that: The step of obtaining the speech signal Mel spectrum comprises: Applying a fundamental frequency extraction module in a neural network vocoder to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determining voiced frames and non-voiced frames in the Mel spectrum of the speech signal; Applying the reference amplitude spectrum generation module in the neural network vocoder, determining the energy information of each frame in the mel spectrum of the speech signal, and distributing the energy information of the non-voiced frame, distributing the energy information of the voiced frame based on the fundamental frequency of the voiced frame, and obtaining a reference amplitude spectrum; Applying the generator network module in the neural network vocoder, based on the speech signal Mel spectrum and the reference amplitude spectrum, predicting the amplitude spectrum and phase spectrum of the speech signal frame by frame, and synthesizing the speech signal based on the amplitude spectrum and the phase spectrum; The generator network module is obtained by jointly training the discriminator based on the sample audio signal by applying the fundamental frequency extraction module and the reference amplitude spectrum generation module.

7. The speech signal synthesis method according to claim 6, characterized in that: The training steps of the generator network module include: Inputting the sample Mel spectrum of the sample audio signal into an initial neural network vocoder to obtain a synthetic audio signal of the sample audio signal output by the initial neural network vocoder; The synthesized audio signal is converted into a frequency domain using a plurality of different short-time Fourier transform parameters to obtain a synthesized amplitude spectrum of the synthesized audio signal, and the synthesized amplitude spectrum is converted into a synthesized Mel spectrum using different Mel transform matrices; Calculating a mel reconstruction loss based on the synthesized mel spectrum and the sample mel spectrum, and calculating a generative adversarial loss and a feature matching loss of an initial discriminator based on the synthesized audio signal and the sample audio signal; Based on the Mel reconstruction loss, the feature matching loss and the generative adversarial loss, the initial neural network vocoder and the initial discriminator are iteratively trained to obtain the neural network vocoder and the generator network module therein.

8. A speech signal synthesis device, characterized in that: include: A mel spectrum acquisition unit, used for acquiring a mel spectrum of a speech signal; A fundamental frequency extraction unit, used to extract the fundamental frequency of each frame in the Mel spectrum of the speech signal, and determine the voiced sound frame and the unvoiced sound frame in the Mel spectrum of the speech signal; A reference amplitude spectrum generating unit, configured to determine the energy information of each frame in the mel spectrum of the speech signal, and to distribute the energy information of the non-voiced frame, and to distribute the energy information of the voiced frame based on the fundamental frequency of the voiced frame to obtain a reference amplitude spectrum; The speech synthesis unit is used to predict the amplitude spectrum and phase spectrum of the speech signal frame by frame based on the Mel spectrum of the speech signal and the reference amplitude spectrum, and synthesize the speech signal based on the amplitude spectrum and the phase spectrum.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech signal synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech signal synthesis method according to any one of claims 1 to 7 is implemented.

Images