Speech enhancement method, device, equipment, and computer program

Abstract

This application relates to the technical field of speech processing, and specifically provides a speech enhancement method, device, equipment, and storage medium, the method including: performing pre-emphasis processing on a target speech frame based on a corresponding complex spectrum of the target speech frame to obtain a first complex spectrum; performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain, and excitation signal of the target speech frame; and performing synthesis processing based on the glottal parameters, gain, and excitation signal to obtain a corresponding enhanced speech signal of the target speech frame. This solution can effectively enhance speech and is applicable to cloud conferences.

Description

[Technical field]

[0001] The present application relates to the technical field of speech processing, and more particularly to a speech enhancement method, device, apparatus and storage medium.

[0002] This application claims priority to a Chinese patent application filed with the China Patent Office on February 8, 2021, bearing application number 202110181389.4 and titled "Speech enhancement method, device, equipment and storage medium," the entire contents of which are incorporated herein by reference. [Background technology]

[0003] Due to the convenience and timeliness of voice communication, the application of voice communication is becoming more and more widespread, for example, voice signals are transmitted between participants in a cloud conference. However, in voice communication, noise may be mixed into the voice signal, which may cause deterioration of communication quality and have a significant impact on the auditory experience of users. Therefore, how to remove noise by performing emphasis processing on the voice is a technical problem that needs to be solved quickly in the prior art. Summary of the Invention [Problem to be solved by the invention]

[0004] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The embodiments of the present application provide a speech enhancement method, device, apparatus and storage medium to realize speech enhancement and improve the quality of a speech signal.

[0005] Additional features and advantages of the present application will become apparent from the following detailed description, or in part may be learned by the practice of the present application. [Means for solving the problem]

[0006] According to one aspect of an embodiment of the present application, a speech enhancement method is provided, comprising the steps of: performing a pre-emphasis process on a target speech frame based on a corresponding complex spectrum of the target speech frame to obtain a first complex spectrum; performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame; and performing a synthesis process based on the glottal parameters, the gain and the excitation signal to obtain a corresponding enhanced speech signal of the target speech frame.

[0007] According to another aspect of an embodiment of the present application, a speech enhancement device is provided, comprising: a pre-emphasis module used for performing pre-emphasis processing on a target speech frame based on a complex spectrum of the target speech frame to obtain a first complex spectrum; a speech decomposition module used for performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame; and a synthesis processing module used for performing synthesis processing based on the glottal parameters, the gain and the excitation signal to obtain a corresponding enhanced speech signal of the target speech frame.

[0008] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implementing the speech enhancement method described above.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided having computer-readable instructions stored thereon, the computer-readable instructions, when executed by a processor, implementing the speech enhancement method described above. Effect of the Invention

[0010] In the solution of the present application, first, a first complex spectrum is obtained by performing pre-emphasis on a target speech frame, and then speech decomposition and synthesis are performed on the target speech frame based on the first complex spectrum, thereby realizing enhancement of the target speech frame in two stages, and thus the speech enhancement effect can be effectively guaranteed. Then, speech decomposition is performed on the target speech frame based on the first complex spectrum obtained by performing pre-emphasis on the target speech frame, and the noise information in the first complex spectrum is less than that of the target speech frame before pre-emphasis. Meanwhile, in the speech decomposition process, noise may affect the accuracy of speech decomposition. Therefore, by using the first complex spectrum as the basis of speech decomposition, the difficulty of speech decomposition can be reduced, the accuracy of glottal parameters, excitation signal and gain obtained by speech decomposition can be improved, and the accuracy of the subsequently obtained enhanced speech signal can be guaranteed. The first complex spectrum obtained by pre-emphasis contains phase information and amplitude information, and speech decomposition and speech synthesis are performed based on the phase information and amplitude information in the first complex spectrum, thereby ensuring the accuracy of the amplitude and phase of the enhanced speech signal corresponding to the obtained target speech frame.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the present application.

[0012] The drawings herein are incorporated in and constitute a part of the specification, show embodiments that match the present application, and are used together with the specification to interpret the principles of the present application. It is apparent that the drawings in the following description are merely some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without requiring creative work. The drawings are as follows: [Brief description of the drawings]

[0013] [Figure 1]FIG. 2 is a schematic diagram of a voice communication link in a VoIP system according to one illustrative embodiment. [Diagram 2] 1 shows a schematic diagram of a digital model in which an audio signal originates. [Diagram 3] 1 shows a schematic diagram of the frequency response of decomposing the excitation signal and the glottal filter based on one original speech signal. [Figure 4] 1 is a flow chart of a speech enhancement method according to one embodiment of the present application; [Diagram 5] FIG. 2 is a schematic diagram illustrating a complex convolution layer according to a specific embodiment performing convolution processing on complex numbers. [Figure 6] FIG. 2 is a structural schematic diagram of a first neural network according to a specific embodiment; [Figure 7] FIG. 4 is a schematic diagram of a second neural network according to one specific embodiment; [Figure 8] FIG. 11 is a schematic diagram of the inputs and outputs of a second neural network according to another embodiment. [Figure 9] FIG. 10 is a schematic diagram of a third neural network according to one specific embodiment; [Figure 10] FIG. 11 is a schematic diagram of a fourth neural network according to a specific embodiment; [Figure 11] 4 is a flow chart of step 430 according to one embodiment. [Figure 12] 1 is a flowchart of a speech enhancement method according to one specific embodiment. [Figure 13] 4 is a flow chart of step 420 according to one embodiment. [Figure 14] 4 is a flow chart of step 430 according to another embodiment. [Figure 15] 4 is a flowchart of a speech enhancement method according to another specific embodiment; [Figure 16]FIG. 2 is a schematic diagram of windowing and overlap in short-time Fourier transform according to one specific embodiment; [Figure 17] FIG. 1 is a block diagram of a voice enhancement device according to one embodiment. [Figure 18] FIG. 1 shows a structural schematic diagram of a computer system suitable for use in an electronic device for implementing an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments will be described more completely with reference to the drawings. However, the exemplary embodiments can be implemented in various forms and should not be understood to be limited to the examples described herein. On the contrary, the provision of these embodiments will make the present application more complete and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0015] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a thorough understanding of the embodiments of the present application. However, one skilled in the art can recognize that the technical means of the present application can be practiced without one or more of the specific details, or by employing other methods, elements, devices, steps, etc. In other circumstances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0016] The block diagrams shown in the drawings are merely functional entities that do not necessarily correspond to physically separate entities, i.e. they can be realised by adopting a form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor and / or microcontroller devices.

[0017] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, and are not necessarily performed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, and therefore the order in which they are actually performed may vary according to actual circumstances.

[0018] It is necessary to clarify that "plurality" as referred to in this specification means two or more than two. "And / or" describes a relationship between related objects and indicates that three types of relationships can exist, for example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0019] Noise in a voice signal can significantly reduce the voice quality and affect the user's hearing experience, so in order to improve the quality of a voice signal, it is necessary to perform enhancement processing on the voice signal to remove noise to the maximum extent and retain the original voice signal in the signal (i.e., a pure signal without noise). In order to realize enhancement processing on voice, the solution of the present application is proposed.

[0020] The solution of the present application can be applied in application scenarios of voice communication, such as voice communication via instant messaging applications, voice communication in game applications, etc. Specifically, voice enhancement can be performed according to the solution of the present application at a voice sending end, a voice receiving end, or a server terminal providing voice communication services.

[0021] Cloud conference is an important process in online business execution. In cloud conference, after the sound collecting device of the cloud conference participant collects the voice signal of the speaker, the collected voice signal needs to be transmitted to other conference participants. The voice signal involved in this process is transmitted and played among multiple participants, and if the noise signal mixed in the voice signal is not processed, it may have a significant impact on the auditory experience of the conference participants. In such a scenario, the solution of the present application can be applied to emphasize the voice signal during the cloud conference, so that the voice signal heard by the conference participant can be the emphasized voice signal, and the quality of the voice signal can be improved.

[0022] Cloud conferencing is an efficient, convenient and low-cost conferencing format based on cloud computing technology, where users can quickly and efficiently synchronize and share voice, data files and videos with global teams and clients through the Internet interface by simply carrying out simple and easy-to-use operations, while complex technologies such as data transmission and processing during the conference can be operated by the cloud conferencing service provider with the assistance of users.

[0023] At present, cloud conferencing in China mainly focuses on the service content based on SaaS (Software as a Service) mode, including telephone, network, video and other service formats, and video conferencing based on cloud computing is called cloud conferencing. In the era of cloud conferencing, data transmission, processing and storage are all handled by the computer resources of the video conferencing provider, and users do not need to purchase additional expensive hardware or install complicated software at all, but can simply open the client terminal and access the corresponding interface to conduct a highly efficient remote conference.

[0024] Cloud conferencing systems support multi-server dynamic cluster configuration and provide multiple high-performance servers, greatly improving the stability, security and availability of conferences. In recent years, video conferencing has greatly improved communication efficiency, continuously reduced communication costs, and brought about an upgrade in internal management level, so it has become popular with many users and has been widely applied in various fields such as government, military, traffic, transportation, finance, operators, education, and enterprises.

[0025] 1 is a schematic diagram of a voice communication link in a VoIP (Voice over Internet Protocol, network telephone) system according to a specific embodiment. As shown in FIG. 1, based on the network connection between a transmitting end 110 and a receiving end 120, the transmitting end 110 and the receiving end 120 can perform voice transmission.

[0026] As shown in Fig. 1, the transmitting end 110 includes a collecting module 111, a pre-enhancement processing module 112 and an encoding module 113, where the collecting module 111 is used for collecting voice signals, which can convert the collected audio signals into digital signals, and the pre-enhancement processing module 112 is used for enhancing the collected voice signals, thereby removing noise in the collected voice signals and improving the quality of the voice signals. The encoding module 113 is used for encoding the enhanced voice signals, thereby improving the interference resistance during the transmission process of the voice signals. The pre-enhancement processing module 112 performs voice enhancement according to the method of the present application, and can further perform encoding, compression and transmission after the voice enhancement, thus ensuring that the signal received by the receiving end is not affected by noise.

[0027] The receiving end 120 includes a decoding module 121, a post-emphasis module 122 and a reproduction module 123. The decoding module 121 is used for decoding the received encoded speech signal to obtain a decoded speech signal, the post-emphasis module 122 is used for enhancement processing on the decoded speech signal, and the reproduction module 123 is used for reproducing the enhanced speech signal. The post-emphasis module 122 can also perform speech enhancement according to the method of the present application. In some embodiments, the receiving end 120 may further include an audio effect adjustment module, which is used for performing audio effect adjustment on the enhanced speech signal.

[0028] In a specific embodiment, only the receiving end 120 or only the transmitting end 110 can perform speech enhancement according to the method of the present application, and of course, both the transmitting end 110 and the receiving end 120 can also perform speech enhancement according to the method of the present application.

[0029] In some application scenarios, the terminal equipment in the VoIP system can support VoIP communication, and can also support other third-party protocols, such as traditional PSTN (Public Switched Telephone Network) circuit domain telephone. Meanwhile, traditional PSTN services cannot perform voice enhancement. In such scenarios, the terminal as the receiving end can perform voice enhancement according to the method of the present application.

[0030] Before providing a detailed explanation of the solution of the present application, it is necessary to explain how a voice signal is generated. A voice signal is generated by the physiological movement of the human body's sound-producing organs under the brain control, that is, a certain energy noise-like impulse signal (corresponding to an excitation signal) is generated at the trachea, the impulse signal impacts the human vocal cords (the vocal cords correspond to the glottal filter), causing a roughly periodic opening and closing, and after being amplified through the oral cavity, a sound is generated (a voice signal is output).

[0031] Figure 2 shows a schematic diagram of the digital model of the voice signal generation, which can describe the process of the voice signal generation. As shown in Figure 2, the excitation signal impacts the glottal filter, and then undergoes gain control, and then outputs the voice signal, where the glottal filter is limited by the glottal parameters. The process can be expressed as follows: x(n) = G r(n) ar(n) (Eq. 1) where x(n) represents the input speech signal, G represents the gain, which can also be called the linear prediction gain, r(n) represents the excitation signal, and ar(n) represents the glottal filter.

[0032] FIG. 3 shows a schematic diagram of the frequency response of decomposing an excitation signal and a glottal filter based on an original speech signal. FIG. 3a shows a schematic diagram of the frequency response of the original speech signal, FIG. 3b shows a schematic diagram of the frequency response of the glottal filter decomposed based on the original speech signal, and FIG. 3c shows a schematic diagram of the frequency response of the excitation signal decomposed based on the original speech signal. As shown in FIG. 3, the waveform part in the schematic diagram of the frequency response of the original speech signal corresponds to the peak position in the schematic diagram of the frequency response of the glottal filter, and the excitation signal corresponds to the residual signal after performing LP (Linear Prediction) analysis on the original speech signal, and therefore its corresponding frequency response is relatively gentle.

[0033] As can be seen from the above, the excitation signal, the glottal filter and the gain can be decomposed based on one original speech signal (i.e., a noise-free speech signal), and the decomposed excitation signal, the glottal filter and the gain can be used to represent the original speech signal, where the glottal filter can be represented by the glottal parameters. Conversely, if the glottal parameters used to determine the corresponding excitation signal, the glottal filter and the gain of one original speech signal are known, the original speech signal can be reconstructed based on the corresponding excitation signal, the glottal filter and the gain.

[0034] Based on this principle, the solution of this application is to reconstruct the original speech signal in a speech frame based on the corresponding glottal parameters, excitation signal and gain of the speech frame, thereby realizing speech enhancement.

[0035] The technical means of the embodiments of the present application will be described in detail below.

[0036] 4 is a flow chart of a speech enhancement method according to an embodiment of the present application, which may be executed by a computer device with processing capabilities, such as a terminal, a server, etc., without specific limitation here. Referring to FIG. 4, the method includes at least steps 410-430, which are described in detail as follows:

[0037] Step 410: Perform pre-emphasis processing on the target speech frame according to a corresponding complex spectrum of the target speech frame to obtain a first complex spectrum.

[0038] Although speech signals do not change slowly and randomly but change over time, speech signals have strong correlation within a short time, i.e., speech signals have short-term correlation. Therefore, in the solution of the present application, speech enhancement is performed in units of speech frames. The target speech frame refers to the speech frame currently being subjected to enhancement processing.

[0039] The corresponding complex spectrum of the target speech frame can be obtained by performing a time-frequency transformation on the time domain signal of the target speech frame, which may be, for example, a Short-term Fourier transform (STFT). The coefficient of the real part in the corresponding complex spectrum of the target speech frame is used to indicate the amplitude information of the target speech frame, and the coefficient of the imaginary part is used to indicate the phase information of the target speech frame.

[0040] By performing pre-emphasis on the target speech frame, some noise in the target speech frame can be removed, and thus the noise content in the first complex spectrum obtained by pre-emphasis is less than that in the corresponding complex spectrum of the target speech frame.

[0041] In some embodiments of the present application, a deep learning method can be adopted to perform pre-emphasis on a target voice frame. A neural network model is trained to predict the complex spectrum of noise in a voice frame based on the corresponding complex spectrum of the voice frame, and then the complex spectrum of the voice frame is subtracted from the predicted complex spectrum of noise to obtain a first complex spectrum. For convenience of description, the neural network model used to predict the complex spectrum of noise in a voice frame is called a noise prediction model. After training, the noise prediction model can output a predicted complex spectrum of noise based on the complex spectrum of the input voice frame, and then the complex spectrum of the voice frame is subtracted from the complex spectrum of noise to obtain a first complex spectrum.

[0042] In some embodiments of the present application, a neural network model can be further trained to predict the first complex spectrum of the speech frame after enhancement according to the complex spectrum of the speech frame. For convenience of description, the neural network model used to predict the complex spectrum after enhancement is called an enhancement complex spectrum prediction model. In the training process, the complex spectrum of the sample speech frame is input into the enhancement complex spectrum prediction model, the enhancement complex spectrum prediction model predicts the complex spectrum after enhancement, and the parameters of the enhancement complex spectrum prediction model are adjusted according to the predicted enhanced complex spectrum and the label information of the sample speech frame, until the difference between the predicted enhanced complex spectrum and the complex spectrum indicated by the label information meets a predetermined requirement. The label information of the sample speech frame is used to indicate the complex spectrum of the original speech signal in the sample speech frame. After the training is completed, the enhancement complex spectrum prediction model can output a first complex spectrum according to the complex spectrum of the target speech frame.

[0043] Step 420: Perform speech decomposition on the target speech frame based on the first complex spectrum, and obtain corresponding glottal parameters, gain and excitation signal of the target speech frame.

[0044] The corresponding glottal parameters, the corresponding gains and the corresponding excitation signals of the target speech frame obtained by speech decomposition are used to reconstruct the original speech signal in the target speech frame according to the process shown in FIG.

[0045] As described above, an original speech signal is obtained by applying an excitation signal to the glottal filter and then performing gain control. The first complex spectrum contains the information of the original speech signal of the target speech frame, so that linear prediction analysis is performed based on the first complex spectrum to inversely determine the glottal parameters, excitation signal and gain used to reconstruct the original speech signal in the target speech frame.

[0046] The glottal parameters refer to parameters used to construct a glottal filter, and when the glottal parameters are determined, the glottal filter is correspondingly determined, and the glottal filter is a digital filter. The glottal parameters may be Linear Prediction Coefficients (LPC) coefficients, and may also be Line Spectral Frequency (LSF) parameters. The number of glottal parameters corresponding to a target speech frame is related to the order of the glottal filter, and when the glottal filter is a K-th order filter, the glottal parameters include K-th order LSF parameters or K-th order LPC coefficients, where LSF parameters and LPC coefficients can be converted to each other.

[0047] A single pth-order glottal filter is A p (z)=1+a1z -1 +a2z -2 +…+a p z -p It may be expressed as (Equation 2). Here, a1, a2, …, a p are the LPC coefficients, p is the order of the glottal filter, and z is the input signal to the glottal filter.

[0048] Based on Equation 2, P(z)=A p (z)-z -(p+1) A p (z -1 ) (Formula 3) Q(z)=A p (z)+z -(p+1)A p (z -1 ) When set as in (Equation 4), The following [Mathematical formula 1] (Equation 5) can be obtained.

[0049]

number

[0050] Physically, P(z) and Q(z) represent the periodic change rules of glottal opening and glottal closure, respectively. The roots of the polynomials P(z) and Q(z) alternate on the complex plane, which are a series of angular frequencies distributed on the complex plane unit circle, and the LSF parameters are the corresponding angular frequencies on the complex plane unit circle of the roots of P(z) and Q(z), and the corresponding LSF parameters LSF(n) of the nth speech frame can be expressed as ωn. Of course, the corresponding LSF parameters LSF(n) of the nth speech frame can also be directly represented by the roots of P(z) and the roots of Q(z) corresponding to the nth speech frame.

[0051] The roots of P(z) and Q(z) in the complex plane corresponding to the nth speech frame are θ n Then, the corresponding LSF parameters of the nth speech frame are This is expressed as follows (Equation 6):

[0052]

number

[0053] Here, Rel{θ n} is a complex number θ n represents the real part of Imag{θ n} is a complex number θ n Represents the imaginary part of.

[0054] In some embodiments of the present application, a deep learning approach can be adopted to perform speech decomposition. First, neural network models used for performing glottal parameter prediction, excitation signal prediction, and gain prediction can be trained, so that the three neural network models can respectively predict the corresponding glottal parameters, excitation signal, and gain of a target speech frame according to a first complex spectrum.

[0055] In some embodiments of the present application, signal processing can be further performed according to the principle of linear predictive analysis based on the first complex spectrum, and corresponding glottal parameters, excitation signals and gains of the target speech frame can be calculated. The specific process is described below.

[0056] Step 430: Perform a synthesis process based on the glottal parameters, the gain and the excitation signal to obtain a corresponding enhanced speech signal of the target speech frame.

[0057] When the corresponding glottal parameters of the target speech frame are determined, the corresponding glottal filter is determined accordingly. On that basis, according to the generation process of the original speech signal shown in Fig. 2, the corresponding excitation signal of the target speech frame is impacted on the determined glottal filter, and the gain control is performed on the signal obtained by filtering according to the corresponding gain of the target speech frame, so that the reconstruction of the original speech signal can be realized, and the signal obtained by reconstruction is the corresponding enhanced speech signal of the target speech frame.

[0058] In the solution of the present application, first, a first complex spectrum is obtained by performing pre-emphasis on a target speech frame, and then speech decomposition and synthesis are performed on the target speech frame based on the first complex spectrum, thereby realizing enhancement of the target speech frame in two stages, and effectively ensuring the effect of speech enhancement. Then, speech decomposition is performed on the target speech frame based on the first complex spectrum obtained by performing pre-emphasis on the target speech frame, and the noise information in the first complex spectrum is less than that of the spectrum before the target speech frame is pre-emphasized. In the speech decomposition process, noise may affect the accuracy of speech decomposition, so that the difficulty of speech decomposition is reduced by using the first complex spectrum as the basis of speech decomposition, and the accuracy of the glottal parameters, excitation signal and gain obtained by speech decomposition is improved, and the accuracy of the subsequently obtained enhanced speech signal can be guaranteed. The first complex spectrum obtained by pre-emphasis contains phase information and amplitude information, and speech decomposition and speech synthesis are performed based on the phase information and amplitude information in the first complex spectrum, thereby ensuring the accuracy of the amplitude and phase of the enhanced speech signal corresponding to the obtained target speech frame.

[0059] In some embodiments of the present application, step 410 includes inputting the corresponding complex spectrum of the target speech frame into a first neural network, the first neural network being obtained by training based on the corresponding complex spectrum of a sample speech frame and the corresponding complex spectrum of an original speech signal in the sample speech frame; and outputting the first complex spectrum by the first neural network based on the corresponding complex spectrum of the target speech frame.

[0060] The first neural network may be a model constructed by a long-short-term memory neural network, a convolutional neural network, a recurrent neural network, a fully connected neural network, a gated recurrent unit, etc., and no specific limitations are imposed here.

[0061] In some embodiments of the present application, a plurality of sample audio frames can be obtained by performing frame division on a sample audio signal, where the sample audio signal can be obtained by combining a known original audio signal and a known noise signal, and thus, when the original audio signal is known, a time-frequency transform can be performed on the original audio signal in the sample audio frame correspondingly to obtain a corresponding complex spectrum of the original audio signal in the sample audio frame, where the corresponding complex spectrum of the sample audio frame can be obtained by performing a time-frequency transform on the time domain signal of the sample audio frame.

[0062] In the training process, the corresponding complex spectrum of the sample speech frame is input to the first neural network, and the first neural network makes a prediction based on the corresponding complex spectrum of the sample speech frame, and outputs a predicted first complex spectrum, and then compares the predicted first complex spectrum with the corresponding complex spectrum of the original speech signal in the sample speech frame, and if the similarity between the two does not meet a predetermined requirement, adjust the parameters of the first neural network, and continue until the similarity between the predicted first complex spectrum output by the first neural network and the corresponding complex spectrum of the original speech signal in the sample speech frame meets the predetermined requirement. Here, the predetermined requirement may be that the similarity between the predicted first complex spectrum and the corresponding complex spectrum of the original speech signal in the sample speech frame is equal to or greater than a similarity threshold, and the similarity threshold can be set according to needs, for example, 100%, 98%, etc. Through the above training process, the first neural network can learn the ability to predict the first complex spectrum based on the input complex spectrum.

[0063] In some embodiments of the present application, the first neural network includes a complex convolution layer, a gated recurrent unit layer and a fully connected layer. The above-mentioned step of outputting the first complex spectrum according to the complex spectrum of the target speech frame by the first neural network further includes the steps of performing complex convolution processing by the complex convolution layer according to the real part and the imaginary part of the complex spectrum corresponding to the target speech frame, performing conversion processing on the output of the complex convolution layer by the gated recurrent unit layer, and performing fully connected processing on the output of the gated recurrent unit by the fully connected layer, and outputting the first complex spectrum.

[0064] In a specific embodiment, the first neural network may include one or more complex convolution layers; similarly, the gated recurrent unit layer and the fully connected layer may also be one or more layers; specifically, the number of complex convolution layers, gated recurrent unit layers and fully connected layers can be set according to actual needs.

[0065] Figure 5 is a schematic diagram of a complex convolution layer according to a specific embodiment performing convolution processing on complex numbers. Assume that the input complex number of the complex convolution layer is E+jF, and the weight of the complex convolution layer is A+jB. As shown in Figure 5, the complex convolution layer includes a two-dimensional convolution layer (Real_conv, Imag_conv), a concatenation layer (Concat) and an activation layer (Leaky_Relu). After the real part E and the imaginary part F of the input complex number are input to the two-dimensional convolution layer, the two-dimensional convolution layer performs convolution according to the weight of the complex convolution layer, and the process of performing the convolution operation is shown in the following formula: (E+jF)*(A+jB)=(E*AF*B)+j(E*B+F*A) (Formula 7) If we set C=E*AF*B and D=E*B+F*A, then Equation 7 above can be further reduced to: Transform into (E+jF)*(A+jB)=C+jD (Equation 8).

[0066] As shown in Figure 5, after the two-dimensional convolution layer outputs the real and imaginary parts after convolution, the real and imaginary parts are combined by the combination layer to obtain the combined result, and then the activation layer activates the combined result. In Figure 5, the activation function used in the activation layer is the Leaky_Relu activation function. The expression of the Leaky_Relu activation function is f(x) = max(ax,x) (where a is a constant) (Equation 9).

[0067] In other embodiments, the activation function used in the activation layer may be other functions, such as the zRelu function, and no specific limitation is given here.

[0068] FIG. 6 is a structural diagram of a first neural network according to a specific embodiment. As shown in FIG. 6, the first neural network includes six cascaded complex convolution layers (Conv), one gated recurrent unit (GRU) layer, and two fully connected (FC) layers. After a complex spectrum S(n) corresponding to a target voice frame is input to the first neural network, the complex spectrum S(n) is first complex convolutionally processed by six complex convolution layers, then transformed by a GRU layer, and then fully connected by two FC layers, and the first complex spectrum is output by the last FC layer. Here, the numbers in parentheses of each layer represent the dimensions of the variables output by the layer. In the first neural network shown in FIG. 6, the dimensions output by the last FC layer are 322 dimensions, which are used to represent the real and imaginary parts of the 161 STFT coefficients.

[0069] In some embodiments of the present application, step 420 includes the steps of: performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain a corresponding glottal parameter of the target speech frame; performing excitation signal prediction for the target speech frame based on the first complex spectrum to obtain a corresponding excitation signal of the target speech frame; and performing gain prediction for the target speech frame based on corresponding gains of historical speech frames previous to the target speech frame to obtain a corresponding gain of the target speech frame.

[0070] In some embodiments of the present application, a neural network model (assumed to be a second neural network) used for glottal parameter prediction, a neural network model (assumed to be a third neural network) for gain prediction, and a neural network model (assumed to be a fourth neural network) for excitation signal prediction can be trained, where the three neural network models can be models constructed by long-short-term memory neural networks, convolutional neural networks, recurrent neural networks, fully connected neural networks, etc., and are not specifically limited here.

[0071] In some embodiments of the present application, the step of performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain the corresponding glottal parameter of the target speech frame further includes the steps of inputting the first complex spectrum into a second neural network, the second neural network being obtained by training based on the corresponding complex spectrum of a sample speech frame and the corresponding glottal parameter of the sample speech frame, and outputting the corresponding glottal parameter of the target speech frame by the second neural network based on the first complex spectrum.

[0072] The corresponding complex spectrum of the sample speech frame is obtained by performing a time-frequency transform on the time domain signal of the sample speech frame. In some embodiments of the present application, the sample speech signal can be divided into frames to obtain a plurality of sample speech frames. The sample speech signal can be obtained by combining a known original speech signal and a known noise signal. In this way, when the original speech signal is known, the corresponding glottal parameters of the sample speech frame can be obtained by performing a linear prediction analysis on the original speech signal, in other words, the corresponding glottal parameters of the sample speech frame refer to the glottal parameters used to reconstruct the original speech signal in the sample speech frame.

[0073] In the training process, the complex spectrum of the sample speech frame is inputted into the second neural network, and then the second neural network performs glottal parameter prediction according to the complex spectrum of the sample speech frame, and outputs a predicted glottal parameter, and then compares the predicted glottal parameter with the corresponding glottal parameter of the sample speech frame, and if they are not consistent, adjust the parameters of the second neural network until the predicted glottal parameter outputted by the second neural network according to the complex spectrum of the sample speech frame is consistent with the corresponding glottal parameter of the sample speech frame. After the training is completed, the second neural network has learned the ability to accurately predict the glottal parameters used in reconstructing the original speech signal in the speech frame according to the complex spectrum of the input speech frame.

[0074] FIG. 7 is a schematic diagram of a second neural network according to a specific embodiment. As shown in FIG. 7, the second neural network includes one LSTM (Long-Short Term Memory) layer and three cascaded FC (Fully Connected) layers. Here, the LSTM layer is a hidden layer, which includes 256 units, and the input of the LSTM layer is the first complex spectrum S'(n) corresponding to the nth speech frame. In this embodiment, the input of the LSTM layer is 321-dimensional. In the three cascaded FC layers, the activation function σ() is set in the first two FC layers, and the set activation function is used to increase the nonlinear expression ability of the second neural network, and no activation function is set in the last FC layer, and the last FC layer serves as a classifier to output classification. As shown in FIG. 7, in the direction from input to output, three FC layers contain 512, 512, and 16 units, respectively, and the output of the last FC layer is a 16-dimensional line spectral frequency coefficient LSF(n) corresponding to the n-th speech frame, i.e., a 16th-order line spectral frequency parameter.

[0075] In some embodiments of the present application, there is correlation between speech frames, and the similarity of frequency domain features between two adjacent speech frames is relatively high, so that the corresponding glottal parameters of the target speech frame can be predicted in combination with the corresponding glottal parameters of the historical speech frame before the target speech frame. In one embodiment, the step of performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain the corresponding glottal parameters of the target speech frame further includes the steps of inputting the first complex spectrum and the corresponding glottal parameters of the historical speech frame before the target speech frame into a second neural network, the second neural network being obtained by training based on the corresponding complex spectrum of a sample speech frame, the corresponding glottal parameters of the historical speech frame before the sample speech frame, and the corresponding glottal parameters of the sample speech frame; and the step of outputting the corresponding glottal parameters of the target speech frame by the first neural network based on the first complex spectrum and the corresponding glottal parameters of the historical speech frame before the target speech frame.

[0076] Since there is a correlation between the historical speech frame and the target speech frame, and there is a similarity between the glottal parameters corresponding to the historical speech frame of the target speech frame and the corresponding glottal parameters of the target speech frame, the glottal parameters of the target speech frame are used as a reference to monitor the prediction process of the glottal parameters of the target speech frame, so that the accuracy rate of the glottal parameter prediction can be improved.

[0077] In some embodiments of the present application, since the closer the speech frames are in time, the higher the similarity of the glottal parameters is, the corresponding glottal parameters of the historical speech frames relatively close to the target speech frame are used as reference, so that the prediction accuracy rate can be further ensured, for example, the corresponding glottal parameters of the speech frame immediately before the target speech frame can be used as reference. In a specific embodiment, the number of historical speech frames used as reference can be one frame or multiple frames, and can be specifically selected according to actual needs.

[0078] The glottal parameters corresponding to the historical speech frames of the target speech frame may be glottal parameters obtained by performing glottal parameter prediction on the historical speech frames. In other words, in the process of predicting the glottal parameters, the predicted glottal parameters for the historical speech frames are multiplexed as references for the glottal parameter prediction process of the current speech frame.

[0079] The training process of the second neural network in this embodiment is similar to that of the second neural network in the previous embodiment, so the training process will not be described again here.

[0080] FIG. 8 is a schematic diagram of the input and output of the second neural network shown according to another embodiment. Here, the structure of the second neural network in FIG. 8 is the same as that in FIG. 7, and compared with FIG. 7, the input of the second neural network in FIG. 8 further includes the line spectrum frequency parameter LSF(n-1) of the voice frame immediately before the voice frame of the nth frame (i.e., the n-1th frame). As shown in FIG. 8, the line spectrum frequency parameter LSF(n-1) of the voice frame immediately before the voice frame of the nth frame is embedded in the FC layer of the second layer as reference information. The similarity of the LSF parameters of two adjacent voice frames is very high, so that the corresponding LSF parameters of the historical voice frame of the voice frame of the nth frame can be used as reference information to improve the prediction accuracy of the LSF parameters.

[0081] In some embodiments of the present application, the step of performing gain prediction for the target speech frame based on the corresponding gain of a historical speech frame preceding the target speech frame to obtain the corresponding gain of the target speech frame may further include the steps of inputting the corresponding gain of the historical speech frame preceding the target speech frame to a third neural network, the third neural network being obtained by training based on the corresponding gain of the historical speech frame preceding the sample speech frame and the corresponding gain of the sample speech frame; and outputting the corresponding gain of the target speech frame by the third neural network based on the corresponding gain of the historical speech frame preceding the target speech frame.

[0082] The corresponding gain of the historical speech frame of the target speech frame may be obtained by the third neural network performing gain prediction of the historical speech frame, in other words, the predicted gain for the historical speech frame is multiplexed as an input of the third neural network model in the process of performing gain prediction for the target speech frame.

[0083] The sample speech frame may be obtained by performing frame division on the sample speech signal, and the sample speech signal may be obtained by combining a known original speech signal and a known noise signal. In this way, when the original speech signal in the sample speech is known, a linear predictive analysis can be performed on the original speech signal to obtain glottal parameters used to reconstruct the original speech signal, i.e., the corresponding glottal parameters of the sample speech frame.

[0084] FIG. 9 is a schematic diagram of a third neural network shown based on one specific embodiment. As shown in FIG. 9, the third neural network includes one layer of LSTM layer and one layer of FC layer. Here, the LSTM layer is one hidden layer, which includes 128 units. The input dimension of the FC layer is 512, and the output is a one-dimensional gain. In one specific embodiment, the corresponding gain G_pre(n) of the historical audio frames of the audio frame of the nth frame can be defined as the gain corresponding to the first four audio frames of the audio frame of the nth frame, that is, G_pre(n) = {G(n - 1), G(n - 2), G(n - 3), G(n - 4)}.

[0085] Of course, the number of historical audio frames selected for gain prediction is not limited to the above example. Specifically, it can be selected and used according to actual needs.

[0086] The second neural network and the third neural network shown as above generally exhibit an M-to-N mapping relationship (N << M), that is, the dimension of the input information of the neural network model is M, the dimension of the output information is N, and the structure of the neural network model is extremely greatly simplified, reducing the complexity of the neural network model.

[0087] In some embodiments of the present application, the step of predicting an excitation signal for the target audio frame based on the above-mentioned first complex spectrum and obtaining the corresponding excitation signal of the target audio frame may further include inputting the first complex spectrum into a fourth neural network, where the fourth neural network is obtained by training based on the corresponding complex spectrum of the sample audio frame and the frequency-domain representation of the excitation signal corresponding to the sample audio frame, and the step of outputting, by the fourth neural network, the frequency-domain representation of the excitation signal corresponding to the target audio frame based on the first complex spectrum.

[0088] The corresponding excitation signal of the sample speech frame may be obtained by performing a linear predictive analysis on the known original speech signal in the sample speech frame. The frequency domain representation may be an amplitude spectrum or a complex spectrum, and no specific limitation is made here.

[0089] In the process of training the fourth neural network, the complex spectrum of the sample speech frame is input into the fourth neural network model, and then the fourth neural network performs excitation signal prediction based on the complex spectrum of the input sample speech frame, and outputs a frequency domain representation of the predicted excitation signal, and then adjusts the parameters of the fourth neural network based on the frequency domain representation of the predicted excitation signal and the frequency domain representation of the excitation signal corresponding to the sample speech frame. That is, if the similarity between the frequency domain representation of the predicted excitation signal and the frequency domain representation of the excitation signal corresponding to the sample speech frame does not meet the predetermined requirement, adjust the parameters of the fourth neural network, and continue until the similarity between the frequency domain representation of the predicted excitation signal output by the fourth neural network for the sample speech frame and the frequency domain representation of the excitation signal corresponding to the sample speech frame meets the predetermined requirement. Through the above training process, the fourth neural network can learn the ability to predict the frequency domain representation of the corresponding excitation signal of the speech frame based on the amplitude spectrum of the speech frame, so as to accurately perform excitation signal prediction.

[0090] FIG. 10 is a schematic diagram of a fourth neural network according to a specific embodiment. As shown in FIG. 10, the fourth neural network includes one LSTM layer and three FC layers, where the LSTM layer is a hidden layer and includes 256 units, and the input of the LSTM is the first complex spectrum S'(n) corresponding to the nth speech frame, and the dimension may be 321 dimensions. The number of units included in the three FC layers is 512, 512 and 321 respectively, and the last FC layer outputs a frequency domain representation R(n) of the excitation signal corresponding to the nth speech frame of 321 dimensions. Along the direction from input to output, activation functions are set in the first two FC layers of the three FC layers, which are used to enhance the nonlinear expression ability of the model, and there is no activation function in the last FC layer, which is used to perform classification output.

[0091] The structures of the first neural network, the second neural network, the third neural network and the fourth neural network shown above are merely exemplary, and in other embodiments, neural network models with corresponding structures can be set in a deep learning open source platform and trained accordingly.

[0092] In some embodiments of the present application, as shown in FIG. 11, step 430 includes steps 1110 and 1120,

[0093] Step 1110: Filter the corresponding excitation signal of the target speech frame with a glottal filter to obtain a filtered output signal, where the glottal filter is constructed based on the corresponding glottal parameters of the target speech frame.

[0094] Step 1120: Perform an amplification process on the filtering output signal according to a corresponding gain of the target speech frame to obtain a corresponding enhanced speech signal of the target speech frame.

[0095] If the glottal parameters are LPC coefficients, the glottal filter can be constructed directly according to the above formula (2). If the glottal filter is a p-th order filter, the corresponding glottal parameters of the target speech frame are the p-th order LPC coefficients, i.e., a1, a2, ..., a in the above formula (2). p In other embodiments, the constant 1 in equation (2) above may be an LPC coefficient.

[0096] If the glottal parameters are LSF parameters, the LSF parameters can be converted to LPC coefficients, and then a glottal filter can be constructed correspondingly according to equation (2) above.

[0097] The filtering process is a convolution in the time domain, so the filtering process of the excitation signal by the glottal filter as described above can be performed by converting it to the time domain. In addition to predicting and obtaining a frequency domain representation of the excitation signal corresponding to the target speech frame, the frequency domain representation of the excitation signal is converted to the time domain to obtain a time domain signal of the excitation signal corresponding to the target speech frame.

[0098] In the present solution, a target speech frame includes multiple sample points, and the excitation signal is filtered by a glottal filter, i.e., a sample point is convolved with a previous history sample point by the glottal filter to obtain a corresponding target signal value of the sample point.

[0099] In some embodiments of the present application, the target speech frame includes a plurality of sample points, the glottal filter is a p-th order filter, where p is a positive integer, and the excitation signal includes corresponding excitation signal values ​​of the plurality of sample points in the target speech frame. According to the above filtering process, step 1120 further includes convolving the p-th order filter with excitation signal values ​​corresponding to p sample points preceding each sample point in the target speech frame to obtain target signal values ​​of each sample point in the target speech frame, and combining corresponding target signal values ​​of all sample points in the target speech frame according to time order to obtain the first speech signal. Here, the expression of the p-th order filter can refer to the above formula (1). That is, for each sample point in the target speech frame, the excitation signal values ​​corresponding to the previous p sample points are used to convolve with the p-th order filter to obtain corresponding target signal values ​​of each sample point.

[0100] It can be understood that for the first sample point in the target voice frame, the excitation signal values ​​of the last p sample points in the voice frame immediately preceding the target voice frame need to be used to calculate the corresponding target signal value of the first sample point; similarly, for the second sample point in the target voice frame, the excitation signal values ​​of the last (p-1) sample points in the voice frame immediately preceding the target voice frame and the excitation signal value of the first sample point in the target voice frame need to be convolved with a p-th filter to obtain the target signal value corresponding to the second sample point in the target voice frame.

[0101] In summary, step 1120 further requires the inclusion of excitation signal values ​​corresponding to historical speech frames of the target speech frame. The number of sample points in the required historical speech frame is related to the order of the glottal filter, i.e., a glottal filter of order p requires the inclusion of excitation signal values ​​corresponding to the last p sample points in the speech frame immediately preceding the target speech frame.

[0102] In the related art, there exists a method of performing speech enhancement by using the methods of spectrum estimation and spectrum regression prediction. The speech enhancement method of spectrum estimation is to estimate the noise by using a statistical model, etc., since it is considered that a mixed speech includes a speech part and a noise part, and the noise can be estimated by subtracting the corresponding spectrum of the noise from the corresponding spectrum of the mixed speech, and the remaining spectrum is the speech spectrum, so that the spectrum obtained by subtracting the corresponding spectrum of the noise based on the corresponding spectrum of the mixed speech restores a clean speech signal. The speech enhancement method of spectrum regression prediction is to predict the corresponding masking threshold of a speech frame by a neural network, which reflects the ratio of the speech component and the noise component at each frequency point in the speech frame, and then perform gain control on the mixed signal spectrum based on the masking threshold to obtain an enhanced spectrum.

[0103] The above-mentioned speech enhancement method using spectrum estimation and spectrum regression prediction is based on noise spectrum posterior probability, and the estimated noise is inaccurate. For example, there may be transient noise such as keyboard tapping, which occurs instantaneously, so the estimated noise spectrum is very inaccurate, which causes poor noise suppression effect. When the noise spectrum prediction is inaccurate, if the original mixed speech signal is processed according to the estimated noise spectrum, it may cause speech distortion in the mixed speech signal or cause deterioration of the noise suppression effect, so in such a situation, a compromise must be made between speech fidelity and noise suppression.

[0104] In the above embodiment of realizing speech enhancement based on glottal parameters, excitation signal and gain prediction, since the glottal parameters have a strong correlation with the glottal features in the physical process of speech production, the predicted glottal parameters effectively guarantee the speech structure of the original speech signal in the target speech frame; therefore, obtaining the enhanced speech signal of the target speech frame by performing synthesis on the glottal parameters, excitation signal and gain obtained by speech decomposition can effectively avoid reducing the original speech, effectively protect the speech structure, and there is no need to make a compromise between both speech fidelity and noise suppression, since processing is no longer performed on the original noisy speech after obtaining the corresponding glottal parameters, excitation signal and gain of the target speech frame.

[0105] FIG. 12 is a flow chart of a speech enhancement method according to another specific embodiment. In the embodiment shown in FIG. 12, the second neural network, the third neural network and the fourth neural network are combined to perform speech decomposition. Assuming that the speech frame of the nth frame is a target speech frame, the time domain signal of the speech frame of the nth frame is s(n). As shown in FIG. 12, the speech enhancement method includes steps 1210 to 1270.

[0106] Step 1210: Time-frequency transformation: transforming the time domain signal s(n) of the nth speech frame into a corresponding complex spectrum S(n) of the nth speech frame.

[0107] Step 1220: Pre-emphasis: performing pre-emphasis on the n-th speech frame based on the complex spectrum S(n) to obtain a first complex spectrum S'(n).

[0108] Step 1230: predict glottal parameters by a second neural network. In this step, the input of the second neural network may only have the first complex spectrum S'(n), or may include the first complex spectrum S'(n) and the corresponding glottal parameters P_pre(n) of the historical speech frame of the nth speech frame, and the second neural network outputs the corresponding glottal parameters ar(n) of the nth speech frame, which may be LPC coefficients or LSF parameters.

[0109] Step 1240: Predict the excitation signal by a third neural network. The input of the third neural network is the first complex spectrum S'(n), and the output is a frequency domain representation R(n) of the excitation signal corresponding to the nth speech frame. Then, in step 1250, a frequency-time transform is performed on R(n) to obtain a time domain signal r(n) of the excitation signal corresponding to the nth speech frame.

[0110] Step 1260: Predict a gain by a fourth neural network. The input of the fourth neural network is the gain G_pre(n) corresponding to the historical speech frame of the nth speech frame, and the output is the corresponding gain G(n) of the nth speech frame.

[0111] After obtaining the corresponding glottal parameter ar(n), the corresponding excitation signal r(n) and the corresponding gain G_(n) of the nth speech frame, synthesis filtering is performed in step 1270 according to the three parameters to obtain the time domain signal s_e(n) of the enhanced speech signal corresponding to the nth speech frame. The synthesis filtering process in step 1270 can be performed with reference to the process shown in FIG.

[0112] In some other embodiments of the present application, as shown in FIG.

[0113] Step 1310: Calculate and obtain a power spectrum based on the first complex spectrum.

[0114] If the first complex spectrum is S′(n), the power spectrum Pa(n) obtained in step 1310 is Pa(n) = Real(S'(n))² + Imag(S'(n))² (Equation 10).

[0115] Here, Real(S'(n)) represents the real part of the first complex spectrum S'(n), and Imag(S'(n)) represents the imaginary part of the first complex spectrum S'(n). The power spectrum calculated and obtained in step 1310 is the power spectrum of the signal after pre-emphasis is performed on the target speech frame.

[0116] Step 1320: Calculate and obtain an autocorrelation coefficient based on the power spectrum.

[0117] According to the Wiener-Khinchine theorem: the power spectrum of a stationary random process and its autocorrelation function are a pair of Fourier transform relationships. In this solution, a voice frame is regarded as a stationary random signal. Therefore, in addition to obtaining the pre-emphasized power spectrum corresponding to the target voice frame, we can also perform an inverse Fourier transform on the pre-emphasized power spectrum corresponding to the target voice frame to obtain the corresponding autocorrelation coefficient of the pre-emphasized power spectrum.

[0118] Specifically, step 1320 includes performing an inverse Fourier transform on the power spectrum to obtain an inverse transform result, extracting a real part in the inverse transform result, and obtaining the autocorrelation coefficient. That is, AC(n)=Real(iFFT(Pa(n))) (Equation 11) AC(n) represents the corresponding autocorrelation coefficient of the nth speech frame, iFFT (Inverse Fast Fourier Transform) refers to the inverse transformation of FFT (Fast Fourier Transform), and Real represents the real part of the result obtained by inverse fast Fourier transform. AC(n) contains p parameters, where p is the order of the glottal filter, and the coefficients in AC(n) are further divided into AC j (n), where 1≦j≦p.

[0119] Step 1330: Calculate and obtain the glottal parameters based on the autocorrelation coefficients.

[0120] According to the Yule-Walker equation, for the nth speech frame, the following relationship exists between its corresponding autocorrelation coefficient and the corresponding glottal parameters: k-KA=0 (Equation 12) Here, k is the autocorrelation vector, K is the autocorrelation matrix, and A is the LPC coefficient matrix. Specifically, it is expressed as follows:

[0121]

number

[0122] Here, A.C. j (n)=E[s(n)s(nj)],0≦j≦p (Equation 13)

[0123] p is the order of the glottal filter, a1(n), a2(n), …, a p (n) are LPC coefficients corresponding to the nth speech frame, and are a1, a2, ..., a p Since a0(n) is a constant 1, a0(n) can also be regarded as one LPC coefficient corresponding to the nth speech frame.

[0124] In addition to obtaining the autocorrelation coefficients, the autocorrelation vector and the autocorrelation matrix can be determined correspondingly, and then the LPC coefficients can be obtained by obtaining Equation 12. In a specific embodiment, the Levinson-Durbin algorithm can be adopted to obtain Equation 12, which utilizes the symmetry of the autocorrelation matrix and uses an iterative manner to calculate and obtain the autocorrelation coefficients.

[0125] The LSF parameters and the LPC coefficients can be converted to each other, so that when the LPC coefficients are calculated and obtained, the LSF parameters can be determined correspondingly. In other words, whether the glottal parameters are the LPC coefficients or the LSF parameters, they can be determined by the above process.

[0126] Step 1340: Calculate and obtain the gain based on the glottal parameters and the set of autocorrelation parameters.

[0127] The corresponding gain of the nth speech frame can be calculated according to the following equation: [Math 4] (Formula 14)

[0128]

number

[0129] G(n), calculated according to Equation 14, is the squared gain corresponding to the target speech frame on the time domain representation.

[0130] Step 1350: Calculate and obtain a power spectrum of the excitation signal based on the gain and a power spectrum of a glottal filter, the glottal filter being a filter constructed based on the glottal parameters.

[0131] Assuming that the corresponding complex spectrum of the target speech frame is obtained by Fourier transforming m (m is a positive integer) sample points, to calculate the power spectrum of the glottal filter, first construct an all-zero array s_AR(n) of dimension m for the nth speech frame, then construct a (p+1)-dimensional array a j . . p. Assign m(n) to the first (p+1) dimension of the all-zero array, where j=0, 1, 2, ... p, and obtain the FFT coefficients by invoking the Fast Fourier Transform (FFT) of the m sample points. S_AR(n)=FFT(s_AR(n)) (Eq. 15) In addition to obtaining the FFT coefficient S_AR(n), the power spectrum of the glottal filter corresponding to the nth speech frame can be obtained for each sample according to the following Equation 16: AR_LPS(n,k)=(Real(S_AR(n,k))) 2 +(Imag(S_AR(n,k))) 2 (Formula 16) Here, Real(S_AR(n,k)) represents the real part of S_AR(n,k), Imag(S_AR(n,k)) represents the imaginary part of S_AR(n,k), k represents the sequence of FFT coefficients, 0≦k≦m, and k is a positive integer.

[0132] After obtaining the frequency response AR_LPS(n) of the glottal filter corresponding to the nth speech frame, for convenience of calculation, convert the power spectrum AR_LPS(n) of the glottal filter from the natural number domain to the logarithm domain according to Equation 17; AR_LPS1(n)=log 10 (AR_LPS(n)) (Eq. 17) The above AR_LPS1(n) is inverted according to the following equation 18, that is, the inverse corresponding power spectrum of the glottal filter AR_LPS2(n) is obtained, AR_LPS2(n)=-1*AR_LPS1(n) (Equation 18) Then, the power spectrum R(n) of the excitation signal corresponding to the target speech frame can be calculated and obtained according to the following Equation 19. R(n)=Pa(n)*(G1(n)) 2 *AR_LPS3(n) (Equation 19) Here, [Number 5] (Equation 20) [Number 6] (Formula 21)

[0133]

number

number

[0134] Through the above process, the glottal parameters corresponding to the target speech frame, the gain and frequency response of the excitation signal, and the frequency response of the glottal filter limited by the glottal parameters are calculated and obtained.

[0135] After obtaining the gain corresponding to the target speech frame, the power spectrum of the corresponding excitation signal, and the power spectrum of the glottal filter limited to the glottal parameters, the synthesis process can be performed according to the process shown in Fig. 14. As shown in Fig. 14, step 430 includes steps 1410 to 1430.

[0136] Step 1410: Generate a first amplitude spectrum based on the power spectrum of the glottal filter and the power spectrum of the excitation signal.

[0137] The first amplitude spectrum S_filt(n) can be obtained by calculation according to the following Equation 22. [Number 7] (Formula 22)

[0138]

number

[0139] Here, R1(n)=10*log10 (R(n)) (Equation 23)

[0140] Step 1420: Amplify the first amplitude spectrum according to the gain to obtain a second amplitude spectrum.

[0141] The second amplitude spectrum S_e(n) can be obtained according to the following formula: S_e(n)=G2(n)*S_filt(n) (Equation 24) Here, [Equation 8] (Equation 25)

[0142]

number

[0143] Step 1430: Determine a corresponding enhanced speech signal of the target speech frame based on the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

[0144] In some embodiments of the present application, step 1430 further includes a step of combining the second amplitude spectrum with the phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum, in other words, the second amplitude spectrum is the real part of the second complex spectrum and the phase spectrum extracted from the first complex spectrum is the imaginary part of the second complex spectrum, and transforming the second complex spectrum into a time domain to obtain a time domain signal of the enhanced speech signal corresponding to the target speech frame.

[0145] 15 is a flowchart of a speech enhancement method according to a specific embodiment, in which the n-th speech frame is a target speech frame, and the time domain signal of the n-th speech frame is s(n). As shown in FIG. 15, the method specifically includes steps 1510 to 1560.

[0146] Step 1510: Time-frequency transformation, in which the time domain signal s(n) of the nth speech frame is transformed through step 1510 to obtain the corresponding complex spectrum S(n) of the nth speech frame.

[0147] Step 1520: Pre-emphasis: performing pre-emphasis processing on the n-th speech frame based on the corresponding complex spectrum S(n) of the n-th speech frame to obtain a first complex spectrum S'(n) of the n-th speech frame.

[0148] Step 1530: Spectral decomposition. Performing spectral decomposition on the first complex spectrum S'(n) to obtain a corresponding power spectrum Pa(n) and a corresponding phase spectrum Ph(n) of the first complex spectrum S'(n).

[0149] Step 1540: speech decomposition, performing speech decomposition according to the power spectrum Pa(n) of the speech frame of n-th frame, determining the corresponding glottal parameter set P(n) of the speech frame of n-th frame and the frequency domain representation R(n) of the excitation signal corresponding to the speech frame of n-th frame. The glottal parameter set P(n) includes glottal parameter ar(n) and gain G(n). A specific speech decomposition process may be shown in Figure 13, obtaining the glottal parameters, and correspondingly obtaining the power spectrum AR_LPS(n) of the glottal filter, the power spectrum R(n) of the excitation signal and the gain G(n).

[0150] Step 1550: synthesize speech. A specific speech synthesis process may be shown in Fig. 14, and synthesize the frequency response AR_LPS(n) of the glottal filter corresponding to the nth speech frame, the frequency response R(n) of the excitation signal, and the gain G(n) to obtain a second amplitude spectrum S_e(n).

[0151] Step 1560: Frequency-time transform. The phase spectrum Ph(n) extracted from the first complex spectrum S'(n) is multiplexed, and the phase spectrum Ph(n) is combined with the second amplitude spectrum S_e(n) to obtain an enhanced complex spectrum corresponding to the nth speech frame. The obtained enhanced complex spectrum is transformed into the time domain to obtain a time domain signal s_e(n) of an enhanced speech signal corresponding to the nth speech frame.

[0152] In the solution of this embodiment, speech decomposition is performed based on the first complex spectrum obtained by performing pre-emphasis on the target speech frame, and in the process of pre-emphasis, some noise information is removed, so that the noise information in the first complex spectrum is less. Therefore, by performing speech decomposition based on the first complex spectrum, the influence of noise on speech decomposition is reduced, the difficulty of speech decomposition is reduced, the accuracy of glottal parameters, excitation signal and gain obtained by speech decomposition is improved, and the accuracy of the subsequently obtained emphasized speech signal is guaranteed. In addition, in the speech synthesis process, the solution can only focus on the amplitude spectrum, and does not need to focus on phase information, and the phase spectrum extracted from the first complex spectrum is directly multiplexed, thereby reducing the amount of calculation in the speech synthesis process. The first complex spectrum is obtained by performing pre-emphasis, and its noise content is less, so that the accuracy of the phase information is guaranteed to a certain extent.

[0153] In the embodiment shown in FIG. 15, in step 1510, pre-emphasis can be realized by a first neural network. Step 1540 can be realized according to the process shown in FIG. 13, and step 1550 can be realized according to the process shown in FIG. 14, thereby achieving a deep combination of traditional signal processing and deep learning, and performing secondary emphasis on the target speech frame. Therefore, the embodiment of the present application realizes performing multi-stage emphasis on the target speech frame. That is, in the first stage, a deep learning method is adopted to perform pre-emphasis based on the amplitude spectrum of the target speech frame, which can reduce the difficulty of obtaining glottal parameters, excitation signals and gains through speech decomposition in the second stage, and in the second stage, a signal processing method is used to obtain glottal parameters, excitation signals and gains used to reconstruct the original speech signal. And in the second stage, speech synthesis is performed according to a digital model where speech is generated, and processing is not directly performed on the signal of the target speech frame, so that the occurrence of speech reduction situations in the second stage can be avoided.

[0154] In some embodiments of the present application, before step 410, the method further includes the steps of obtaining a time-domain signal of the target speech frame, and performing a time-frequency transformation on the time-domain signal of the target speech frame to obtain a complex spectrum of the target speech frame.

[0155] The time-frequency transform may be a short-term Fourier transform (STFT). In the short-term Fourier transform, a windowing and overlapping operation is adopted to eliminate the unevenness between frames. FIG. 16 is a schematic diagram of the windowing and overlapping in the short-term Fourier transform shown according to a specific embodiment. In FIG. 16, a 50% windowing and overlapping operation is adopted, and if the short-term Fourier transform is for 640 sample points, the number of overlapped samples (hop-size) of the window function is 320. The window function used for the windowing may be a Hanning window, a Hamming window, etc., and of course, other window functions may be adopted, and no specific limitation is made here.

[0156] In other embodiments, a windowing / overlap operation other than 50% may be used, for example if the short-time Fourier transform is for 512 sample points, then if there are 320 sample points in one audio frame, it is only necessary to overlap 192 sample points of the previous audio frame.

[0157] In some embodiments of the present application, the step of obtaining the time-domain signal of the target audio frame further includes the steps of obtaining an audio signal to be processed, the audio signal to be processed being an audio signal obtained by decoding a collected audio signal or an encoded audio, and performing frame division on the audio signal to be processed to obtain the time-domain signal of the target audio frame.

[0158] In some examples, the speech signal to be processed can be divided into frames according to a set frame length, and the frame length can be set according to actual needs, for example, the frame length is set to 20 ms. By dividing the frames, multiple speech frames can be obtained, and each of the speech frames can be the target speech frame in the present application.

[0159] As described above, the solution of the present application can be applied to a transmitting end to perform voice enhancement, and can also be applied to a receiving end to perform voice enhancement. When the solution of the present application is applied to a transmitting end, the voice signal to be processed is a voice signal collected by a transmitting end, and in this case, frame division is performed on the voice signal to be processed to obtain a plurality of voice frames. After frame division, the voice signal to be processed is divided into a plurality of voice frames, and then each voice frame is taken as a target voice frame, and enhancement can be performed on the target voice frame according to the above process of steps 410 to 430. Furthermore, after obtaining the enhanced voice signal corresponding to the target voice frame, the enhanced voice signal can also be further encoded, and voice transmission is performed based on the obtained encoding.

[0160] In one embodiment, the directly collected voice signal is an analog signal, so that in order to facilitate signal processing, the voice signal needs to be further digitized before frame division, and the time-continuous voice signal needs to be converted into a time-discrete voice signal. In the digitization process, the collected voice signal can be sampled according to a set sampling rate, and the set sampling rate can be 16000Hz, 8000Hz, 32000Hz, 48000Hz, etc., and can be specifically set according to actual needs.

[0161] When the solution of the present application is applied to the receiving end, the speech signal to be processed is a speech signal obtained by decoding the received encoded speech. In such a case, the transmitting end may not have performed enhancement on the speech signal to be transmitted, so that the receiving end needs to perform enhancement on the speech signal in order to improve the signal quality. After obtaining a plurality of speech frames by performing frame division on the speech signal to be processed, the speech frames are set as target speech frames, and enhancement is performed on the target speech frames according to the above-mentioned steps 410 to 430 to obtain an enhanced speech signal of the target speech frame. In addition, the corresponding enhanced speech signal of the target speech frame can also be reproduced, and the obtained enhanced speech signal has a higher quality of the speech signal than the signal before the enhancement of the target speech frame because noise has already been removed, and therefore the user has a better auditory experience.

[0162] Hereinafter, an apparatus embodiment of the present application will be described, which can be used to perform the method in the above embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above method embodiment of the present application.

[0163] Fig. 17 is a block diagram of a speech enhancement device according to an embodiment. As shown in Fig. 17, the speech enhancement device includes: a pre-emphasis module 1710 for performing pre-emphasis processing on a target speech frame according to a complex spectrum of the target speech frame to obtain a first complex spectrum, a speech decomposition module 1720 for performing speech decomposition on the target speech frame according to the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame, and a synthesis processing module 1730 for performing synthesis processing according to the glottal parameters, the gain and the excitation signal to obtain corresponding enhanced speech signal of the target speech frame.

[0164] In some embodiments of the present application, the pre-emphasis module 1710 includes: a first input unit used for inputting a corresponding complex spectrum of the target speech frame into a first neural network, the first neural network being obtained by training based on a corresponding complex spectrum of a sample speech frame and a corresponding complex spectrum of an original speech signal in the sample speech frame; and a first output unit used for outputting the first complex spectrum by the first neural network based on the corresponding complex spectrum of the target speech frame.

[0165] In some embodiments of the present application, the first neural network includes a complex convolutional layer, a gated recurrent unit layer and a fully connected layer, and the first output unit includes: a complex convolutional unit used by the complex convolutional layer to perform complex convolution processing based on the real part and the imaginary part of the complex spectrum corresponding to the target speech frame; a transformation unit used by the gated recurrent unit layer to perform transformation processing on the output of the complex convolutional layer; and a fully connected unit used by the fully connected layer to perform fully connected processing on the output of the gated recurrent unit and output the first complex spectrum.

[0166] In some embodiments of the present application, the speech decomposition module 1720 includes: a first amplitude spectrum acquisition unit used in a glottal parameter prediction unit, which is used to perform glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain a corresponding glottal parameter of the target speech frame; an excitation signal prediction unit, which is used to perform excitation signal prediction for the target speech frame based on the first complex spectrum to obtain a corresponding excitation signal of the target speech frame; and a gain prediction unit, which is used to perform gain prediction for the target speech frame based on a corresponding gain of a historical speech frame before the target speech frame to obtain a corresponding gain of the target speech frame.

[0167] In some embodiments of the present application, the glottal parameter prediction unit includes: a second input unit, used for inputting the first complex spectrum into a second neural network, where the second neural network is obtained by training based on the corresponding complex spectrum of a sample speech frame and the corresponding glottal parameter of the sample speech frame; and a second output unit, used for outputting the corresponding glottal parameter of the target speech frame by the second neural network based on the first complex spectrum.

[0168] In some other embodiments of the present application, the glottal parameter prediction unit includes: a third input unit, used for inputting the first complex spectrum and the corresponding glottal parameter of a historical speech frame before the target speech frame to a second neural network, where the second neural network is obtained by training based on the corresponding complex spectrum of a sample speech frame, the corresponding glottal parameter of a historical speech frame before the sample speech frame, and the corresponding glottal parameter of the sample speech frame; and a third output unit, used for outputting the corresponding glottal parameter of the target speech frame by the first neural network based on the first complex spectrum and the corresponding glottal parameter of a historical speech frame before the target speech frame.

[0169] In some embodiments of the present application, the gain prediction unit includes: a fourth input unit, used for inputting a corresponding gain of a historical speech frame preceding the target speech frame to a third neural network, where the third neural network is obtained by training based on the corresponding gain of a historical speech frame preceding the sample speech frame and the corresponding gain of the sample speech frame; and a fourth output unit, used for outputting a corresponding gain of the target speech frame by the third neural network based on the corresponding gain of a historical speech frame preceding the target speech frame.

[0170] In some embodiments of the present application, the excitation signal prediction unit includes: a fifth input unit used for inputting the first complex spectrum into a fourth neural network, where the fourth neural network is obtained by training based on the corresponding complex spectrum of a sample speech frame and a frequency domain representation of an excitation signal corresponding to the sample speech frame; and a fifth output unit used for outputting, by the fourth neural network, a frequency domain representation of an excitation signal corresponding to the target speech frame based on the first complex spectrum.

[0171] In some embodiments of the present application, the synthesis processing module 1730 includes a filtering unit used for filtering a corresponding excitation signal of the target speech frame by a glottal filter to obtain a filtered output signal, where the glottal filter is constructed based on a corresponding glottal parameter of the target speech frame, and an amplification processing unit used for performing an amplification processing on the filtered output signal according to a corresponding gain of the target speech frame to obtain a corresponding enhanced speech signal of the target speech frame.

[0172] In some embodiments of the present application, the speech decomposition module 1720 includes: a power spectrum calculation unit used for calculating and obtaining a power spectrum based on the first complex spectrum; an autocorrelation coefficient calculation unit used for calculating and obtaining an autocorrelation coefficient based on the power spectrum; a glottal parameter calculation unit used for calculating and obtaining the glottal parameter based on the autocorrelation coefficient; a gain calculation unit used for calculating and obtaining the gain based on the glottal parameter and the autocorrelation parameter set; and an excitation signal determination unit used for calculating and obtaining a power spectrum of the excitation signal based on the gain and a power spectrum of a glottal filter, where the glottal filter is a filter constructed based on the glottal parameter.

[0173] In some embodiments of the present application, the synthesis processing module 1730 includes a second amplitude spectrum generation unit used for generating a first amplitude spectrum based on the power spectrum of the glottal filter and the power spectrum of the excitation signal, a third amplitude spectrum determination unit used for performing an amplification process on the first amplitude spectrum according to the gain to obtain a second amplitude spectrum, and an enhanced speech signal determination unit used for determining a corresponding enhanced speech signal of the target speech frame based on the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

[0174] In some embodiments of the present application, the enhanced speech signal determination unit includes: a second complex spectrum calculation unit used for combining the second amplitude spectrum with the phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum; and a time domain transformation unit used for transforming the second complex spectrum into a time domain to obtain a time domain signal of the enhanced speech signal corresponding to the target speech frame.

[0175] FIG. 18 shows a structural schematic diagram of a computer system suitable for use in an electronic device for implementing the embodiments of the present application.

[0176] It should be clarified that the electronic device computer system 1800 shown in FIG. 18 is merely an example and should not pose any limitation to the functionality and scope of use of the embodiments of the present application.

[0177] As shown in Fig. 18, the computer system 1800 includes a central processing unit (CPU) 1801, which can execute various suitable operations and processes based on programs stored in a read-only memory (ROM) 1802 or programs uploaded from a storage portion 1808 into a random access memory (RAM) 1803, for example, executing the method in the above embodiment. Various programs and data required for the system operation are also stored in the RAM 1803. The CPU 1801, the ROM 1802, and the RAM 1803 are connected to each other via a bus 1804. An input / output (I / O) interface 1805 is also connected to the bus 1804.

[0178] The following members are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 executes communication processing via a network such as the Internet. A driver 1810 is also connected to the I / O interface 1805 according to needs. A removable medium 1811, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the driver 1810 according to needs, and a computer program read from the medium is installed in the storage section 1808 according to needs.

[0179] In particular, according to an embodiment of the present application, the process described with reference to the above flowcharts can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code used to execute the method illustrated in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network by the communication portion 1809 and / or installed from a removable medium 1811. When the computer program is executed by the central processing unit (CPU) 1801, it performs various functions limited to the system of the present application.

[0180] It is necessary to explain that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium, or a computer-readable storage medium, or any combination of both. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, which can be used in or in combination with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such propagated data signals may take a number of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program used for use in or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, wireless, wired, etc., or any suitable combination of the above.

[0181] The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Here, each block in the flowcharts or block diagrams may represent a module, program segment or part of code, which includes one or more executable instructions used to realize the specified logic function. It should also be noted that in some alternative implementation forms, the functions marked in the blocks may occur in a different order from the order marked in the drawings. For example, two blocks shown in succession may actually be executed essentially in parallel, and in some cases, they may be executed in the reverse order, which is determined by the related functions. It should also be noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts, may be realized in a system based on dedicated hardware that executes the specified functions or operations, or may be realized in a combination of dedicated hardware and computer instructions.

[0182] The units described and mentioned in the embodiments of the present application may be implemented in the form of software or hardware, and the described units may be installed in a processor, where the names of these units do not constitute limitations on the units themselves.

[0183] In another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist separately and not be assembled in the electronic device, the computer-readable storage medium carrying computer-readable instructions, which, when executed by a processor, implements the method in any of the above embodiments.

[0184] According to an aspect of the present application, there is further provided an electronic device including a processor and a memory having computer readable instructions stored thereon that, when executed by the processor, implements the method of any of the above embodiments.

[0185] According to one aspect of the present invention, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable storage medium, the processor executing the computer instructions and causing the computing device to perform the method of any of the above embodiments.

[0186] It should be noted that although the above detailed description refers to multiple modules or units of an apparatus for performing operations, such division is not mandatory. In fact, according to an embodiment of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided to be embodied by multiple modules or units.

[0187] From the above description of the embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein may be realized by software, or may be realized by a combination of software and necessary hardware. Therefore, the technical means according to the embodiments of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a U disk, a mobile disk, etc.) or on a network, and includes some instructions to make a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the present application.

[0188] Those skilled in the art can easily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present application, which modifications, uses or adaptations follow the general principles of the present application and include known knowledge or common technical means in the art that are not disclosed herein.

[0189] It is to be understood that the present application is not limited to the exact construction described above and illustrated in the drawings, and various modifications and variations can be made without departing from the scope thereof, which is limited only by the appended claims. [Explanation of symbols]

[0190] 110 Transmitting end 111 Collection Module 112 Pre-enhancement processing module 113 Encoding Module 120 Receiving end 121 Decryption Module 122 Post-emphasis module 123 Playback Module 1710 Pre-emphasis module 1720 Audio Decomposition Module 1730 Synthesis Processing Module 1800 Computer Systems 1801 Central Processing Unit (CPU) 1804 Bus 1805 I / O Interface 1805 Output (Input / Output, I / O) Interface 1806 Input section 1807 Output part 1808 Memory part 1809 Communication section 1810 Driver 1811 Medium

Claims

1. 1. A method for speech enhancement performed by a computing device, comprising: performing a pre-emphasis process on the target speech frame based on a corresponding complex spectrum of the target speech frame to obtain a first complex spectrum; performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame; performing a synthesis process based on the glottal parameters, the gain and the excitation signal to obtain a corresponding enhanced speech signal of the target speech frame; A method for enhancing speech, comprising:

2. The step of performing pre-emphasis processing on the target speech frame based on a corresponding complex spectrum of the target speech frame to obtain a first complex spectrum includes: inputting the corresponding complex spectrum of the target speech frame into a first neural network, the first neural network being obtained by training based on the corresponding complex spectrum of a sample speech frame and the corresponding complex spectrum of an original speech signal in the sample speech frame, the sample speech frame being obtained by combining the original speech signal and a noise signal; outputting, by the first neural network, the first complex spectrum based on a corresponding complex spectrum of the target speech frame; The method of claim 1 , comprising:

3. The first neural network includes a complex convolutional layer, a gated recurrent unit layer, and Includes a fully connected layer, The first neural network generates a corresponding complex space of the target speech frame. The step of outputting the first complex spectrum based on a vector comprises: performing complex convolution processing by the complex convolution layer based on a real part and an imaginary part of a complex spectrum corresponding to the target speech frame; performing a transform process on the output of the complex convolution layer by the gated recurrent unit layer; performing a fully connected process on the output of the gated recurrent unit by the fully connected layer, and outputting the first complex spectrum; The method of claim 2 , comprising:

4. The step of performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame comprises: performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters of the target speech frame; performing excitation signal prediction for the target speech frame based on the first complex spectrum to obtain a corresponding excitation signal for the target speech frame; performing gain prediction for the target speech frame based on corresponding gains of historical speech frames preceding the target speech frame to obtain corresponding gains of the target speech frame; The method of claim 1 , comprising:

5. performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters of the target speech frame, inputting the first complex spectrum into a second neural network, the second neural network being trained based on the corresponding complex spectrum of a sample speech frame and the corresponding glottal parameters of the sample speech frame; outputting, by the second neural network, corresponding glottal parameters of the target speech frame based on the first complex spectrum; The method of claim 4 , comprising:

6. performing glottal parameter prediction for the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters of the target speech frame, inputting the first complex spectrum and a corresponding glottal parameter of a historical speech frame preceding the target speech frame into a second neural network, the second neural network being obtained by training based on the corresponding complex spectrum of a sample speech frame, the corresponding glottal parameter of a historical speech frame preceding the sample speech frame, and the corresponding glottal parameter of the sample speech frame; outputting, by the second neural network, corresponding glottal parameters of the target speech frame based on the first complex spectrum and corresponding glottal parameters of a historical speech frame preceding the target speech frame; The method of claim 4 , comprising:

7. The step of performing a gain prediction for the target speech frame based on a corresponding gain of a historical speech frame preceding the target speech frame to obtain a corresponding gain of the target speech frame includes: inputting a corresponding gain of a historical speech frame preceding the target speech frame into a third neural network, the third neural network being trained based on the corresponding gain of a historical speech frame preceding the sample speech frame and the corresponding gain of the sample speech frame; outputting, by the third neural network, a corresponding gain of the target speech frame based on a corresponding gain of a historical speech frame preceding the target speech frame; The method of claim 4 , comprising:

8. The step of performing excitation signal prediction for the target speech frame based on the first complex spectrum to obtain a corresponding excitation signal for the target speech frame comprises: inputting the first complex spectrum into a fourth neural network, the fourth neural network being trained based on a corresponding complex spectrum of a sample speech frame and a frequency domain representation of an excitation signal corresponding to the sample speech frame; outputting, by the fourth neural network, a frequency domain representation of an excitation signal corresponding to the target speech frame based on the first complex spectrum; The method of claim 4 , comprising:

9. said step of performing a synthesis process based on said glottal parameters, said gain and said excitation signal to obtain a corresponding enhanced speech signal of said target speech frame comprises: filtering the corresponding excitation signal of the target speech frame with a glottal filter to obtain a filtered output signal, the glottal filter being constructed based on the corresponding glottal parameters of the target speech frame; amplifying the filtering output signal according to a corresponding gain of the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame; The method of claim 4 , comprising:

10. The step of performing speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gain and excitation signal of the target speech frame comprises: calculating and obtaining a power spectrum based on the first complex spectrum; calculating and obtaining an autocorrelation coefficient based on the power spectrum; calculating and obtaining the glottal parameters based on the autocorrelation coefficients; calculating and obtaining the gain based on the glottal parameters and the autocorrelation coefficients; calculating and obtaining a power spectrum of the excitation signal based on the gain and a power spectrum of a glottal filter, the glottal filter being a filter constructed based on the glottal parameters; The method of claim 1 , comprising:

11. said step of performing a synthesis process based on said glottal parameters, said gain and said excitation signal to obtain a corresponding enhanced speech signal of said target speech frame comprises: generating a first amplitude spectrum based on a power spectrum of the glottal filter and a power spectrum of the excitation signal; amplifying the first amplitude spectrum in accordance with the gain to obtain a second amplitude spectrum; and determining a corresponding enhanced speech signal of the target speech frame based on the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

12. determining a corresponding enhanced speech signal of the target speech frame based on the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum, combining the second amplitude spectrum with a phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum; transforming the second complex spectrum into a time domain to obtain a time domain signal of an enhanced speech signal corresponding to the target speech frame; The method of claim 11 , comprising:

13. A voice enhancement device, comprising: A pre-emphasis module is used for performing a pre-emphasis process on the target speech frame according to the complex spectrum of the target speech frame to obtain a first complex spectrum; a speech decomposition module adapted to perform speech decomposition on the target speech frame based on the first complex spectrum to obtain corresponding glottal parameters, gains and excitation signals of the target speech frame; a synthesis processing module, which is used for performing synthesis processing based on the glottal parameters, the gain and the excitation signal to obtain a corresponding enhanced speech signal of the target speech frame; A voice enhancement device comprising:

14. An electronic device, A processor; A memory, on which computer readable instructions are stored, the computer readable instructions being adapted to implement a method according to any one of claims 1 to 12 when executed by the processor; Including electronic devices.

15. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images