Speech enhancement method, device, equipment and storage medium

By pre-enhancing and decomposing the voice signal, the glottal parameters and excitation signals are obtained, efficient enhancement of the voice signal is achieved, noise interference problem is solved, and communication quality is improved.

CN113571080BActive Publication Date: 2024-11-08TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110181389.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-08
Publication Date
2024-11-08
Estimated Expiration
2041-02-08

AI Technical Summary

Technical Problem

There is noise interference in voice communication, resulting in poor communication quality and affecting the user's auditory experience. It is difficult for the prior art to effectively remove noise and retain the original quality of the voice signal.

Method used

By pre-enhancing the target speech frame, the complex spectrum is obtained, and then speech decomposition is performed to obtain glottal parameters, gain and excitation signals, and synthesis processing is performed based on these parameters to achieve enhanced speech signals.

Benefits of technology

It effectively removes noise, improves the quality of the voice signal, ensures the accuracy of the amplitude and phase of the enhanced voice signal, and improves the user's auditory experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113571080B_ABST
    Figure CN113571080B_ABST
Patent Text Reader

Abstract

The present application relates to the field of speech processing technology, and specifically provides a speech enhancement method, device, equipment and storage medium, the method comprising: pre-enhancement processing of the target speech frame according to the complex spectrum corresponding to the target speech frame to obtain a first complex spectrum; speech decomposition of the target speech frame according to the first complex spectrum to obtain the glottal parameters, gain and excitation signal corresponding to the target speech frame; synthesis processing according to the glottal parameters, the gain and the excitation signal to obtain the enhanced speech signal corresponding to the target speech frame. This solution can effectively enhance speech and can be applied to cloud conferences.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech enhancement method, apparatus, device and storage medium. Background Art

[0002] Due to the convenience and timeliness of voice communication, voice communication is increasingly used, for example, voice signals are transmitted between participants in a cloud conference. In voice communication, the voice signal may be mixed with noise, which will lead to poor communication quality and greatly affect the user's auditory experience. Therefore, how to enhance the voice to remove noise is a technical problem that needs to be solved in the prior art. Summary of the invention

[0003] The embodiments of the present application provide a speech enhancement method, apparatus, device and storage medium to achieve speech enhancement.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a speech enhancement method is provided, comprising: performing pre-enhancement processing on a target speech frame according to a complex spectrum corresponding to the target speech frame to obtain a first complex spectrum; performing speech decomposition on the target speech frame according to the first complex spectrum to obtain glottal parameters, gain and excitation signal corresponding to the target speech frame; performing synthesis processing according to the glottal parameters, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

[0006] According to one aspect of an embodiment of the present application, a speech enhancement device is provided, comprising: a pre-enhancement module, used to perform pre-enhancement processing on a target speech frame according to the complex spectrum of the target speech frame to obtain a first complex spectrum; a speech decomposition module, used to perform speech decomposition on the target speech frame according to the first complex spectrum to obtain glottal parameters, gain and excitation signal corresponding to the target speech frame; and a synthesis processing module, used to perform synthesis processing according to the glottal parameters, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

[0007] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the speech enhancement method as described above is implemented.

[0008] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the speech enhancement method as described above is implemented.

[0009] In the scheme of the present application, the target speech frame is first pre-enhanced to obtain the first complex spectrum, and then the target speech frame is subjected to speech decomposition and synthesis based on the first complex spectrum, so that the target speech frame is enhanced in two stages, which can effectively ensure the speech enhancement effect. Moreover, based on the first complex spectrum obtained by pre-enhancing the target speech frame, the target speech frame is subjected to speech decomposition. Compared with the target speech frame before pre-enhancement, the first complex spectrum contains less noise information, and in the speech decomposition process, noise will affect the accuracy of speech decomposition. Therefore, using the first complex spectrum as the basis for speech decomposition can reduce the difficulty of speech decomposition, improve the accuracy of the glottal parameters, excitation signals and gains obtained by speech decomposition, and thus ensure the accuracy of the subsequent enhanced speech signal. Moreover, the first complex spectrum obtained by pre-enhancement includes phase information and amplitude information, and speech decomposition and speech synthesis are performed based on the phase information and amplitude information in the first complex spectrum, which ensures the accuracy of the amplitude and phase of the enhanced speech signal corresponding to the obtained target speech frame.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 The diagram is a schematic diagram of a voice communication link in a VoIP (Voice over Internet Protocol) system according to a specific embodiment.

[0013] Figure 2 Schematic diagram showing a digital model of speech signal generation.

[0014] Figure 3 The figure shows a schematic diagram of decomposing an excitation signal and a frequency response of a glottal filter according to an original speech signal.

[0015] Figure 4 It is a flow chart of a speech enhancement method according to an embodiment of the present application.

[0016] Figure 5 It is a schematic diagram of a complex convolution layer performing convolution processing on a complex number according to a specific embodiment.

[0017] Figure 6 It is a schematic diagram of the structure of a first neural network according to a specific embodiment.

[0018] Figure 7 is a schematic diagram of a second neural network according to a specific embodiment.

[0019] Figure 8 is a schematic diagram of input and output of a second neural network according to another embodiment.

[0020] Fig. 9 is a schematic diagram of a third neural network according to a specific embodiment.

[0021] Fig.10 is a schematic diagram of a fourth neural network according to a specific embodiment.

[0022] Fig.11 is a flow chart showing step 430 according to an embodiment.

[0023] Fig.12 The figure is a flow chart of a speech enhancement method according to a specific embodiment.

[0024] Fig.13 is a flow chart of step 420 according to another embodiment.

[0025] Fig.14 Is a flowchart of step 430 according to another embodiment.

[0026] Fig.15 is a flow chart of a speech enhancement method according to another specific embodiment.

[0027] Fig.16 The invention is a schematic diagram of windowing and overlapping in short-time Fourier transform according to a specific embodiment.

[0028] Fig.17 is a block diagram of a speech enhancement device according to an embodiment.

[0029] Fig.18 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0031] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0035] Noise in speech signals can greatly reduce speech quality and affect the user's auditory experience. Therefore, in order to improve the quality of speech signals, it is necessary to enhance the speech signals to remove noise as much as possible and retain the original speech signals in the signals (i.e., pure signals without noise). In order to achieve speech enhancement processing, the solution of the present application is proposed.

[0036] The solution of the present application can be applied to application scenarios of voice calls, such as voice communication through instant messaging applications and voice calls in game applications. Specifically, voice enhancement can be performed according to the solution at the voice sending end, the voice receiving end, or the server providing voice communication services.

[0037] Cloud conferencing is an important part of online office work. In a cloud conference, after the sound collection device of the participants of the cloud conference collects the voice signal of the speaker, it needs to send the collected voice signal to other conference participants. This process involves the transmission and playback of voice signals between multiple participants. If the noise signal mixed in the voice signal is not processed, it will greatly affect the auditory experience of the conference participants. In this scenario, the solution of this application can be used to enhance the voice signal in the cloud conference, so that the voice signal heard by the conference participants is the enhanced voice signal, thereby improving the quality of the voice signal.

[0038] Cloud conferencing is an efficient, convenient and low-cost conferencing format based on cloud computing technology. Users only need to use the Internet interface and perform simple and easy-to-use operations to quickly and efficiently share voice, data files and videos with teams and customers around the world. The cloud conferencing service provider helps users operate complex technologies such as data transmission and processing during the meeting.

[0039] At present, domestic cloud conferences are mainly concentrated in the service content based on the SaaS (Software as a Service) model, including telephone, network, video and other service forms. Video conferences based on cloud computing are called cloud conferences. In the era of cloud conferences, data transmission, processing and storage are all handled by the computer resources of the video conference provider. Users no longer need to purchase expensive hardware and install cumbersome software. They only need to open the client and enter the corresponding interface to conduct efficient remote conferences.

[0040] The cloud conference system supports dynamic cluster deployment of multiple servers and provides multiple high-performance servers, which greatly improves the stability, security and availability of conferences. In recent years, video conferencing has been welcomed by many users because it can greatly improve communication efficiency, continuously reduce communication costs, and bring about internal management upgrades. It has been widely used in various fields such as government, transportation, finance, operators, education, and enterprises.

[0041] Figure 1 FIG. 1 is a schematic diagram of a voice communication link in a VoIP (Voice over Internet Protocol) system according to a specific embodiment. Figure 1 As shown, based on the network connection between the sending end 110 and the receiving end 120, the sending end 110 and the receiving end 120 can perform voice transmission.

[0042] like Figure 1As shown, the transmitting end 110 includes an acquisition module 111, a pre-enhancement processing module 112 and a coding module 113, wherein the acquisition module 111 is used to acquire voice signals, which can convert the acquired acoustic signals into digital signals; the pre-enhancement processing module 112 is used to enhance the acquired voice signals to remove noise in the acquired voice signals and improve the quality of the voice signals. The coding module 113 is used to encode the enhanced voice signals to improve the anti-interference of the voice signals during transmission. The pre-enhancement processing module 112 can perform voice enhancement according to the method of the present application, and after the voice is enhanced, it is encoded, compressed and transmitted, so as to ensure that the signal received by the receiving end is no longer affected by noise.

[0043] The receiving end 120 includes a decoding module 121, a post-enhancement module 122 and a playing module 123. The decoding module 121 is used to decode the received encoded speech to obtain a decoded signal; the post-enhancement module 122 is used to enhance the decoded speech signal; and the playing module 123 is used to play the enhanced speech signal. The post-enhancement module 122 can also perform speech enhancement according to the method of the present application. In some embodiments, the receiving end 120 can also include a sound effect adjustment module, which is used to adjust the sound effect of the enhanced speech signal.

[0044] In a specific embodiment, speech enhancement may be performed only at the receiving end 120 or only at the transmitting end 110 according to the method of the present application. Of course, speech enhancement may also be performed at both the transmitting end 110 and the receiving end 120 according to the method of the present application.

[0045] In some application scenarios, the terminal equipment in the VoIP system can not only support VoIP communication, but also support other third-party protocols, such as traditional PSTN (Public Switched Telephone Network) circuit domain telephone. However, traditional PSTN services cannot perform voice enhancement. In this scenario, voice enhancement can be performed in the terminal serving as the receiving end according to the method of the present application.

[0046] Before describing the scheme of this application, it is necessary to introduce the generation of speech signals. Speech signals are generated by the physiological movement of the human vocal organs under the control of the brain, that is, at the trachea, a noise-like impact signal (equivalent to the excitation signal) with a certain energy is generated; the impact signal impacts the human vocal cords (the vocal cords are equivalent to the glottal filter), generating a quasi-periodic opening and closing; after being amplified by the mouth, a sound is emitted (output speech signal).

[0047] Figure 2 FIG. 1 shows a schematic diagram of a digital model for speech signal generation, through which the generation process of speech signals can be described. Figure 2 As shown, after the excitation signal impacts the glottal filter, the speech signal is output after gain control, wherein the glottal filter is limited by the glottal parameters. This process can be expressed by the following formula:

[0048] x(n)=G·r(n)·ar(n); (Formula 1)

[0049] Among them, x(n) represents the input speech signal; G represents the gain, which can also be called the linear prediction gain; r(n) represents the excitation signal; ar(n) represents the glottal filter.

[0050] Figure 3 FIG. 4 is a schematic diagram showing a frequency response of an excitation signal and a glottal filter decomposed from an original speech signal. Figure 3 a shows the frequency response diagram of the original speech signal. Figure 3 b shows a schematic diagram of the frequency response of the glottal filter decomposed from the original speech signal, Figure 3 c shows a schematic diagram of the frequency response of the excitation signal decomposed from the original speech signal. Figure 3 As shown, the undulating part of the frequency response diagram of the original speech signal corresponds to the peak position in the frequency response diagram of the glottal filter, and the excitation signal is equivalent to the residual signal after LP (Linear Prediction) analysis of the original speech signal, so its corresponding frequency response is relatively flat.

[0051] It can be seen from the above that an excitation signal, a glottal filter and a gain can be decomposed according to an original speech signal (i.e., a speech signal without noise), and the decomposed excitation signal, glottal filter and gain can be used to express the original speech signal, wherein the glottal filter can be expressed by glottal parameters. Conversely, if an excitation signal corresponding to an original speech signal, a glottal parameter and a gain for determining a glottal filter are known, the original speech signal can be reconstructed according to the corresponding excitation signal, glottal filter and gain.

[0052] The solution of the present application is based on this principle, and reconstructs the original speech signal in the speech frame according to the glottal parameters, excitation signal and gain corresponding to the speech frame to achieve speech enhancement.

[0053] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0054] Figure 4 1 is a flow chart of a method for speech enhancement according to an embodiment of the present application. The method can be executed by a computer device with processing capabilities, such as a terminal, a server, etc., which is not specifically limited here. Figure 4As shown, the method at least includes steps 410 to 430, which are described in detail as follows:

[0055] Step 410: pre-enhance the target speech frame according to the complex spectrum corresponding to the target speech frame to obtain a first complex spectrum.

[0056] The speech signal changes over time rather than steadily and randomly, but the speech signal is strongly correlated in a short time, that is, the speech signal has short-term correlation. Therefore, in the solution of the present application, speech enhancement is performed in units of speech frames. The target speech frame refers to the speech frame to be enhanced.

[0057] The complex spectrum corresponding to the target speech frame can be obtained by performing a time-frequency transformation on the time domain signal of the target speech frame, such as a short-term Fourier transform (STFT). The coefficient of the real part of the complex spectrum corresponding to the target speech frame is used to indicate the amplitude information of the target speech frame, and the coefficient of the imaginary part is used to indicate the phase information of the target speech frame.

[0058] By pre-enhancing the target speech frame, part of the noise in the target speech frame can be removed. Therefore, compared with the complex spectrum corresponding to the target speech frame, the first complex spectrum obtained by pre-enhancement contains less noise.

[0059] In some embodiments of the present application, a deep learning method can be used to pre-enhance the target speech frame. A neural network model is trained to predict the complex spectrum of the noise in the speech frame based on the complex spectrum corresponding to the speech frame, and then the complex spectrum of the speech frame is subtracted from the complex spectrum of the predicted noise to obtain a first complex spectrum. For ease of description, the neural network model used to predict the complex spectrum of the noise in the speech frame is called a noise prediction model. After the training is completed, the noise prediction model can output the complex spectrum of the predicted noise based on the complex spectrum of the input speech frame, and then the complex spectrum of the speech frame is subtracted from the complex spectrum of the noise to obtain a first complex spectrum.

[0060] In some embodiments of the present application, a neural network model can also be trained to predict the first complex spectrum of the enhanced speech frame based on the complex spectrum of the speech frame. For ease of description, the neural network model used to predict the enhanced complex spectrum is called an enhanced complex spectrum prediction model. During the training process, the complex spectrum of the sample speech frame is input into the enhanced complex spectrum prediction model, and the enhanced complex spectrum is predicted by the enhanced complex spectrum prediction model, and the parameters of the enhanced complex spectrum prediction model are adjusted according to the predicted enhanced complex spectrum and the label information of the sample speech frame, until the difference between the predicted enhanced complex spectrum and the complex spectrum indicated by the label information meets the preset requirements. The label information of the sample speech frame is used to indicate the complex spectrum of the original speech signal in the sample speech frame. After the training is completed, the amplitude spectrum prediction model can output the first complex spectrum according to the complex spectrum of the target speech frame.

[0061] Step 420: Perform speech decomposition on the target speech frame according to the first complex spectrum to obtain glottal parameters, gain, and excitation signal corresponding to the target speech frame.

[0062] The glottal parameters, the corresponding gain and the corresponding excitation signal corresponding to the target speech frame obtained by speech decomposition are used according to Figure 2 The process shown reconstructs the original speech signal in the target speech frame.

[0063] As described above, an original speech signal is obtained by impacting a glottal filter with an excitation signal and then performing gain control. The first complex spectrum includes information of the original speech signal of the target speech frame, and therefore, a linear prediction analysis is performed based on the first complex spectrum to reversely determine the glottal parameters, excitation signal and gain for reconstructing the original speech signal in the target speech frame.

[0064] The glottal parameters refer to the parameters used to construct the glottal filter. When the glottal parameters are determined, the glottal filter is determined accordingly, and the glottal filter is a digital filter. The glottal parameters can be linear prediction coding (Linear Prediction Coefficients, LPC) coefficients, and can also be line spectral frequency (Line Spectral Frequency, LSF) parameters. The number of glottal parameters corresponding to the target speech frame is related to the order of the glottal filter. If the glottal filter is a K-order filter, the glottal parameters include K-order LSF parameters or K-order LPC coefficients, wherein the LSF parameters and LPC coefficients can be converted to each other.

[0065] A p-order glottal filter can be expressed as:

[0066] A p (z) = 1 + a1z -1 +a2z -2+…+a p z -p ; (Formula 2)

[0067] Among them, a1, a2, ..., a p is the LPC coefficient; p is the order of the glottal filter; z is the input signal of the glottal filter.

[0068] Based on formula 2, if:

[0069] P(z)=A p (z)-z -(p+1) A p (z -1 ); (Formula 3)

[0070] Q(z)=A p (z)+z -(p+1) A p (z -1 ); (Formula 4)

[0071] You can get:

[0072]

[0073] In a physical sense, P(z) and Q(z) represent the periodic variation of glottal opening and glottal closing, respectively. The roots of the polynomials P(z) and Q(z) appear alternately on the complex plane, and are distributed as a series of angular frequencies on the unit circle of the complex plane. The LSF parameter is the angular frequency corresponding to the roots of P(z) and Q(z) on the unit circle of the complex plane. The LSF parameter LSF(n) corresponding to the nth frame of speech can be expressed as ω n Of course, the LSF parameter LSF(n) corresponding to the n-th speech frame can also be directly represented by the root of P(z) and the root of Q(z) corresponding to the n-th speech frame.

[0074] The roots of P(z) and Q(z) corresponding to the nth speech frame in the complex plane are defined as θ n , then the LSF parameters corresponding to the nth speech frame are expressed as:

[0075]

[0076] Among them, Rel{θ n} represents the complex number θ n The real part of Imag{θ n} represents the complex number θ n The imaginary part of .

[0077] In some embodiments of the present application, speech decomposition can be performed in a deep learning manner. Neural network models for glottal parameter prediction, excitation signal prediction, and gain prediction can be trained first, so that the three neural network models can respectively predict the glottal parameters, excitation signal, and gain corresponding to the target speech frame based on the first complex spectrum.

[0078] In some embodiments of the present application, signal processing can also be performed according to the principle of linear prediction analysis based on the first complex spectrum, and the glottal parameters, excitation signals and gains corresponding to the target speech frame can be calculated. The specific process is described below.

[0079] Step 430: Perform synthesis processing according to the glottal parameter, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

[0080] When the glottal parameters corresponding to the target speech frame are determined, the corresponding glottal filter is determined accordingly. On this basis, Figure 2 The generation process of the original speech signal shown in the figure is to reconstruct the original speech signal by impacting the glottal filter determined by the excitation signal corresponding to the target speech frame, and to perform gain control on the signal obtained by filtering according to the gain corresponding to the target speech frame. The reconstructed signal is the enhanced speech signal corresponding to the target speech frame.

[0081] In the scheme of the present application, the target speech frame is first pre-enhanced to obtain a first complex spectrum, and then the target speech frame is speech decomposed and synthesized based on the first complex spectrum, so that the target speech frame is enhanced in two stages, which can effectively ensure the speech enhancement effect. Moreover, based on the first complex spectrum obtained by pre-enhancing the target speech frame, the target speech frame is speech decomposed based on the first complex spectrum, compared with the spectrum before the target speech frame is pre-enhanced, the first complex spectrum has less noise information, and in the speech decomposition process, the noise will affect the accuracy of speech decomposition. Therefore, using the first complex spectrum as the basis for speech decomposition can reduce the difficulty of speech decomposition, improve the accuracy of the glottal parameters, excitation signals and gains obtained by speech decomposition, and thus ensure the accuracy of the subsequent enhanced speech signal. The first complex spectrum obtained by pre-enhancement includes phase information and amplitude information, and speech decomposition and speech synthesis are performed based on the phase information and amplitude information in the first complex spectrum, which ensures the accuracy of the amplitude and phase of the enhanced speech signal corresponding to the obtained target speech frame.

[0082] In some embodiments of the present application, step 410 includes: inputting the complex spectrum corresponding to the target speech frame into a first neural network, the first neural network is trained based on the complex spectrum corresponding to the sample speech frame and the complex spectrum corresponding to the original speech signal in the sample speech frame; the first neural network outputs the first complex spectrum based on the complex spectrum corresponding to the target speech frame.

[0083] The first neural network may be a model constructed by a long short-term memory neural network, a convolutional neural network, a recurrent neural network, a fully connected neural network, a gated recurrent unit, etc., which is not specifically limited here.

[0084] In some embodiments of the present application, a plurality of sample speech frames can be obtained by framing a sample speech signal. The sample speech signal can be obtained by combining a known original speech signal with a known noise signal. Then, when the original speech signal is known, the original speech signal in the sample speech frame can be subjected to time-frequency transformation to obtain a complex spectrum corresponding to the original speech signal in the sample speech frame. The complex spectrum corresponding to the sample speech frame can be obtained by performing time-frequency transformation on the time domain signal of the sample speech frame.

[0085] During the training process, the complex spectrum corresponding to the sample speech frame is input into the first neural network, and the first neural network predicts based on the complex spectrum corresponding to the sample speech frame and outputs the predicted first complex spectrum; then the predicted first complex spectrum is compared with the complex spectrum corresponding to the original speech signal in the sample speech frame. If the similarity between the two does not meet the preset requirements, the parameters of the first neural network are adjusted until the similarity between the predicted first complex spectrum output by the first neural network and the complex spectrum corresponding to the original speech signal in the sample speech frame meets the preset requirements. Among them, the preset requirement can be that the similarity between the predicted first complex spectrum and the complex spectrum corresponding to the original speech signal in the sample speech frame is not less than the similarity threshold, and the similarity threshold can be set as needed, for example, 100%, 98%, etc. Through the above training process, the first neural network can learn the ability to predict the first complex spectrum based on the input complex spectrum.

[0086] In some embodiments of the present application, the first neural network includes a complex convolution layer, a gated recurrent unit layer and a fully connected layer; the above-mentioned step of outputting the first complex spectrum according to the complex spectrum of the target speech frame by the first neural network further includes: the complex convolution layer performs complex convolution processing according to the real part and the imaginary part in the complex spectrum corresponding to the target speech frame; the gated recurrent unit layer performs transformation processing on the output of the complex convolution layer; the fully connected layer performs fully connection processing on the output of the gated recurrent unit, and outputs the first complex spectrum.

[0087] In a specific embodiment, the first neural network may include one or more complex convolutional layers. Similarly, the gated recurrent unit layer and the fully connected layer may also be one or more layers. The specific number of complex convolutional layers, gated recurrent unit layers and fully connected layers can be set according to actual needs.

[0088] Figure 5 FIG. 1 is a schematic diagram of a complex convolution layer performing convolution processing on a complex number according to a specific embodiment. If the input complex number of the complex convolution layer is E+jF, the weight of the complex convolution layer is A+jB. Figure 5 As shown in the figure, the complex convolution layer includes a two-dimensional convolution layer (Real_conv, Imag_conv), a connection layer (Concat) and an activation layer (Leaky_Relu). After the real part E and the imaginary part F of the input complex number are input into the two-dimensional convolution layer, the two-dimensional convolution layer performs convolution according to the weights of the complex convolution layer. The process of convolution operation is shown in the following formula:

[0089] (E+jF)*(A+jB)=(E*AF*B)+j(E*B+F*A); (Formula 7)

[0090] Let C = E*AF*B; D = E*B+F*A; then the above formula 7 is further transformed into:

[0091] (E+jF)*(A+jB)=C+jD; (Formula 8)

[0092] like Figure 5 As shown in FIG. 1 , after the two-dimensional convolutional layer outputs the real part and the imaginary part after the convolution, the connection layer connects the real part and the imaginary part to obtain a connection result; and then the activation layer activates the connection result. Figure 5 In the activation layer, the activation function used is the Leaky_Relu activation function. The expression of the Leaky_Relu activation function is:

[0093] f(x)=max(ax,x); a is a constant; (Formula 9)

[0094] In other embodiments, the activation function used by the activation layer may also be other functions, such as a zRelu function, etc., which is not specifically limited here.

[0095] Figure 6 is a schematic diagram of the structure of a first neural network according to a specific embodiment. Figure 6As shown, the first neural network includes 6 layers of complex convolutional layers (Conv) cascaded in sequence, 1 layer of gated recurrent unit (GRU) layer and 2 layers of fully connected (FC) layers. After the complex spectrum S(n) corresponding to the target speech frame is input into the first neural network, the 6 layers of complex convolutional layers perform complex convolution processing in sequence, then the GRU layer performs transformation, and then the two FC layers perform full connection in sequence, and the last FC layer outputs the first complex spectrum. The numbers in brackets of each layer represent the dimensions of the variables output by the layer. Figure 6 In the first neural network shown, the dimension of the output of the last FC layer is 322 dimensions, which is used to represent the real and imaginary parts of the 161 STFT coefficients.

[0096] In some embodiments of the present application, step 420 includes: predicting glottal parameters of the target speech frame according to the first complex spectrum to obtain glottal parameters corresponding to the target speech frame; predicting an excitation signal of the target speech frame according to the first complex spectrum to obtain an excitation signal corresponding to the target speech frame; predicting a gain of the target speech frame according to a gain corresponding to a historical speech frame before the target speech frame to obtain a gain corresponding to the target speech frame.

[0097] In some embodiments of the present application, a neural network model for glottal parameter prediction (assuming the second neural network), a neural network model for gain prediction (assuming the third neural network), and a neural network model for excitation signal prediction (assuming the fourth neural network) can be trained separately. Among them, the three neural network models can be models constructed by long short-term memory neural network, convolutional neural network, recurrent neural network, fully connected neural network, etc., which are not specifically limited here.

[0098] In some embodiments of the present application, the step of predicting the glottal parameters of the target speech frame according to the first complex spectrum to obtain the glottal parameters corresponding to the target speech frame further includes: inputting the first complex spectrum into a second neural network, the second neural network is trained according to the complex spectrum corresponding to the sample speech frame and the glottal parameters corresponding to the sample speech frame; and the second neural network outputs the glottal parameters corresponding to the target speech frame according to the first complex spectrum.

[0099] The complex spectrum corresponding to the sample speech frame is obtained by performing a time-frequency transformation on the time domain signal of the sample speech frame. In some embodiments of the present application, the sample speech signal can be framed to obtain multiple sample speech frames. The sample speech signal can be obtained by combining a known original speech signal with a known noise signal. Then, when the original speech signal is known, the glottal parameters corresponding to the sample speech frame can be obtained by performing a linear prediction analysis on the original speech signal. In other words, the glottal parameters corresponding to the sample speech frame refer to the glottal parameters used to reconstruct the original speech signal in the sample speech frame.

[0100] During the training process, after the complex spectrum of the sample speech frame is input into the second neural network, the second neural network predicts the glottal parameters according to the complex spectrum of the sample speech frame and outputs the predicted glottal parameters; then the predicted glottal parameters are compared with the glottal parameters corresponding to the sample speech frame. If the two are inconsistent, the parameters of the second neural network are adjusted until the predicted glottal parameters output by the second neural network according to the complex spectrum of the sample speech frame are consistent with the glottal parameters corresponding to the sample speech frame. After the training is completed, the second neural network learns the ability to accurately predict the glottal parameters used to reconstruct the original speech signal in the speech frame according to the complex spectrum of the input speech frame.

[0101] Figure 7 is a schematic diagram of a second neural network according to a specific embodiment, such as Figure 7 As shown, the second neural network includes one LSTM (Long-Short Term Memory) layer and three cascaded FC (Full Connected) layers. Among them, the LSTM layer is 1 hidden layer, which includes 256 units, and the input of the LSTM layer is the first complex spectrum S'(n) corresponding to the nth speech frame. In this embodiment, the input of the LSTM layer is 321 dimensions. In the three-layer cascaded FC layer, the first two FC layers are provided with an activation function σ(), and the set activation function is used to increase the nonlinear expression ability of the second neural network. The last FC layer is not provided with an activation function, and the last FC layer is used as a classifier for classification output. As shown Figure 7 As shown, along the direction from input to output, the three FC layers include 512, 512, and 16 units respectively, and the output of the last FC layer is the 16-dimensional line spectrum frequency coefficient LSF(n) corresponding to the n-th speech frame, that is, the 16-order line spectrum frequency parameter.

[0102] In some embodiments of the present application, since there is correlation between speech frames, the frequency domain feature similarity between two adjacent speech frames is relatively high. Therefore, the glottal parameters corresponding to the target speech frame can be predicted in combination with the glottal parameters corresponding to the historical speech frame before the target speech frame. In one embodiment, the above-mentioned step of predicting the glottal parameters of the target speech frame according to the first complex spectrum to obtain the glottal parameters corresponding to the target speech frame further includes: inputting the first complex spectrum and the glottal parameters corresponding to the historical speech frame before the target speech frame into a second neural network, the second neural network is trained based on the complex spectrum corresponding to the sample speech frame, the glottal parameters corresponding to the historical speech frame before the sample speech frame, and the glottal parameters corresponding to the sample speech frame; the first neural network outputs the glottal parameters corresponding to the target speech frame based on the first complex spectrum and the glottal parameters corresponding to the historical speech frame before the target speech frame.

[0103] Since there is a correlation between the historical speech frames and the target speech frames, the glottal parameters corresponding to the historical speech frames of the target speech frames are similar to the glottal parameters corresponding to the target speech frames. Therefore, the glottal parameters corresponding to the historical speech frames of the target speech frames are used as a reference to supervise the prediction process of the glottal parameters of the target speech frames, which can improve the accuracy of glottal parameter prediction.

[0104] In some embodiments of the present application, since the glottal parameters of speech frames that are closer in time have higher similarity, the glottal parameters corresponding to the historical speech frames that are closer to the target speech frame can be used as a reference, which can further ensure the prediction accuracy. For example, the glottal parameters corresponding to the previous speech frame of the target speech frame can be used as a reference. In a specific embodiment, the number of historical speech frames used as a reference can be one frame or multiple frames, which can be selected according to actual needs.

[0105] The glottal parameters corresponding to the historical speech frame of the target speech frame can be the glottal parameters obtained by performing glottal parameter prediction on the historical speech frame. In other words, in the process of glottal parameter prediction, the glottal parameters predicted for the historical speech frame are reused as a reference for the glottal parameter prediction process of the current speech frame.

[0106] The training process of the second neural network in this embodiment is similar to the training process of the second neural network in the previous embodiment, and the training process will not be repeated here.

[0107] Figure 8 is a schematic diagram of input and output of a second neural network according to another embodiment, wherein: Figure 8 The structure of the second neural network in Figure 7 The same as Figure 7 , Figure 8The input of the third neural network also includes the line spectrum frequency parameter LSF(n-1) of the previous speech frame (i.e., the n-1th frame) of the nth speech frame. Figure 8 As shown, the line spectrum frequency parameter LSF(n-1) of the previous speech frame of the nth speech frame is embedded in the second FC layer as reference information. Since the LSF parameters of two adjacent speech frames are very similar, if the LSF parameters corresponding to the historical speech frame of the nth speech frame are used as reference information, the LSF parameter prediction accuracy can be improved.

[0108] In some embodiments of the present application, the step of predicting the gain of the target speech frame based on the gains corresponding to the historical speech frames before the target speech frame to obtain the gain corresponding to the target speech frame may further include: inputting the gains corresponding to the historical speech frames before the target speech frame into a third neural network, the third neural network being trained based on the gains corresponding to the historical speech frames before the sample speech frame and the gains corresponding to the sample speech frame; and the third neural network outputting the gain corresponding to the target speech frame based on the gains corresponding to the historical speech frames before the target speech frame.

[0109] The gain corresponding to the historical speech frame of the target speech frame can be obtained by the third neural network performing gain prediction for the historical speech frame. In other words, the gain predicted for the historical speech frame is reused as the input of the third neural network model in the process of gain prediction for the target speech frame.

[0110] The sample speech frame can be obtained by dividing the sample speech signal into frames, and the sample speech signal can be obtained by combining a known original speech signal and a known noise signal. Then, when the original speech signal in the sample speech is known, a linear prediction analysis can be performed on the original speech signal to obtain the glottal parameters used to reconstruct the original speech signal, that is, the glottal parameters corresponding to the sample speech frame.

[0111] Fig. 9 is a schematic diagram of a third neural network according to a specific embodiment, such as Fig. 9 As shown, the third neural network includes an LSTM layer and an FC layer, wherein the LSTM layer is a hidden layer including 128 units; the input dimension of the FC layer is 512, and the output is a 1-dimensional gain. In a specific embodiment, the gain G_pre(n) corresponding to the historical speech frame of the n-th speech frame can be defined as the gain corresponding to the first 4 speech frames of the n-th speech frame, that is:

[0112] G_pre(n)={G(n-1),G(n-2),G(n-3),G(n-4)};

[0113] Of course, the number of historical speech frames selected for gain prediction is not limited to the above example, and can be selected according to actual needs.

[0114] As shown above, the second neural network and the third neural network present an M-to-N mapping relationship as a whole ( N<<M ), that is, the dimension of the input information of the neural network model is M, and the dimension of the output information is N, which greatly simplifies the structure of the neural network model and reduces the complexity of the model.

[0115] In some embodiments of the present application, the step of predicting the excitation signal for the target speech frame based on the first complex spectrum to obtain the excitation signal corresponding to the target speech frame may further include: inputting the first complex spectrum into a fourth neural network, the fourth neural network being trained based on the complex spectrum corresponding to the sample speech frame and the frequency domain representation of the excitation signal corresponding to the sample speech frame; and the fourth neural network outputting the frequency domain representation of the excitation signal corresponding to the target speech frame based on the first complex spectrum.

[0116] The excitation signal corresponding to the sample speech frame can be obtained by performing linear prediction analysis on the known original speech signal in the sample speech frame. The frequency domain representation can be an amplitude spectrum or a complex spectrum, which is not specifically limited here.

[0117] In the process of training the fourth neural network, the complex spectrum of the sample speech frame is input into the fourth neural network model, and then the fourth neural network predicts the excitation signal according to the complex spectrum of the input sample speech frame, and outputs the frequency domain representation of the predicted excitation signal; then the parameters of the fourth neural network are adjusted according to the frequency domain representation of the predicted excitation signal and the frequency domain representation of the excitation signal corresponding to the sample speech frame, that is: if the similarity between the frequency domain representation of the predicted excitation signal and the frequency domain representation of the excitation signal corresponding to the sample speech frame does not meet the preset requirements, the parameters of the fourth neural network are adjusted until the similarity between the frequency domain representation of the predicted excitation signal output by the fourth neural network for the sample speech frame and the frequency domain representation of the excitation signal corresponding to the sample speech frame meets the preset requirements. Through the above training process, the fourth neural network can learn the ability to predict the frequency domain representation of the excitation signal corresponding to a speech frame based on the amplitude spectrum of the speech frame, so as to accurately predict the excitation signal.

[0118] Fig.10 is a schematic diagram of a fourth neural network according to a specific embodiment, such as Fig.10As shown, the fourth neural network includes one LSTM layer and three FC layers, wherein the LSTM layer is one hidden layer, including 256 units, and the input of the LSTM is the first complex spectrum S'(n) corresponding to the n-th frame of speech, and its dimension can be 321 dimensions. The number of units included in the three FC layers is 512, 512 and 321 respectively, and the last FC layer outputs the frequency domain representation R(n) of the excitation signal corresponding to the n-th frame of speech of 321 dimensions. In the direction from input to output, the first two FC layers of the three FC layers are provided with activation functions to enhance the nonlinear expression ability of the model, and the last FC layer has no activation function and is used for classification output.

[0119] It is worth mentioning that the structures of the first neural network, the second neural network, the third neural network, and the fourth neural network shown above are merely illustrative examples. In other embodiments, a neural network model with a corresponding structure can also be set in an open source platform for deep learning and trained accordingly.

[0120] In some embodiments of the present application, Fig.11 As shown, step 430 includes:

[0121] Step 1110, filtering the excitation signal corresponding to the target speech frame through a glottal filter to obtain a filtered output signal; the glottal filter is constructed according to the glottal parameters corresponding to the target speech frame.

[0122] Step 1120: amplify the filtered output signal according to the gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0123] If the glottal parameters are LPC coefficients, the glottal filter can be constructed directly according to the above formula (2). If the glottal filter is a p-order filter, the glottal parameters corresponding to the target speech frame include the p-order LPC coefficients, that is, a1, a2, ..., a in the above formula (2). p In other embodiments, the constant 1 in the above formula (2) can also be used as an LPC coefficient.

[0124] If the glottal parameters are LSF parameters, the LSF parameters can be converted into LPC coefficients, and then the glottal filter is constructed according to the above formula (2).

[0125] The filtering process is the convolution in the time domain, so the process of filtering the excitation signal through the glottal filter can be converted to the time domain. Then, on the basis of predicting the frequency domain representation of the excitation signal corresponding to the target speech frame, the frequency domain representation of the excitation signal is transformed into the time domain to obtain the time domain signal of the excitation signal corresponding to the target speech frame.

[0126] In the solution of the present application, the target speech frame includes a plurality of sample points. The excitation signal is filtered by a glottal filter, that is, a historical sample point before a sample point is convolved with the glottal filter to obtain a target signal value corresponding to the sample point.

[0127] In some embodiments of the present application, the target speech frame includes multiple sample points; the glottal filter is a p-order filter, p is a positive integer; the excitation signal includes the excitation signal values ​​corresponding to the multiple sample points in the target speech frame; according to the filtering process as above, step 1120 further includes: convolving the excitation signal values ​​corresponding to the first p sample points of each sample point in the target speech frame with the p-order filter to obtain the target signal value of each sample point in the target speech frame; combining the target signal values ​​corresponding to all sample points in the target speech frame in time order to obtain the first speech signal. Among them, the expression of the p-order filter can refer to the above formula (1). That is, for each sample point in the target speech frame, the excitation signal values ​​corresponding to the first p sample points are used to convolve with the p-order filter to obtain the target signal value corresponding to each sample point.

[0128] It can be understood that for the first sample point in the target speech frame, it is necessary to use the excitation signal values ​​of the last p sample points in the previous speech frame of the target speech frame to calculate the target signal value corresponding to the first sample point. Similarly, for the second sample point in the target speech frame, it is necessary to use the excitation signal values ​​of the last (p-1) sample points in the previous speech frame of the target speech frame and the excitation signal value of the first sample point in the target speech frame to convolve with a p-order filter to obtain the target signal value corresponding to the second sample point in the target speech frame.

[0129] In summary, step 1120 also requires the participation of the excitation signal value corresponding to the historical speech frame of the target speech frame. The number of sample points in the required historical speech frame is related to the order of the glottal filter, that is, if the glottal filter is of order p, the participation of the excitation signal value corresponding to the last p sample points in the previous speech frame of the target speech frame is required.

[0130] In the related art, there are methods of speech enhancement through spectral estimation and spectral regression prediction. The spectral estimation speech enhancement method assumes that a mixed speech contains a speech part and a noise part. Therefore, the noise can be estimated through statistical models, etc., and the spectrum corresponding to the mixed speech is subtracted from the spectrum corresponding to the noise. The remaining is the speech spectrum. In this way, the clean speech signal can be restored by subtracting the spectrum corresponding to the noise from the spectrum corresponding to the mixed speech. The spectral regression prediction speech enhancement method predicts the masking threshold corresponding to the speech frame through a neural network. The masking threshold reflects the proportion of speech components and noise components in each frequency point in the speech frame; then the mixed signal spectrum is gain controlled according to the masking threshold to obtain an enhanced spectrum.

[0131] The above speech enhancement method through spectrum estimation and spectrum regression prediction is based on the estimation of the posterior probability of the noise spectrum, which may have inaccurate estimated noise. For example, transient noise such as keyboard typing occurs instantaneously, and the estimated noise spectrum is very inaccurate, resulting in poor noise suppression effect. In the case of inaccurate noise spectrum prediction, if the original mixed speech signal is processed according to the estimated noise spectrum, it may cause speech distortion in the mixed speech signal, or lead to poor noise suppression effect; therefore, in this case, it is necessary to compromise between speech fidelity and noise suppression.

[0132] In the above embodiment of realizing speech enhancement based on glottal parameters, excitation signal and gain prediction, since the glottal parameters are strongly correlated with the glottal features in the physical process of sound generation, the predicted glottal parameters effectively guarantee the speech structure of the original speech signal in the target speech frame. Therefore, synthesizing the glottal parameters, excitation signal and gain obtained based on speech decomposition to obtain the enhanced speech signal of the target speech frame can effectively avoid the reduction of the original speech and effectively protect the speech structure. At the same time, after obtaining the glottal parameters, excitation signal and gain corresponding to the target speech frame, since the original noisy speech will not be processed again, there is no need to compromise between speech fidelity and noise suppression.

[0133] Fig.12 is a flow chart of a speech enhancement method according to another specific embodiment. Fig.12 In the embodiment shown, the second neural network, the third neural network and the fourth neural network are combined to perform speech decomposition. Assume that the nth speech frame is taken as the target speech frame, and the time domain signal of the nth speech frame is s(n). Fig.12 As shown, the speech enhancement method includes steps 1210-1270.

[0134] Step 1210, time-frequency conversion: convert the time domain signal s(n) of the n-th speech frame into a complex spectrum S(n) corresponding to the n-th speech frame.

[0135] Step 1220, pre-enhancement: pre-enhance the nth speech frame based on the complex spectrum S(n) to obtain a first complex spectrum S'(n).

[0136] Step 1230, predicting the glottal parameters through the second neural network. In this step, the input of the second neural network may be only the first complex spectrum S'(n), or may include the first complex spectrum S'(n) and the glottal parameters P_pre(n) corresponding to the historical speech frames of the n-th speech frame, and the second neural network outputs the glottal parameters ar(n) corresponding to the n-th speech frame, and the glottal parameters may be LPC coefficients or LSF parameters.

[0137] Step 1240, predict the excitation signal through the third neural network. The input of the third neural network is the first complex spectrum S'(n), and the output is the frequency domain representation R(n) of the excitation signal corresponding to the n-th speech frame. Then, through step 1250, frequency-time transformation is performed on R(n), and the time domain signal r(n) of the excitation signal corresponding to the n-th speech frame can be obtained.

[0138] Step 1260: predict the gain through a fourth neural network. The fourth neural network inputs the gain G_pre(n) corresponding to the historical speech frame of the n-th speech frame, and outputs the gain G(n) corresponding to the n-th speech frame.

[0139] After obtaining the glottal parameter ar(n), the corresponding excitation signal r(n), and the corresponding gain G_(n) corresponding to the n-th speech frame, synthesis filtering is performed according to step 1270 based on the three parameters to obtain the time domain signal s_e(n) of the enhanced speech signal corresponding to the n-th speech frame. The synthesis filtering process of step 1270 can refer to Fig.11 The process shown is carried out.

[0140] In other embodiments of the present application, Fig.13 As shown, step 420 includes:

[0141] Step 1310: Calculate a power spectrum based on the first complex spectrum.

[0142] If the first complex spectrum is S′(n), the power spectrum Pa(n) obtained in step 1310 is:

[0143] Pa(n)=Real(S′(n)) 2 +Imag(S′(n)) 2 ; (Formula 10)

[0144] Wherein, Real(S′(n)) represents the real part of the first complex spectrum S′(n), and Imag(S′(n)) represents the imaginary part of the first complex spectrum S′(n). The power spectrum calculated in step 1310 is the power spectrum of the signal after pre-enhancement of the target speech frame.

[0145] Step 1320: Calculate the autocorrelation coefficient based on the power spectrum.

[0146] According to the Wiener-Hinchin theorem: the power spectrum of a stationary random process and its autocorrelation function are a pair of Fourier transform relationships. In this solution, a speech frame is regarded as a stationary random signal. Therefore, on the basis of obtaining the pre-enhanced power spectrum corresponding to the target speech frame, the pre-enhanced power spectrum corresponding to the target speech frame can be subjected to inverse Fourier transform to obtain the autocorrelation coefficient corresponding to the pre-enhanced power spectrum.

[0147] Specifically, step 1320 includes: performing an inverse Fourier transform on the power spectrum to obtain an inverse transform result; extracting the real part of the inverse transform result to obtain the autocorrelation coefficient. That is:

[0148] AC(n)=Real(iFFT(Pa(n))); (Formula 11)

[0149] AC(n) represents the autocorrelation coefficient corresponding to the nth speech frame, iFFT (Inverse Fast Fourier Transform) refers to the inverse transform of FFT (Fast Fourier Transform), and Real represents the real part of the result obtained by the inverse fast Fourier transform. AC(n) includes p parameters, where p is the order of the glottal filter. The coefficients in AC(n) can be further expressed as AC j (n), 1≤j≤p.

[0150] Step 1330: Calculate the glottal parameter according to the autocorrelation coefficient.

[0151] According to the Yule-Walker equation, for the nth speech frame, there is the following relationship between the corresponding autocorrelation coefficient and the corresponding glottal parameter:

[0152] k-KA=0; (Formula 12)

[0153] Among them, k is the autocorrelation vector; K is the autocorrelation matrix; A is the LPC coefficient matrix. Specifically,

[0154]

[0155]

[0156] Among them, AC j (n) = E[s(n)s(nj)], 0≤j≤p; (Formula 13)

[0157] p is the order of the glottal filter; a1(n), a2(n), ..., a p (n) are the LPC coefficients corresponding to the nth speech frame, which are a1, a2, ..., a1 in the above formula 2. p ; Since a0(n) is a constant 1, a0(n) can also be regarded as an LPC coefficient corresponding to the nth speech frame.

[0158] On the basis of obtaining the autocorrelation coefficient, the autocorrelation vector and the autocorrelation matrix can be determined correspondingly, and then the LPC coefficient can be obtained by solving formula 12. In a specific embodiment, the Levinson-Durbin algorithm can be used to solve formula 12. The Levinson-Durbin algorithm uses the symmetry of the autocorrelation matrix and uses an iterative method to calculate the autocorrelation coefficient.

[0159] Since LSF parameters and LPC coefficients can be converted to each other, when the LPC coefficients are calculated, the LSF parameters can be determined accordingly. In other words, whether the glottal parameters are LPC coefficients or LSF parameters, they can be determined through the above process.

[0160] Step 1340: Calculate the gain according to the glottal parameter and the autocorrelation parameter set.

[0161] The gain corresponding to the nth speech frame may be calculated according to the following formula.

[0162]

[0163] It is worth mentioning that G(n) calculated according to Formula 14 is the square of the gain corresponding to the target speech frame in the time domain representation.

[0164] Step 1350: Calculate the power spectrum of the excitation signal based on the gain and the power spectrum of the glottal filter, where the glottal filter is a filter constructed based on the glottal parameters.

[0165] Assume that the complex spectrum corresponding to the target speech frame is obtained by Fourier transform of m (m is a positive integer) sample points. In order to calculate the power spectrum of the glottal filter, first construct an m-dimensional all-zero array, s_AR(n), for the nth speech frame; then, convert the (p+1)-dimensional a j(n) assign values ​​to the first (p+1) dimensions of the all-zero array, where j = 0, 1, 2, ... p; obtain the FFT coefficients by calling the Fast Fourier Transform (FFT) of m sample points:

[0166] S_AR(n)=FFT(s_AR(n)); (Formula 15)

[0167] On the basis of obtaining the FFT coefficient S_AR(n), the power spectrum of the glottal filter corresponding to the n-th speech frame can be obtained sample by sample according to the following formula 16:

[0168] AR_LPS(n,k)=(Real(S_AR(n,k))) 2 +(Imag(S_AR(n,k))) 2 ; (Formula 16) Wherein, Real(S_AR(n, k)) represents the real part of S_AR(n, k), Imag(S_AR(n, k)) represents the imaginary part of S_AR(n, k), k represents the order of FFT coefficients, 0≤k≤m, and k is a positive integer.

[0169] After obtaining the frequency response AR_LPS(n) of the glottal filter corresponding to the nth speech frame, for the convenience of calculation, the power spectrum AR_LPS(n) of the glottal filter is converted from the natural number domain to the logarithmic domain according to formula 17:

[0170] AR_LPS1(n)=log 10 (AR_LPS(n)); (Formula 17)

[0171] The above AR_LPS1(n) is inverted according to the following formula 18 to obtain the power spectrum AR_LPS2(n) corresponding to the inverse of the glottal filter:

[0172] AR_LPS2(n)=-1*AR_LPS1(n); (Formula 18)

[0173] Then, the power spectrum R(n) of the excitation signal corresponding to the target speech frame can be calculated according to the following formula 19:

[0174] R(n)=Pa(n)*(G1(n)) 2 *AR_LPS3(n); (Formula 19)

[0175] in,

[0176]

[0177] Through the above process, the glottal parameters, gain and frequency response of the excitation signal corresponding to the target speech frame, and the frequency response of the glottal filter defined by the glottal parameters are calculated accordingly.

[0178] After obtaining the gain corresponding to the target speech frame, the power spectrum of the corresponding excitation signal and the power spectrum of the glottal filter defined by the glottal parameters, the following can be used: Fig.14 The synthesis process is performed according to the process shown in the figure. Fig.14 As shown, step 430 includes:

[0179] Step 1410: Generate a first amplitude spectrum according to the power spectrum of the glottal filter and the power spectrum of the excitation signal.

[0180] The first amplitude spectrum S_filt(n) can be calculated according to the following formula 22:

[0181]

[0182] Where R1(n)=10*log 10 (R(n)); (Formula 23)

[0183] Step 1420: amplify the first amplitude spectrum according to the gain to obtain a second amplitude spectrum.

[0184] The second amplitude spectrum S_e(n) can be obtained according to the following formula:

[0185] S_e(n)=G2(n)*S_filt(n); (Formula 24)

[0186] in,

[0187] Step 1430: Determine an enhanced speech signal corresponding to the target speech frame according to the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

[0188] In some embodiments of the present application, step 1430 further includes: combining the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum; in other words, taking the second amplitude spectrum as the real part of the second complex spectrum, and taking the phase spectrum extracted from the first complex spectrum as the imaginary part of the second complex spectrum; transforming the second complex spectrum to the time domain to obtain a time domain signal of the enhanced speech signal corresponding to the target speech frame.

[0189] Fig.15 is a flow chart of a speech enhancement method according to a specific embodiment, wherein the nth speech frame is taken as the target speech frame, and the time domain signal of the nth speech frame is s(n). Fig.15As shown, it specifically includes steps 1510-1550.

[0190] Step 1510, time-frequency conversion: through step 510, the time domain signal s(n) of the n-th speech frame is converted to obtain the complex spectrum S(n) corresponding to the n-th speech frame.

[0191] Step 1520, pre-enhancement: pre-enhance the nth speech frame based on the complex spectrum S(n) corresponding to the nth speech frame to obtain a first complex spectrum S'(n) of the nth speech frame.

[0192] Step 1530, spectrum decomposition: by performing spectrum decomposition on the first complex spectrum S'(n), a power spectrum Pa(n) and a corresponding phase spectrum Ph(n) corresponding to the first complex spectrum S'(n) are obtained.

[0193] Step 1540, speech decomposition: perform speech decomposition based on the power spectrum Pa(n) of the nth speech frame, and determine the glottal parameter set P(n) corresponding to the nth speech frame and the frequency domain representation R(n) of the excitation signal corresponding to the nth speech frame. The glottal parameter set P(n) includes the glottal parameter ar(n) and the gain G(n). The specific process of speech decomposition can be as follows: Fig.13 The process shown obtains the glottal parameters, and correspondingly obtains the power spectrum AR_LPS(n) of the glottal filter, the power spectrum R(n) of the excitation signal, and the gain G(n).

[0194] Step 1550: speech synthesis. The specific speech synthesis process can be as follows: Fig.14 As shown, the second amplitude spectrum S_e(n) is obtained by synthesizing the frequency response AR_LPS(n) of the glottal filter corresponding to the nth speech frame, the frequency response R(n) of the excitation signal and the gain G(n).

[0195] Step 1560, frequency-time conversion. Multiplex the phase spectrum Ph(n) extracted from the first complex spectrum S′(n), combine the phase spectrum Ph(n) and the second amplitude spectrum S_e(n) to obtain the enhanced complex spectrum corresponding to the n-th speech frame. Transform the obtained enhanced complex spectrum into the time domain, that is, obtain the time domain signal se(n) of the enhanced speech signal corresponding to the n-th speech frame.

[0196] In the scheme of this embodiment, speech decomposition is performed based on the first complex spectrum obtained by pre-enhancing the target speech frame. During the pre-enhancement process, part of the noise information is filtered out, so the noise information in the first complex spectrum is less; then speech decomposition based on the first complex spectrum can reduce the influence of noise on speech decomposition, reduce the difficulty of speech decomposition, and improve the accuracy of the glottal parameters, excitation signals and gains obtained by speech decomposition, thereby ensuring the accuracy of the subsequent enhanced speech signal. In addition, in this scheme, during the speech synthesis process, only the amplitude spectrum can be paid attention to, without paying attention to the phase information, and the phase spectrum extracted from the first complex spectrum can be directly reused, thereby reducing the amount of calculation in the speech synthesis process. The first complex spectrum is obtained by pre-enhancement, and its noise content is less, so the accuracy of the phase information is guaranteed to a certain extent.

[0197] exist Fig.15 In the embodiment shown, step 1510 can be implemented and enhanced by the first neural network, and step 1540 can be implemented according to Fig.13 The process shown in FIG. 15 can be implemented as follows: Fig.14 The process shown in the figure realizes the deep combination of traditional signal processing and deep learning, performs secondary enhancement on the target speech frame, and realizes multi-stage enhancement on the target speech frame, that is, the first stage uses deep learning to perform pre-enhancement according to the amplitude spectrum of the target speech frame, which can reduce the difficulty of obtaining glottal parameters, excitation signals and gains by speech decomposition in the second stage; in the second stage, the glottal parameters, excitation signals and gains for reconstructing the original speech signal are obtained by signal processing. Moreover, in the second stage, speech synthesis is performed according to the digital model of speech generation, and the signal of the target speech frame is not directly processed, so the situation of speech clipping in the second stage can be avoided.

[0198] In some embodiments of the present application, before step 410, the method further includes: acquiring a time domain signal of the target speech frame; performing a time-frequency transform on the time domain signal of the target speech frame to obtain a complex spectrum of the target speech frame.

[0199] The time-frequency transform may be a short-term Fourier transform (STFT), in which a windowing and overlapping operation is used to eliminate unevenness between frames. Fig.16 is a schematic diagram of windowing overlap in a short-time Fourier transform according to a specific example. Fig.16In the example, a 50% windowing overlap operation is used. If the short-time Fourier transform is for 640 sample points, the number of overlapping samples (hop-size) of the window function is 320. The window function used for windowing can be a Hanning window, a Hamming window, etc. Of course, other window functions can also be used, which are not specifically limited here.

[0200] In other embodiments, non-50% windowing and overlapping operations may also be used. For example, if the short-time Fourier transform is for 512 sample points, in this case, if a speech frame includes 320 sample points, only 192 sample points of the previous speech frame need to be overlapped.

[0201] In some embodiments of the present application, the step of obtaining the time domain signal of the target speech frame further includes: obtaining a speech signal to be processed, wherein the speech signal to be processed is a collected speech signal or a speech signal obtained by decoding the encoded speech; and framing the speech signal to be processed to obtain the time domain signal of the target speech frame.

[0202] In some examples, the speech signal to be processed can be framed according to a set frame length, which can be set according to actual needs, for example, the frame length is set to 20ms. By framing, multiple speech frames can be obtained, and each speech frame can be used as a target speech frame in this application.

[0203] As described above, the scheme of the present application can be applied to the transmitting end for speech enhancement, and can also be applied to the receiving end for speech enhancement. In the case where the scheme of the present application is applied to the transmitting end, the voice signal to be processed is the voice signal collected by the transmitting end, and the voice signal to be processed is framed to obtain multiple voice frames. After framing, the voice signal to be processed is divided into multiple voice frames, and then each voice frame can be used as a target voice frame and the target voice frame can be enhanced according to the process of the above steps 410-430. Furthermore, after obtaining the enhanced voice signal corresponding to the target voice frame, the enhanced voice signal can also be encoded for transmission based on the obtained encoded voice.

[0204] In one embodiment, since the directly collected voice signal is an analog signal, in order to facilitate signal processing, before framing, the signal needs to be further digitized to convert the time-continuous voice signal into a time-discrete voice signal. During the digitization process, the collected voice signal can be sampled according to a set sampling rate, and the set sampling rate can be 16000Hz, 8000Hz, 32000Hz, 48000Hz, etc., which can be set according to actual needs.

[0205] In the case where the solution of the present application is applied to the receiving end, the voice signal to be processed is a voice signal obtained by decoding the received encoded voice. In this case, it may be because the transmitting end does not enhance the voice signal to be transmitted. Therefore, in order to improve the signal quality, the voice needs to be enhanced at the receiving end. After obtaining multiple voice frames by framing the voice signal to be processed, it is used as a target voice frame and the target voice frame is enhanced according to the process of steps 410-430 above to obtain an enhanced voice signal of the target voice frame. Furthermore, the enhanced voice signal corresponding to the target voice frame can also be played. Since the obtained enhanced voice signal is compared with the signal before the target voice frame is enhanced, the noise is removed and the quality of the voice signal is higher. Therefore, for the user, the auditory experience is better.

[0206] The following describes an apparatus embodiment of the present application, which can be used to execute the method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned method embodiment of the present application.

[0207] Fig.17 is a block diagram of a speech enhancement device according to an embodiment. Fig.17 As shown, the speech enhancement device includes: a pre-enhancement module 1710, used to perform pre-enhancement processing on the target speech frame according to the complex spectrum of the target speech frame to obtain a first complex spectrum; a speech decomposition module 1720, used to perform speech decomposition on the target speech frame according to the first complex spectrum to obtain the glottal parameters, gain and excitation signal corresponding to the target speech frame; a synthesis processing module 1730, used to perform synthesis processing according to the glottal parameters, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

[0208] In some embodiments of the present application, the pre-enhancement module 1710 includes: a first input unit, used to input the complex spectrum corresponding to the target speech frame into a first neural network, the first neural network is trained based on the complex spectrum corresponding to the sample speech frame and the complex spectrum corresponding to the original speech signal in the sample speech frame; a first output unit, used by the first neural network to output the first complex spectrum based on the complex spectrum corresponding to the target speech frame.

[0209] In some embodiments of the present application, the first neural network includes a complex convolution layer, a gated recurrent unit layer and a fully connected layer; the first output unit includes: a complex convolution unit, which is used for the complex convolution layer to perform complex convolution processing according to the real part and the imaginary part in the complex spectrum corresponding to the target speech frame; a transformation unit, which is used for the gated recurrent unit layer to perform transformation processing on the output of the complex convolution layer; and a fully connected unit, which is used for the fully connected layer to perform fully connected processing on the output of the gated recurrent unit and output the first complex spectrum.

[0210] In some embodiments of the present application, the speech decomposition module 1720 includes: a first amplitude spectrum acquisition unit, a glottal parameter prediction unit, which is used to predict the glottal parameters of the target speech frame according to the first complex spectrum to obtain the glottal parameters corresponding to the target speech frame; an excitation signal prediction unit, which is used to predict the excitation signal of the target speech frame according to the first complex spectrum to obtain the excitation signal corresponding to the target speech frame; a gain prediction unit, which is used to predict the gain of the target speech frame according to the gain corresponding to the historical speech frame before the target speech frame to obtain the gain corresponding to the target speech frame.

[0211] In some embodiments of the present application, the glottal parameter prediction unit includes: a second input unit, used to input the first complex spectrum into a second neural network, the second neural network is trained according to the complex spectrum corresponding to the sample speech frame and the glottal parameters corresponding to the sample speech frame; a second output unit, used for the second neural network to output the glottal parameters corresponding to the target speech frame according to the first complex spectrum.

[0212] In some other embodiments of the present application, the glottal parameter prediction unit includes: a third input unit, used to input the glottal parameters corresponding to the first complex spectrum and the historical speech frame before the target speech frame into a second neural network, the second neural network is trained according to the complex spectrum corresponding to the sample speech frame, the glottal parameters corresponding to the historical speech frame before the sample speech frame, and the glottal parameters corresponding to the sample speech frame; a third output unit, used to output the glottal parameters corresponding to the target speech frame according to the first complex spectrum and the glottal parameters corresponding to the historical speech frame before the target speech frame by the first neural network.

[0213] In some embodiments of the present application, the gain prediction unit includes: a fourth input unit, used to input the gain corresponding to the historical speech frame before the target speech frame into a third neural network, and the third neural network is trained based on the gain corresponding to the historical speech frame before the sample speech frame and the gain corresponding to the sample speech frame; a fourth output unit, used to output the gain corresponding to the target speech frame by the third neural network based on the gain corresponding to the historical speech frame before the target speech frame.

[0214] In some embodiments of the present application, the excitation signal prediction unit includes: a fifth input unit, used to input the first complex spectrum into a fourth neural network, and the fourth neural network is trained based on the complex spectrum corresponding to the sample speech frame and the frequency domain representation of the excitation signal corresponding to the sample speech frame; a fifth output unit, used by the fourth neural network to output the frequency domain representation of the excitation signal corresponding to the target speech frame based on the first complex spectrum.

[0215] In some embodiments of the present application, the synthesis processing module 1730 includes: a filtering unit, which is used to filter the excitation signal corresponding to the target speech frame through a glottal filter to obtain a filtered output signal; the glottal filter is constructed according to the glottal parameters corresponding to the target speech frame; an amplification processing unit, which is used to amplify the filtered output signal according to the gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0216] In some embodiments of the present application, the speech decomposition module 1720 includes: a power spectrum calculation unit, used to calculate the power spectrum according to the first complex spectrum; an autocorrelation coefficient calculation unit, used to calculate the autocorrelation coefficient according to the power spectrum; a glottal parameter calculation unit, used to calculate the glottal parameters according to the autocorrelation coefficient; a gain calculation unit, used to calculate the gain according to the glottal parameters and the autocorrelation parameter set; an excitation signal determination unit, used to calculate the power spectrum of the excitation signal according to the gain and the power spectrum of the glottal filter, wherein the glottal filter is a filter constructed according to the glottal parameters.

[0217] In some embodiments of the present application, the synthesis processing module 1730 includes: a second amplitude spectrum generating unit, used to generate a first amplitude spectrum according to the power spectrum of the glottal filter and the power spectrum of the excitation signal; a third amplitude spectrum determining unit, used to amplify the first amplitude spectrum according to the gain to obtain a second amplitude spectrum; an enhanced speech signal determining unit, used to determine the enhanced speech signal corresponding to the target speech frame according to the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

[0218] In some embodiments of the present application, the enhanced speech signal determination unit includes: a second complex spectrum calculation unit, used to combine the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum; a time domain transformation unit, used to transform the second complex spectrum to the time domain to obtain a time domain signal of the enhanced speech signal corresponding to the target speech frame.

[0219] Fig.18 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0220] It should be noted that Fig.18 The computer system 1800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0221] like Fig.18 As shown, the computer system 1800 includes a central processing unit (CPU) 1801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1802 or the program loaded from the storage part 1808 to the random access memory (RAM) 1803, such as executing the method in the above embodiment. In RAM 1803, various programs and data required for system operation are also stored. CPU 1801, ROM 1802 and RAM 1803 are connected to each other through bus 1804. Input / output (I / O) interface 1805 is also connected to bus 1804.

[0222] The following components are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed so that a computer program read therefrom is installed into the storage section 1808 as needed.

[0223] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1809, and / or installed from a removable medium 1811. When the computer program is executed by a central processing unit (CPU) 1801, various functions defined in the system of the present application are executed.

[0224] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0225] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0226] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0227] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0228] According to one aspect of the present application, an electronic device is also provided, which includes: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method in any of the above embodiments is implemented.

[0229] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in any of the above embodiments.

[0230] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0231] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0232] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0233] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A speech enhancement method, characterized in that: include: Performing pre-enhancement processing on the target speech frame according to the complex spectrum corresponding to the target speech frame to obtain a first complex spectrum, wherein the first complex spectrum includes phase information and amplitude information; Performing speech decomposition on the target speech frame according to the first complex spectrum to obtain glottal parameters, gain and excitation signal corresponding to the target speech frame; A synthesis process is performed according to the glottal parameter, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

2. The method according to claim 1, characterized in that The pre-enhancement processing is performed on the target speech frame according to the complex spectrum corresponding to the target speech frame to obtain a first complex spectrum, including: Inputting the complex spectrum corresponding to the target speech frame into a first neural network, wherein the first neural network is trained based on the complex spectrum corresponding to the sample speech frame and the complex spectrum corresponding to the original speech signal in the sample speech frame; The first neural network outputs the first complex spectrum according to the complex spectrum corresponding to the target speech frame.

3. The method according to claim 2, characterized in that The first neural network includes a complex convolutional layer, a gated recurrent unit layer, and a fully connected layer; The step of outputting the first complex spectrum according to the complex spectrum of the target speech frame by the first neural network includes: The complex convolution layer performs complex convolution processing according to the real part and the imaginary part of the complex spectrum corresponding to the target speech frame; The gated recurrent unit layer transforms the output of the complex convolutional layer; The fully connected layer performs fully connected processing on the output of the gated recurrent unit and outputs the first complex spectrum.

4. The method according to claim 1, characterized in that: The step of performing speech decomposition on the target speech frame according to the first complex spectrum to obtain a glottal parameter, a gain and an excitation signal corresponding to the target speech frame includes: Predicting glottal parameters of the target speech frame according to the first complex spectrum to obtain glottal parameters corresponding to the target speech frame; Performing excitation signal prediction on the target speech frame according to the first complex spectrum to obtain an excitation signal corresponding to the target speech frame; The gain of the target speech frame is predicted according to the gain corresponding to the historical speech frame before the target speech frame to obtain the gain corresponding to the target speech frame.

5. The method according to claim 4, characterized in that The step of predicting the glottal parameters of the target speech frame according to the first complex spectrum to obtain the glottal parameters corresponding to the target speech frame includes: Inputting the first complex spectrum into a second neural network, wherein the second neural network is trained according to the complex spectrum corresponding to the sample speech frame and the glottal parameters corresponding to the sample speech frame; The second neural network outputs the glottal parameters corresponding to the target speech frame according to the first complex spectrum.

6. The method according to claim 4, characterized in that The step of predicting the glottal parameters of the target speech frame according to the first complex spectrum to obtain the glottal parameters corresponding to the target speech frame includes: Inputting the first complex spectrum and the glottal parameters corresponding to the historical speech frame before the target speech frame into a second neural network, wherein the second neural network is trained according to the complex spectrum corresponding to the sample speech frame, the glottal parameters corresponding to the historical speech frame before the sample speech frame, and the glottal parameters corresponding to the sample speech frame; The second neural network outputs the glottal parameters corresponding to the target speech frame based on the first complex spectrum and the glottal parameters corresponding to the historical speech frame before the target speech frame.

7. The method according to claim 4, characterized in that The step of performing gain prediction on the target speech frame according to the gains corresponding to the historical speech frames before the target speech frame to obtain the gain corresponding to the target speech frame includes: Inputting the gain corresponding to the historical speech frame before the target speech frame into a third neural network, wherein the third neural network is trained based on the gain corresponding to the historical speech frame before the sample speech frame and the gain corresponding to the sample speech frame; The third neural network outputs the gain corresponding to the target speech frame based on the gains corresponding to the historical speech frames before the target speech frame.

8. The method according to claim 4, characterized in that The step of predicting the excitation signal of the target speech frame according to the first complex spectrum to obtain the excitation signal corresponding to the target speech frame includes: Inputting the first complex spectrum into a fourth neural network, wherein the fourth neural network is trained according to the complex spectrum corresponding to the sample speech frame and the frequency domain representation of the excitation signal corresponding to the sample speech frame; The fourth neural network outputs a frequency domain representation of the excitation signal corresponding to the target speech frame according to the first complex spectrum.

9. The method according to claim 4, characterized in that The synthesizing process is performed according to the glottal parameter, the gain and the excitation signal to obtain the enhanced speech signal corresponding to the target speech frame, including: The excitation signal corresponding to the target speech frame is filtered by a glottal filter to obtain a filtered output signal; the glottal filter is constructed according to the glottal parameters corresponding to the target speech frame; The filtered output signal is amplified according to the gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

10. The method according to claim 1, characterized in that The step of performing speech decomposition on the target speech frame according to the first complex spectrum to obtain a glottal parameter, a gain and an excitation signal corresponding to the target speech frame includes: Calculate a power spectrum according to the first complex spectrum; Calculate the autocorrelation coefficient according to the power spectrum; Calculate the glottal parameter according to the autocorrelation coefficient; The gain is calculated according to the glottal parameter and the autocorrelation coefficient; The power spectrum of the excitation signal is calculated based on the gain and the power spectrum of the glottal filter, where the glottal filter is a filter constructed based on the glottal parameters.

11. The method according to claim 10, characterized in that The synthesizing process is performed according to the glottal parameter, the gain and the excitation signal to obtain the enhanced speech signal corresponding to the target speech frame, including: generating a first amplitude spectrum according to the power spectrum of the glottal filter and the power spectrum of the excitation signal; amplifying the first amplitude spectrum according to the gain to obtain a second amplitude spectrum; An enhanced speech signal corresponding to the target speech frame is determined according to the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum.

12. The method according to claim 11, characterized in that The step of determining the enhanced speech signal corresponding to the target speech frame according to the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum includes: combining the second amplitude spectrum and the phase spectrum extracted from the first complex spectrum to obtain a second complex spectrum; The second complex spectrum is transformed into the time domain to obtain a time domain signal of the enhanced speech signal corresponding to the target speech frame.

13. A speech enhancement device, characterized in that: include: A pre-enhancement module, configured to perform pre-enhancement processing on the target speech frame according to the complex spectrum of the target speech frame to obtain a first complex spectrum, wherein the first complex spectrum includes phase information and amplitude information; A speech decomposition module, used for performing speech decomposition on the target speech frame according to the first complex spectrum to obtain glottal parameters, gain and excitation signal corresponding to the target speech frame; The synthesis processing module is used to perform synthesis processing according to the glottal parameter, the gain and the excitation signal to obtain an enhanced speech signal corresponding to the target speech frame.

14. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the method according to any one of claims 1 to 12.