Speech enhancement method, speech enhancement device, electronic device, and computer program

Abstract

A speech enhancement method, device, equipment, and storage medium are provided. The method includes: performing glottal parameter prediction based on a frequency domain representation of the target speech frame to obtain glottal parameters corresponding to the target speech frame (410); performing gain prediction for the target speech frame based on gains corresponding to past speech frames of the target speech frame to obtain gains corresponding to the target speech frame (420); performing excitation signal prediction based on a frequency domain representation of the target speech frame to obtain an excitation signal corresponding to the target speech frame (430); and performing synthesis processing on the glottal parameters corresponding to the target speech frame, the gain corresponding to the target speech frame, and the excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame (440). The present invention can effectively enhance a speech signal and improve the quality of the speech signal. The present invention is applicable to cloud conferences to improve the quality of the speech signal.

Description

[Technical field]

[0001] This application claims priority from a Chinese patent application filed with the China Patent Office on February 8, 2021, bearing application number 202110171244.6 and titled "Speech enhancement method, device, equipment, and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of speech processing, and in particular to a speech enhancement method, device, apparatus, and storage medium. [Background technology]

[0003] Due to the convenience and timeliness of voice communication, the application of voice communication is becoming more and more widespread, for example, voice signals are transmitted between participants in a cloud conference. However, in voice communication, noise may be mixed into the voice signal. The noise mixed into the voice signal causes deterioration of communication quality and has a significant impact on the user's hearing experience. Therefore, how to enhance the voice and remove the noise is a technical problem that needs to be urgently solved in the prior art. Summary of the Invention [Problem to be solved by the invention]

[0004] SUMMARY OF THE DISCLOSURE The embodiments of the present application provide a speech enhancement method, an apparatus, a device, and a storage medium for achieving speech enhancement to improve the quality of a speech signal. [Means for solving the problem]

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, there is provided a method for speech enhancement, the method comprising: obtaining glottal parameters corresponding to a target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame; obtaining a gain corresponding to the target speech frame by performing gain prediction for the target speech frame based on gains corresponding to past speech frames of the target speech frame; obtaining an excitation signal corresponding to the target speech frame by performing excitation signal prediction based on a frequency domain representation of the target speech frame; and performing a synthesis process on glottal parameters corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0007] According to another aspect of the present invention, there is provided a speech enhancement device, the device comprising: a glottal parameter prediction module for performing glottal parameter prediction based on a frequency domain representation of a target speech frame to obtain glottal parameters corresponding to the target speech frame; a gain prediction module for performing gain prediction on the target speech frame based on gains corresponding to past speech frames of the target speech frame to obtain a gain corresponding to the target speech frame; an excitation signal prediction module for performing excitation signal prediction based on a frequency domain representation of the target speech frame to obtain an excitation signal corresponding to the target speech frame; a synthesis module that performs synthesis processing on glottal parameters corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0008] According to another aspect of an embodiment of the present application, there is provided an electronic device comprising a processor and a memory having stored thereon computer readable instructions which, when executed by the processor, cause the electronic device to implement the speech enhancement method as described above.

[0009] According to another aspect of an embodiment of the present application, there is provided a computer readable storage medium having computer readable instructions stored thereon, the computer readable instructions, when executed by a processor, causing implementation of the speech enhancement method as described above.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. [Brief description of the drawings]

[0011] The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments conforming to the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings in the following description only illustrate some embodiments of the present application, and those skilled in the art can derive other drawings from these drawings without creative labor. In the drawings, [Figure 1] 1 is a schematic diagram of a voice communication link in a VoIP system according to one illustrative embodiment; [Diagram 2] 1 shows a schematic diagram of a digital model of audio signal generation. [Diagram 3] 2 shows a schematic diagram of an excitation signal decomposed from an original speech signal and the frequency response of a glottal filter. [Figure 4] 1 is a flowchart of a speech enhancement method according to one embodiment of the present application; [Diagram 5] 5 is a flow chart of one embodiment of step 440 of the embodiment corresponding to FIG. 4. [Figure 6]FIG. 2 is a schematic diagram of performing short-time Fourier transform on an audio frame with windowing and overlapping as illustrated by one embodiment of the present application; [Figure 7] 1 is a flowchart of speech enhancement according to one embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of a first neural network according to an embodiment of the present application; [Figure 9] FIG. 2 is a schematic diagram of the input and output of a first neural network according to another embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of a second neural network according to an embodiment of the present application; [Figure 11] FIG. 2 is a schematic diagram of a third neural network according to an embodiment of the present application; [Figure 12] FIG. 1 is a block diagram of a voice enhancement device according to one embodiment of the present application; [Figure 13] FIG. 1 is a schematic diagram showing the configuration of a computer system of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Exemplary embodiments will be described more completely with reference to the drawings. However, exemplary embodiments can be implemented in various forms and should not be understood as being limited to the exemplary examples described herein. On the contrary, by providing these embodiments, the present application will be more thorough and complete, and the concept of exemplary embodiments will be fully conveyed to those skilled in the art.

[0013] It should be noted that the described features, configurations, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a thorough understanding of the embodiments of the present application. However, it should be recognized by those skilled in the art that the configurations of the present application may be implemented without one or more of the specific details, or by employing other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0014] The block diagrams shown in the drawings are merely functional entities which do not necessarily correspond to physically separate entities, i.e. these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor and / or microcontroller devices.

[0015] The flowcharts illustrated in the drawings are merely illustrative and do not necessarily include all contents and operations / steps, and do not necessarily have to be performed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be merged or partially merged, so that the order in which they are actually performed may be changed according to actual circumstances.

[0016] As a matter of clarification, "plurality" as referred to herein refers to two or more. "And / or" describes an association relationship between related objects and represents three possible relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the related objects before and after it.

[0017] Noise in a voice signal significantly reduces the voice quality and affects the user's hearing experience. Therefore, in order to improve the quality of a voice signal, it is necessary to perform a voice signal enhancement process. This removes noise as much as possible and preserves the original voice signal in the signal (i.e., a pure signal without noise). In order to realize the voice enhancement process, the present invention is proposed.

[0018] The present invention is applicable to application scenarios of voice communication, such as voice communication by instant messaging applications, voice communication in game applications, etc. In particular, voice enhancement can be performed according to the present invention on the voice sending side, the voice receiving side, or the service side providing voice communication services.

[0019] Cloud conference is an important part of online work. In a cloud conference, the voice collecting device of a cloud conference participant needs to collect the voice signal of the speaker and then transmit the collected voice signal to other conference participants. This process involves the transmission and playback of voice signals among multiple participants, and if the noise signal mixed in the voice signal is not processed, it will have a significant impact on the hearing experience of the conference participants. In such a scenario, the present invention can be applied to enhance the voice signal of the cloud conference, so that the voice signal heard by the conference participant is the enhanced voice signal, and the quality of the voice signal is improved.

[0020] Cloud conferencing is a highly efficient, convenient, and low-cost conferencing format based on cloud computing technology. Users can synchronously share voice, data files, and video with teams and customers around the world quickly and efficiently through simple and easy-to-use operations via the Internet interface. Meanwhile, complex technologies such as data transmission and processing during the conference are operated by the cloud conferencing service provider with the assistance of the users.

[0021] At present, cloud conferencing in China mainly focuses on the service content based on Software as a Service (SaaS) mode, including telephone, network, video and other service formats. Video conferencing based on cloud computing is called cloud conferencing. In the era of cloud conferencing, data transmission, processing and storage are all handled by the computer resources of the video conferencing provider, and users do not need to purchase additional expensive hardware or install complicated software at all. They can conduct highly efficient remote conferencing by simply opening a client and entering the corresponding interface.

[0022] Cloud conferencing systems support multi-server dynamic cluster deployment, provide multiple high-performance servers, and greatly improve the stability, security, and availability of conferences. In recent years, video conferencing has become popular with many users because it can greatly improve communication efficiency, continuously reduce communication costs, and bring about an upgrade in internal management level. It has been widely used in various fields such as government, military, transportation, finance, business, education, and enterprises.

[0023] 1 is a schematic diagram of a voice communication link in a network telephone (VoIP: Voice over Internet Protocol) system according to a specific embodiment. As shown in FIG. 1, based on the network connection between a sender 110 and a receiver 120, the sender 110 and the receiver 120 can perform voice transmission.

[0024] As shown in FIG. 1, the transmitting side 110 includes a collecting module 111, a pre-enhancement processing module 112, and an encoding module 113. The collecting module 111 can collect voice signals and convert the collected acoustic signals into digital signals, and the pre-enhancement processing module 112 can enhance the collected voice signals to eliminate noise in the collected voice signals and improve the quality of the voice signals. The encoding module 113 can encode the enhanced voice signals to improve the interference resistance during the transmission of the voice signals. The pre-enhancement processing module 112 can perform voice enhancement according to the method of the present application, and after enhancing the voice, it can be encoded, compressed, and transmitted, thus ensuring that the signal received at the receiving side is not affected by noise.

[0025] The receiving side 120 includes a decoding module 121, a post-emphasis module 122, and a reproduction module 123. The decoding module 121 obtains a decoded speech signal by decoding the received encoded speech signal, the post-emphasis module 122 performs enhancement processing on the decoded speech signal, and the reproduction module 123 reproduces the speech signal after enhancement processing. The post-emphasis module 122 can also perform speech enhancement according to the method of the present application. In some embodiments, the receiving side 120 may further include an audio effect adjustment module, which performs audio effect adjustment on the enhanced speech signal.

[0026] In a specific embodiment, speech enhancement may be performed according to the method of the present application only at the receiving side 120 or only at the transmitting side 110, and of course, speech enhancement may be performed according to the method of the present application at both the transmitting side 110 and the receiving side 120.

[0027] In some application scenarios, the terminal equipment in the VoIP system can support other third party protocols, such as traditional Public Switched Telephone Network (PSTN) line telephone, in addition to supporting VoIP communication, while the traditional PSTN service cannot perform voice enhancement, in such scenarios, the terminal as the receiving side may perform voice enhancement according to the method of the present application.

[0028] Before describing the present invention in detail, it is necessary to introduce the generation of voice signals. Voice signals are generated by the physiological movement of the human body's sound-producing organs under the control of the brain. That is, a certain energy noise-like impulse signal (corresponding to an excitation signal) is generated in the trachea, and the impulse signal impacts the human vocal cords (the vocal cords correspond to the glottal filter), causing a roughly periodic opening and closing, which is amplified by the oral cavity, and then a voice is produced (a voice signal is output).

[0029] Figure 2 shows a schematic diagram of the digital model of speech signal generation. This digital model can describe the process of generating a speech signal. As shown in Figure 2, the excitation signal impacts the glottal filter, and then further gain control is performed to output the speech signal. Here, the glottal filter is limited by the glottal parameters. This process can be expressed by the following equation:

[0030]

number

[0031] Fig. 3 shows a schematic diagram of an excitation signal and a frequency response of a glottal filter decomposed from one original speech signal, Fig. 3a shows a schematic diagram of the frequency response of the original speech signal, Fig. 3b shows a schematic diagram of the frequency response of the glottal filter decomposed from the original speech signal, and Fig. 3c shows a schematic diagram of the frequency response of the excitation signal decomposed from the original speech signal. As shown in Fig. 3, the undulating part in the schematic diagram of the frequency response of the original speech signal corresponds to the peak position in the schematic diagram of the frequency response of the glottal filter, and the excitation signal corresponds to a residual signal obtained by performing a linear prediction (LP) analysis on the original speech signal, so that the corresponding frequency response is gentle.

[0032] As can be seen from the above, an excitation signal, a glottal filter, and a gain can be decomposed from an original speech signal (i.e., a speech signal without noise), and the decomposed excitation signal, glottal filter, and gain can be used to represent the original speech signal, where the glottal filter can be represented by a glottal parameter. Conversely, when an excitation signal, a glottal parameter for determining a glottal filter, and a gain corresponding to an original speech signal are known, the original speech signal can be reconstructed based on the corresponding excitation signal, glottal filter, and gain.

[0033] According to the principle, the present invention predicts glottal parameters, excitation signals, and gains corresponding to an original speech signal in a speech signal to be processed based on one speech signal to be processed, and then performs speech synthesis based on the obtained glottal parameters, excitation signals, and gains. The synthesized speech signal corresponds to the original speech signal in the speech signal to be processed. Therefore, the synthesized signal corresponds to a signal from which noise has been removed. In the process, the enhancement of the speech signal to be processed is realized, so the synthesized signal is also called an enhanced speech signal corresponding to the speech signal to be processed.

[0034] FIG. 4 is a flow chart of a speech enhancement method according to an embodiment of the present application. The method may be executed by a computer device having processing capability, such as a server or a terminal. No specific limitation is made here. As shown in FIG. 4, the method includes at least steps 410 to 440. The following is a detailed introduction.

[0035] In step 410, glottal parameters corresponding to a target speech frame are obtained by performing glottal parameter prediction based on a frequency domain representation of the target speech frame.

[0036] Although speech signals do not change randomly, but rather change over time, there is a strong correlation within a short period of time, i.e., speech signals have short-term correlation. For this reason, in the present invention, speech enhancement is performed on a speech frame basis. The target speech frame refers to the speech frame that is currently the target of enhancement processing.

[0037] The frequency domain representation of the target audio frame (also called "frequency domain representation") can be obtained by performing a time-frequency transformation on the time domain signal of the target audio frame, which may be, for example, a Short-term Fourier transform (STFT). The frequency domain representation may be an amplitude spectrum, a complex spectrum, or the like, and is not specifically limited here.

[0038] The glottal parameters refer to parameters for constructing a glottal filter, and when the glottal parameters are determined, the glottal filter is determined accordingly, and the glottal filter is a digital filter. The glottal parameters may be Linear Predictive Coding (LPC) coefficients or Line Spectral Frequency (LSF) parameters. The number of glottal parameters corresponding to a target speech frame correlates with the order of the glottal filter. When the glottal filter is a K-th order filter, the glottal parameters include K-th order LSF parameters or K-th order LPC coefficients, where the LSF parameters and the LPC coefficients can be converted to each other.

[0039] A p-th order glottal filter can be expressed by Equation 2.

[0040]

number

[0041] Based on Equation 2,

[0042]

number

[0043]

number

[0044]

number

[0045] Physically, P(z) and Q(z) represent the periodic change law of glottal opening and closing, respectively. The roots of the polynomials P(z) and Q(z) are a series of angular frequencies that alternate on the complex plane and are distributed on the unit circle of the complex plane. The LSF parameters are the angular frequencies on the unit circle of the complex plane that correspond to the roots of P(z) and Q(z), and the LSF parameters LSF(n) corresponding to the nth speech frame are expressed as ω n Of course, the LSF parameter LSF(n) corresponding to the n-th speech frame can also be directly expressed by the root of P(z) corresponding to the n-th speech frame and the root of the corresponding Q(z). The roots of P(z) and Q(z) in the complex plane corresponding to the n-th speech frame can be expressed as θ n Then, the LSF parameters corresponding to the n-th speech frame are expressed by Equation 6.

[0046]

number

[0047] The glottal parameter prediction performed in step 410 refers to predicting the glottal parameters for reconstructing the original speech signal in the target speech frame. In one embodiment, the glottal parameters corresponding to the target speech frame may be predicted by a trained neural network model.

[0048] In some embodiments of the present application, step 410 includes inputting a frequency domain representation of the target speech frame into a first neural network, the first neural network being trained based on the frequency domain representations of sample speech frames and glottal parameters corresponding to the sample speech frames; and outputting, by the first neural network, glottal parameters corresponding to the target speech frame based on the frequency domain representation of the target speech frame.

[0049] The first neural network refers to a neural network model for predicting glottal parameters, and may be a model constructed by a long-short-term memory neural network, a convolutional neural network, a recurrent neural network, a fully connected neural network, or the like, and is not specifically limited here.

[0050] The frequency domain representation of the sample audio frame is obtained by performing a time-frequency transformation on the time domain signal of the sample audio frame, and the frequency domain representation may be an amplitude spectrum, a complex spectrum, etc., and no specific limitation is imposed here.

[0051] In some embodiments of the present application, the signal represented by the sample speech frames can be obtained by combining a known original speech signal and a known noise signal. If the original speech signal is known, the glottal parameters corresponding to each sample speech frame can be obtained by performing a linear predictive analysis on the original speech signal.

[0052] In the training process, the frequency domain representation of the sample speech frame is input to the first neural network, and then the first neural network performs glottal parameter prediction based on the frequency domain representation of the sample speech frame, and outputs a predicted glottal parameter, and then compares the predicted glottal parameter with the glottal parameter corresponding to the original speech signal in the sample speech frame, and if they are not consistent, adjust the parameters of the first neural network until the predicted glottal parameter output by the first neural network based on the frequency domain representation of the sample speech frame is consistent with the glottal parameter corresponding to the original speech signal in the sample speech frame. After the training is completed, the first neural network has learned the ability to accurately predict the glottal parameter corresponding to the original speech signal in the speech frame based on the frequency domain representation of the input speech frame.

[0053] In some embodiments of the present application, since there is correlation between speech frames and the similarity of frequency domain features between two adjacent speech frames is high, glottal parameters corresponding to a target speech frame may be predicted by referring to glottal parameters corresponding to a previous speech frame before the target speech frame. In this embodiment, step 410 includes: obtaining glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame by referring to glottal parameters corresponding to a previous speech frame before the target speech frame.

[0054] Since there is a correlation between the past speech frame and the target speech frame, and there is a similarity between the glottal parameters corresponding to the past speech frame of the target speech frame and the glottal parameters corresponding to the target speech frame, the glottal parameters corresponding to the original speech signal in the past speech frame of the target speech frame are used as reference to supervise the prediction process of the glottal parameters of the target speech frame, thereby improving the accuracy of the glottal parameter prediction.

[0055] In one embodiment of the present application, since the closer the speech frame, the higher the similarity of the glottal parameters, the more the accuracy of prediction can be guaranteed by referring to the glottal parameters corresponding to the past speech frame close to the target speech frame. For example, the glottal parameters corresponding to the speech frame one frame before the target speech frame can be referred to. In a specific embodiment, the number of past speech frames referred to can be one frame or multiple frames, and can be selected and used according to actual needs.

[0056] The glottal parameters corresponding to the past speech frames of the target speech frame may be glottal parameters obtained by performing glottal parameter prediction on the past speech frames, in other words, the glottal parameter prediction process reuses the glottal parameters predicted for the past speech frames to supervise the glottal parameter prediction process for the current speech frame.

[0057] In some embodiments of the present application, in a scenario where a first neural network is used to predict glottal parameters, in addition to a frequency domain representation of a target speech frame, glottal parameters corresponding to past speech frames of the target speech frame are also input to the first neural network to perform glottal parameter prediction. In this embodiment, step 410 includes inputting the frequency domain representation of the target speech frame and the glottal parameters corresponding to past speech frames of the target speech frame to a first neural network, the first neural network being trained based on frequency domain representations of sample speech frames, the glottal parameters corresponding to the sample speech frames, and the glottal parameters corresponding to past speech frames of the sample speech frames; and performing a prediction by the first neural network based on the frequency domain representation of the target speech frame and the glottal parameters corresponding to the past speech frames of the target speech frame, and outputting the glottal parameters corresponding to the target speech frame.

[0058] In the training process of the first neural network in this embodiment, the frequency domain representation of the sample speech frame and the glottal parameters corresponding to the past speech frames of the sample speech frame are input to the first neural network, the predicted glottal parameters are output by the first neural network, and if the predicted glottal parameters are not consistent with the glottal parameters corresponding to the original speech signal in the sample speech frame, the parameters of the first neural network are adjusted until the predicted glottal parameters are consistent with the glottal parameters corresponding to the original speech signal in the sample speech frame. After the training is completed, the first neural network has learned the ability to predict the glottal parameters for reconstructing the original speech signal in the speech frame according to the frequency domain representation of the speech frame and the glottal parameters corresponding to the past speech frames of the speech frame.

[0059] Continuing with reference to FIG. 4, in step 420, a gain corresponding to the target speech frame is obtained by performing gain prediction for the target speech frame based on gains corresponding to past speech frames of the target speech frame.

[0060] The gain corresponding to the past speech frame refers to the gain for reconstructing the original speech signal in the past speech frame. Similarly, the gain corresponding to the target speech frame predicted in step 420 is for reconstructing the original speech signal in the target speech frame.

[0061] In some embodiments of the present application, gain prediction may be performed for the target speech frame by deep learning. That is, gain prediction is performed by a constructed neural network model. For convenience of explanation, the neural network model for performing gain prediction is referred to as a second neural network. The second neural network may be a model constructed by a long-short-term memory neural network, a convolutional neural network, a fully connected neural network, etc.

[0062] In one embodiment of the present application, step 420 may include the steps of inputting gains corresponding to past speech frames of the target speech frame into a second neural network, the second neural network being trained based on gains corresponding to sample speech frames and gains corresponding to past speech frames of the sample speech frame, and outputting, by the second neural network, gains corresponding to the target speech frame based on gains corresponding to past speech frames of the target speech frame.

[0063] The signal represented by the sample audio frames can be obtained by combining a known original audio signal with a known noise signal, so that, when the original audio signal is known, a gain corresponding to each sample audio frame, i.e., a gain for reconstructing the original audio signal in the sample audio frame, can be determined according to a linear predictive analysis performed on the original audio signal.

[0064] The gains corresponding to the past speech frames of the target speech frame may be obtained by performing gain prediction on the past speech frames by the second neural network, in other words, the gains predicted on the past speech frames are reused as inputs of the second neural network in the process of performing gain prediction on the target speech frame.

[0065] In the process of training the second neural network, the gain corresponding to the past speech frame of the sample speech frame is input to the second neural network, and then the second neural network performs gain prediction according to the gain corresponding to the past speech frame of the input sample speech frame, and outputs a predicted gain, and further adjusts the parameters of the second neural network according to the predicted gain and the gain corresponding to the sample speech frame, that is, if the predicted gain does not match the gain corresponding to the sample speech frame, adjusts the parameters of the second neural network until the predicted gain output by the second neural network for the sample speech frame matches the gain corresponding to the sample speech frame. After the above training process, the second neural network learns the ability to predict the gain corresponding to a speech frame according to the gain corresponding to the past speech frame of the speech frame, and can accurately perform gain prediction.

[0066] In step 430, an excitation signal corresponding to the target speech frame is obtained by performing excitation signal prediction based on a frequency domain representation of the target speech frame.

[0067] The excitation signal prediction performed in step 430 refers to prediction of the excitation signal for reconstructing the original speech signal in the target speech frame, so that the excitation signal corresponding to the obtained target speech frame can be used to reconstruct the original speech signal in the target speech frame.

[0068] In some embodiments of the present application, the excitation signal may be predicted by deep learning, that is, the excitation signal may be predicted by a constructed neural network model. For convenience of explanation, the neural network model for performing the excitation signal prediction is called a third neural network. The third neural network may be a model constructed by a long-short-term memory neural network, a convolutional neural network, a fully connected neural network, etc.

[0069] In some embodiments of the present application, step 430 includes inputting a frequency domain representation of the target speech frame into a third neural network, the third neural network being trained based on the frequency domain representations of sample speech frames and the frequency domain representations of excitation signals corresponding to the sample speech frames; and outputting, by the third neural network, a frequency domain representation of an excitation signal corresponding to the target speech frame based on the frequency domain representation of the target speech frame.

[0070] The excitation signal corresponding to the sample audio frame refers to an excitation signal that can be used to reconstruct the original audio signal in the sample audio frame. The excitation signal corresponding to the sample audio frame can be determined by performing linear predictive analysis on the original audio signal in the sample audio frame. The frequency domain expression of the excitation signal can be the amplitude spectrum or complex spectrum of the excitation signal, and is not specifically limited herein.

[0071] In the process of training the third neural network, the frequency domain representation of the sample speech frame is input to the third neural network, and then the third neural network performs excitation signal prediction based on the frequency domain representation of the input sample speech frame, and outputs a frequency domain representation of the predicted excitation signal, and further adjusts the parameters of the third neural network based on the frequency domain representation of the predicted excitation signal and the frequency domain representation of the excitation signal corresponding to the sample speech frame, i.e., if the frequency domain representation of the predicted excitation signal does not match the frequency domain representation of the excitation signal corresponding to the sample speech frame, adjusts the parameters of the third neural network until the frequency domain representation of the predicted excitation signal output by the third neural network for the sample speech frame matches the frequency domain representation of the excitation signal corresponding to the sample speech frame. After the above training process, the third neural network learns the ability to predict the excitation signal corresponding to a speech frame based on the frequency domain representation of the speech frame, and can accurately perform excitation signal prediction.

[0072] In step 440, a synthesis process is performed on the glottal parameters corresponding to the target speech frame, the gain corresponding to the target speech frame, and the excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0073] After obtaining the glottal parameters corresponding to the target speech frame, the gain corresponding to the target speech frame, and the excitation signal corresponding to the target speech frame, linear prediction analysis may be performed based on these three parameters to realize synthesis processing to obtain the enhanced speech signal corresponding to the target speech frame. Specifically, a glottal filter may be first constructed based on the glottal parameters corresponding to the target speech frame, and then the gain corresponding to the target speech frame and the corresponding excitation signal may be referred to and speech synthesis may be performed according to the above formula 1 to obtain the enhanced speech signal corresponding to the target speech frame.

[0074] In some embodiments of the present application, as shown in FIG. 5, step 440 includes steps 510 to 530.

[0075] In step 510, a glottal filter is constructed based on the glottal parameters corresponding to the target speech frame.

[0076] If the glottal parameters are LPC coefficients, the glottal filter can be constructed directly according to the above Equation 2. If the glottal filter is a K-th order filter, the glottal parameters corresponding to the target speech frame can be the K-th order LPC coefficients, i.e., (outside 2) JPEG0007615510000008.jpg15143. In another embodiment, the constant 1 in the above Equation 2 may also be an LPC coefficient.

[0077] If the glottal parameters are LSF parameters, the LSF parameters may be converted to LPC coefficients and then the glottal filter may be constructed according to Equation 2 above.

[0078] In step 520, a first speech signal is obtained by filtering an excitation signal corresponding to the target speech frame with the glottal filter.

[0079] The filtering process is a convolution in the time domain. Therefore, the process of filtering the excitation signal by the glottal filter as described above can be performed by transforming it into the time domain. The frequency domain representation of the excitation signal corresponding to the target speech frame is predicted, and then the frequency domain representation of the excitation signal is transformed into the time domain to obtain the time domain signal of the excitation signal corresponding to the target speech frame.

[0080] In the present invention, the target speech frame is a digital signal and includes a plurality of sample points. Filtering the excitation signal by a glottal filter means convolving a previous sample point before a certain sample point with the glottal filter to obtain a target signal value corresponding to the sample point. In some embodiments of the present application, the target speech frame includes a plurality of sample points, the glottal filter is a K-th order filter (K ​​is a positive integer), and the excitation signal includes excitation signal values ​​corresponding to each of the plurality of sample points in the target speech frame. According to the above filtering process, step 520 includes: convolving excitation signal values ​​corresponding to K sample points before each sample point in the target speech frame with the K-th order filter to obtain a target signal value of each sample point in the target speech frame; and combining target signal values ​​corresponding to all sample points in the target speech frame in time order to obtain the first speech signal. Here, the expression of the K-th order filter can be referred to the above Equation 1. That is, for each sample point in the target speech frame, the excitation signal values ​​corresponding to the previous K sample points are convolved with a K-th order filter to obtain the target signal value corresponding to each sample point.

[0081] As can be seen, for the first sample point in a target speech frame, the excitation signal values ​​of the last K sample points in the speech frame preceding the target speech frame need to be used to calculate the target signal value corresponding to the first sample point. Similarly, for the second sample point in the target speech frame, the excitation signal values ​​of the last (K-1) sample points in the speech frame preceding the target speech frame and the excitation signal value of the first sample point in the target speech frame need to be convolved with a K-th order filter to obtain the target signal value corresponding to the second sample point in the target speech frame.

[0082] In summary, step 502 also requires excitation signal values ​​corresponding to past speech frames of the target speech frame. The number of sample points in the past speech frames required is correlated with the order of the glottal filter, i.e., if the glottal filter is of order K, excitation signal values ​​corresponding to the last K sample points in the speech frame preceding the target speech frame are required.

[0083] In step 530, the first audio signal is amplified with a gain corresponding to the target audio frame to obtain an enhanced audio signal corresponding to the target audio frame.

[0084] Through steps 510 to 530 as described above, speech synthesis is realized for the glottal parameters, excitation signal, and gain predicted for the target speech frame, and an enhanced speech signal of the target speech frame is obtained.

[0085] In the present invention, based on the frequency domain representation of the target speech frame, glottal parameters and excitation signals for reconstructing the original speech signal in the target speech frame are predicted, and based on the gains of the past speech frames of the target speech frame, gains for reconstructing the original speech signal in the target speech frame are predicted. Then, speech synthesis is performed on the glottal parameters, corresponding excitation signals, and corresponding gains corresponding to the predicted target speech frame, which corresponds to the reconstruction of the original speech signal in the target speech frame. The signal obtained by the synthesis process is, i.e., an enhanced speech signal corresponding to the target speech frame, which realizes the enhancement of the speech frame and improves the quality of the speech signal.

[0086] In the related art, there are speech enhancement methods using the methods of spectrum estimation and spectrum regression prediction. In the speech enhancement method using spectrum estimation, since it is considered that a mixed speech includes a speech part and a noise part, the noise can be estimated by a statistical model or the like. The spectrum corresponding to the noise is subtracted from the spectrum corresponding to the mixed speech, and the remaining spectrum is the speech spectrum. In this way, a clean speech signal is restored based on the spectrum obtained by subtracting the spectrum corresponding to the noise from the spectrum corresponding to the mixed speech. In the speech enhancement method using spectrum regression prediction, a masking threshold corresponding to a speech frame is predicted by a neural network, and then a gain control is performed on the mixed signal spectrum based on the masking threshold to obtain an enhanced spectrum. The masking threshold reflects the ratio of the speech component and the noise component at each frequency point in the speech frame.

[0087] The above-mentioned speech enhancement method using spectrum estimation and spectrum regression prediction is an estimation based on the posterior probability of the noise spectrum, and the estimated noise may be inaccurate. For example, the estimated noise spectrum is very inaccurate because of the instantaneous occurrence of transient noise such as typing on a keyboard. This results in poor noise suppression effect. When the prediction of the noise spectrum is inaccurate, processing the original mixed speech signal according to the estimated noise spectrum may cause speech distortion in the mixed speech signal or cause deterioration of the noise suppression effect. Therefore, in this case, a compromise between speech fidelity and noise suppression is required.

[0088] In the present invention, since the glottal parameters have a strong correlation with the glottal features in the physical process of speech production, synthesizing speech based on the predicted glottal parameters effectively guarantees the speech structure of the original speech signal in the target speech frame. Therefore, obtaining the enhanced speech signal of the target speech frame by synthesizing the predicted glottal parameters, excitation signal, and gain can effectively avoid reducing the original speech signal in the target speech frame, and the speech structure is effectively protected. And, since the glottal parameters, excitation signal, and gain corresponding to the target speech frame are predicted, the original noisy speech is no longer processed, and therefore there is no need to compromise between speech fidelity and noise suppression.

[0089] In some embodiments of the present application, before step 410, the method further includes the steps of obtaining a time-domain signal of the target audio frame, and obtaining a frequency-domain representation of the target audio frame by time-frequency transforming the time-domain signal of the target audio frame.

[0090] The time-frequency transform may be a short-term Fourier transform (STFT). The expression in the frequency domain may be an amplitude spectrum, a complex spectrum, or the like, and is not specifically limited here.

[0091] In the short-time Fourier transform, the windowing and overlapping operation is adopted to eliminate the unevenness between frames. FIG. 6 is a schematic diagram of the windowing and overlapping in the short-time Fourier transform shown in one specific embodiment. In FIG. 6, when the windowing and overlapping operation of 50% is adopted and the short-time Fourier transform is for 640 sample points, the overlapping sample number (hop-size) of the window function is 320. The window function used for the windowing may be a Hanning window, and of course, other window functions may be adopted, and no specific limitation is made here.

[0092] In other embodiments, windowing and overlapping operations other than 50% may be used, for example if the short-time Fourier transform is for 512 sample points, and one audio frame contains 320 sample points, then it is only necessary to overlap 192 sample points of the previous audio frame.

[0093] In some embodiments of the present application, the step of acquiring the time-domain signal of the target audio frame includes the steps of acquiring a second audio signal, the second audio signal being a collected audio signal or an audio signal decoded from an encoded audio signal, and acquiring the time-domain signal of the target audio frame by framing the second audio signal.

[0094] In some examples, the second audio signal may be framed with a set frame length. The frame length may be set according to practical needs. For example, the frame length may be set to 20 ms.

[0095] As mentioned above, the present invention may be applied on the transmitting side for speech enhancement, or on the receiving side for speech enhancement.

[0096] When the present invention is applied to the transmitting side, the second voice signal is a voice signal collected at the transmitting side. A plurality of voice frames are obtained by framing the second voice signal. After the voice frames are obtained by framing, each voice frame may be set as a target voice frame, and the target voice frame may be emphasized in the above process of steps 410 to 440. Furthermore, after obtaining an emphasized voice signal corresponding to the target voice frame, the emphasized voice signal may be encoded, and transmission may be performed based on the obtained encoded voice signal.

[0097] In one embodiment, since the directly collected voice signal is an analog signal, the voice signal needs to be further digitized before framing in order to facilitate signal processing. The collected voice signal may be sampled at a set sampling rate. The set sampling rate may be 16000Hz, 8000Hz, 32000Hz, 48000Hz, etc., and may be specifically set according to actual needs.

[0098] When the present invention is applied to a receiving side, the second audio signal is an audio signal obtained by decoding a received encoded audio signal. After obtaining a plurality of audio frames by framing the second audio signal, the plurality of audio frames are set as a target audio frame, and the target audio frame is enhanced in the above process of steps 410 to 440 to obtain an enhanced audio signal of the target audio frame. Furthermore, the enhanced audio signal corresponding to the target audio frame may be reproduced. Compared with the signal before enhancement of the target audio frame, the acquired enhanced audio signal has already been noise-removed and has a higher quality of the audio signal, so that the user can have a better hearing experience.

[0099] The present invention will now be described in further detail with reference to specific examples.

[0100] FIG. 7 is a flow chart of a speech enhancement method according to a specific embodiment. Suppose the nth speech frame is a target speech frame, the time domain signal of the nth speech frame is s(n). As shown in FIG. 7, in step 710, the nth speech frame is subjected to a time-frequency transform to obtain a frequency domain representation S(n) of the nth speech frame. S(n) may be an amplitude spectrum or a complex spectrum, and no specific limitation is made here.

[0101] After obtaining the frequency domain representation S(n) of the nth speech frame, the glottal parameters corresponding to the nth speech frame can be predicted in step 720, and the excitation signal corresponding to the target speech frame can be obtained in steps 730 and 740.

[0102] In step 720, the first neural network may receive only the frequency domain representation S(n) of the nth speech frame, or may receive the glottal parameters P_pre(n) corresponding to the previous speech frames of the target speech frame and the frequency domain representation S(n) of the nth speech frame. The first neural network may obtain the glottal parameters ar(n) corresponding to the nth speech frame by performing glottal parameter prediction based on the input information.

[0103] In step 730, the frequency domain representation S(n) of the nth speech frame is input to a third neural network, which performs excitation signal prediction based on the input information and outputs a frequency domain representation R(n) of the excitation signal corresponding to the nth speech frame. On this basis, in step 740, the frequency domain representation R(n) of the excitation signal corresponding to the nth speech frame can be transformed into a time domain signal r(n) by performing a frequency-time transformation.

[0104] The gain corresponding to the n-th speech frame is obtained by step 750. In step 750, the gain G_pre(n) of the previous speech frame of the n-th speech frame is input to the second neural network. In response, the second neural network performs gain prediction to obtain the gain G_(n) corresponding to the n-th speech frame.

[0105] After obtaining the glottal parameters ar(n), the corresponding excitation signal r(n), and the corresponding gain G_(n) corresponding to the nth speech frame, synthesis filtering is performed in step 760 based on these three parameters to obtain the enhanced speech signal s_e(n) corresponding to the nth speech frame. Specifically, speech synthesis may be performed according to the principle of linear predictive analysis. The process of performing speech synthesis according to the principle of linear predictive analysis requires the use of information of past speech frames. Specifically, the process of filtering the excitation signal by the glottal filter is, for the tth sample point, to obtain a target signal value corresponding to the sample point by convolving with a pth order glottal filter using the excitation signal values ​​of the previous p past sample points. If the glottal filter is a 16th order digital filter, the process of performing synthesis processing on the nth speech frame also requires the use of information of the last p sample points in the n-1th frame.

[0106] Hereinafter, the above steps 720, 730, and 750 will be further described with reference to a specific embodiment. Assuming that the sampling frequency Fs of the speech signal to be processed is 16000Hz and the frame length is 20ms, each speech frame contains 320 sample points. The short-time Fourier transform performed in the method adopts 640 sample points, and assumes that the number of overlapping sample points is 320. Furthermore, it is assumed that the glottal parameters are line spectrum frequency coefficients, that is, the glottal parameters corresponding to the nth speech frame are ar(n), and the corresponding LSF parameters are LSF(n), and the glottal filter is a 16th order filter.

[0107] FIG. 8 is a schematic diagram of a first neural network according to a specific embodiment. As shown in FIG. 8, the first neural network includes one long-short-term memory (LSTM) layer and three cascaded fully connected (FC) layers. The LSTM layer is a hidden layer and includes 256 units. The input of the LSTM layer is the frequency domain representation S(n) of the n-th speech frame. In this embodiment, the input of the LSTM layer is a 321-dimensional STFT coefficient. Among the three cascaded FC layers, the first two FC layers are set with an activation function σ(), which is for increasing the nonlinear expression ability of the first neural network, and the last FC layer is not set with an activation function, and the last FC layer performs classification output as a classifier. As shown in FIG. 8, the three FC layers from bottom to top contain 512, 512, and 16 units, respectively, and the output of the last FC layer is a 16-dimensional line spectral frequency coefficient LSF(n) corresponding to the n-th speech frame, i.e., a 16-order line spectral frequency coefficient.

[0108] FIG. 9 is a schematic diagram of the input and output of a first neural network shown in another embodiment. Here, the structure of the first neural network in FIG. 9 is the same as that in FIG. 8. Compared with FIG. 8, the input of the first neural network in FIG. 9 further includes the line spectral frequency coefficient LSF(n-1) of the voice frame one before the n-th voice frame (i.e., the n-1-th frame). As shown in FIG. 9, the line spectral frequency coefficient LSF(n-1) of the voice frame one before the n-th voice frame is embedded in the second FC layer as reference information. Since the similarity of the LSF parameters of two adjacent voice frames is very high, if the LSF parameters corresponding to the past voice frame of the n-th voice frame are used as reference information, the accuracy of prediction of the LSF parameters can be improved.

[0109] FIG. 10 is a schematic diagram of a second neural network shown by one specific embodiment. As shown in FIG. 10, the second neural network includes one LSTM layer and one FC layer. Among them, the LSTM layer is a hidden layer and includes 128 units. The FC layer has an input that is a 512-dimensional vector and an output that is a one-dimensional gain. In one specific embodiment, the past speech frame gain G_pre(n) of the nth speech frame is the gain corresponding to the previous four speech frames of the nth speech frame, that is, G_pre(n)={G(n - 1), G(n - 2), G(n - 3), G(n - 4)} may be defined as such.

[0110] Of course, the number of past speech frames for gain prediction to be selected is not limited to the examples listed above. Specifically, it may be selected and used according to actual needs.

[0111] In the structures of the first neural network and the second neural network as shown above, the network exhibits an M-to-N mapping relationship (N << M). That is, the neural network has an input information dimension of M and an output information dimension of N. The structures of the first neural network and the second neural network are extremely simplified, and the complexity of the neural network model is reduced.

[0112] FIG. 11 is a schematic diagram of a third neural network shown in a specific embodiment. As shown in FIG. 11, the third neural network includes one LSTM layer and three FC layers. The LSTM layer is a hidden layer and includes 256 units, and the input of the LSTM is the 321-dimensional STFT coefficient S(n) corresponding to the n-th voice frame. The number of units included in the three FC layers is 512, 512, and 321, respectively, and the last FC layer outputs a 321-dimensional frequency domain representation R(n) of the excitation signal corresponding to the n-th voice frame. Among the three FC layers from bottom to top, the first two FC layers are set with activation functions to improve the nonlinear expression ability of the model, and the last FC layer for performing classification output does not have an activation function.

[0113] The structures of the first neural network, the second neural network, and the third neural network shown in Figures 8 to 11 are merely illustrative examples. In other embodiments, training may be performed according to setting corresponding network structures in a deep learning open source platform.

[0114] Hereinafter, an apparatus embodiment of the present application will be introduced, which can be used to perform the method in the above embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above embodiment of the method of the present application.

[0115] FIG. 12 is a block diagram of a voice enhancement device according to an embodiment. As shown in FIG. 12, the voice enhancement device includes: a glottal parameter prediction module 1210 for performing glottal parameter prediction based on a frequency domain representation of a target speech frame to obtain glottal parameters corresponding to the target speech frame; a gain prediction module 1220 for performing gain prediction on the target speech frame based on gains corresponding to past speech frames of the target speech frame to obtain a gain corresponding to the target speech frame; an excitation signal prediction module 1230 for performing excitation signal prediction based on a frequency domain representation of the target speech frame to obtain an excitation signal corresponding to the target speech frame; and a synthesis module 1240 for performing synthesis processing on glottal parameters corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0116] In some embodiments of the present application, the synthesis module 1240 includes: a glottal filter construction unit for constructing a glottal filter based on glottal parameters corresponding to the target speech frame; a filtering unit for obtaining a first speech signal by filtering an excitation signal corresponding to the target speech frame with the glottal filter; and an amplification unit for amplifying the first speech signal with a gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame.

[0117] In some embodiments of the present application, the target speech frame includes a plurality of sample points, the glottal filter is a K-th order filter (K ​​is a positive integer), the excitation signal includes excitation signal values ​​corresponding to each of the plurality of sample points in the target speech frame, and the filtering unit includes: a convolution unit for convolving the K-th order filter with excitation signal values ​​corresponding to K sample points preceding each sample point in the target speech frame to obtain a target signal value for each sample point in the target speech frame, and a combination unit for combining target signal values ​​corresponding to all sample points in the target speech frame in time order to obtain the first speech signal. In some embodiments of the present application, the glottal filter is a K-th order filter, and the glottal parameters include K-th order line spectral frequency parameters or K-th order linear prediction coefficients.

[0118] In some embodiments of the present application, the glottal parameter prediction module 1210 includes a first input unit for inputting a frequency domain representation of the target speech frame to a first neural network, the first neural network being trained based on the frequency domain representations of sample speech frames and glottal parameters corresponding to the sample speech frames, and a first output unit for outputting glottal parameters corresponding to the target speech frame by the first neural network based on the frequency domain representation of the target speech frame.

[0119] In some embodiments of the present application, the glottal parameter prediction module 1210 is further configured to obtain glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame with reference to glottal parameters corresponding to past speech frames of the target speech frame.

[0120] In some embodiments of the present application, the glottal parameter prediction module 1210 includes a second input unit for inputting the frequency domain representation of the target speech frame and glottal parameters corresponding to past speech frames of the target speech frame to a first neural network, the first neural network being trained based on the frequency domain representation of sample speech frames, the glottal parameters corresponding to the sample speech frames, and the glottal parameters corresponding to past speech frames of the sample speech frames; and a second output unit for making a prediction by the first neural network based on the frequency domain representation of the target speech frame and the glottal parameters corresponding to past speech frames of the target speech frame, and outputting the glottal parameters corresponding to the target speech frame.

[0121] In some embodiments of the present application, the gain prediction module 1220 includes a third input unit for inputting gains corresponding to past speech frames of the target speech frame to a second neural network, the second neural network being trained based on gains corresponding to sample speech frames and gains corresponding to past speech frames of the sample speech frame, and a third output unit for outputting a gain corresponding to the target speech frame by the second neural network based on gains corresponding to past speech frames of the target speech frame.

[0122] In some embodiments of the present application, the excitation signal prediction module 1230 includes a fourth input unit for inputting a frequency domain representation of the target speech frame into a third neural network, the third neural network being trained based on the frequency domain representation of a sample speech frame and the frequency domain representation of an excitation signal corresponding to the sample speech frame, and a fourth output unit for outputting a frequency domain representation of an excitation signal corresponding to the target speech frame by the third neural network based on the frequency domain representation of the target speech frame.

[0123] In some embodiments of the present application, the speech enhancement device further includes an acquisition module for acquiring a time-domain signal of the target speech frame, and a time-frequency transformation module for acquiring a frequency-domain representation of the target speech frame by time-frequency transforming the time-domain signal of the target speech frame.

[0124] In some embodiments of the present application, the acquisition module is further configured to acquire a time-domain signal of the target audio frame by acquiring a second audio signal and framing the second audio signal, where the second audio signal is a collected audio signal or a decoded audio signal of an encoded audio.

[0125] In some embodiments of the present application, the voice enhancement device further comprises a processing module for reproducing or encoding and transmitting an enhanced voice signal corresponding to said target voice frame.

[0126] FIG. 13 is a schematic diagram showing the configuration of a computer system of an electronic device suitable for implementing an embodiment of the present application.

[0127] It should be understood that the electronic device computer system 1300 shown in FIG. 13 is merely an example and should not pose any limitations on the functionality and scope of use of the embodiments of the present application.

[0128] As shown in Fig. 13, the computer system 1300 includes a central processing unit (CPU) 1301, which can execute various appropriate operations and processes, for example, the method in the above embodiment, based on a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage unit 1308 to a random access memory (RAM) 1303. The RAM 1303 further stores various programs and data necessary for the system operation. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0129] To the I / O interface 1305, an input unit 1306 including a keyboard, a mouse, etc., an output unit 1307 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, a storage unit 1308 including a hard disk, etc., and a communication unit 1309 including, for example, a local area network (LAN) card, a modem, etc., are connected. The communication unit 1309 executes communication processing via a network such as the Internet. A driver 1310 is also connected to the I / O interface 1305 as necessary. For example, a removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the driver 1310 as necessary so that a computer program read from the removable medium 1311 is installed in the storage unit 1308 as necessary.

[0130] In particular, according to an embodiment of the present application, the processes described above with reference to the flowcharts may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product including a computer program carried on a computer-readable medium, the computer program including program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication unit 1309 and / or installed from a removable medium 1311. When executed by the central processing unit (CPU) 1301, the computer program causes the system of the present application to perform various functions defined therein.

[0131] For illustrative purposes, the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection having one or more conductors, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in combination with an instruction execution system, apparatus, or device. However, in this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier, the data signal carrying a computer-readable program code. Such a propagated data signal may take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code included in the computer-readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the above.

[0132] The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. Each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code, which includes one or more executable instructions for implementing a certain logical function. It should also be noted that in some alternative implementations, the functions described in the blocks may be performed in a different order from the order described in the drawings. For example, two blocks shown in succession may actually be essentially executed in parallel, or may be executed in the reverse order depending on the related functions. It should also be noted that each block in the block diagrams or flowcharts, and combinations of blocks in the block diagrams or flowcharts, may be implemented in a dedicated hardware-based system for executing a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of the present application may be implemented in software or hardware, and the described units may be installed in a processor, where the names of the units do not limit the units themselves in some cases.

[0134] In another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being assembled to the electronic device. The computer-readable storage medium is loaded with computer-readable instructions, which, when executed by a processor, cause the method in any of the above embodiments to be realized.

[0135] According to an aspect of the present application, there is further provided an electronic device comprising: a processor; and a memory having stored thereon computer-readable instructions that, when executed by the processor, cause the electronic device to implement the method of any of the above embodiments.

[0136] According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable storage medium and, when executed by the processor, causing the computing device to perform the method of any of the above embodiments.

[0137] It should be noted that, although the above detailed description refers to some modules or units of an apparatus for performing operations, such division is not mandatory. In fact, according to an embodiment of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided to be embodied by multiple modules or units.

[0138] According to the above description of the embodiments, it is easily understood by those skilled in the art that the exemplary embodiments described herein may be realized by software or by a combination of software and necessary hardware. Therefore, the configuration according to the embodiments of the present application may be embodied in the form of a software product. The software product may be stored in a non-volatile storage medium (which may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or a network, and includes some instructions that cause a computing device (which may be a personal computer, a server, a touch terminal, a network device, etc.) to execute the method according to the embodiments of the present application.

[0139] Those skilled in the art may easily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any modifications, uses, or adaptations of the present application. These modifications, uses, or adaptations are in accordance with the general principles of the present application and include common general knowledge or customary technical means in the art that are not disclosed in the present application.

[0140] It should be understood that the present application is not limited to the exact construction described above and illustrated in the drawings, and various modifications and changes are possible without departing from the scope thereof, which is limited only by the appended claims.

Claims

1. 1. A computing device implemented method for speech enhancement, comprising: obtaining glottal parameters corresponding to a target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame; obtaining a gain corresponding to the target speech frame by performing gain prediction on the target speech frame based on gains corresponding to past speech frames of the target speech frame; obtaining an excitation signal corresponding to the target speech frame by performing excitation signal prediction based on a frequency domain representation of the target speech frame; performing a synthesis process on a glottal parameter corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame; Including, performing a synthesis process on a glottal parameter corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame, the synthesis process comprising: constructing a glottal filter based on glottal parameters corresponding to the target speech frame; obtaining a first speech signal by filtering an excitation signal corresponding to the target speech frame with the glottal filter; and amplifying the first speech signal with a gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame. Speech enhancement methods.

2. the target speech frame includes a plurality of sample points, the glottal filter is a K-th order filter, where K is a positive integer, and the excitation signal includes excitation signal values ​​corresponding to each of the plurality of sample points in the target speech frame; obtaining a first speech signal by filtering an excitation signal corresponding to the target speech frame with the glottal filter, obtaining a target signal value for each sample point in the target speech frame by convolving the K-th order filter with excitation signal values ​​corresponding to K sample points preceding each sample point in the target speech frame; obtaining the first speech signal by combining target signal values ​​corresponding to all sample points in the target speech frame in time order. The method of claim 1 .

3. The glottal filter is a K-th order filter (K ​​is a positive integer), and the glottal parameters include K-th order line spectral frequency parameters or K-th order linear prediction coefficients. The method of claim 1 .

4. obtaining glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame, comprising: inputting the frequency domain representation of the target speech frame into a first neural network, the first neural network being trained based on the frequency domain representation of sample speech frames and glottal parameters corresponding to the sample speech frames; and outputting, by the first neural network, glottal parameters corresponding to the target speech frame based on a frequency domain representation of the target speech frame. The method of claim 1 .

5. obtaining glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame, comprising: and performing glottal parameter prediction based on a frequency domain representation of the target speech frame by referring to glottal parameters corresponding to past speech frames of the target speech frame to obtain glottal parameters corresponding to the target speech frame. The method of claim 1 .

6. obtaining glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame with reference to glottal parameters corresponding to past speech frames of the target speech frame, inputting the frequency domain representation of the target speech frame and glottal parameters corresponding to past speech frames of the target speech frame into a first neural network, the first neural network being trained with the frequency domain representation of sample speech frames, the glottal parameters corresponding to the sample speech frames, and the glottal parameters corresponding to past speech frames of the sample speech frame; and performing a prediction using the first neural network based on a frequency domain representation of the target speech frame and glottal parameters corresponding to previous speech frames of the target speech frame, and outputting glottal parameters corresponding to the target speech frame. The speech enhancement method according to claim 5.

7. obtaining a gain corresponding to the target speech frame by performing gain prediction on the target speech frame based on gains corresponding to past speech frames of the target speech frame, inputting gains corresponding to past speech frames of the target speech frame into a second neural network, the second neural network being trained based on gains corresponding to sample speech frames and gains corresponding to past speech frames of the sample speech frame; and outputting, by the second neural network, a gain corresponding to the target speech frame based on gains corresponding to past speech frames of the target speech frame. The method of claim 1 .

8. obtaining an excitation signal corresponding to the target speech frame by performing excitation signal prediction based on a frequency domain representation of the target speech frame, the step comprising: inputting the frequency domain representation of the target speech frame into a third neural network, the third neural network being trained based on the frequency domain representation of a sample speech frame and on a frequency domain representation of an excitation signal corresponding to the sample speech frame; and outputting, by the third neural network, a frequency domain representation of an excitation signal corresponding to the target speech frame based on the frequency domain representation of the target speech frame. The method of claim 1 .

9. prior to obtaining glottal parameters corresponding to the target speech frame by performing glottal parameter prediction based on a frequency domain representation of the target speech frame, obtaining a time domain signal of the target speech frame; and obtaining a frequency domain representation of the target speech frame by performing a time-to-frequency transform on the time domain signal of the target speech frame. The method of claim 1 .

10. The step of obtaining a time domain signal of the target speech frame comprises: acquiring a second audio signal, the second audio signal being a collected audio signal or a decoded audio signal from an encoded audio signal; and obtaining a time-domain signal of the target audio frame by framing the second audio signal. The method of claim 9 .

11. performing a synthesis process on the glottal parameters corresponding to the target speech frame, the gain corresponding to the target speech frame, and the excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame; and further comprising the step of: reproducing or encoding and transmitting an enhanced speech signal corresponding to the target speech frame. The method of claim 1 .

12. A voice enhancement device, comprising: a glottal parameter prediction module for performing glottal parameter prediction based on a frequency domain representation of a target speech frame to obtain glottal parameters corresponding to the target speech frame; a gain prediction module for performing gain prediction on the target speech frame based on gains corresponding to past speech frames of the target speech frame to obtain a gain corresponding to the target speech frame; an excitation signal prediction module for performing excitation signal prediction based on a frequency domain representation of the target speech frame to obtain an excitation signal corresponding to the target speech frame; a synthesis module performing a synthesis process on a glottal parameter corresponding to the target speech frame, a gain corresponding to the target speech frame, and an excitation signal corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame; Including, The synthesis module comprises: constructing a glottal filter based on glottal parameters corresponding to the target speech frame; obtaining a first speech signal by filtering an excitation signal corresponding to the target speech frame with the glottal filter; and amplifying the first speech signal with a gain corresponding to the target speech frame to obtain an enhanced speech signal corresponding to the target speech frame; Speech enhancement device.

13. An electronic device, A processor; A memory having stored thereon computer readable instructions which, when executed by the processor, implement the speech enhancement method of any one of claims 1 to 11; An electronic device comprising:

14. A program for causing a computer to execute the speech enhancement method according to any one of claims 1 to 11.

Images