Audio signal enhancement method, device, computer device and computer program
The audio signal enhancement method addresses slow processing and limited quality improvement by using sequential decoding, filtering, and a pre-trained model to enhance audio signals quickly and effectively, reducing quantization noise and ensuring timely output.
Patent Information
- Application Number
- JP2023535590
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-30
- Filing Date
- 2022-04-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-15
AI Technical Summary
Conventional audio signal enhancement methods suffer from slow processing speeds, significant delays, and limited voice quality improvement, with poor timeliness in addressing quantization noise during encoding/decoding.
An audio signal enhancement method involving sequential decoding of packets to obtain residual and filtering parameters, followed by feature extraction and conversion to a filtered excitation signal, enhanced using linear and long-term filtering parameters, and finally synthesized into an improved audio signal through a pre-trained signal enhancement model.
The method achieves rapid and effective enhancement of audio signals, improving voice quality by reducing quantization noise and ensuring timely output, even in scenarios with packet errors.
Smart Images

Figure 0007584662000016 
Figure 0007584662000017 
Figure 0007584662000018
Abstract
Description
[Technical field]
[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on April 30, 2021, bearing application number 2021104841966, and entitled "Audio signal enhancement method, apparatus, computer equipment, and storage medium," the entire contents of which are incorporated herein by reference.
[0002] The present application relates to the field of computer technology, and in particular to an audio signal enhancement method, apparatus, computer device, storage medium and computer program product. [Background technology]
[0003] During encoding / decoding, quantization noise is usually introduced into the audio signal, which causes distortion of the decoded and synthesized voice. In conventional methods, post-processing using a pitch filter or a neural network is generally adopted to enhance the audio signal and reduce the impact of quantization noise on the voice quality. Summary of the Invention [Problem to be solved by the invention]
[0004] However, the signal processing speed of the conventional manner is low, there is a large delay, the achievable voice quality improvement effect is limited, and the timeliness of the audio signal enhancement is poor.
[0005] Various embodiments of the present application provide audio signal enhancement methods, apparatus, computer devices, storage media and computer program products. [Means for solving the problem]
[0006] 1. A computer device implemented method for audio signal enhancement, comprising: decoding the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; if the audio signal is a forward error correction frame signal, extracting feature parameters from the audio signal; transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal; performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0007] In one embodiment, the linear filtering parameters include linear filtering coefficients and energy gain values, and the step of parameterizing a linear predictive filter based on the linear filtering parameters and linearly synthesizing filtering the enhanced speech excitation signal with the parameterized linear predictive filter comprises: configuring parameters for a linear prediction filter based on the linear filtering coefficients; obtaining an energy gain value corresponding to a decoded historical voice packet before the voice packet is decoded; determining an energy adjustment parameter based on the energy gain values corresponding to the historical voice packets and the energy gain values corresponding to the voice packets; performing energy adjustment on a historical long-term filtered excitation signal corresponding to the historical voice packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtered excitation signal; inputting the adjusted historical long-term filtered excitation signal and the enhanced speech excitation signal into a parameter configured linear prediction filter, where the linear prediction filter linearly synthetically filters the enhanced speech excitation signal based on the adjusted historical long-term filtered excitation signal.
[0008] 1. An audio signal enhancement device, comprising: a voice packet processing module for sequentially decoding the received voice packets to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; a feature parameter extraction module for extracting feature parameters from the audio signal when the audio signal is a forward error correction frame signal; a signal transformation module for transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; a voice enhancement module for performing voice enhancement processing on the filtered voice excitation signal according to the feature parameters, the long-term filtering parameters, and the linear filtering parameters to obtain an enhanced voice excitation signal; and a speech synthesis module that performs speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0009] A computer device including a memory having a computer program stored therein and a processor, the processor executing the computer program comprising: decoding the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; if the audio signal is a forward error correction frame signal, extracting feature parameters from the audio signal; transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal; performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0010] A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, decoding the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; if the audio signal is a forward error correction frame signal, extracting feature parameters from the audio signal; transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal; performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0011] A computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium, the computer program being configured to cause a processor of a computing device to read the computer instructions from the computer-readable storage medium and execute the computer instructions to: decoding the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; if the audio signal is a forward error correction frame signal, extracting feature parameters from the audio signal; transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal; performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0012] The details of one or more embodiments of the application are set forth in the drawings and description below. Other features and advantages of the application will become apparent from the description, drawings, and claims.
[0013] The drawings described herein provide further understanding of the present application and constitute a part of the present application. The schematic examples and the description thereof are intended to be illustrative and not limiting of the present application. [Brief description of the drawings]
[0014] [Figure 1] FIG. 2 is a schematic diagram of a speech production model using an excitation signal according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an application environment of an audio signal enhancement method according to an embodiment. [Diagram 3] 1 is a flow diagram of a method for enhancing an audio signal according to an embodiment; [Figure 4] FIG. 2 is a schematic flow diagram of audio signal transmission according to one embodiment. [Diagram 5] FIG. 4 is a magnitude frequency response diagram of a long-term prediction filter according to an embodiment. [Figure 6] FIG. 2 is a flow diagram of the steps of decoding and filtering voice packets according to one embodiment; [Figure 7] FIG. 2 is a magnitude frequency response diagram of a long-term inverse filter according to an embodiment. [Figure 8] FIG. 2 is a schematic diagram of a signal enhancement model according to one embodiment. [Figure 9] 4 is a flow diagram of an audio signal enhancement method according to another embodiment; [Figure 10] 4 is a flow diagram of an audio signal enhancement method according to another embodiment; [Figure 11] 1 is a structural block diagram of an audio signal enhancement device according to an embodiment; [Figure 12] FIG. 4 is a structural block diagram of an audio signal enhancing device according to another embodiment. [Figure 13] FIG. 2 is a diagram showing the internal structure of a computer device according to an embodiment. [Figure 14] FIG. 11 is a diagram showing the internal structure of a computer device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in more detail below in combination with drawings and examples. Note that the specific examples described herein are merely for the purpose of interpreting the present application and are not intended to limit the present application.
[0016] Before describing the audio signal enhancement method of the present application, a voice generation model is first described. Referring to the voice generation model by excitation signal in Fig. 1, the physical theory basis of the voice generation model by excitation signal is the human voice generation process, which includes the following steps: (1) In the trachea, a noise-like impulse signal having a certain energy is generated, and this impulse signal corresponds to the excitation signal in the voice production model based on the excitation signal. (2) The impulse signal impacts the human vocal cords, generating periodic opening and closing, which are then amplified by the oral cavity, producing a sound, and the produced sound corresponds to the filter in the model of speech production by the excitation signal.
[0017] In practice, taking into account the characteristics of voice, the filter in the excitation signal-based voice generation model is subdivided into a Long Term Prediction (LTP) filter and a Linear Predictive Coding (LPC) filter, and the LTP filter strengthens the audio signal according to the long-term correlation of voice, and the LPC filter strengthens the audio signal according to the short-term correlation of voice. Specifically, for voiced periodic signals, in the excitation signal-based voice generation model, the excitation signal impacts the LTP filter and the LPC filter respectively, and for unvoiced non-periodic signals, the excitation signal only impacts the LPC filter.
[0018] An aspect according to the embodiment of the present application relates to techniques such as machine learning of artificial intelligence, and will be specifically described by the following embodiment. The present application provides an audio signal enhancement method executed by a computer device, and is specifically applied to the application environment of Fig. 2. The terminal 202 communicates with the server 204 via a network, and receives voice packets transmitted from the server 204 or voice packets transferred from other devices via the server 204, and the server 204 receives the voice packets transmitted from the terminal or voice packets transmitted from other devices. The above-mentioned audio signal enhancement method is applied to the terminal 202 or the server 204. Take the terminal 202 for example, which executes it. The terminal 202 sequentially decodes the received voice packets to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, filters the residual signal to obtain an audio signal, if the audio signal is a forward error correction frame signal, extracts feature parameters from the audio signal, converts the audio signal into a filtered voice excitation signal based on the linear filtering parameters, performs voice enhancement processing on the filtered voice excitation signal based on the feature parameters, the long-term filtering parameter and the linear filtering parameter to obtain an enhanced voice excitation signal, and performs voice synthesis based on the enhanced voice excitation signal and the linear filtering parameter to obtain a voice enhancement signal.
[0019] The terminal 202 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and the server 204 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0020] In one embodiment, a method for enhancing an audio signal is provided, as shown in Fig. 3. Taking the method as an example applied to the computer device (terminal or server) of Fig. 2, the method includes the following steps S302, S304, S306, S308, and S310.
[0021] In S302, the received voice packet is decoded in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and the residual signal is filtered to obtain an audio signal.
[0022] The received voice packets may be voice packets in a scenario where packet loss is prevented by a Forward Error Correction (FEC) technique.
[0023] Forward error correction technology is an error control method in which a signal is encoded according to a certain algorithm before being sent to a transmission channel, and a redundant code that has the characteristics of the signal itself is added. At the receiving end, the received signal is decoded according to the corresponding algorithm to find and correct the error code generated during the transmission process.
[0024] The redundant code is also called redundant information. In the embodiment of the present application, referring to FIG. 4, when the signal transmitting side encodes the audio signal of the current voice frame (abbreviated as current frame), the signal transmitting side encodes the audio signal information of the immediately preceding voice frame (abbreviated as immediately preceding frame) as redundant information into a voice packet corresponding to the audio signal of the current frame, and when the encoding is completed, transmits the voice packet corresponding to the audio signal of the current frame to the receiving side, and the receiving side receives the voice packet. Thus, even if an abnormality occurs during the signal transmission process and the receiving side does not receive a certain voice packet or an error code appears in a certain voice packet, the audio signal corresponding to the lost or error code voice packet can be obtained by decoding the voice packet corresponding to the audio signal of the next voice frame (abbreviated as next frame), thereby improving the reliability of the signal transmission. The receiving side may be the terminal 202 of FIG. 2.
[0025] Specifically, when the terminal receives a voice packet, it stores the received voice packet in a cache, retrieves from the cache a voice packet corresponding to a voice frame to be played back, performs decoding and filtering on the voice packet to obtain an audio signal, and if the voice packet is an adjacent packet of the historical voice packet decoded immediately before and there is no abnormality in the historical voice packet decoded immediately before, it directly outputs the obtained audio signal, or performs audio signal enhancement processing on the audio signal to obtain and output an audio enhancement signal. If the voice packet is not an adjacent packet of the historical voice packet decoded immediately before, or if the voice packet is an adjacent packet of the historical voice packet decoded immediately before but there is an abnormality in the historical voice packet decoded immediately before, it performs audio signal enhancement processing on the audio signal to obtain and output an audio enhancement signal. Here, the audio enhancement signal includes an audio signal corresponding to the adjacent packet of the historical voice packet decoded immediately before.
[0026] Specifically, the decoding may be entropy decoding, which is a decoding mode corresponding to entropy coding. Specifically, when the transmitting side encodes an audio signal, the transmitting side encodes the audio signal according to the entropy coding mode to obtain a voice packet. Thus, when the receiving side receives a voice packet, the receiving side decodes the received voice packet according to the entropy decoding mode.
[0027] In one embodiment, when a terminal receives a voice packet, the terminal decodes the received voice packet to obtain a residual signal and filter parameters, and performs signal synthesis filtering on the residual signal according to the filter parameters to obtain an audio signal, where the filter parameters include a long-term filtering parameter and a linear filtering parameter.
[0028] Specifically, when the transmitting side encodes the audio signal of the current frame, it analyzes the audio signal of the immediately preceding frame to obtain filter parameters, performs parameter configuration for the filter based on the obtained filter parameters, performs analysis filtering on the audio signal of the current frame using the configured filter to obtain a residual signal of the audio signal of the current frame, encodes the audio signal using the residual signal and the analyzed filter parameters to obtain a voice packet, and transmits the voice packet to the receiving side. Thus, when the receiving side receives the voice packet, it performs decoding processing on the received voice packet to obtain the residual signal and filter parameters, and performs signal synthesis filtering on the residual signal based on the filter parameters to obtain the audio signal.
[0029] In one embodiment, the filter parameters include linear filtering parameters and long-term filtering parameters. When encoding an audio signal of a current frame, the transmitting side obtains the linear filtering parameters and the long-term filtering parameters by analyzing an audio signal of a previous frame, linearly analyzes and filters the audio signal of the current frame according to the linear filtering parameters to obtain a linear filtered excitation signal, and long-term analyzes and filters the linear filtered excitation signal according to the long-term filtering parameters to obtain a residual signal corresponding to the audio signal of the current frame. The transmitting side encodes the audio signal of the current frame according to the residual signal, the linear filtering parameters and the long-term filtering parameters based on the analysis, and obtains a voice packet to send to the receiving side.
[0030] Specifically, the step of linearly analyzing and filtering the audio signal of the current frame according to the linear filtering parameters specifically includes: configuring parameters for a linear predictive filter according to the linear filtering parameters; and linearly analyzing and filtering the audio signal through the parameter-configured linear predictive filter to obtain a linear filtering excitation signal, where the linear filtering parameters include a linear filtering coefficient and an energy gain value, the linear filtering coefficient is denoted as LPC AR, the energy gain value is denoted as LPC gain, and the equation of the linear predictive filter is as follows:
[0031]
number
[0032] where e(n) is the linear filtering excitation signal corresponding to the audio signal of the current frame, s(n) is the audio signal of the current frame, p is the number of sampling points contained in the audio signal of each frame, and a i is the linear filtering coefficient obtained by analyzing the audio signal of the previous frame, and s adj(ni) is the energy adjusted state of the audio signal s(ni) of the frame immediately preceding the audio signal s(n) of the current frame, and s adj (ni) is obtained from the following formula:
[0033]
number
[0034] where s(ni) is the audio signal of the frame immediately before the current frame audio signal s(n), and the gain adj is the energy adjustment parameter of the audio signal s(ni) of the previous frame, and gain adj is obtained from the following formula:
[0035]
number
[0036] where gain(n) is the energy gain value corresponding to the audio signal of the current frame, and gain(ni) is the energy gain value corresponding to the audio signal of the immediately preceding frame.
[0037] According to the long-term filtering parameters, the long-term analysis filtering of the linear filtering excitation signal specifically includes: according to the long-term filtering parameters, performing parameter configuration for a long-term prediction filter; and performing long-term analysis filtering of the residual signal by the parameter-configured long-term prediction filter to obtain a corresponding residual signal of the audio signal of the current frame, where the long-term filtering parameters include a pitch period and a corresponding amplitude gain value, the pitch period is denoted as LTP pitch, the corresponding amplitude gain value is denoted as LTP gain, and the frequency domain of the long-term prediction filter is shown as follows, and the frequency domain is denoted as Z domain:
[0038]
number
[0039] In the above equation, p(z) is the magnitude frequency response of the long-term prediction filter, z is the twiddle factor of the frequency domain transform, γ is the magnitude gain value LTP gain, and T is the pitch period LTP pitch. FIG. 5 shows the corresponding magnitude frequency response diagram of the long-term prediction filter when γ=1 and T=80 in one embodiment.
[0040] The time domain of the long term prediction filter is given as follows:
[0041]
number
[0042] where δ(n) is a residual signal corresponding to an audio signal of a current frame, e(n) is a linear filtering excitation signal corresponding to an audio signal of the current frame, γ is an amplitude gain value LTP gain, T is a pitch period LTP pitch, and e(nT) is a linear filtering excitation signal corresponding to an audio signal of a pitch period immediately preceding the audio signal of the current frame.
[0043] In one embodiment, the filter parameters obtained by the terminal performing decoding include long-term filtering parameters and linear filtering parameters, and the signal synthesis filtering includes long-term synthesis filtering according to the long-term filtering parameters and linear synthesis filtering according to the linear filtering parameters. After the terminal decodes the voice packet to obtain a residual signal, the long-term filtering parameters and the linear filtering parameters, the terminal performs long-term synthesis filtering on the residual signal according to the long-term filtering parameters to obtain a long-term filtered excitation signal, and then performs linear synthesis filtering on the long-term filtered excitation signal according to the linear filtering parameters to obtain an audio signal.
[0044] In one embodiment, after obtaining a residual signal, the terminal divides the obtained residual signal into multiple subframes to obtain multiple sub-residual signals, and performs long-term synthesis filtering on each sub-residual signal according to a corresponding long-term filtering parameter to obtain a long-term filtered excitation signal corresponding to each subframe, and then combines the long-term filtered excitation signals corresponding to each subframe according to the sequence of each subframe to obtain a corresponding long-term filtered excitation signal.
[0045] For example, if one voice packet corresponds to a 20 ms audio signal, that is, the obtained residual signal is 20 ms, the residual signal is divided into four subframes to obtain four 5 ms sub-residual signals, and long-term synthesis filtering is performed on each 5 ms sub-residual signal according to its corresponding long-term filtering parameter to obtain four 5 ms long-term filtered excitation signals, and then the four 5 ms long-term filtered excitation signals are combined according to the sequence of each subframe to obtain a 20 ms long-term filtered excitation signal.
[0046] In one embodiment, after obtaining a long-term filtered excitation signal, the terminal divides the obtained long-term filtered excitation signal into multiple subframes to obtain multiple sub-long-term filtered excitation signals, and then performs linear synthesis filtering on each sub-long-term filtered excitation signal according to a corresponding linear filtering parameter to obtain a sub-linear filtered excitation signal corresponding to each subframe, and then combines the linear filtered excitation signals corresponding to each subframe according to the sequence of each subframe to obtain a corresponding linear filtered excitation signal.
[0047] For example, if one voice packet corresponds to a 20 ms audio signal, that is, the obtained long-term filtered excitation signal is 20 ms, the long-term filtered excitation signal is divided into two sub-frames to obtain two 10 ms sub-long-term filtered excitation signals, and linear synthesis filtering is performed on each 10 ms sub-long-term filtered excitation signal according to corresponding linear filtering parameters to obtain two 10 ms sub-audio signals, and then the two 10 ms sub-audio signals are combined according to the sequence of each sub-frame to obtain a 20 ms audio signal.
[0048] In S304, if the audio signal is a forward error correction frame signal, feature parameters are extracted from the audio signal.
[0049] The audio signal being a forward error correction frame signal means that an abnormality exists in the audio signal of the historical adjacent frame of the audio signal, and the abnormality exists in the audio signal of the historical adjacent frame specifically includes that a voice packet corresponding to the audio signal of the historical adjacent frame is not received, or that a voice packet corresponding to the received audio signal of the historical adjacent frame is not normally decoded. The feature parameters include cepstrum feature parameters.
[0050] In one embodiment, the terminal decodes and filters a received voice packet to obtain an audio signal, and then determines whether there is a data abnormality in the decoded historical voice packet before the voice packet is decoded. If there is a data abnormality in the decoded historical voice packet, it determines that the decoded and filtered current audio signal is a forward error correction frame signal.
[0051] Specifically, the terminal determines whether or not the historical audio signal corresponding to the historical voice packet decoded at the timing immediately before decoding the voice packet is the audio signal of the frame immediately before the audio signal obtained by decoding the voice packet, and if YES, determines that there is no data abnormality in the historical voice packet, and if NO, determines that there is a data abnormality in the historical voice packet.
[0052] In this embodiment, the terminal determines whether the current decoded and filtered audio signal is a forward error correction frame signal by determining whether there is a data abnormality in the historical voice packet decoded before the current voice packet is decoded, and if the audio signal is a forward error correction frame signal, performs enhancement processing on the audio signal to further improve the quality of the audio signal.
[0053] In one embodiment, when the decoded audio signal is a forward error correction frame signal, extract feature parameters from the decoded audio signal, and the extracted feature parameters may be cepstral feature parameters. Specifically, the method includes: performing Fourier transform on the audio signal to obtain a Fourier transformed audio signal; performing logarithmic processing on the Fourier transformed audio signal to obtain a logarithmic result; and performing inverse Fourier transform on the obtained logarithmic result to obtain cepstral feature parameters. Specifically, the following formula is used to realize the extraction of cepstral feature parameters from the audio signal:
[0054]
number
[0055] where C(n) is the cepstral feature parameter of the decoded and filtered audio signal S(n), and S(F) is the Fourier transformed audio signal obtained by Fourier transforming the audio signal S(n).
[0056] In the above embodiment, the terminal extracts cepstral feature parameters from the audio signal, and enhances the audio signal based on the extracted cepstral feature parameters to improve the quality of the audio signal.
[0057] In one embodiment, if the audio signal is not a forward error correction frame signal, i.e., if there is no abnormality in the audio signal of the frame immediately preceding the current decoded and filtered audio signal, feature parameters may be extracted from the current decoded and filtered audio signal to perform audio signal enhancement processing on the current decoded and filtered audio signal.
[0058] At S306, the audio signal is transformed into a filtered audio excitation signal based on the linear filtering parameters.
[0059] Specifically, after the terminal decodes and filters the voice packet to obtain an audio signal, the terminal may further obtain a linear filtering parameter obtained by decoding the voice packet, and perform linear analysis filtering on the obtained audio signal according to the linear filtering parameter, thereby realizing the conversion of the audio signal into a filtered voice excitation signal.
[0060] In one embodiment, S306 specifically includes: performing parameter configuration for a linear prediction filter according to the linear filtering parameters; and linearly analyzing and filtering the audio signal through the parameter-configured linear prediction filter to obtain a filtered audio excitation signal.
[0061] In the embodiment of the present application, when performing linear analysis filtering on an audio signal, it is not necessary to perform subframe division processing on the audio signal of the entire frame, and linear analysis filtering is directly performed on the audio signal of the entire frame.
[0062] Specifically, the terminal performs linear analysis filtering on the audio signal according to the following formula to obtain a filtered voice excitation signal:
[0063]
number
[0064] where D(n) is a filtered audio excitation signal corresponding to an audio signal S(n) obtained by decoding and filtering audio packets, and S(n) is an audio signal obtained by decoding and filtering audio packets, and S adj (ni) is the energy adjusted state of the audio signal S(ni) of the frame immediately preceding the acquired audio signal S(n), p is the number of sampling points included in the audio signal of each frame, and A i is the linear filtering coefficient obtained by decoding the voice packet.
[0065] In the above embodiment, the terminal converts the audio signal into a filtered voice excitation signal according to the linear filtering parameters, and enhances the filtered voice excitation signal to realize enhancement of the audio signal, and improve the quality of the audio signal.
[0066] In S308, the filtered speech excitation signal is speech enhanced according to the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal.
[0067] The long-term filtering parameters include the pitch period and the amplitude gain value.
[0068] In one embodiment, S308 includes performing speech enhancement processing on the filtered speech excitation signal based on the pitch period, the amplitude gain value, the linear filtering parameters and the cepstral feature parameters to obtain an enhanced speech excitation signal.
[0069] Specifically, the step of processing the audio signal for voice enhancement is specifically realized by a pre-trained signal enhancement model, which is a Neural Network (NN) model, and the neural network model can be specifically an LSTM and CNN level structure.
[0070] In the above embodiment, the terminal performs voice enhancement processing on the filtered voice excitation signal according to the pitch period, the amplitude gain value, the linear filtering parameters and the cepstrum feature parameters, obtains an enhanced voice excitation signal, and realizes audio signal enhancement according to the enhanced voice excitation signal, so as to improve the quality of the audio signal.
[0071] In one embodiment, the terminal inputs the acquired feature parameters, the long-term filtering parameters, the linear filtering parameters, and the filtered voice excitation signal into a pre-trained signal enhancement model, so that the signal enhancement model performs voice enhancement processing on the filtered voice excitation signal based on the feature parameters, and obtains an enhanced voice excitation signal.
[0072] In the above embodiment, the terminal realizes enhancement of the voice excitation signal through a pre-trained signal enhancement model, and realizes enhancement of the audio signal based on the enhanced voice excitation signal, so as to improve the quality of the audio signal and the efficiency of the audio signal enhancement processing.
[0073] In addition, in the embodiment of the present application, in the process of performing speech enhancement processing on the filtered speech excitation signal by the pre-trained signal enhancement model, there is no need to perform subframe division processing on the filtered speech excitation signal of the entire frame, and speech enhancement processing is performed on the filtered speech excitation signal of the entire frame.
[0074] In S310, speech synthesis is performed based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0075] The speech synthesis may be linear synthesis filtering performed based on linear filtering parameters.
[0076] In one embodiment, after obtaining an enhanced speech excitation signal, the terminal performs parameter configuration on a linear prediction filter based on the linear filtering parameters, and linearly synthesizes and filters the enhanced speech excitation signal by the parameter-configured linear prediction filter to obtain a speech enhancement signal.
[0077] The linear filtering parameters include linear filtering coefficients and energy gain values, where the linear filtering coefficients are denoted as LPC AR and the energy gain values are denoted as LPC gain. Since linear synthesis filtering is the inverse process of linear analysis filtering performed when the transmitting side encodes an audio signal, the linear predictive filter that performs linear synthesis filtering is also called a linear inverse filter, and the time domain of the linear predictive filter is expressed as follows:
[0078]
number
[0079] However, S enh (n) is the voice reinforcement signal, and D enh (n) is the enhanced speech excitation signal obtained by performing speech enhancement processing on the filtered speech excitation signal D(n), and S adj (ni) is the energy adjusted state of the audio signal S(ni) of the frame immediately preceding the acquired audio signal S(n), p is the number of sampling points included in the audio signal of each frame, and A i is the linear filtering coefficient obtained by decoding the voice packet.
[0080] The energy-adjusted state of the audio signal S(ni) of the frame immediately preceding the audio signal S(n) is S adj (ni) is obtained from the following formula:
[0081]
number
[0082] In the above formula, S adj (ni) is the state after energy adjustment of the audio signal S(ni) of the previous frame, and gain adj is the energy adjustment parameter of the audio signal S(ni) of the previous frame.
[0083] In this embodiment, the terminal obtains a voice enhancement signal by performing linear synthesis filtering on the enhanced voice excitation signal, that is, realizes enhancement processing on the audio signal, and improves the quality of the audio signal.
[0084] It should be noted that in the embodiment of the present application, the process of speech synthesis does not need to perform subframe division processing on the enhanced speech excitation signal of the entire frame, but performs speech synthesis on the enhanced speech excitation signal of the entire frame.
[0085] According to the above audio signal enhancement method, when a terminal receives a voice packet, the terminal sequentially performs decoding and filtering on the voice packet to obtain an audio signal, and when the audio signal is a forward error correction frame signal, the terminal extracts feature parameters from the audio signal, and converts the audio signal into a filter voice excitation signal according to the linear filtering coefficient obtained by decoding the voice packet, and performs voice enhancement processing on the filter voice excitation signal according to the feature parameters and the long-term filtering parameter obtained by decoding the voice packet to obtain an enhanced voice excitation signal, and performs voice synthesis according to the enhanced voice excitation signal and the linear filtering parameter to obtain an enhanced voice signal, thereby completing the audio signal enhancement processing within a short time, achieving a good signal enhancement effect, and improving the timeliness of the audio signal enhancement.
[0086] In one embodiment, as shown in FIG. 6, S302 specifically includes the following steps S602 and S604.
[0087] In S602, a long-term prediction filter is parameterized according to the long-term filtering parameters, and a residual signal is long-term synthesis filtered by the parameterized long-term prediction filter to obtain a long-term filtered excitation signal.
[0088] The long-term filtering parameters include a pitch period and a corresponding amplitude gain value, the pitch period is denoted as LTP pitch, the LTP pitch is also called pitch period, and the corresponding amplitude gain value is denoted as LTP gain, and the residual signal is long-term synthesis filtered by the parameter-configured long-term prediction filter. Since the long-term synthesis filtering is the inverse process of the long-term analysis filtering performed when the transmitting side encodes the audio signal, the long-term prediction filter performing the long-term synthesis filtering is also called a long-term inverse filter, that is, the long-term inverse filter is adopted to process the residual signal, and the frequency domain of the long-term inverse filter corresponding to the formula (1) is shown as follows:
[0089]
number
[0090] However, p -1 (z) is the magnitude frequency response of the long-term inverse filter, z is the twiddle factor of the frequency domain transform, γ is the magnitude gain value LTP gain, and T is the pitch period LTP pitch. FIG. 7 shows the magnitude frequency response diagram of the corresponding long-term inverse prediction filter when γ=1 and T=80 in one embodiment.
[0091] The time domain of the long-term inverse filter corresponding to equation (10) is given as follows:
[0092]
number
[0093] In the above formula, E(n) is the long-term filtering excitation signal corresponding to the voice packet, δ(n) is the residual signal corresponding to the voice packet, γ is the amplitude gain value LTP gain, T is the pitch period LTP pitch, and E(nT) is the long-term filtering excitation signal corresponding to the audio signal of the pitch period immediately before the voice packet. Here, in this embodiment, the long-term filtering excitation signal E(n) obtained by the receiving side performing long-term synthesis filtering on the residual signal by the long-term inverse filter is the same as the linear filtering excitation signal e(n) obtained by the transmitting side performing linear analysis filtering on the audio signal by the linear filter when encoding.
[0094] In S604, a linear prediction filter is parameterized based on the linear filtering parameters, and the long-term filtering excitation signal is linearly synthesized filtered by the parameterized linear prediction filter to obtain an audio signal.
[0095] The linear filtering parameters include linear filtering coefficients and energy gain values, where the linear filtering coefficients are denoted as LPC AR and the energy gain values are denoted as LPC gain. Since linear synthesis filtering is the inverse process of linear analysis filtering performed when the transmitting side encodes an audio signal, the linear predictive filter that performs linear synthesis filtering is also called a linear inverse filter, and the time domain of the linear predictive filter is expressed as follows:
[0096]
number
[0097] In the above equation, S(n) is the audio signal corresponding to the voice packet, E(n) is the long-term filtering excitation signal corresponding to the voice packet, and S adj(ni) is the energy adjusted state of the audio signal S(ni) of the frame immediately preceding the acquired audio signal S(n), p is the number of sampling points included in the audio signal of each frame, and A i is the linear filtering coefficient obtained by decoding the voice packet.
[0098] The energy-adjusted state of the audio signal S(ni) of the frame immediately preceding the audio signal S(n) is S adj (ni) is obtained from the following formula:
[0099]
number
[0100] However, gain adj is the energy adjustment parameter of the audio signal S(ni) of the previous frame, gain(n) is the energy gain value obtained by decoding the voice packet, and gain(ni) is the energy gain value corresponding to the audio signal of the previous frame.
[0101] In the above embodiment, the terminal performs long-term synthesis filtering on the residual signal according to the long-term filtering parameters to obtain a long-term filtered excitation signal, and performs linear synthesis filtering on the long-term filtered excitation signal according to the decoded linear filtering parameters to obtain an audio signal, so that if the audio signal is not a forward error correction frame signal, the audio signal is directly output, and if the audio signal is a forward error correction frame signal, the audio signal is enhanced before being output, thereby improving the timeliness of the audio signal output.
[0102] In one embodiment, S604 specifically includes: dividing the long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal; dividing the decoded linear filtering parameters into groups to obtain at least two linear filtering parameter sets; respectively performing parameter configuration for at least two linear predictive filters according to the linear filtering parameter sets; respectively inputting the obtained sub-long-term filtered excitation signals into the parameter-configured linear predictive filters, and the linear predictive filters linearly synthesize and filter the sub-long-term filtered excitation signals according to the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe; and combining the sub-audio signals according to the sequence of each subframe to obtain an audio signal.
[0103] The linear filtering parameter set has two types: a linear filtering coefficient set and an energy gain value set.
[0104] Specifically, when the linear synthesis filtering is performed on the sub long-term filtering excitation signal corresponding to each subframe by the linear inverse filter corresponding to equation (12), S(n) in equation (12) is the sub audio signal corresponding to any one subframe, E(n) is the long-term filtering excitation signal corresponding to the subframe, and S adj (ni) is the energy adjusted state of the sub-audio signal S(ni) of the sub-frame immediately preceding the acquired sub-audio signal S(n), p is the number of sampling points included in the audio signal of each sub-frame, and A i is the set of linear filtering coefficients corresponding to the subframe, and the gain adjis an energy adjustment parameter of a sub-audio signal of a subframe immediately preceding the sub-audio signal, gain(n) is an energy gain value of the sub-audio signal, and gain(ni) is an energy gain value of a sub-audio signal of a subframe immediately preceding the sub-audio signal.
[0105] In the above embodiment, the terminal divides the long-term filtering excitation signal into at least two subframes to obtain a sub-long-term filtering excitation signal, divides the decoded linear filtering parameters into groups to obtain at least two linear filtering parameter sets, and performs parameter configuration for at least two linear predictive filters according to the linear filtering parameter sets, respectively inputs the obtained sub-long-term filtering excitation signals into the parameter-configured linear predictive filters, and the linear predictive filters perform linear synthesis filtering on the sub-long-term filtering excitation signal according to the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe, and combines the sub-audio signals according to the sequence of each subframe to obtain an audio signal, thereby ensuring that the obtained audio signal can well restore the audio signal transmitted by the transmitting side, and improving the quality of the restored audio signal.
[0106] In one embodiment, the linear filtering parameters include linear filtering coefficients and energy gain values, and S604 further includes: for a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, obtaining an energy gain value of a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal; determining an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal based on the energy gain value corresponding to the historical sub-long-term filtered excitation signal and the energy gain value of the sub-long-term filtered excitation signal corresponding to the first subframe; and performing energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter to obtain an energy adjusted historical sub-long-term filtered excitation signal.
[0107] The historical long-term filtered excitation signal is the long-term filtered excitation signal of the frame immediately preceding the long-term filtered excitation signal of the current frame, and the historical sub-long-term filtered excitation signal of the subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal is the sub-long-term filtered excitation signal corresponding to the last subframe of the long-term filtered excitation signal of the immediately preceding frame.
[0108] For example, when a long-term filtered excitation signal of a current frame is divided into two subframes to obtain a sub-long-term filtered excitation signal corresponding to a first subframe and a sub-long-term filtered excitation signal corresponding to a second subframe, the sub-long-term filtered excitation signal corresponding to the second subframe of the long-term filtered excitation signal of the previous frame and the sub-long-term filtered excitation signal corresponding to the first subframe of the current frame are adjacent subframes.
[0109] In one embodiment, after the terminal obtains an energy-adjusted historical sub-long-term filtered excitation signal, the terminal inputs the obtained sub-long-term filtered excitation signal and the energy-adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe based on the linear filtering coefficients and the energy-adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to the first subframe.
[0110] For example, if one voice packet corresponds to a 20 ms audio signal, i.e., the acquired long-term filtering excitation signal is 20 ms, the AR coefficients obtained by decoding the voice packet are {A1, A2, …, A p-1 , A p , A p+1 , … A 2p-1 , A 2p}, the energy gain value obtained by decoding the voice packet is {gain1(n), gain2(n)}, the long-term filtering excitation signal is divided into two subframes to obtain a first sub-filtering excitation signal E1(n) corresponding to the previous 10 ms and a second sub-filtering excitation signal E2(n) corresponding to the next 10 ms, and the AR coefficients are divided into groups to obtain an AR coefficient set 1{A1, A2, …, A p-1 , A p} and AR coefficient set 2{A p+1 , … A 2p-1 , A 2p}, and divide the energy gain values into groups to obtain an energy gain value set 1 {gain1(n)} and an energy gain value set 2 {gain2(n)}, so that the sub-filtered excitation signal of the sub-frame immediately preceding the first sub-filtered excitation signal E1(n) is E2(ni), the energy gain value set of the sub-frame immediately preceding the first sub-filtered excitation signal E1(n) is {gain2(ni)}, and the energy gain value set of the sub-frame immediately preceding the second sub-filtered excitation signal E2(n) is {gain2(ni)}. The sub-filtering excitation signal is E1(n), and the energy gain value set of the subframe immediately preceding the second sub-filtering excitation signal E2(n) is {gain1(n)}, then the sub-audio signal corresponding to the first sub-filtering excitation signal E1(n) is obtained by substituting the corresponding parameters into equations (12) and (13), and the sub-audio signal corresponding to the second sub-filtering excitation signal E2(n) is obtained by substituting the corresponding parameters into equations (12) and (13).
[0111] In the above embodiment, the terminal obtains, for a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, an energy gain value of a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal, and determines an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal according to the energy gain value corresponding to the historical sub-long-term filtered excitation signal and the energy gain value of the sub-long-term filtered excitation signal corresponding to the first subframe, and performs energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter, inputs the obtained sub-long-term filtered excitation signal and the energy adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe according to the linear filtering coefficient and the energy adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to the first subframe. This ensures that the obtained audio signal of each subframe can well restore the audio signal of each subframe transmitted by the transmitting side, and improves the quality of the restored audio signal.
[0112] In one embodiment, the feature parameters include cepstral feature parameters, and S308 includes the steps of vectorizing the cepstral feature parameters, the long-term filtering parameters and the linear filtering parameters, and stitching the result of the vectorization process to obtain a feature vector; inputting the feature vector and the filtered speech excitation signal into a pre-trained signal enhancement model; performing feature extraction on the feature vector by the signal enhancement model to obtain a target feature vector; and enhancing the filtered speech excitation signal based on the target feature vector to obtain an enhanced speech excitation signal.
[0113] The signal enhancement model is a multi-level network structure, specifically including a first feature stitching layer, a second feature stitching layer, a first neural network layer and a second neural network layer. The target feature vector is the enhanced feature vector.
[0114] Specifically, the terminal vectorizes the cepstral feature parameters, the long-term filtering parameters, and the linear filtering parameters through a first feature stitching layer of the signal enhancement model, stitches the result of the vectorization process to obtain a feature vector, inputs the obtained feature vector into a first neural network layer of the signal enhancement model, performs feature extraction on the feature vector through the first neural network layer to obtain a primary feature vector, inputs envelope information obtained by performing a Fourier transform on the primary feature vector and the linear filtering coefficients in the linear filtering parameters into a second feature stitching layer of the signal enhancement model to obtain a stitched primary feature vector, inputs the stitched primary feature vector into a second neural network layer of the signal enhancement model, performs feature extraction on the stitched primary feature vector by the second neural network layer to obtain a target feature vector, and enhances the filtered voice excitation signal according to the target feature vector to obtain an enhanced voice excitation signal.
[0115] In the above embodiment, the terminal performs vectorization processing on the cepstral feature parameters, the long-term filtering parameters and the linear filtering parameters, stitches the result of the vectorization processing to obtain a feature vector, inputs the feature vector and the filter voice excitation signal into a pre-trained signal enhancement model, performs feature extraction on the feature vector by the signal enhancement model to obtain a target feature vector, and enhances the filter voice excitation signal according to the target feature vector to obtain an enhanced voice excitation signal, thereby realizing the enhancement processing on the audio signal by the signal enhancement model, and improving the quality of the audio signal and the efficiency of the enhancement processing of the audio signal.
[0116] In one embodiment, the step of the terminal enhancing the filtered speech excitation signal based on the target feature vector to obtain an enhanced speech excitation signal includes the steps of: performing a Fourier transform on the filtered speech excitation signal to obtain a frequency-domain speech excitation signal; enhancing an amplitude value feature of the frequency-domain speech excitation signal based on the target feature vector; and performing an inverse Fourier transform on the frequency-domain speech excitation signal with the enhanced amplitude value feature to obtain an enhanced speech excitation signal.
[0117] Specifically, the terminal performs a Fourier transform on the filtered voice excitation signal to obtain a frequency-domain voice excitation signal, enhances the amplitude value feature of the frequency-domain voice excitation signal according to the target feature vector, and then combines the phase feature of the unenhanced frequency-domain voice excitation signal to perform an inverse Fourier transform on the frequency-domain voice excitation signal with the enhanced amplitude value feature to obtain an enhanced voice excitation signal.
[0118] As shown in FIG. 8, the two feature stitching layers are concat1 and concat2, and the two neural network layers are NN part1 and NN part2. The cepstrum feature parameter Cepstrum with a dimension of 40, the pitch period LTP pitch with a dimension of 1, and the amplitude gain value LTP Gain with a dimension of 1 are stitched by concat1 to form a feature vector with a dimension of 42, and the feature vector with a dimension of 42 is input to NN part1. NN part1 is composed of a two-layer convolutional neural network and a two-layer fully connected network, and the dimensions of the convolutional core of the first layer are (1, 128, 3, 1), the dimensions of the convolutional core of the second layer are (128, 128, 3, 1), the number of nodes of the fully connected layer are 128 and 8, and the activation function of the end of each layer is a Tanh function. NN Extract high-level features from the feature vector through part1 to obtain a primary feature vector with a dimension of 1024, then stitch the primary feature vector with a dimension of 1024 and the envelope information Envelope with a dimension of 161 obtained by performing a Fourier transform on the linear filtering coefficient LPC AR in the linear filtering parameter through concat2 to obtain a stitched primary feature vector with a dimension of 1185, and input the stitched primary feature vector with a dimension of 1185 into NN part2, which is a two-layer fully connected network, the number of nodes is 256 and 161 respectively, and the activation function at the end of each layer is a Tanh function, obtain a target feature vector through NN part2, and then enhance the amplitude value feature Excitation of the frequency domain speech excitation signal obtained by performing a Fourier transform on the filter speech excitation signal according to the target feature vector, and perform an inverse Fourier transform on the filter speech excitation signal with the enhanced amplitude value feature Excitation to obtain an enhanced speech excitation signal D enh Get (n).
[0119] In the above embodiment, the terminal performs Fourier transform on the filtered audio excitation signal to obtain a frequency-domain audio excitation signal, enhances the amplitude value feature of the frequency-domain audio excitation signal according to the target feature vector, and performs inverse Fourier transform on the frequency-domain audio excitation signal with the enhanced amplitude value feature to obtain an enhanced audio excitation signal, thereby realizing audio signal enhancement processing and improving the quality of the audio signal while ensuring that the phase information of the audio signal does not change.
[0120] In one embodiment, the linear filtering parameters include linear filtering coefficients and energy gain values, and the terminal performs parameter configuration on the linear predictive filter based on the linear filtering parameters, and the step of linearly synthesizing filtering the enhanced speech excitation signal by the parameter-configured linear predictive filter includes: performing parameter configuration on the linear predictive filter based on the linear filtering coefficients; obtaining an energy gain value corresponding to a historical speech packet decoded before the speech packet is decoded; determining an energy adjustment parameter based on the energy gain value corresponding to the historical speech packet and the energy gain value corresponding to the speech packet; performing energy adjustment on the historical long-term filtering excitation signal corresponding to the historical speech packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtering excitation signal; inputting the adjusted historical long-term filtering excitation signal and the enhanced speech excitation signal into the parameter-configured linear predictive filter, and the linear predictive filter linearly synthesizes filtering the enhanced speech excitation signal based on the adjusted historical long-term filtering excitation signal.
[0121] The historical audio signal corresponding to the historical voice packet is the audio signal of the frame immediately before the audio signal of the current frame corresponding to the current voice packet, and the energy gain value corresponding to the historical voice packet may be the energy gain value corresponding to the audio signal of the entire frame of the historical voice, or may be the energy gain value corresponding to the audio signal of some subframes of the historical voice packet.
[0122] Specifically, if the audio signal is not a forward error correction frame signal, i.e., if the audio signal of the frame immediately preceding the audio signal of the current frame is obtained by the terminal by successfully decoding the historical voice packet, the terminal obtains an energy gain value of the historical voice packet obtained by decoding the historical voice packet, and determines an energy adjustment parameter based on the energy gain value of the historical voice packet; if the audio signal is a forward error correction frame signal, i.e., if the audio signal of the frame immediately preceding the audio signal of the current frame is not obtained by the terminal by successfully decoding the historical voice packet, a compensation energy gain value corresponding to the audio signal of the immediately preceding frame is determined based on a predetermined energy gain compensation mechanism, the compensation energy gain value is determined as the energy gain value of the historical voice packet, and the energy adjustment parameter is determined based on the energy gain value of the historical voice packet.
[0123] In one embodiment, if the audio signal is not a forward error correction frame signal, the energy adjustment parameter gain of the audio signal S(ni) of the previous frame is adj is calculated and obtained by the following formula:
[0124]
number
[0125] However, gain adjis the energy adjustment parameter of the audio signal S(ni) of the previous frame, gain(ni) is the energy gain value of the audio signal S(ni) of the previous frame, and gain(n) is the energy gain value of the audio signal of the current frame. Equation (14) calculates the energy adjustment parameter based on the energy gain values corresponding to the audio signals of the entire frames of the historical speech.
[0126] In one embodiment, if the audio signal is not a forward error correction frame signal, the energy adjustment parameter gain of the audio signal S(ni) of the previous frame is adj is obtained from the following formula:
[0127]
number
[0128] However, gain adj is the energy adjustment parameter of the audio signal S(ni) of the previous frame, and gain m (ni) is the energy gain value of the mth subframe of the audio signal S(ni) of the previous frame, and gain m (n) is the energy gain value of the mth subframe of the audio signal of the current frame, m is the number of subframes corresponding to each audio signal, and {gain1(n)+...+gain(n)} / m is the energy gain value of the audio signal of the current frame. Equation (15) calculates the energy adjustment parameter based on the energy gain values corresponding to the audio signals of some subframes of the historical speech.
[0129] In the above embodiment, the terminal performs parameter configuration for the linear prediction filter based on the linear filtering coefficient, obtains an energy gain value corresponding to the decoded historical voice packet before the voice packet is decoded, determines an energy adjustment parameter based on the energy gain value corresponding to the historical voice packet and the energy gain value corresponding to the voice packet, performs energy adjustment on the historical long-term filtering excitation signal corresponding to the historical voice packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtering excitation signal, and inputs the adjusted historical long-term filtering excitation signal and the enhanced voice excitation signal into the parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the enhanced voice excitation signal based on the adjusted historical long-term filtering excitation signal to smooth the audio signal between different frames and improve the quality of the voice composed of the audio signals of different frames.
[0130] In one embodiment, as shown in FIG. 9, an audio signal enhancement method is provided. The method is applied to the computer device (terminal or server) of FIG. 2 as an example. S902: decoding the voice packet to obtain a residual signal, a long-term filtering parameter, and a linear filtering parameter; S904: configure a parameter for a long-term prediction filter according to the long-term filtering parameters, and perform long-term synthesis filtering on the residual signal through the parameter-configured long-term prediction filter to obtain a long-term filtered excitation signal; S906: dividing the long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal; S908, dividing the linear filtering parameters into groups to obtain at least two linear filtering parameter sets; S910: configuring parameters for at least two linear prediction filters based on the linear filtering parameter set; S912: respectively inputting the obtained sub-long-term filtered excitation signals to parameter-configured linear prediction filters, and the linear prediction filters linearly synthesize and filter the sub-long-term filtered excitation signals according to a set of linear filtering parameters to obtain sub-audio signals corresponding to each subframe; S914 combining the sub-audio signals according to the sequence of each subframe to obtain an audio signal; S916, determining whether there is a data anomaly in the decoded historical voice packet before the voice packet is decoded; S918, if there is a data anomaly in the historical voice packet, determining that the decoded and filtered audio signal is a forward error correction frame signal; S920, if the audio signal is a forward error correction frame signal, performing a Fourier transform on the audio signal to obtain a Fourier transformed audio signal, performing a logarithmic process on the Fourier transformed audio signal to obtain a logarithmic result, and performing an inverse Fourier transform on the logarithmic result to obtain a cepstral feature parameter; S922: parameterizing a linear prediction filter based on the linear filtering parameters, and linearly analyzing and filtering the audio signal through the parameterized linear prediction filter to obtain a filtered audio excitation signal; S924: inputting the feature parameters, the long-term filtering parameters, the linear filtering parameters, and the filtered speech excitation signal into a pre-trained signal enhancement model, so that the signal enhancement model performs speech enhancement processing on the filtered speech excitation signal based on the feature parameters to obtain an enhanced speech excitation signal; and parameterizing a linear prediction filter based on the linear filtering parameters, and linearly synthesis filtering the enhanced speech excitation signal with the parameterized linear prediction filter to obtain an speech enhancement signal S926.
[0131] The present application further provides application scenarios in which the above audio signal enhancement method is applied. Specifically, the application of the audio signal enhancement method in the application scenarios is as follows:
[0132] Although a wideband signal with Fs of 16000 Hz is taken as an example, the present application also applies to other sampling rate scenarios, for example, Fs may be 8000 Hz, 32000 Hz, or 48000 Hz. When the frame length of an audio signal is set to 20 ms and Fs=16000 Hz, each frame contains 320 sample points. Referring to FIG. 10, when a terminal receives a voice packet corresponding to an audio signal of one frame, the terminal performs entropy decoding on the voice packet to obtain δ(n), LTP pitch, LTP gain, LPC AR and LPC gain, performs LTP synthesis filtering on δ(n) according to the LTP pitch and LTP gain to obtain E(n), performs LPC synthesis filtering on each subframe of E(n) according to the LPC AR and LPC gain, respectively, combines the LPC synthesis filtering results to obtain one frame S(n), and then performs cepstrum analysis on S(n) to obtain C(n), performs LPC analysis filtering on the entire frame S(n) according to the LPC AR and LPC gain to obtain D(n) of the entire frame, inputs the Fourier transformed envelope information of the LTP pitch, LTP gain and LPC AR, C(n) and D(n) into a pre-trained signal enhancement model NN postfilter, enhances D(n) of the entire frame through the NN postfilter, and obtains D(n) of the entire frame. enh (n) is obtained, and the D of the entire frame is calculated based on the LPC AR and LPC gain. enh (n) is filtered by LPC synthesis to obtain S enh Get (n).
[0133] Note that, although each step in the flowcharts of Figures 3, 4, 6, 9, and 10 is shown in order according to the direction of the arrows, these steps are not necessarily performed in the order shown by the arrows. Unless explicitly described in this specification, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Furthermore, at least some of the steps in Figures 3, 4, 6, 9, and 10 may include multiple steps or multiple stages, and these steps or stages are not necessarily performed and completed at the same time, but may be performed at different times, and the order of execution of these steps or stages is not necessarily sequential, and they may be performed in order or alternately with other steps or at least some of the steps or stages in other steps.
[0134] In one embodiment, as shown in Fig. 11, an audio signal enhancing device is provided. The device is a part of a computer device, which is a software module or a hardware module, or a combination of both. The device specifically includes: a voice packet processing module 1102 for decoding and filtering the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; a feature parameter extraction module 1104 for extracting feature parameters from the audio signal if the audio signal is a forward error correction frame signal; a signal transformation module 1106 for transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; a speech enhancement module 1108 for performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters, and the linear filtering parameters to obtain an enhanced speech excitation signal; and a speech synthesis module 1110 for performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
[0135] In the above embodiment, the computer device sequentially decodes the received voice packet to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, filters the residual signal to obtain an audio signal, and when the audio signal is a forward error correction frame signal, extracts feature parameters from the audio signal, converts the audio signal into a filter voice excitation signal according to the linear filtering coefficient obtained by decoding the voice packet, performs voice enhancement processing on the filter voice excitation signal according to the feature parameters and the long-term filtering parameter obtained by decoding the voice packet to obtain an enhanced voice excitation signal, and performs voice synthesis according to the enhanced voice excitation signal and the linear filtering parameter to obtain an enhanced voice signal, thereby completing the enhancement processing of the audio signal within a short time, achieving a good signal enhancement effect, and improving the timeliness of the audio signal enhancement.
[0136] In one embodiment, the voice packet processing module 1102 further performs parameter configuration on a long-term prediction filter based on the long-term filtering parameters, long-term synthesis filtering of the residual signal by the parameter-configured long-term prediction filter to obtain a long-term filtered excitation signal, and performs parameter configuration on a linear prediction filter based on the linear filtering parameters, and linear synthesis filtering of the long-term filtered excitation signal by the parameter-configured linear prediction filter to obtain an audio signal.
[0137] In the above embodiment, the terminal performs long-term synthesis filtering on the residual signal according to the long-term filtering parameters to obtain a long-term filtered excitation signal, and performs linear synthesis filtering on the long-term filtered excitation signal according to the decoded linear filtering parameters to obtain an audio signal, so that if the audio signal is not a forward error correction frame signal, the audio signal is directly output, and if the audio signal is a forward error correction frame signal, the audio signal is enhanced before being output, thereby improving the timeliness of the audio signal output.
[0138] In one embodiment, the voice packet processing module 1102 further divides the long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal, divides the linear filtering parameters into groups to obtain at least two linear filtering parameter sets, respectively performs parameter configuration for at least two linear predictive filters based on the linear filtering parameter sets, respectively inputs the obtained sub-long-term filtered excitation signals to the parameter-configured linear predictive filters, and the linear predictive filters linearly synthesize and filter the sub-long-term filtered excitation signals based on the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe, and combines the sub-audio signals according to the sequence of each subframe to obtain an audio signal.
[0139] In the above embodiment, the terminal divides a long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal, divides linear filtering parameters into groups to obtain at least two linear filtering parameter sets, and performs parameter configuration for at least two linear predictive filters according to the linear filtering parameter sets, respectively inputs the obtained sub-long-term filtered excitation signals into the parameter-configured linear predictive filters, and the linear predictive filters perform linear synthesis filtering on the sub-long-term filtered excitation signal according to the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe, and combines the sub-audio signals according to the sequence of each subframe to obtain an audio signal, thereby ensuring that the obtained audio signal can well restore the audio signal transmitted by the transmitting side, and improving the quality of the restored audio signal.
[0140] In one embodiment, the linear filtering parameters include linear filtering coefficients and energy gain values, and the audio packet processing module 1102 further obtains, for a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, an energy gain value corresponding to a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal; determines an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal according to the energy gain value corresponding to the historical sub-long-term filtered excitation signal and the energy gain value of the sub-long-term filtered excitation signal corresponding to the first subframe; performs energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter; inputs the obtained sub-long-term filtered excitation signal and the energy adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe according to the linear filtering coefficients and the energy adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to the first subframe.
[0141] In the above embodiment, the terminal obtains, for a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, an energy gain value of a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal, and determines an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal according to the energy gain value corresponding to the historical sub-long-term filtered excitation signal and the energy gain value of the sub-long-term filtered excitation signal corresponding to the first subframe, and performs energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter, inputs the obtained sub-long-term filtered excitation signal and the energy adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe according to the linear filtering coefficient and the energy adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to the first subframe. This ensures that the obtained audio signal of each subframe can restore the audio signal of each subframe transmitted from the transmitting side, and improves the quality of the restored audio signal.
[0142] In one embodiment, as shown in FIG. 12, the apparatus further includes a data anomaly determination module 1112, which determines whether there is a data anomaly in the decoded historical voice packet before the voice packet is decoded, and a forward error correction frame signal determination module 1114, which determines that the decoded and filtered audio signal is a forward error correction frame signal if there is a data anomaly in the historical voice packet.
[0143] In the above embodiment, the terminal determines whether the current decoded and filtered audio signal is a forward error correction frame signal by determining whether there is a data abnormality in the past voice packet decoded before the current voice packet is decoded, and if the audio signal is a forward error correction frame signal, performs enhancement processing on the audio signal to further improve the quality of the audio signal.
[0144] In one embodiment, the feature parameters include cepstral feature parameters, and the feature parameter extraction module 1104 further includes: Fourier transforming the audio signal to obtain a Fourier transformed audio signal, logarithmically processing the Fourier transformed audio signal to obtain a logarithmic result, and inverse Fourier transforming the logarithmic result to obtain the cepstral feature parameters.
[0145] In the above embodiment, the terminal extracts cepstral feature parameters from the audio signal, and enhances the audio signal based on the extracted cepstral feature parameters to improve the quality of the audio signal.
[0146] In one embodiment, the long-term filtering parameters include a pitch period and an amplitude gain value, and the speech enhancement module 1108 further performs speech enhancement processing on the filtered speech excitation signal based on the pitch period, the amplitude gain value, the linear filtering parameters and the cepstral feature parameters to obtain an enhanced speech excitation signal.
[0147] In the above embodiment, the terminal performs voice enhancement processing on the filtered voice excitation signal according to the pitch period, the amplitude gain value, the linear filtering parameters and the cepstrum feature parameters, obtains an enhanced voice excitation signal, and realizes audio signal enhancement according to the enhanced voice excitation signal, so as to improve the quality of the audio signal.
[0148] In one embodiment, the signal transformation module 1106 further performs parameter configuration on a linear prediction filter based on the linear filtering parameters, and linearly analyses and filters the audio signal with the parameterized linear prediction filter to obtain a filtered audio excitation signal.
[0149] In the above embodiment, the terminal converts the audio signal into a filtered voice excitation signal according to the linear filtering parameters, and enhances the filtered voice excitation signal to realize enhancement of the audio signal, and improve the quality of the audio signal.
[0150] In one embodiment, the voice enhancement module 1108 further inputs the feature parameters, the long-term filtering parameters, the linear filtering parameters, and the filtered voice excitation signal into a pre-trained signal enhancement model, and the signal enhancement model performs voice enhancement processing on the filtered voice excitation signal based on the feature parameters to obtain an enhanced voice excitation signal.
[0151] In the above embodiment, the terminal realizes enhancement of the voice excitation signal through a pre-trained signal enhancement model, and realizes audio signal enhancement based on the enhanced voice excitation signal, so as to improve the quality of the audio signal and the efficiency of the audio signal enhancement processing.
[0152] In one embodiment, the feature parameters include cepstral feature parameters, and the voice enhancement module 1108 further performs a vectorization process on the cepstral feature parameters, the long-term filtering parameters and the linear filtering parameters, stitches the result of the vectorization process to obtain a feature vector, inputs the feature vector and the filtered voice excitation signal into a pre-trained signal enhancement model, performs feature extraction on the feature vector by the signal enhancement model to obtain a target feature vector, and enhances the filtered voice excitation signal based on the target feature vector to obtain an enhanced voice excitation signal.
[0153] In the above embodiment, the terminal performs vectorization processing on the cepstral feature parameters, the long-term filtering parameters and the linear filtering parameters, stitches the result of the vectorization processing to obtain a feature vector, inputs the feature vector and the filter voice excitation signal into a pre-trained signal enhancement model, performs feature extraction on the feature vector by the signal enhancement model to obtain a target feature vector, and enhances the filter voice excitation signal according to the target feature vector to obtain an enhanced voice excitation signal, thereby realizing the enhancement processing on the audio signal by the signal enhancement model, and improving the quality of the audio signal and the efficiency of the enhancement processing of the audio signal.
[0154] In one embodiment, the audio enhancement module 1108 further performs a Fourier transform on the filtered audio excitation signal to obtain a frequency-domain audio excitation signal, enhances amplitude value features of the frequency-domain audio excitation signal based on the target feature vector, and performs an inverse Fourier transform on the frequency-domain audio excitation signal with the enhanced amplitude value features to obtain an enhanced audio excitation signal.
[0155] In the above embodiment, the terminal performs Fourier transform on the filtered audio excitation signal to obtain a frequency domain audio excitation signal, enhances the amplitude value feature of the frequency domain audio excitation signal according to the target feature vector, and performs inverse Fourier transform on the frequency domain audio excitation signal with the enhanced amplitude value feature to obtain an enhanced audio excitation signal, thereby realizing the enhancement processing of the audio signal and improving the quality of the audio signal while ensuring that the phase information of the audio signal does not change.
[0156] In one embodiment, the speech synthesis module 1110 further performs parameter configuration on a linear predictive filter based on the linear filtering parameters, and linearly synthesizes and filters the enhanced speech excitation signal with the parameterized linear predictive filter to obtain a speech enhancement signal.
[0157] In this embodiment, the terminal performs linear synthesis filtering on the enhanced voice excitation signal to obtain a voice enhancement signal, that is, realizes enhancement processing on the audio signal, and improves the quality of the audio signal.
[0158] In one embodiment, the linear filtering parameters include linear filtering coefficients and energy gain values, and the speech synthesis module 1110 further performs parameter configuration on a linear predictive filter based on the linear filtering coefficients, obtains an energy gain value corresponding to a decoded historical speech packet before the speech packet is decoded, determines an energy adjustment parameter based on the energy gain value corresponding to the historical speech packet and the energy gain value corresponding to the speech packet, performs energy adjustment on a historical long-term filtered excitation signal corresponding to the historical speech packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtered excitation signal, inputs the adjusted historical long-term filtered excitation signal and the enhanced speech excitation signal into a parameter-configured linear predictive filter, and the linear predictive filter linearly synthesizes and filters the enhanced speech excitation signal based on the adjusted historical long-term filtered excitation signal.
[0159] In the above embodiment, the terminal performs parameter configuration for the linear prediction filter based on the linear filtering coefficient, obtains an energy gain value corresponding to the decoded historical voice packet before the voice packet is decoded, determines an energy adjustment parameter based on the energy gain value corresponding to the historical voice packet and the energy gain value corresponding to the voice packet, performs energy adjustment on the historical long-term filtering excitation signal corresponding to the historical voice packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtering excitation signal, and inputs the adjusted historical long-term filtering excitation signal and the enhanced voice excitation signal into the parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the enhanced voice excitation signal based on the adjusted historical long-term filtering excitation signal to smooth the audio signal between different frames and improve the quality of the voice composed of the audio signals of different frames.
[0160] The specific limitations of the audio signal enhancing device can be referred to the limitations of the audio signal enhancing method, and will not be described here. All or part of the modules in the above audio signal enhancing device may be realized by software, hardware, or a combination thereof. Each of the above modules may be embedded in a processor in a computer device in the form of hardware, or may exist separately, or may be stored in a memory in a computer device in the form of software to perform operations corresponding to each of the above modules when called by the processor.
[0161] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram can be seen in FIG. 13. The computer device includes a processor, a memory, and a network interface connected by a system bus. The processor of the computer device provides calculation and control functions. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment in which the operating system and the computer program in the non-volatile storage medium can be executed. The database of the computer device stores voice packet data. The network interface of the computer device connects and communicates with an external terminal via a network. When the computer readable instructions are executed by the processor, the method for enhancing an audio signal is realized.
[0162] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram can be seen in FIG. 14. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. The processor of the computer device provides calculation and control functions. The memory of the computer device includes a non-volatile storage medium and a built-in memory. The non-volatile storage medium stores an operating system and a computer program. The built-in memory provides an environment in which the operating system and the computer program in the non-volatile storage medium can be executed. The communication interface of the computer device communicates with an external terminal in a wired or wireless manner, and the wireless manner can be realized by WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, it realizes the audio signal enhancement method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touch pad disposed on the housing of the computer device, or an external keyboard, a touch pad, a mouse, etc.
[0163] As will be understood by those skilled in the art, the structures of Figures 13 and 14 do not limit the computer devices to which aspects of the present application can be applied, but are merely block diagrams of some structures related to aspects of the present application, and a specific computer device may include more or fewer components than those shown, or may combine some components, or have a different component arrangement.
[0164] In one embodiment, there is further provided a computing device including a memory having a computer program stored therein and a processor, the processor executing the computer program to implement the steps in each of the method embodiments above.
[0165] In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, the computer program being adapted to implement the steps of each of the method embodiments described above when executed by a processor.
[0166] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable storage medium and executed to cause the computing device to perform the steps of each of the method embodiments described above.
[0167] As can be understood by those skilled in the art, all or part of the flow of the above embodiment method may be realized by a computer program instructing the relevant hardware, and the computer program may be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, it may include the flow of each embodiment of the above method. Any reference to memory, storage, database, or other medium in each embodiment provided in the present application includes at least one of non-volatile and volatile memory. Non-volatile memory includes Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, etc. Volatile memory includes Random Access Memory (RAM) or external cache memory. By way of explanation and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), etc.
[0168] The technical features of the above embodiments may be combined in any manner. For the sake of brevity, not all possible combinations of the technical features of the above embodiments are described. However, any combination of these technical features falls within the scope of the present specification, provided that they are not contradictory.
[0169] The above examples only show some embodiments of the present application, and the description is specific and detailed, but does not limit the scope of the invention patent. Here, those skilled in the art may make some modifications and improvements without departing from the concept of the present application, and all of them belong to the scope of protection of the present application. Therefore, the scope of protection of the present application patent is subject to the appended claims. [Explanation of symbols]
[0170] 102 Terminals 104 Server
Claims
1. 1. A computer device implemented method for audio signal enhancement, comprising: decoding the received voice packets in order to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; if the audio signal is a forward error correction frame signal, extracting feature parameters from the audio signal; transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters and the linear filtering parameters to obtain an enhanced speech excitation signal; and performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
2. The step of filtering the residual signal to obtain an audio signal comprises: parameterizing a long-term prediction filter according to the long-term filtering parameters, and long-term synthesis filtering the residual signal through the parameterized long-term prediction filter to obtain a long-term filtered excitation signal; 2. The method of claim 1, further comprising: parameterizing a linear predictive filter based on the linear filtering parameters; and linearly synthesizing filtering the long-term filtered excitation signal with the parameterized linear predictive filter to obtain an audio signal.
3. The step of parameterizing a linear prediction filter based on the linear filtering parameters and linearly synthesizing filtering the long-term filtering excitation signal by the parameterized linear prediction filter to obtain an audio signal includes: Splitting the long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal; grouping the linear filtering parameters to obtain at least two linear filtering parameter sets; performing parameter configuration for at least two linear prediction filters based on the linear filtering parameter set; respectively inputting the obtained sub-long-term filtered excitation signals to a parameter-configured linear prediction filter, and the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signals according to the linear filtering parameter set to obtain sub-audio signals corresponding to each subframe; and combining the sub-audio signals according to a sequence of each of the sub-frames to obtain an audio signal.
4. the linear filtering parameters include linear filtering coefficients and energy gain values; For a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, obtaining an energy gain value of a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal; determining an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal based on an energy gain value corresponding to the historical sub-long-term filtered excitation signal and an energy gain value of a sub-long-term filtered excitation signal corresponding to the first subframe; performing an energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter; The step of inputting the obtained sub-long-term filtered excitation signals to parameter-configured linear prediction filters, respectively, so that the linear prediction filters perform linear synthesis filtering on the sub-long-term filtered excitation signals according to the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe, 4. The method of claim 3, further comprising: inputting the obtained sub-long-term filtered excitation signal and the energy-adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, so that the linear prediction filter linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe based on the linear filtering coefficients and the energy-adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to a first subframe.
5. determining whether there is a data anomaly in the decoded historical voice packet before the voice packet is decoded; 2. The method of claim 1, further comprising: determining that the decoded and filtered audio signal is a forward error correction frame signal if there is a data abnormality in the historical voice packet.
6. the feature parameters include cepstral feature parameters; The step of extracting feature parameters from the audio signal comprises: performing a Fourier transform on the audio signal to obtain a Fourier transformed audio signal; logarithmically processing the Fourier transformed audio signal to obtain a logarithmic result; and performing an inverse Fourier transform on the logarithmic result to obtain cepstral feature parameters.
7. the long-term filtering parameters include a pitch period and an amplitude gain value; The step of performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters, and the linear filtering parameters to obtain an enhanced speech excitation signal includes: The method of claim 6, further comprising: subjecting the filtered speech excitation signal to speech enhancement processing based on the pitch period, the amplitude gain value, the linear filtering parameters and the cepstral feature parameters to obtain an enhanced speech excitation signal.
8. Transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters, 2. The method of claim 1, further comprising the steps of: parameterizing a linear prediction filter based on the linear filtering parameters; and linearly analyzing and filtering the audio signal with the parameterized linear prediction filter to obtain a filtered audio excitation signal.
9. The step of performing speech enhancement processing on the filtered speech excitation signal based on the feature parameters, the long-term filtering parameters, and the linear filtering parameters to obtain an enhanced speech excitation signal includes:
2. The method of claim 1, further comprising: inputting the feature parameters, the long-term filtering parameters, the linear filtering parameters, and the filtered speech excitation signal into a pre-trained signal enhancement model, so that the signal enhancement model performs speech enhancement processing on the filtered speech excitation signal based on the feature parameters to obtain an enhanced speech excitation signal.
10. the feature parameters include cepstral feature parameters; The step of inputting the feature parameters, the long-term filtering parameters, the linear filtering parameters, and the filtered speech excitation signal into a pre-trained signal enhancement model, and the signal enhancement model performs speech enhancement processing on the filtered speech excitation signal based on the feature parameters to obtain an enhanced speech excitation signal, comprises: vectorizing the cepstral feature parameters, the long-term filtering parameters, and the linear filtering parameters, and stitching a result of the vectorization to obtain a feature vector; inputting the feature vector and the filtered speech excitation signal into a pre-trained signal enhancement model; performing feature extraction on the feature vector by the signal enhancement model to obtain a target feature vector; and enhancing the filtered audio excitation signal based on the target feature vector to obtain an enhanced audio excitation signal.
11. said step of enhancing the filtered audio excitation signal based on the target feature vector to obtain an enhanced audio excitation signal comprises: Fourier transforming the filtered audio excitation signal to obtain a frequency domain audio excitation signal; enhancing an amplitude value feature of the frequency domain audio excitation signal based on the target feature vector; and performing an inverse Fourier transform on the amplitude value feature enhanced frequency domain audio excitation signal to obtain an enhanced audio excitation signal.
12. The step of performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal includes:
2. The method of claim 1, further comprising: parameterizing a linear predictive filter based on the linear filtering parameters; and linearly synthesizing filtering the enhanced speech excitation signal with the parameterized linear predictive filter to obtain a speech enhancement signal.
13. the linear filtering parameters include linear filtering coefficients and energy gain values; said step of parameterizing a linear prediction filter based on said linear filtering parameters and linearly synthesis filtering said enhanced audio excitation signal with said parameterized linear prediction filter; configuring parameters for a linear prediction filter based on the linear filtering coefficients; obtaining an energy gain value corresponding to a decoded historical voice packet before the voice packet is decoded; determining an energy adjustment parameter based on the energy gain values corresponding to the historical voice packets and the energy gain values corresponding to the voice packets; performing energy adjustment on a historical long-term filtered excitation signal corresponding to the historical voice packet according to the energy adjustment parameter to obtain an adjusted historical long-term filtered excitation signal; and inputting the adjusted historical long-term filtered excitation signal and the enhanced speech excitation signal into a parameter configured linear prediction filter, which linearly synthesis filters the enhanced speech excitation signal based on the adjusted historical long-term filtered excitation signal.
14. 1. An audio signal enhancement device, comprising: a voice packet processing module for sequentially decoding the received voice packets to obtain a residual signal, a long-term filtering parameter and a linear filtering parameter, and filtering the residual signal to obtain an audio signal; a feature parameter extraction module for extracting feature parameters from the audio signal when the audio signal is a forward error correction frame signal; a signal transformation module for transforming the audio signal into a filtered audio excitation signal based on the linear filtering parameters; a voice enhancement module for performing voice enhancement processing on the filtered voice excitation signal according to the feature parameters, the long-term filtering parameters, and the linear filtering parameters to obtain an enhanced voice excitation signal; a speech synthesis module for performing speech synthesis based on the enhanced speech excitation signal and the linear filtering parameters to obtain a speech enhancement signal.
15. The voice packet processing module further comprises: Parameterize a long-term prediction filter according to the long-term filtering parameters, and perform long-term synthesis filtering of the residual signal through the parameterized long-term prediction filter to obtain a long-term filtered excitation signal; The audio signal enhancing apparatus of claim 14, characterized in that, based on the linear filtering parameters, a linear predictive filter is parameterized, and the long-term filtered excitation signal is linearly synthesis filtered by the parameterized linear predictive filter to obtain an audio signal.
16. The voice packet processing module further comprises: Splitting the long-term filtered excitation signal into at least two subframes to obtain a sub-long-term filtered excitation signal; grouping the linear filtering parameters to obtain at least two linear filtering parameter sets; Perform parameter configuration for at least two linear prediction filters based on the linear filtering parameter set; Input the obtained sub-long-term filtered excitation signals to respective parameter-configured linear prediction filters, and the linear prediction filters perform linear synthesis filtering on the sub-long-term filtered excitation signals according to the linear filtering parameter sets to obtain sub-audio signals corresponding to each subframe; The audio signal enhancing apparatus of claim 15, characterized in that the sub-audio signals are combined according to the sequence of each of the sub-frames to obtain an audio signal.
17. the linear filtering parameters include linear filtering coefficients and energy gain values; The voice packet processing module further comprises: For a sub-long-term filtered excitation signal corresponding to a first subframe in the long-term filtered excitation signal, obtain an energy gain value of a historical sub-long-term filtered excitation signal of a subframe adjacent to the sub-long-term filtered excitation signal corresponding to the first subframe in the historical long-term filtered excitation signal; Determine an energy adjustment parameter corresponding to the sub-long-term filtered excitation signal based on an energy gain value corresponding to the historical sub-long-term filtered excitation signal and an energy gain value of the sub-long-term filtered excitation signal corresponding to the first subframe; performing an energy adjustment on the historical sub-long-term filtered excitation signal according to the energy adjustment parameter; 17. The audio signal enhancement device of claim 16, further comprising: inputting the obtained sub-long-term filtered excitation signal and the energy-adjusted historical sub-long-term filtered excitation signal into a parameter-configured linear prediction filter, which linearly synthesizes and filters the sub-long-term filtered excitation signal corresponding to the first subframe based on the linear filtering coefficients and the energy-adjusted historical sub-long-term filtered excitation signal to obtain a sub-audio signal corresponding to a first subframe.
18. A computing device comprising a memory having a computer program stored therein and a processor, the computer program comprising instructions which, when executed by the processor, cause the computing device to implement a method for enhancing an audio signal according to any one of claims 1 to 13.
19. A computer program which, when executed by a processor, implements the method for enhancing an audio signal according to any one of claims 1 to 13.
Citation Information
Patent Citations
Voice processing method, device and equipment and storage medium
CN111554323A
Audio decoder and method for providing decoded audio information using error concealment to modify a time-domain excitation signal
JP2016535867A
Encoder and method for encoding an audio signal having reduced background noise using linear prediction coding
JP2018528480A
Speech encoding
US20100174538A1
