Speech enhancement method and device, medium, equipment and program product

Abstract

The invention discloses a speech enhancement method and device, a medium, equipment and a program product, and belongs to the technical field of audio coding and decoding. The method mainly comprises the following steps: when a current frame contains voiced sound, determining a cepstrum coefficient interval corresponding to the voiced sound in a first cepstrum coefficient according to the cepstrum coefficient corresponding to the maximum cepstrum value of the first cepstrum coefficient in a fundamental tone range; modifying a cepstrum coefficient in the cepstrum coefficient interval into a preset value to obtain a second cepstrum coefficient, and calculating the second cepstrum coefficient to obtain a second audio spectrum coefficient magnitude spectrum; calculating to obtain a noise magnitude spectrum corresponding to the current frame by using the first audio spectrum coefficient and the second audio spectrum coefficient magnitude spectrum, and calculating to obtain sub-band noise energy corresponding to the current frame by using the noise magnitude spectrum; and performing noise elimination on the first audio spectrum coefficient by using the sub-band noise energy. The voiced sound is suppressed before noise estimation, the noise estimation value is prevented from being larger than the actual value, and therefore it is ensured that the tone quality is not affected.

Description

Technical Field

[0001] The present application relates to the field of audio coding and decoding technology, and in particular to a speech enhancement method, device, storage medium, electronic device and computer program product. Background Art

[0002] In the process of Bluetooth voice communication, it is often necessary to perform noise reduction on the audio signal. In the prior art, spectral subtraction is usually used to achieve this. The specific implementation process is as follows: Figure 1 That is, first, the time domain signal is converted to the frequency domain through time-frequency transformation, and then the noise spectrum is estimated based on the spectral coefficients, and the noise spectrum is subtracted to obtain updated spectral coefficients. After that, the updated spectral coefficients are converted to the time domain through inverse time-frequency transformation, and then the noise-eliminated audio signal is audio-encoded and transmitted through the Bluetooth transmitter. Finally, the audio data is received through the Bluetooth receiver, and decoding and playback are performed.

[0003] The use of existing technologies has the following disadvantages: First, the requirements for time-frequency transformation and inverse time-frequency transformation on computing power are relatively high, which puts high requirements on the storage space and computing power of the system, and the time-frequency transformation and inverse time-frequency transformation of the speech enhancement module are repeated with the time-frequency transformation in audio encoding and the time-frequency inverse transformation in audio decoding, resulting in repeated computing power consumption, increased power consumption, and reduced user experience; secondly, the speech enhancement processing in the existing technology is often performed in the form of a separate module. In order to avoid discontinuity of the enhanced speech, it is usually based on the overlap-addition method to eliminate this effect, but the delay caused by the algorithm introduced will increase the overall delay of the system, and LE Audio has strict requirements on end-to-end delay, which makes it difficult to optimize the system delay; finally, the noise estimation of the existing technology is inaccurate, especially when the speech duration is long, the noise estimation will be affected by voiced sounds, which causes the noise estimation value to be greater than the actual value, causing the enhanced speech to be distorted. Summary of the invention

[0004] In response to the problems of high computing power requirements, system delays and voice distortion in the prior art, the present application mainly provides a speech enhancement method, device, storage medium, electronic device and computer program product.

[0005] In order to achieve the above-mentioned purpose, the first technical solution adopted in the present application is: to provide a speech enhancement method, which includes: performing cepstrum calculation using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to obtain the first cepstrum coefficient of the current frame; when the current frame contains voiced sound, determining the cepstrum coefficient interval corresponding to the voiced sound in the first cepstrum coefficient according to the cepstrum coefficient corresponding to the cepstrum maximum value of the first cepstrum coefficient within the fundamental tone range; modifying the cepstrum coefficient in the cepstrum coefficient interval to a preset value to obtain a second cepstrum coefficient, and calculating the second cepstrum coefficient to obtain a second audio spectrum coefficient amplitude spectrum; using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; and using the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient.

[0006] Optionally, the speech enhancement method also includes, during the encoding process, using a long-term post-filter module to obtain a voiced speech flag of the current frame; judging whether the current frame contains voiced speech based on the voiced speech flag of the current frame; and / or determining whether the current frame contains voiced speech based on the relationship between the maximum value of the inverse spectrum and a preset threshold value.

[0007] Optionally, the noise amplitude spectrum corresponding to the current frame is calculated using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, including: calculating the baseline deviation between the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, and calculating the noise amplitude spectrum corresponding to the current frame using the baseline deviation and the second audio spectrum coefficient amplitude spectrum.

[0008] Optionally, the subband noise energy corresponding to the current frame is calculated using the noise amplitude spectrum, including: dividing the noise amplitude spectrum into subbands according to a predetermined rule to obtain all subbands corresponding to the current frame; calculating the subband average energy of each subband; and obtaining the subband noise energy corresponding to each subband using the subband average energy calculation.

[0009] Optionally, the speech enhancement method also includes, during the encoding process, performing cepstrum calculation using the first audio spectrum coefficient calculated during the audio encoding or audio decoding process to obtain the first cepstrum coefficient of the current frame; when the current frame contains voiced sound, determining the cepstrum coefficient interval corresponding to the voiced sound in the first cepstrum coefficient based on the cepstrum coefficient corresponding to the cepstrum maximum value of the first cepstrum coefficient within the fundamental tone range; clearing the cepstrum coefficients in the cepstrum coefficient interval to obtain the second cepstrum coefficient, and calculating the second cepstrum coefficient to obtain the amplitude spectrum of the second audio spectrum coefficient; using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; using the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient, and encoding the enhanced audio spectrum coefficient to obtain the corresponding bit stream; and during the audio decoding process, decoding the bit stream to obtain an enhanced audio signal.

[0010] The second technical solution adopted in the present application is: to provide a speech enhancement device, which includes: a first cepstral coefficient acquisition module, which is used to perform cepstral calculation using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to obtain the first cepstral coefficient of the current frame; an interval acquisition module, which is used to determine the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the fundamental range when the current frame contains voiced sound; a second audio spectrum coefficient acquisition module, which is used to modify the cepstral coefficient in the cepstral coefficient interval to a preset value to obtain the second cepstral coefficient, and calculate the second cepstral coefficient to obtain the amplitude spectrum of the second audio spectrum coefficient; a sub-band noise energy calculation module, which uses the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame, and uses the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; and a signal enhancement module, which uses the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient.

[0011] Optionally, the speech enhancement device also includes a voiced sound judgment module, which, during the encoding process, uses a long-term post-filter module to obtain a voiced sound flag of the current frame; determines whether the current frame contains voiced sound based on the voiced sound flag of the current frame; and / or determines whether the current frame contains voiced sound based on the size relationship between the maximum value of the inverse spectrum and a preset threshold value.

[0012] Optionally, the noise energy calculation module includes: calculating a baseline deviation between a first audio spectrum coefficient and a second audio spectrum coefficient amplitude spectrum, and using the baseline deviation and the second audio spectrum coefficient amplitude spectrum to calculate a noise amplitude spectrum corresponding to the current frame.

[0013] Optionally, the noise energy calculation module includes: dividing the noise amplitude spectrum into sub-bands according to a predetermined rule to obtain all sub-bands corresponding to the current frame; calculating the sub-band average energy of each sub-band; and using the sub-band average energy calculation to obtain the sub-band noise energy corresponding to each sub-band.

[0014] Optionally, the speech enhancement device also includes, during the encoding process, performing cepstrum calculation using the first audio spectrum coefficient calculated during the audio encoding or audio decoding process to obtain the first cepstrum coefficient of the current frame; when the current frame contains voiced sound, determining the cepstrum coefficient interval corresponding to the voiced sound in the first cepstrum coefficient based on the cepstrum coefficient corresponding to the cepstrum maximum value of the first cepstrum coefficient within the fundamental tone range; clearing the cepstrum coefficients in the cepstrum coefficient interval to obtain the second cepstrum coefficient, and calculating the second cepstrum coefficient to obtain the amplitude spectrum of the second audio spectrum coefficient; using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; using the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient, and encoding the enhanced audio spectrum coefficient to obtain the corresponding bit stream; and during the audio decoding process, decoding the bit stream to obtain an enhanced audio signal.

[0015] The third technical solution adopted in the present application is: providing a computer-readable storage medium storing a computer program / instruction, which is operated to execute the speech enhancement method in solution one or solution two.

[0016] The fourth technical solution adopted in the present application is: providing a computer device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the speech enhancement method in solution one or solution two.

[0017] The fifth technical solution adopted in the present application is: to provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the speech enhancement method in Solution 1 or Solution 2.

[0018] The technical solution of the present application can achieve the following beneficial effects: using cepstrum processing to reduce the influence of voicing in noise estimation, and reusing the existing time-frequency transformation processing in the encoding and decoding process, thereby saving computing power, saving data storage space and reducing power consumption while ensuring sound quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 is a schematic diagram of speech enhancement processing steps in the prior art;

[0021] Figure 2 It is a schematic diagram of a specific implementation of a speech enhancement method of the present application;

[0022] Figure 3 It is a schematic diagram of a specific implementation of a speech enhancement device of the present application.

[0023] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present application.

[0025] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0026] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0027] Figure 2 An implementation of a speech enhancement method of the present application is shown.

[0028] Figure 2 The speech enhancement method shown includes: step S201, performing cepstrum calculation using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to obtain the first cepstrum coefficient of the current frame;

[0029] Step S202, when the current frame contains voiced sound, determining the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range;

[0030] Step S203, modifying the cepstral coefficients in the cepstral coefficient interval to preset values ​​to obtain second cepstral coefficients, and calculating the second cepstral coefficients to obtain a second audio spectrum coefficient amplitude spectrum;

[0031] Step S204, using the amplitude spectrum of the first audio spectrum coefficient and the second audio spectrum coefficient to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; and

[0032] Step S205: De-noise the first audio spectrum coefficient using the sub-band noise energy to obtain enhanced audio spectrum coefficients.

[0033] This specific implementation method suppresses the influence of turbidity before noise estimation, avoids the noise estimation value being greater than the actual value, and thus ensures that the sound quality is not affected; by completing noise elimination during the audio encoding process, system delay is avoided, and the number of time-frequency conversion calculations is reduced, thereby reducing the amount of calculations in the audio enhancement process.

[0034] exist Figure 2 In the illustrated implementation, the speech enhancement method includes step S201, performing cepstrum calculation using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to obtain the first cepstrum coefficient of the current frame. This step obtains the first audio spectrum coefficient using the original calculation steps in the encoding and decoding process, which can reduce repeated calculation processes in the speech enhancement process, thereby reducing the amount of calculation and reducing the system's demand for storage space.

[0035] Specifically, in the audio encoding and decoding process, the audio signal is processed by dividing into frames, and each frame of the audio signal is encoded and decoded. In the audio encoding process or the audio decoding process, the first audio spectrum coefficient is obtained, and the first audio spectrum coefficient is cepstrum-calculated to obtain the first cepstrum coefficient of the current frame.

[0036] For example, during the encoding process, the low-delay improved discrete cosine transform module of the LC3 encoder is used to perform time-frequency conversion on the audio signal frame to obtain the first audio spectrum coefficient, and the corresponding calculation process is as follows:

[0037] t(n)=x m (ZN F +n), for n=0…2·N F -1-Z

[0038] t(2N F -Z+n)=0,for n=0…Z-1

[0039]

[0040] for k=0…N F -1

[0041] where x m (n) is the input m-th frame time domain audio PCM signal, X m (k) is the frequency domain spectrum coefficient after discrete cosine transform.

[0042] The first audio spectrum coefficient obtained in the encoding process is calculated according to the following calculation process to obtain the first cepstrum coefficient of the current frame, that is, the first audio spectrum coefficient is calculated in the order of first calculating the logarithm of the amplitude spectrum and then calculating the cepstrum to obtain the first audio spectrum coefficient. The calculation process is as follows:

[0043]

[0044]

[0045] where Y m (k) is the logarithm of the amplitude spectrum of the first audio spectrum coefficient in the mth frame, C m (n) is the first cepstral coefficient at the mth frame.

[0046] exist Figure 2 In the illustrated embodiment, the speech enhancement method includes step S202, when the current frame contains voiced sound, determining the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range.

[0047] Specifically, the envelope, excitation and noise of speech in noisy speech can be separated according to the cepstral coefficients, so the present invention suppresses voiced sounds in the cepstral domain and then estimates the noise, which is different from the method of noise estimation based on Fourier transform amplitude spectrum or discrete cosine transform amplitude spectrum in the prior art. The technical means adopted in the prior art are easily affected by voiced sounds when estimating noise, thereby affecting the sound quality of the audio, while the present application can improve the sound quality while reducing computing power.

[0048] When it is determined that the current frame contains voiced sounds, the present application first suppresses voiced sounds and then estimates noise; when it is determined that the current frame does not contain voiced sounds, the noise estimation is performed directly. That is, when the current frame contains voiced sounds, the maximum value of the cepstral coefficient is first searched within the pitch range of the first cepstral coefficient to obtain the cepstral maximum value, and then the cepstral coefficient interval corresponding to the voiced sounds in the first cepstral coefficient is determined based on the cepstral coefficient corresponding to the cepstral maximum value.

[0049] For example, the fundamental frequency range is usually between 50 and 500 Hz, so the fundamental frequency range corresponding to the cepstral coefficient is [BW / 500, BW / 50], that is, [16, 160], where BW is the Nyquist bandwidth. At this time, the maximum value of the cepstral is I max =max(C m (n)), n = 16 ~ 159, where I max is the maximum value of the cepstrum within the fundamental frequency range, C m (n) is the first cepstral coefficient at the mth frame, and n is the cepstral coefficient. The cepstral coefficient corresponding to the voiced sound is determined by using the cepstral maximum value, that is, the corresponding interval is [I max -2,I max +2].

[0050] exist Figure 2 In the illustrated implementation, the speech enhancement method includes step S203, modifying the cepstral coefficients in the cepstral coefficient interval to a preset value to obtain a second cepstral coefficient, and calculating the second cepstral coefficient to obtain a second audio spectrum coefficient amplitude spectrum. This step uses cepstral processing to reduce the influence of voicing in noise estimation, thereby ensuring sound quality.

[0051] Specifically, the cepstral coefficients related to voiced sounds in the cepstral coefficients are set to zero to obtain the second cepstral coefficients, thereby eliminating the influence of voiced harmonics. That is, the second cepstral coefficients are calculated by the following formula:

[0052]

[0053] Among them, C m (n) is the first cepstral coefficient at the mth frame, is the second cepstral coefficient at the mth frame.

[0054] The second cepstrum coefficient is transformed back to the discrete cosine transform spectrum coefficient amplitude spectrum using the following formula:

[0055]

[0056]

[0057] in, is the amplitude spectrum of the second audio spectrum coefficient, NF is the frame length.

[0058] In particular, the preset value may also adopt other easily identifiable numerical values, and this application does not make any specific limitations.

[0059] exist Figure 2 In the illustrated embodiment, the speech enhancement method includes step S204, using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame. This step suppresses voiced sounds before noise estimation, thereby preventing the noise estimation value from being greater than the actual value, thereby ensuring that the sound quality is not affected.

[0060] In a specific embodiment of the present application, the noise amplitude spectrum corresponding to the current frame is calculated using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, including: calculating the baseline deviation between the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, and using the baseline deviation and the second audio spectrum coefficient amplitude spectrum to calculate the noise amplitude spectrum corresponding to the current frame.

[0061] Specifically, the baseline deviation between the first audio spectrum coefficient and the second audio spectrum coefficient is calculated using the amplitude spectrum of the second audio spectrum coefficient and the amplitude spectrum of the first audio spectrum coefficient. The baseline deviation is the amplitude corresponding to the trough between the voiced sound harmonics of the first audio spectrum coefficient and the second audio spectrum coefficient. The specific calculation process is as follows:

[0062]

[0063] Among them, BD min (k) is the baseline deviation, |Y(k)| is the amplitude spectrum of the first audio spectrum coefficient, is the amplitude spectrum of the second audio spectrum coefficient.

[0064] The noise amplitude spectrum corresponding to the current frame is calculated using the baseline deviation and the amplitude spectrum of the second audio spectrum coefficient. The specific calculation process is as follows:

[0065]

[0066] in, is the noise amplitude spectrum corresponding to the current frame, BD min (k) is the baseline deviation, is the amplitude spectrum of the second audio spectrum coefficient.

[0067] In a specific embodiment of the present application, the subband noise energy corresponding to the current frame is calculated using the noise amplitude spectrum, including: dividing the noise amplitude spectrum into subbands according to a predetermined rule to obtain all subbands corresponding to the current frame; calculating the subband average energy of each subband; and, using the subband average energy to calculate the subband noise energy corresponding to each subband.

[0068] Specifically, the noise amplitude spectrum is divided into sub-bands according to a predetermined rule to obtain all sub-bands corresponding to the current frame. Specifically, the noise amplitude spectrum can be divided into sub-bands using uniform sub-bands or Bark sub-bands. For example, when the current frame is configured with a sampling rate of 16kHz and a frame length of 10ms, the sub-band division is performed according to uniform sub-bands, and 16 noise amplitude spectra can be obtained for each sub-band, that is, sub-band 1: Subband 2: ...Subband 16:

[0069] The sub-band average energy is calculated using the divided sub-bands. The specific calculation process is as follows:

[0070]

[0071] for b=1…SUBBAND_NUM

[0072] SUBBAND NUM =16

[0073] SUBBAND wIDTH =10;

[0074] Wherein, Energy(b) is the subband average energy of the current subband.

[0075] Then, the subband average energy is used to calculate the inter-frame smoothed subband noise energy corresponding to each subband, where the subband noise energy calculation process of the b-th subband of the m-th frame is as follows:

[0076] N(m,b)=max(Nmin,β*N(k-1,b)+(1-β)*Energy(b))for b=0...SUBBAND NUM -1

[0077] Where β is the energy smoothing factor, which is usually between 0 and 1, and preferably 0.5; Nmin is the minimum energy, and preferably its normalized value is 0.05.

[0078] exist Figure 2 In the illustrated implementation, the speech enhancement method includes step S205, performing noise elimination on the first audio spectrum coefficients by using the sub-band noise energy to obtain enhanced audio spectrum coefficients.

[0079] Specifically, the sub-band noise energy is used to perform sub-band updating on the first audio spectrum coefficient of each frame of audio, so as to eliminate the noise and obtain the enhanced audio spectrum coefficient. The specific calculation process is as follows:

[0080]

[0081]

[0082] Wherein, sign() is the sign function, max() is the maximum value, μ is the noise energy adjustment factor, and N(b) is an estimate based on the average energy on the subband. In particular, since N(b) is an estimate based on the average energy on the subband, there will be a slight deviation between it and the actual average energy value, so μ is usually taken as a value less than 1 and greater than 0.9, and the preferred value of μ is 0.94.

[0083] In a specific embodiment of the present application, the speech enhancement method also includes, during the encoding process, using a long-term post-filter module to obtain a voiced speech flag of the current frame; judging whether the current frame contains voiced speech based on the voiced speech flag of the current frame; and / or determining whether the current frame contains voiced speech based on the size relationship between the maximum value of the cepstrum and a preset threshold value.

[0084] Specifically, when the present application is applied to the encoder end, the long-term post filter module (LTPF) in the encoding process is used to obtain parameters indicating whether the LTPF of the previous frame is activated, parameters indicating whether the LTPF of the current frame is activated, and parameters indicating whether the current frame has voiced sound. The parameters indicating whether the LTPF of the previous frame is activated, the parameters indicating whether the LTPF of the current frame is activated, and the parameters indicating whether the current frame has voiced sound are used to calculate the voiced sound flag of the current frame. According to the voiced sound flag of the current frame, it is determined whether the current frame contains voiced sound, that is, when one of the three is true, the voiced sound flag is displayed as containing voiced sound. This judgment method judges voiced sound in the time domain, which can reduce the amount of calculation in the speech enhancement process.

[0085] When the present application is applied to the decoder or encoder, whether the current frame contains voiced sound is determined based on the size relationship between the maximum value of the cepstrum and the preset threshold value. Based on typical speech statistics, the preset threshold value can be set to 0.02. This judgment method judges voiced sound in the frequency domain, which can reduce the amount of calculation in the speech enhancement process.

[0086] The above two voiced sound judgment methods can be used separately or simultaneously. When the two are used simultaneously, missed detection caused by too large a preset threshold value can be avoided, thereby further improving the correctness and accuracy of the judgment result.

[0087] In a specific embodiment of the present application, the speech enhancement method also includes: in the encoding process, using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to perform cepstrum calculation to obtain the first cepstrum coefficient of the current frame; when the current frame contains voiced sound, according to the cepstrum coefficient corresponding to the cepstrum maximum value of the first cepstrum coefficient within the fundamental frequency range, determining the cepstrum coefficient interval corresponding to the voiced sound in the first cepstrum coefficient; clearing the cepstrum coefficients in the cepstrum coefficient interval to obtain second cepstrum coefficients, and calculating the second cepstrum coefficients to obtain second audio spectrum coefficients; using the first audio spectrum coefficients and the second audio spectrum coefficients to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; using the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient, and encoding the enhanced audio spectrum coefficient to obtain the corresponding bit stream; and in the audio decoding process, decoding the bit stream to obtain an enhanced audio signal. This specific embodiment completes noise elimination during the audio encoding process, eliminating the time-frequency transformation and inverse time-frequency transformation in the prior art, reducing computing power, and at the same time distributes the noise elimination steps on the encoder and decoder ends, while balancing computing power requirements and avoiding algorithm delays caused by traditional overlap-addition.

[0088] Specifically, during the encoding process, the low-delay improved discrete cosine transform module in the encoding process is used to complete the time-frequency conversion to obtain the first audio spectrum coefficient, and the long-term post-filter module is used to obtain the fundamental pitch delay (pitch_delay) and the fundamental pitch frequency index is obtained by using the fundamental pitch delay calculation. The calculation formula is: Then, voiced sound removal, noise estimation and noise removal processing are performed on the encoder side, and then encoding is performed according to the standard encoding process to obtain an audio code stream, that is, the application flow of the present application in the encoding process is: use a low-latency improved discrete cosine transform module to complete the time-frequency conversion to obtain the first audio spectrum coefficient, and then use the first audio spectrum coefficient to perform voiced sound removal, noise estimation and noise removal processing, and then use the enhanced audio spectrum coefficient obtained by noise removal to perform transform domain noise shaping, time domain noise shaping and quantization and other encoding processing processes.

[0089] In particular, the calculation process of baseline deviation in voiced speech removal k=0…N F The k in -1 needs to satisfy

[0090] In the decoding module, decoding is completed according to the standard decoding process, and the low-delay improved inverse discrete cosine transform module in the decoding module completes the conversion from frequency domain to time domain in the process of speech enhancement processing, thereby completing the audio enhancement processing.

[0091] In a specific embodiment of the present application, when the present application is used alone in a decoder, taking the LC3 encoder as an example, noise estimation and noise elimination are performed after transform domain noise shaping and before low-delay improved inverse discrete cosine transform.

[0092] Figure 3 A specific implementation of a speech enhancement device of the present application is shown.

[0093] exist Figure 3 In the specific implementation shown, the speech enhancement device mainly includes: a first cepstral coefficient acquisition module 301, which is used to perform cepstral calculation using the first audio spectrum coefficient calculated during the audio encoding or audio decoding process to obtain the first cepstral coefficient of the current frame;

[0094] The interval acquisition module 302 is used to determine the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range when the current frame contains voiced sound;

[0095] A second audio spectrum coefficient acquisition module 303 is used to modify the cepstral coefficients in the cepstral coefficient interval to preset values ​​to obtain second cepstral coefficients, and calculate the second cepstral coefficients to obtain second audio spectrum coefficients;

[0096] The sub-band noise energy calculation module 304 calculates the noise amplitude spectrum corresponding to the current frame using the first audio spectrum coefficient and the second audio spectrum coefficient, and calculates the sub-band noise energy corresponding to the current frame using the noise amplitude spectrum; and

[0097] The signal enhancement module 305 is configured to perform noise elimination on the first audio spectrum coefficient by using the sub-band noise energy to obtain enhanced audio spectrum coefficients.

[0098] In a specific embodiment of the present application, the speech enhancement device also includes a voiced sound judgment module, which uses a long-term post-filter module to obtain a voiced sound flag of the current frame during the encoding process; judges whether the current frame contains voiced sound based on the voiced sound flag of the current frame; and / or determines whether the current frame contains voiced sound based on the size relationship between the maximum value of the cepstrum and a preset threshold value.

[0099] In a specific embodiment of the present application, the noise energy calculation module includes: calculating a baseline deviation between a first audio spectrum coefficient and a second audio spectrum coefficient, and using the baseline deviation and the second audio spectrum coefficient to calculate a noise amplitude spectrum corresponding to a current frame.

[0100] In a specific embodiment of the present application, the noise energy calculation module includes: dividing the noise amplitude spectrum into sub-bands according to a predetermined rule to obtain the sub-bands corresponding to the current frame; calculating the sub-band average energy of the sub-band; and using the sub-band average energy calculation to obtain the sub-band noise energy corresponding to each sub-band.

[0101] In a specific embodiment of the present application, the speech enhancement device also includes: in the encoding process, using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to perform cepstrum calculation to obtain the first cepstrum coefficient of the current frame; when the current frame contains voiced sound, according to the cepstrum coefficient corresponding to the cepstrum maximum value of the first cepstrum coefficient within the fundamental frequency range, determining the cepstrum coefficient interval corresponding to the voiced sound in the first cepstrum coefficient; clearing the cepstrum coefficients in the cepstrum coefficient interval to obtain the second cepstrum coefficient, and calculating the second cepstrum coefficient to obtain the amplitude spectrum of the second audio spectrum coefficient; using the first audio spectrum coefficient and the second audio spectrum coefficient to calculate the noise amplitude spectrum corresponding to the current frame, and using the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; using the sub-band noise energy to eliminate the noise of the first audio spectrum coefficient to obtain the enhanced audio spectrum coefficient, and encoding the enhanced audio spectrum coefficient to obtain the corresponding bit stream; and in the audio decoding process, decoding the bit stream to obtain an enhanced audio signal.

[0102] The speech enhancement device provided in the present application can be used to execute the speech enhancement method described in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0103] In a specific embodiment of the present application, each functional module in a speech enhancement device of the present application may be directly in hardware, in a software module executed by a processor, or in a combination of the two.

[0104] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.

[0105] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0106] In another specific embodiment of the present application, a computer-readable storage medium stores a computer program / instruction, and the computer program / instruction is operated to execute the speech enhancement method described in the above embodiment.

[0107] In a specific embodiment of the present application, a computer device includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the speech enhancement method described in the above embodiment.

[0108] In a specific embodiment of the present application, a computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the speech enhancement method described in the above embodiment is implemented.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structural transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech enhancement method, characterized in that: include: Performing cepstrum calculation using the first audio spectrum coefficient calculated during the audio encoding or audio decoding process to obtain a first cepstrum coefficient of the current frame; When the current frame contains voiced sound, determining the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range; Modifying the cepstral coefficients in the cepstral coefficient interval to preset values ​​to obtain second cepstral coefficients, and calculating the second cepstral coefficients to obtain a second audio spectrum coefficient amplitude spectrum; Using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, calculate the noise amplitude spectrum corresponding to the current frame, and use the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; as well as The sub-band noise energy is used to perform noise elimination on the first audio spectrum coefficient to obtain enhanced audio spectrum coefficients.

2. The speech enhancement method according to claim 1, characterized in that: Also includes: During the encoding process, using a long-term post-filter module, obtaining a voiced sound flag of the current frame; judging whether the current frame contains voiced sound according to the voiced sound flag of the current frame; and / or Whether the current frame contains voiced sound is determined according to the magnitude relationship between the cepstrum maximum value and a preset threshold value.

3. The speech enhancement method according to claim 1, characterized in that: The step of calculating the noise amplitude spectrum corresponding to the current frame by using the amplitude spectrum of the first audio spectrum coefficient and the second audio spectrum coefficient includes: A baseline deviation between the amplitude spectra of the first audio spectrum coefficient and the second audio spectrum coefficient is calculated, and the noise amplitude spectrum corresponding to the current frame is calculated using the baseline deviation and the amplitude spectrum of the second audio spectrum coefficient.

4. The speech enhancement method according to claim 1, characterized in that: The step of calculating the sub-band noise energy corresponding to the current frame by using the noise amplitude spectrum includes: Dividing the noise amplitude spectrum into sub-bands according to a predetermined rule to obtain all sub-bands corresponding to the current frame; calculating a sub-band average energy for each of the sub-bands; and, The sub-band noise energy corresponding to each sub-band is obtained by calculating the sub-band average energy.

5. The speech enhancement method according to claim 1, characterized in that: Also includes, During the encoding process, the first audio spectrum coefficient calculated during the audio encoding or audio decoding process is used to perform cepstrum calculation to obtain the first cepstrum coefficient of the current frame; When the current frame contains voiced sound, determining the cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range; Clearing the cepstral coefficients in the cepstral coefficient interval to obtain second cepstral coefficients, and calculating the second cepstral coefficients to obtain a second audio spectrum coefficient amplitude spectrum; Using the first audio spectrum coefficient and the second audio spectrum coefficient amplitude spectrum, calculate the noise amplitude spectrum corresponding to the current frame, and use the noise amplitude spectrum to calculate the sub-band noise energy corresponding to the current frame; Using the sub-band noise energy to perform noise elimination on the first audio spectrum coefficient to obtain enhanced audio spectrum coefficients, and encoding the enhanced audio spectrum coefficients to obtain a corresponding bit stream; as well as During the audio decoding process, the code stream is decoded to obtain an enhanced audio signal.

6. A speech enhancement device, characterized in that: include: A first cepstral coefficient acquisition module, used to perform cepstral calculation using the first audio spectrum coefficient calculated in the audio encoding or audio decoding process to obtain the first cepstral coefficient of the current frame; an interval acquisition module, configured to determine, when the current frame contains voiced sound, a cepstral coefficient interval corresponding to the voiced sound in the first cepstral coefficient according to the cepstral coefficient corresponding to the cepstral maximum value of the first cepstral coefficient within the pitch range; A second audio spectrum coefficient acquisition module, used for modifying the cepstral coefficients in the cepstral coefficient interval to preset values ​​to obtain second cepstral coefficients, and calculating the second cepstral coefficients to obtain a second audio spectrum coefficient amplitude spectrum; a sub-band noise energy calculation module, which calculates the noise amplitude spectrum corresponding to the current frame by using the amplitude spectrum of the first audio spectrum coefficient and the second audio spectrum coefficient, and calculates the sub-band noise energy corresponding to the current frame by using the noise amplitude spectrum; as well as The signal enhancement module is used to use the sub-band noise energy to perform noise elimination on the first audio spectrum coefficient to obtain enhanced audio spectrum coefficients.

7. The speech enhancement device according to claim 6, characterized in that: Also includes: The voiced sound judgment module is used to obtain the voiced sound flag of the current frame using the long-term post-filter module during the encoding process; judge whether the current frame contains voiced sound according to the voiced sound flag of the current frame; and / or determine whether the current frame contains voiced sound according to the size relationship between the cepstrum maximum value and a preset threshold value.

8. A computer-readable storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are operated to perform the speech enhancement method as claimed in any one of claims 1-5.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the speech enhancement method according to any one of claims 1 to 5.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the speech enhancement method according to any one of claims 1 to 5 is implemented.

Images