Automatic detection and attenuation of speech articulation noise events.

Abstract

A method for automatic audio enhancement of an input audio signal containing at least one vocal articulation noise event is disclosed, comprising: segmenting the input audio signal into a plurality of audio frames, obtaining at least one feature parameter from the audio frames, and determining, at least in part based on the obtained feature parameter, respective types of vocal articulation noise events and respective time-frequency ranges associated with the vocal articulation noise events within the input audio signal.

Description

[Technical Field]

[0001] References to related applications This application claims priority under Spanish Patent Application No. P202030864 (reference number D20066ES) filed on 12 August 2020 and U.S. Provisional Patent Application No. 61 / 107,012 (reference number D20066USP1) filed on 29 October 2020, with all disclosures of each application incorporated herein.

[0002] This disclosure relates to a general art of automatic audio enhancement, such as automatic detection and attenuation, of speech articulation noise events (e.g., mouth clicks, speech plosives, etc.). [Background technology]

[0003] The volume of audio content, often of varying quality, is increasing across various media platforms, to the point where relying solely on manual editing is no longer feasible. Automated speech enhancement, when implemented correctly, can reduce editing effort while maintaining the naturalness of the audio.

[0004] Generally speaking, speech enhancement algorithms can handle two types of unwanted "noise": noise generated by background sources and noise generated by articulation.

[0005] Plosives belong to the second type and generally occur when air is forcefully blown out of the mouth (for example, when pronouncing syllables containing "p" or "t"), causing the diaphragm of a microphone to vibrate significantly when the blown air strikes it. In this disclosure, the term “plosive” is used broadly to include any blown air from the mouth that causes significant vibration of the microphone diaphragm (for example, including short fricatives such as “f” and “z”).

[0006] Even in audio content recorded in a well-controlled acoustic environment, plosives can frequently produce sudden low-frequency spikes, or so-called "pops," resulting in an unpleasant listening experience.

[0007] Several recording techniques have been proposed to reduce the intensity of plosives, including the use of pop filters or windbreaks, and speaking off-axis. However, "pop" reduction is not as effective as expected for the following practical reasons: for example, it may be difficult to fix the speaker's or actor's posture, and physical filters can reduce the emotional connection with the listener. Therefore, signal processing tools are needed to improve the quality of such recordings. The process of detecting and attenuating plosives is often called "deplosive" (or sometimes referred to as "deplosive" or "deplosive processing").

[0008] Clicking sounds are another type of transient sound produced by vocal articulation using the tongue / teeth / lips in combination with saliva. Clicking sounds can occur in both vocal and non-vocal parts and are often audible through headphones / earphones in high-SNR recordings. Clicking sounds are generally short, often lasting 10-100 ms, and can appear as several consecutive transients.

[0009] In professional recordings for TV, movies, and games, achieving audio quality free from tongue clicks is a very stringent requirement. Even in user-generated content today, tongue clicks are becoming increasingly noticeable due to the prevalence of earphone / headphone listening.

[0010] Several recording techniques have been proposed to reduce tongue-click sounds for professional male / female voice actors. However, in most cases, there is no way to control the speaker's mouth / lip state. Regarding post-processing, manual editing can be a tedious task, so it is not practical to handle hundreds / thousands of lines of dialogue. Therefore, signal processing tools are needed to more efficiently correct tongue-click sounds. The process of detecting and attenuating tongue-click sounds is often also called "mouth de-click" (or sometimes simply "de-click") or "de-click processing".

[0011] Thus, broadly speaking, the subject matter of the present disclosure is to propose a technique for automatically enhancing an audio signal (including but not limited to detecting and attenuating) that includes one or more speech articulation noise events (such as tongue-click sounds, voice plosives, etc.).

Summary of the Invention

Problems to be Solved by the Invention

[0012] In view of the above, the present disclosure generally provides a method for automatically enhancing an input audio signal including at least one speech articulation noise event, each having the characteristics of an independent claim, as well as a corresponding apparatus, program, and computer-readable storage medium.

Means for Solving the Problems

[0013] In one aspect of this disclosure, a method is provided for performing automatic audio enhancement on an input audio signal containing at least one speech articulation noise event. As will be understood and recognized by those skilled in the art, automatic audio enhancement may include, but not limited to, any suitable audio enhancement means including, automatic detection and attenuation of speech articulation noise events in the input audio signal. Here, the term “speech articulation noise event” may be understood, for example, to refer, in a broad sense to any noise event that is caused for any reason by (i.e., results from) speech articulation and is related for any reason to speech articulation.

[0014] In particular, the above method may include the step of segmenting the input audio signal into multiple audio frames (e.g., having a size of 100 ms) (e.g., by using one or more appropriate windows). The above method may further include the step of obtaining at least one feature parameter from the (segmented) audio frames (e.g., determination, calculation, extraction, etc.). In some possible implementations, the feature parameter thus obtained may be thought to be related to the type of speech articulation noise event (to be detected). That is, in some possible implementations, it may be necessary to obtain different feature parameters from the audio frames depending on the type of speech articulation noise event (to be detected) (e.g., the feature parameter may be selected according to the speech articulation noise event to be detected). The above method may further include the step of determining (e.g., detection, calculation, etc.) each type of speech articulation noise event and each range (e.g., time and / or frequency range) related to the speech articulation noise event in the input audio signal, at least in part based on the obtained feature parameter.

[0015] According to the above configuration, the proposed method can provide an efficient and flexible mechanism for determining (detecting) potential speech articulation noise events (e.g., artifacts) contained within the input audio signal. This can facilitate appropriate further enhancement (post) processing (e.g., attenuation). This largely avoids the tedious manual editing / processing that was previously required to identify and attenuate noise events in the audio signal. At the same time, it can greatly improve the listening experience (from the listener's perspective).

[0016] In some implementations, the determined range may include at least one boundary of the determined speech articulation noise event in the time and / or spectral domain. That is, the range thus determined by the proposed method may include information indicating one or more boundaries of the (detected) speech articulation noise event. More specifically, as will be understood and recognized by those skilled in the art, such boundaries may exist in the time domain, the spectral domain, or both.

[0017] In some implementations, the above method may further include a step of attenuating speech articulation noise events according to a determined type and range of speech articulation noise events. As will be understood and recognized by those skilled in the art, attenuation can be performed by any suitable means, for example, by applying an appropriate attenuation gain according to a determined type and range of speech articulation noise events.

[0018] In some implementations, speech articulation noise events may include at least one of either tongue-clicking events or speech plosive events. As described above, broadly speaking, there may typically be two possible types of unwanted / unwanted “noise” that speech enhancement algorithms generally try to address: noise generated by background sources and noise generated by articulation. Plosives belong to the second type. Speech plosives occur when air is forcefully blown out of the mouth (e.g., when pronouncing syllables containing “p” or “t”), causing the diaphragm to vibrate significantly, similar to when wind strikes the diaphragm of a microphone. As described above, in this disclosure, the term “plosive” is used broadly to include any air blown out of the mouth that causes significant vibration of the microphone diaphragm (e.g., including short fricatives such as “f” and “z”). Even with speech content recorded in a well-controlled acoustic environment, plosives can frequently produce a sudden low-frequency rise, the so-called “pop,” resulting in an unpleasant listening experience. On the other hand, tongue clicks are another type of transient sound produced by vocal articulation using the tongue / teeth / lips in conjunction with saliva. Tongue clicks can occur in both vocal and non-vocal parts and are often audible through headphones / earphones in high SNR recordings. Tongue clicks are generally short, often lasting 10-100 ms, and can appear as several consecutive transients. Naturally, as will be understood and recognized by those skilled in the art, the proposed method can also be applied to the detection (and, if necessary, attenuation) of any other suitable vocal articulation noise events.

[0019] In some implementations, a speech articulation noise event may include one or more tongue-clicking events. In particular, one or more tongue-clicking events may include at least one of a non-speech tongue-clicking event, a speech tongue-clicking event, or a tongue-tapping event. Broadly speaking, as understood and recognized by those skilled in the art, tongue-tapping can be considered a special type of non-speech-clicking sound that in some cases often occurs immediately before the start of speech. Tongue-tapping is usually intentionally produced and therefore may appear as a strong and prolonged transient event. In the methods proposed in this disclosure, tongue-tapping events can generally be detected separately from non-speech-clicking events.

[0020] In some implementations, the above method may further include a step of classifying (e.g., determining) the audio frames as either speech frames or non-speech frames, after the step of segmenting the input audio signal into multiple audio frames. That is, the segmented audio frames may be individually determined as speech frames (i.e., containing speech) or non-speech frames (i.e., not containing speech), for example, depending on whether the audio frame contains speech or not. As will be understood and recognized by those skilled in the art, such classification may be carried out in any suitable manner.

[0021] In some implementations (not intended to be limiting), the input audio signal may be identified and segmented into voice and non-voice frames by using a voice activity detector (VAD). That is, the VAD may be used to determine whether each (segmented) audio frame / block (e.g., short audio frame / block) contains voice. Clicks found in the non-voice portion may be referred to as "non-voice clicks." Clicks found in the voice portion may be referred to as "voice clicks." These are detected separately. As described above, clicks are a special type of non-voice click (often occurring immediately before the start of voice) and may be detected separately from non-voice clicks in this disclosure.

[0022] In some implementations, segmentation may be achieved by using two different window sizes. In particular, one of the two window sizes may be shorter (smaller) than the other.

[0023] In some implementations, the shorter (smaller) window size may be used to detect (primarily) speech click events in speech frames. The longer window size may be used to detect (primarily) non-speech click events in non-speech frames. Thus, both short and long transient events can be detected efficiently and reliably. In some possible implementations, it will be understood by those skilled in the art that sufficiently small (one or more) hop sizes may be used as needed to achieve fine temporal resolution.

[0024] In some implementations, the step of obtaining at least one feature parameter from an audio frame may include, for each audio frame, the step of obtaining at least one kurtosis index value based on the time-domain sample amplitude of the audio frame. In addition, the step of determining the respective types and ranges of speech articulation noise events in the input audio signal based on the obtained feature parameters may include comparing the obtained kurtosis index value with a predetermined kurtosis threshold, determining that the audio frame contains a click-tongue event if the kurtosis index value exceeds the predetermined kurtosis threshold, and determining the start and end boundaries of the click-tongue event based on the respective positions where the kurtosis index value is above and below the predetermined kurtosis threshold. In particular, using kurtosis index values ​​allows for efficient estimation (e.g., determination) of a first (coarse) range of click-tongue events, which can then be further refined as needed.

[0025] In some implementations, the step of obtaining at least one feature parameter from an audio frame may include, for each audio frame, obtaining a first kurtosis index value for each approximation of the residual without audio harmonics and for each (time-domain) sample amplitude of the residual approximation. In addition, the step of determining the respective types and ranges of audio articulation noise events in the input audio signal based on the obtained feature parameters may include comparing the obtained first kurtosis index value to a first predetermined kurtosis threshold, determining that if the first kurtosis index value exceeds the first predetermined kurtosis threshold, the audio frame contains an audio click event, and determining the start and end boundaries of the audio click event based on the respective positions where the first kurtosis index value is above and below the first predetermined kurtosis threshold. As described above, by using kurtosis index values, a first (coarse) range of click events can be efficiently estimated (e.g., determined), which allows for further refinement as needed.

[0026] In some implementations, the approximate value of residuals without audio harmonic components may be a quadratic waveform difference.

[0027] In some implementations, the above method may further include the step of obtaining a second kurtosis index value from the residual sample amplitude of the speech frame. In particular, the type and range of speech articulation noise events may be determined based on the second kurtosis index value relative to the first kurtosis index value. As a non-limiting example, the step of determining the type and range of speech articulation noise events based on the second kurtosis index value relative to the first kurtosis index value may include the step of determining the type and range of speech articulation noise events based on the difference between the second kurtosis index value and the first kurtosis index value.

[0028] In some implementations, the method may further include a step of refining (limiting) the determined (coarse) range of the vocal clicking event by: identifying a sample location having the largest quadratic difference within the determined range of the vocal clicking event; and determining a refined range of the vocal clicking event by applying a predetermined vocal clicking event duration (e.g., 5 ms) around the located sample location (e.g., before and after, and possibly around it). As a further non-limiting example, the refined range of the vocal clicking event may be determined as half of the predetermined vocal clicking event duration before the located sample location (e.g., 2.5 ms) and half of the predetermined vocal clicking event duration after the located sample location (e.g., 2.5 ms). Naturally, any other appropriate index value may be adopted depending on the specific implementation.

[0029] In some implementations, the above method may further include a step of determining the range of speech tongue-clicking events based on the maximum / minimum rate of change calculated from the local minimum and maximum values ​​in the speech frame. Broadly speaking, this range determination (or refinement) process can generally be considered as detecting fast modulation within a (coarse) tongue-clicking range. In particular, in some possible implementations, the local minimum / maximum values ​​may be converted to values ​​such as -1 and +1, and the corresponding zero-crossing rates (hereinafter referred to as "maximum / minimum rate of change") may be used to characterize how fast the modulation is.

[0030] In some implementations, the step of obtaining at least one feature parameter from an audio frame may include the step of obtaining a third kurtosis index value for each time-domain sample amplitude in each non-speech frame. In addition, the step of determining the respective types and ranges of speech articulation noise events in the input audio signal based on the obtained feature parameter may include the steps of comparing the obtained third kurtosis index value with a second predetermined kurtosis threshold, determining that the non-speech frame contains a non-speech click event if the third kurtosis index value exceeds the second predetermined kurtosis threshold, and determining the start and end boundaries of the non-speech click event based on the respective positions where the third kurtosis index value is above and below the second predetermined kurtosis threshold.

[0031] In some implementations, the above method may further include the step of merging two neighboring non-audible clicking events into a single audible clicking event (e.g., merging for attenuation purposes) if the two neighboring non-audible clicking events are within a predetermined gap threshold. Broadly speaking, non-audible clicking events typically tend to be relatively long (e.g., 50 ms). Therefore, in some cases, it may be effective to merge neighboring clicking events within a predetermined gap or threshold, e.g., 25 ms.

[0032] In some implementation examples, the above method may further include the steps of: calculating a high / low band peak ratio as the amplitude ratio of the maximum peak above a predetermined frequency and the maximum peak below a predetermined frequency for a non-audio tongue-clicking event determined in the non-audio frame immediately preceding the audio frame; and determining the non-audio tongue-clicking event as a tongue-tapping event if the calculated high / low band peak ratio is above a predetermined ratio threshold.

[0033] In some implementations, the high / low bandwidth peak ratio may be calculated as the amplitude ratio between the maximum peak above a given frequency (e.g., 1.5 kHz) and the maximum peak below that frequency but above a further given low frequency (e.g., 100 Hz). Generally speaking, the given frequency may be selected as the limit frequency from which harmonics become dominant. Naturally, as will be understood and recognized by those skilled in the art, any other suitable calculation method may be employed depending on the various implementations and / or requirements.

[0034] In some implementations, the above method may further include a step of refining the determined range of tongue-tapping events based on the high / low bandwidth peak ratio, spectral slope, and energy envelope.

[0035] In some implementations, the step of refining the determined range of the tummy tic event may include extending the termination position of the tummy tic event determined by using a third kurtosis index value, provided that the high / low band peak ratio is above a predetermined ratio threshold, the spectral slope is below a predetermined slope threshold, and / or the energy in the energy envelope is reduced.

[0036] In some implementations, the above method may further include a step of determining speech articulation noise events according to a further predetermined threshold, based on the centroid (COG) calculated for the speech frame, in order to distinguish between tongue-clicking events and speech transients. Broadly speaking, speech transients can typically share similarities with tongue-clicking in nature, but generally differ in magnitude and / or spectral characteristics. It may be possible to identify speech transients based on the VAD and / or COG (mean time of the signal) of a short-duration speech waveform (waveform of a short-duration frame in the time domain), and thus avoid false-alarm detection as tongue-clicking.

[0037] In some implementations, the method may further include a step of attenuating one or more determined click-sound events based on the spectral envelope of the audio frame containing the detected click-sound event, and the respective spectral gains derived from the target envelope calculated based on the respective reference frame.

[0038] In some implementations, for each detected tongue-clicking event, the reference frame may include the audio frame preceding and following the audio frame containing the detected tongue-clicking event. Furthermore, the target envelope may be calculated by interpolating the spectral envelope of the reference frame. Naturally, as will be understood and recognized by those skilled in the art, any other suitable calculation method may be employed depending on the respective implementation and / or requirements.

[0039] In some implementations, attenuation may be applied to frequency bands higher than a predetermined high-frequency threshold (e.g., 4 kHz). More specifically, in some possible implementations, further constraints may be applied as needed to allow only high-frequency attenuation (e.g., above 4 kHz) for speech tongue clicks in order to avoid unintentionally altering speech harmonics.

[0040] In some implementations, the above method may further include the step of replacing one or more determined click-sound events based on their respective neighboring audio frames. More specifically, in some possible implementations, it may also be possible to use autoregressive modeling or granular approach similar to pitch-synchronous waveform modeling for correcting audio click sounds. That is, given the location of a click sound event, it may be possible to estimate the left and right local periods. By comparing neighboring periods, the click sound can be replaced with a simple crossfade using "waveform slices" that match the relative click sound locations within those periods. In some possible implementations, it may be possible to simply select the one with the smaller waveform difference to choose the left or right period for correction. Naturally, as will be understood and recognized by those skilled in the art, any other suitable means may be employed depending on the respective implementation and / or requirements.

[0041] In some implementations, the speech articulation noise event may include at least one speech plosive event. In addition, the step of obtaining at least one feature parameter from the audio frame may include the step of obtaining the respective LFE index value for each audio frame in order to identify low-frequency energy (LFE) outliers.

[0042] In some implementations, the LFE index value may be calculated in either the time domain or the spectral domain. As will be understood and recognized by those skilled in the art, any suitable means may be employed to calculate the LFE index value, depending on the respective implementation and / or requirements. As a non-limiting example, in some possible implementations, in the time domain, the LFE may be calculated as the mean square (RMS) energy of the signal passed through a low-pass filter. In some possible implementations, the low-pass filter may be, for example, a fourth-order Butterworth filter with a predetermined cutoff frequency of 80 Hz. In some other possible implementations, in the spectral domain, the LFE may be calculated from the spectrum as the RMS energy below the cutoff frequency.

[0043] In some implementations, the above method may further include a step of determining the range of speech plosive events according to outliers identified from the LFE index value and thresholds calculated based on the LFE index value, or according to the LFE ratio calculated from the previous and current audio frames.

[0044] In some implementations, the above method may further include a step of determining the respective Zero Crossing Maximum (ZCM) index value for each audio frame in order to refine the range of speech plosive events determined based on the LFE index value. In particular, the ZCM index value can be considered to represent the length of the longest interval between consecutive zero crossings within the audio frame. In some possible implementations, the ZCM index value may be further normalized by the window size (e.g., the size of the window used for segmenting the audio frame).

[0045] In some implementations, the above method may further include a step of attenuating the determined speech plosive events. Attenuation may be performed in either the time domain or the spectral domain.

[0046] In some implementations, time-domain attenuation may be performed by applying a high-pass filter (e.g., a Butterworth high-pass filter). In particular, in some implementations, the filter's cutoff frequency may be determined based on the ZCM index value for audio frames within the range of determined speech plosive events, and the filter's order may be determined based on the LFE index value for audio frames within the range of determined speech plosive events. Naturally, as will be understood and recognized by those skilled in the art, depending on the various implementations and / or requirements, any other suitable high-pass filter, or more generally, any other suitable time-domain attenuation, may be determined and used.

[0047] In some implementations, spectral domain attenuation may be performed by using a superimposed summation short-time Fourier transform (STFT) with adaptive spectral gradients and frequencies.

[0048] In some implementations, spectral domain attenuation may include the steps of processing audio frames using a Fast Fourier Transform (FFT) to generate an attenuated output audio signal, applying an attenuation gain with an adaptive slope and frequency, and applying an inverse FFT, windowing, and superimposition. In particular, in some implementations, the frequency may be determined based on the ZCM index value for audio frames within the range of determined plosive events, and the slope may be determined based on the LFE index value for audio frames within the range of determined plosive events. Naturally, as will be understood and recognized by those skilled in the art, any other suitable spectral domain attenuation may be employed depending on the respective implementation and / or requirements.

[0049] In some implementations, the above method may further include the step of applying noise spectrum estimation to limit the attenuation gain and prevent excessive suppression. That is, in some possible implementations, noise spectrum estimation may be used to limit the gain reduction so that the attenuation does not affect the overall spectral profile of the noise spectrum, particularly in the low-frequency region.

[0050] According to the above configuration, the proposed method generally attenuates faster pops with a higher cutoff frequency, thus effectively adapting to the speaker's voice pitch. Furthermore, the proposed method also attenuates stronger pops with a steeper cutoff frequency slope, thus effectively adapting to both weak and strong plosives.

[0051] In some implementations, the above method may further include the step of applying a content classifier (e.g., VAD) to audio frames to distinguish between audio and non-audio frames in order to determine audio plosive events. More specifically, in some possible implementations, when the above technique is applied to music, or content containing both audio and music, the proposed algorithm may be susceptible to low-frequency transients, such as those produced by a kick drum or bass. To address this problem, in some possible implementations, it may be possible to ensure that music content is not affected by the plosive removal process by using a content classifier (e.g., a voice / music activity detector) that calculates the probability p(n) that a given frame n contains audio, and then modifying detection or attenuation parameters.

[0052] In some implementations, spectral domain attenuation may include the steps of: generating a plurality of approximately equirectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and a plurality of bands above a predetermined frequency threshold, wherein the predetermined frequency threshold is within the frequency range of determined speech plosive events; applying a plurality of attenuation gains to the audio signal in each of the frequency bands, wherein the attenuation gains are calculated based on the energy calculated for the frequency bands; and providing the attenuated audio samples to a synthesis filter bank to generate an output audio signal. Compared to the spectral domain attenuation illustrated above, this spectral domain attenuation can generally be used when computational complexity allows.

[0053] In some implementations, the attenuation gain in each frequency band may be further constrained so that the energy in that frequency band is not reduced below the estimated noise floor in that frequency band. In other words, in some possible implementations, the (attenuation) gain may be further clipped to ensure that the power in each band is not reduced below the estimated noise floor in each band. Generally speaking, this can avoid an audible dip in noise when a plosive is present in the presence of significant background noise. As will be understood and recognized by those skilled in the art, noise (or noise floor) can be estimated by any suitable means.

[0054] In some implementations, the above method may further include the step of calculating time-smoothed low-frequency energy estimates of audio samples above the estimated noise threshold in order to distinguish between speech plosive events and higher-frequency content in the input audio signal.

[0055] In some implementations, the above method may further include the steps of calculating an audio harmonic protection index in the spectrum of an input audio signal, and calculating an attenuation gain according to the audio harmonic protection index and a time-smoothed low-frequency energy estimate.

[0056] In some implementations, the audio harmonic protection index value may be a periodic or tonal index value.

[0057] In some implementations, the periodicity index value within the spectrum can be calculated from the cepstrum of the audio samples before the final bandwidth calculation of the analysis filter bank.

[0058] In some implementations, the tonal index value in the spectrum may be calculated based on the principal lobes of the spectral peaks compared to the principal lobes of the sinusoidal peaks, prior to the final bandwidth calculation of the analysis filter bank.

[0059] In some implementations, the above method may further include a step of further constraining the calculated attenuation gain based on the frequency band immediately below it. As a non-limiting example, the gain may be further constrained so that it cannot attenuate more with respect to a certain threshold, e.g., a frequency band above 70 Hz than with respect to the frequency band immediately below it. Generally speaking, this can force the reduction or attenuation to follow the physical decrease in plosive energy with frequency. That is, if a lower frequency band has significantly reduced energy, and the next higher frequency band has greater energy, this is more likely to be pure speech energy rather than plosive-related energy. Broadly speaking, the lowest frequency band (e.g., below 70 Hz) may not follow this trend. For example, a surplus 60 Hz hum may make a certain frequency band larger, and a DC block filter may attenuate the lowest frequency band, but this should not limit the attenuation of plosive energy.

[0060] In another aspect of this disclosure, a method is provided for performing automatic audio enhancement on an input audio signal to detect and / or attenuate at least one speech articulation noise event contained in the input audio signal. As will be understood and recognized by those skilled in the art, the automatic audio enhancement may include any other suitable audio enhancement means. In particular, the speech articulation noise event may include, among other things, at least one speech plosive event.

[0061] More specifically, the above method may include the step of generating a plurality of substantially equirectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and a plurality of bands above a predetermined frequency threshold, wherein the predetermined frequency threshold is within the frequency range of speech plosive events, by using an analysis filter bank. The above method may further include the step of applying a plurality of attenuation gains to the audio signal in each of the frequency bands, wherein the attenuation gains are calculated based on the energy calculated for the frequency bands. The above method may further include the step of feeding the attenuated audio samples into a synthesis filter bank to generate an output audio signal.

[0062] Broadly speaking, the proposed method, based on the above configuration, provides an efficient and flexible mechanism for determining (detecting) expected / possible speech articulation noise events (e.g., speech plosive events) contained within the input audio signal. This largely avoids the tedious manual editing / processing that was previously required to identify and attenuate noise (e.g., plosive) events in the audio signal. At the same time, it can significantly improve the listening experience (from the listener's perspective).

[0063] In some implementations, the attenuation gain in each frequency band may be further constrained so that the energy in that frequency band is not reduced below the estimated noise floor in that frequency band. In other words, in some possible implementations, the (attenuation) gain may be further clipped to ensure that the power in each band is not reduced below the estimated noise floor in each band. Generally speaking, this can avoid an audible dip in noise when a plosive is present in the presence of significant background noise. As will be understood and recognized by those skilled in the art, noise (or noise floor) can be estimated by any suitable means.

[0064] In some implementations, the above method may further include the step of calculating time-smoothed low-frequency energy estimates of audio samples above the estimated noise threshold in order to distinguish between speech plosive events and higher-frequency content in the input audio signal.

[0065] In some implementations, the above method may further include the steps of calculating an audio harmonic protection index in the spectrum of an input audio signal, and calculating an attenuation gain according to the audio harmonic protection index and a time-smoothed low-frequency energy estimate.

[0066] In some implementations, the audio harmonic protection index value may be a periodic or tonal index value.

[0067] In some implementations, the periodicity index value within the spectrum can be calculated from the cepstrum of the audio samples before the final bandwidth calculation of the analysis filter bank.

[0068] In some implementations, the tonal index value in the spectrum may be calculated based on the principal lobes of the spectral peaks compared to the principal lobes of the sinusoidal peaks, prior to the final bandwidth calculation of the analysis filter bank.

[0069] In some implementations, the above method may further include a step of further constraining the calculated attenuation gain based on the frequency band immediately below it. As a non-limiting example, the gain may be further constrained so that it cannot attenuate more for a band above a certain threshold, e.g., 70 Hz, than for a band immediately below it. Generally speaking, this can force the reduction or attenuation to follow the physical decrease in plosive energy with increasing frequency. That is, if a lower band has significantly reduced energy, and the next higher band has greater energy, this is more likely to be pure speech energy rather than plosive-related energy. Broadly speaking, the lowest band (e.g., below 70 Hz) may not follow this trend. For example, a surplus 60 Hz hum may make a certain band larger, and a DC block filter may attenuate the lowest band, but this should not limit the attenuation of plosive energy.

[0070] In some implementations, the input audio signal may be processed sequentially using a predetermined look-ahead frame (window) size (e.g., 50ms).

[0071] In another aspect of this disclosure, an apparatus is provided comprising a processor and memory connected to the processor. The processor may be configured to cause the apparatus to perform all steps of the exemplary method described throughout the specification.

[0072] Further aspects of this disclosure provide a computer program. The computer program may include instructions that, when executed by a processor, cause the processor to perform all steps of the exemplary method described throughout the entire specification.

[0073] Further aspects of this disclosure provide a computer-readable storage medium, which may store a computer program as described below.

[0074] It is understood that the apparatus features and method processes are interchangeable in many ways. In particular, it is understood by those skilled in the art that the details of the disclosed method can be implemented by the corresponding apparatus (or system) and vice versa. Furthermore, it is understood that all of the above statements relating to the method apply similarly to the corresponding apparatus (or system) and vice versa. [Brief explanation of the drawing]

[0075] Brief explanation of the drawing Examples of the present disclosure are described below with reference to the attached drawings.

[0076] [Figure 1A] Figure 1A is a schematic diagram showing an example of a non-verbal tongue click sound according to one embodiment of the present disclosure. [Figure 1B] Figure 1B is a schematic diagram showing an example of an audio tongue-clicking sound according to one embodiment of the present disclosure. [Figure 1C] Figure 1C is a schematic diagram showing an example of tongue click according to one embodiment of the present disclosure. [Figure 2] Figure 2 is a schematic diagram showing an example of the detection and refinement of speech tongue-clicking sounds according to one embodiment of the present disclosure. [Figure 3] Figure 3 is a schematic diagram illustrating an example of the detection and refinement of speech tongue-clicking sounds according to another embodiment of the present disclosure. [Figure 4] Figure 4 is a schematic diagram showing an example of tongue click detection according to one embodiment of the present disclosure. [Figure 5] Figure 5 is a schematic diagram showing an example of spectral attenuation according to one embodiment of the present disclosure. [Figure 6] Figure 6 is a schematic block diagram illustrating an example of the functional overview of the technology according to the embodiment of this disclosure. [Figure 7] Figure 7 is a schematic diagram illustrating an example of a comparison between the zero crossing maximum value (ZCM) and the zero crossing ratio (ZCR). [Figure 8]Figure 8 is a schematic diagram showing an example of attenuation of a plosive sound according to the embodiment of this disclosure. [Figure 9] Figure 9 is a schematic block diagram illustrating an example of a functional overview according to the embodiment of this disclosure. [Figure 10] Figure 10 is a schematic block diagram illustrating another example of a functional overview according to the embodiments of this disclosure. [Figure 11] Figure 11 is a schematic flowchart illustrating an example of a method according to one embodiment of the present disclosure. [Figure 12] Figure 12 is a schematic flowchart illustrating an example of a method according to another embodiment of the present disclosure. [Figure 13] Figure 13 is a schematic block diagram illustrating yet another example of a functional overview according to the embodiments of this disclosure. [Figure 14] Figure 14 is a block diagram of an apparatus for performing the method according to the embodiments of this disclosure. [Modes for carrying out the invention]

[0077] Detailed explanation The drawings (figures) and the following description relate to preferred embodiments for illustrative purposes only. It will be readily apparent from the following description that alternative embodiments of the structures and methods described herein are viable alternatives that can be employed without departing from the principles of the claims.

[0078] Several embodiments are described below in detail, examples of which are illustrated in the accompanying drawings. Where possible, similar or identical reference numerals are used in the drawings, and they may indicate similar or identical functions. The drawings illustrate embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily understand from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0079] Furthermore, in drawings, connecting elements such as solid or dashed lines or arrows are used to indicate connections, relationships, or associations between two or more other illustrated elements; however, the absence of such connecting elements does not imply that such connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in order to avoid obscuring the disclosure. In addition, for the sake of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if one connecting element represents the communication of signals, data, or commands, a person skilled in the art should understand that such an element may, as necessary, represent one or more signal paths acting on the communication.

[0080] As mentioned above, the volume of audio content, often of varying quality, across various media platforms is increasing to the point where manual editing is no longer a viable solution. Automated speech enhancement, when done correctly, generally reduces editing effort while maintaining the naturalness of the audio.

[0081] Broadly speaking, speech enhancement algorithms typically attempt to address two types of unwanted "noise" events: noise generated by background sources and noise generated by articulation. In particular, both plosives and clicks of the tongue belong to the second type.

[0082] More specifically, plosives often occur when air is forcefully expelled from the mouth (e.g., when pronouncing syllables containing "p" or "t"), causing the diaphragm to vibrate significantly, similar to when wind strikes the diaphragm of a microphone. As stated above, in this disclosure, the term “plosive” is used broadly to include any air expelled from the mouth that causes significant vibration of the microphone diaphragm (e.g., including short fricatives such as "f" and "z"). Even in audio content recorded in a well-controlled acoustic environment, plosives can frequently produce a sudden low-frequency boost, a so-called “pop,” resulting in an unpleasant listening experience. An example of a plosive event can be seen, for example, from diagram 8200 in Figure 8 (particularly the white area in the low-frequency portion; details of which are described later).

[0083] To reduce plosive intensity, several recording techniques have been proposed, such as the use of pop filters or wind deflectors, and off-axis speech. However, "pop" reduction is not as effective as expected for the following practical reasons: for example, the inability to fix the posture of the speaker or actor (voice actor). Therefore, signal processing tools are needed to improve the quality of such recordings. For automatic plosive detection, there are two main, viable approaches, including simple feature-based detection and phone-based detection (multidimensional features for speech recognition). Phone-based detection may seem to have the advantage of identifying the precise duration of plosive events, but it is more complex and therefore requires more computational resources. Simple feature-based detection is often not powerful and does not refine the boundaries of plosive events. Another possible solution generally involves giving three user parameters (sensitivity / intensity / frequency limit) to its plosive removal module. However, to obtain the best results, users may need to manually edit the automation curve for these parameters. This is because plosives vary in intensity and frequency even within the same recording, and therefore, users may want to attenuate them. As a result, this process can be time-consuming.

[0084] On the other hand, tongue clicks are generally transient sounds produced by vocal articulation using the tongue / teeth / lips in combination with saliva. Tongue clicks typically occur in both vocal and non-vocal parts and are often audible through headphones / earphones in high-SNR recordings. Tongue clicks are generally short, often lasting 10-100ms, and can appear as several consecutive transients. In professional recordings such as TV / movie / game dialogue, achieving audio quality free of tongue clicks can be considered a very stringent requirement. Even in user-created content today, tongue clicks are highly audible because listening with earphones / headphones is common.

[0085] In this disclosure, the proposed method generally attempts to address three types of tongue clicks: 1) non-vocal tongue clicks, 2) vocal tongue clicks, and 3) lip smacks (which may also be considered a special type of non-vocal tongue click).

[0086] Referring to the diagrams, Figure 1A schematically illustrates an example of a non-vocal tongue click (for example, at approximately 0.1 s). Figure 1B schematically illustrates an example of a vocal tongue click (in particular, shown at the end of the leftmost cycle at approximately 0.7056 s, indicated by a circle). Figure 1C schematically illustrates an example of a tongue click (in particular, shown as a strong transient immediately preceding the vocal segment at approximately 2.1 s).

[0087] Several recording techniques have been proposed to reduce tongue clicks for professional male and female voice actors. However, in most cases, there is no way to control the state of the speaker's mouth / lips. Regarding post-processing, manual editing is a tedious task, making it impractical to handle hundreds or thousands of lines of dialogue. Therefore, signal processing tools are needed to more efficiently correct tongue clicks. However, currently, there seems to be little academic research available on tongue click detection. While tongue click detection presents a similar problem, the transient energy is usually much greater, making it impossible to directly apply each method to the small transient as with tongue clicks. Furthermore, in digital audio restoration, "tongue click removal" generally functions to remove impulsive noise that often occurs during the playback of phonograph records. When the damaged audio period is long, the problem becomes a general signal interpolation / extrapolation issue.

[0088] In view of the foregoing, this disclosure provides a method for automatically enhancing an input audio signal that includes one or more noise events related to or causing such speech articulations. More specifically, this disclosure aims to maintain or even improve audio quality for the listener while avoiding manual editing by automatically detecting and attenuating, in particular, speech plosives and tongue clicks among the noise events contained in the input audio signal.

[0089] First, a method relating to "tongue-clicking sound removal" according to the embodiment of this disclosure will be described.

[0090] In a broad sense, the automatic detection and attenuation method for tongue clicks described herein mainly comprises two important aspects. Firstly, the detection algorithm generally targets tongue clicks in the non-vocal domain and tongue clicks in the vocal domain, respectively. Generally, the kurtosis measure of the waveform amplitude is used as the primary criterion. The kurtosis measure is applied to both the original waveform and its second-order difference, where the second-order difference serves as an approximation of the non-harmonic signal portion. The roughly detected tongue click location is further refined to more accurately define the tongue click sample region. Secondly, the attenuation of tongue clicks is generally based on spectral gain attenuation derived from spectral envelope interpolation over short-duration frames containing the (detected) tongue clicks.

[0091] The "tongue-clicking sound removal" method will be described in more detail below with reference to Figure 6. Figure 6 generally provides a schematic functional overview of the (tongue-clicking sound removal) technology according to the embodiment of this disclosure.

[0092] More specifically, as shown in block 6010, the input audio signal may be provided in the form of, for example, an input file or stream (or any other suitable form). Depending on its form (e.g., format), the input audio signal may need to undergo appropriate segmentation processing to be divided, for example, into multiple (short-duration) audio frames (e.g., having equal or different frame sizes).

[0093] In particular, applying noise reduction processing (shown as dashed block 6020) to the input signal as needed before proceeding to the subsequent tongue-clicking sound removal process can better reveal latent tongue-clicking sounds.

[0094] Next, given a voice activity detector (VAD), as exemplified in block 6030, it is possible to determine whether each short block (audio frame) of the speech signal contains speech or not. This allows for separate handling of tongue clicks in speech portions (e.g., frames) and non-speech portions. Tongue clicks found in non-speech portions are generally referred to as "non-speech tongue clicks" (e.g., shown in Figure 1A), and tongue clicks found in speech portions are referred to as "speech tongue clicks" (e.g., shown in Figure 1B), and they are detected separately. As described above, in this disclosure, tongue clicks are generally considered a special type of non-speech tongue click, often occurring immediately before the start of speech. Since tongue clicks are usually uttered intentionally, they appear as strong, long transient events (e.g., shown in Figure 1C). Therefore, it may be considered useful to use two (different) window sizes to detect both short and long transient events. In particular, in some possible implementations, a shorter (smaller) window size may be used to detect (primarily) speech click events in speech frames, and a longer window size may be used to detect (primarily) non-speech click events in non-speech frames. Thus, both short and long transient events can be detected efficiently and reliably. In addition, in some possible implementations, a sufficiently small hop size may also be used to achieve fine temporal resolution.

[0095] On the other hand, regarding the detection of non-vocal clicking events, although non-vocal clicking events have low energy, they are generally stronger than background noise and can therefore be identified by transient detection algorithms. In this disclosure, in general, to identify peak distributions (sometimes called large outliers) and distinguish them from flat distributions, a (first) index value k of the kurtosis of the short-time waveform (time domain) amplitude is used. w We propose using (block 6040). Then, the kurtosis translation degree k wThis is compared with a predetermined threshold (block 6100), and a tongue click (in this case, a non-vocal tongue click) may be detected (or determined), as shown in block 6060. The start and / or end positions of the non-vocal tongue click event detected as described above can then be easily defined as the positions where the kurtosis is above and / or below a predetermined threshold. Generally speaking, non-vocal tongue clicks can tend to be relatively long (e.g., 50 ms), so in some cases it may be effective to merge neighboring tongue click events within a predetermined gap / threshold, e.g., 25 ms (e.g., for the purpose of attenuation).

[0096] On the other hand, with regard to the detection of vocal tongue clicks, tongue clicks in voiced speech tend to appear as fast modulation and are therefore generally considered more difficult to detect. Ideally, if vocal harmonics are well modeled, the detection of vocal tongue clicks can rely on the residual waveform (subtracting harmonics) to detect arbitrary abrupt changes. However, this can generally involve the use of robust F0 (also called fundamental frequency) / harmonic estimation algorithms, which can complicate the detection algorithm. Therefore, in this disclosure, we propose using a quadratic sample difference (block 6050) to approximate the removal of slowly changing signal components (harmonics) so that latent transients can be revealed in general. Similar to the detection of non-vocal tongue clicks, the (second) index value k of short-time kurtosis is used. D This can be calculated for the difference (residual) waveform (again, block 6040). However, as those skilled in the art will understand, other forms of residual signals other than the quadratic sample difference may be used at this stage, insofar as they allow for the identification of potential transients.

[0097] In several possible implementations, the (second) index value of kurtosis is k D k is the (first) index value of kurtosis. W It can be evaluated against (or based on). More specifically,

number

[0098] Since tongue clicks usually occur in the voiced portion, the harmonic energy is very strong, and therefore the amplitude distribution is smooth (generally k W This can appear as a relatively small (meaning k) transient. As a result, this implicitly means a peak amplitude distribution (generally, a peak amplitude distribution, or in other words, k W To avoid detecting (meaning that it is loud) as a click sound. R It is relatively loud for vocal tongue clicks but relatively quiet for vocal transients, so it may be possible to distinguish between the two.

[0099] Furthermore, since vocal tongue clicks can be very short, it may generally be necessary to refine the (coarse) tongue click event location defined above with better sampling accuracy.

[0100] A simple method may be to locate the largest quadratic difference (generally meaning the fastest change) within the coarse clicker range detected by kurtosis. Then, using a predetermined vocal clicker duration (e.g., 5 ms), the refined start and / or end positions around the fastest-changing sample position can be detected. As will be understood and recognized by those skilled in the art, this can be achieved by any suitable means. For example (but not limited to), such a vocal clicker duration (e.g., 5 ms) can be simply divided equally before and after the fastest-changing sample position. That is, the center of the interval corresponding to the vocal clicker duration may lie on the fastest-changing sample position.

[0101] An example of such refinement processing is schematically shown in Figure 2. In particular, in the example in Figure 2, waveform 2100 generally represents the original input audio waveform, and waveform 2200 generally represents the second-order difference waveform obtained from the original waveform 2100. Then, as illustrated above, the refined range 2300 of the non-vocal tongue-clicking sound event can be determined based on the second-order difference waveform 2200.

[0102] Another possible refinement method may be to detect rapid modulations within the rough plosive sound range. In particular, by converting local minima / maxima to, for example, values -1 and +1 (or any other suitable values, e.g., values with different signs but equal magnitudes), the corresponding zero crossing rate (ZCR) (hereinafter also referred to as the "maximum / minimum change rate") can be used to characterize how fast the modulation is.

[0103] An example of this refinement process is schematically shown in FIG. 3. In particular, in the example of FIG. 3, similar to the example shown in FIG. 2, the waveform 3100 generally represents the original input audio waveform. However, in this refinement process, instead of using the second derivative, the maximum / minimum change rate waveform 3200 is obtained from the original waveform 3100. Then, as shown in FIG. 3, based on the maximum / minimum change rate waveform 3200, the refined ranges 3310, 3320, and 3330 of the non-speech plosive sound events can be determined.

[0104] In some possible implementation examples, in order to detect speech plosive sounds more accurately, the kurtosis threshold and the maximum / minimum change rate can be combined and used.

[0105] Regarding the detection of plosive sounds, as described above, plosive sound events generally appear as frequent strong transients just before speech (shown in the example of FIG. 1C). To distinguish plosive sound events from the two plosive sound events described below (i.e., speech plosive sounds and normal non-speech plosive sounds), it may be possible to rely on, for example, using spectral features to confirm a sudden change in resonance. In the present disclosure, generally, a step of using the spectral slope (hereinafter also denoted as "SpS") and also the high / low band peak ratio (hereinafter also denoted as "ratioHL") is proposed.

[0106] Generally speaking, in some possible implementation examples, the feature ratioHL is the maximum peak above a predetermined frequency freq HL (e.g., 1.5 kHz) and freq HLIt can be calculated as the amplitude ratio between the lowest maximum peak and the lowest peak. In some possible implementations, to avoid low-frequency noise, a (predetermined) low frequency freq is used. L It may be preferable to further select the maximum peak in the lower frequency band above (for example, 100 Hz).

[0107] In some possible implementations, for non-audio tongue clicks detected immediately before speech, ratioHL>th R (th R If it is within a certain threshold, it can then be considered a candidate for licking (for example, shown in block 6070 of Figure 6).

[0108] Typically, during licking events, the high / low band peak ratio (ratioHL) may tend to be larger, and the spectral slope may tend to be steeper due to high-frequency resonance. Since licking events are typically very long (e.g., typically 100 ms in duration) compared to small (normal) clicks, it may be suggested to refine the event start / end location based on features including ratioHL, SpS, and energy envelope.

[0109] In some possible implementations, the initial (rough) termination position (i.e., k) W The position detected by can be continuously extended as long as the following condition is met: 1) ratioHL > th R 2) SpS <th S , here, th S This reduces the energy to a predetermined threshold, and 3) the energy is reduced.

[0110] Further verification of the extended end position may be performed by comparing the skewness before and after event position refinement. In other words, event extension may simply involve adding samples with smaller amplitudes so that the sample amplitude distribution becomes "more skewed".

[0111] Naturally, any other suitable implementations may be adopted as appropriate, as will be understood and recognized by those skilled in the art.

[0112] Figure 4 is a schematic diagram illustrating an example of tongue tapping detection according to one embodiment of the present disclosure. In particular, the waveforms in Figure 4 schematically and illustratively show the original waveform, spectral slope (SpS), energy, and high / low bandwidth peak ratio (ratioHL), respectively.

[0113] In some cases, it may be necessary or desirable to avoid detecting speech transients as tongue clicks. In particular, speech transients can typically share certain similarities with tongue clicks in their nature, but on the other hand, they typically differ in magnitude and / or spectral characteristics. Therefore, it may be possible to proactively identify speech transients based on the VAD and / or centroid (COG, which can generally be considered as the average time of the signal) of a short-duration speech waveform, and thus avoid false detection as tongue clicks.

[0114] In some possible implementations, the COG can be calculated as follows:

number

[0115] The transition starting to the right side of the window signifies a positive value, which means COG > th COG (For example, th COG It can be used for transient detection by =0.2). More specifically, if the VAD indicates no voice, non-voiced tongue clicks are processed regardless of the COG. Conversely, if the VAD indicates voice, tongue clicks are processed if either COG is close to the start of the tongue click event and the COG If the value is higher than that, it will not be processed.

[0116] Broadly speaking, the use of "unnormalized" index values ​​(i.e., kurtosis) generally facilitates the selection of various levels of transients for correction, while the reason for using "normalized" index values ​​(i.e., COG) is that they allow for a more equivalently treated approach to speech transients.

[0117] After tongue clicks (including non-vocal tongue clicks, vocal tongue clicks, and tongue licks) are detected, the next step may be to attenuation (or correction) (i.e., tongue click removal) of these tongue clicks.

[0118] More specifically, the tongue click removal process proposed in this disclosure generally involves the observed spectral envelope (hereinafter referred to as "E") and the target envelope (hereinafter referred to as "E"), as exemplified in block 6080 of Figure 6. T This is based on spectral gain attenuation (block 6090 in Figure 6) derived from (denoted as ). More specifically, in some possible implementations, given the start / end positions of a click sound, it is proposed to take one block before (having envelope E0) and one after (having envelope E1) the click sound as reference frames. The spectral envelopes of these two reference frames can then function to estimate the target envelope for each short-time block covering the click sound event. In some possible implementations, the target envelope can then be easily calculated as a linear interpolation of the two reference envelopes. Therefore, the spectral gain is then defined by dividing the target envelope by the observed envelope, with the constraint that only attenuation is allowed. That is, for each bin k in a given frame b over a total of B frames, the attenuation gain can be calculated as follows:

number

[0119] Naturally, any other suitable implementations may be adopted as appropriate, as will be understood and recognized by those skilled in the art.

[0120] In particular, to avoid unintentionally altering speech harmonics, further constraints may be applied as needed to allow only high-frequency attenuation (e.g., above 4 kHz) for speech tongue clicks.

[0121] In some possible implementations, if residual estimation (with harmonic components removed) is available (for example, as illustrated in block 13040 of Figure 13), then envelope attenuation can be applied to the residual signal, and then the harmonic components can be added back as a processed output (for example, as illustrated in block 13090 of Figure 13).

[0122] In some possible implementations, other algorithms such as autoregressive modeling or granular-based approaches similar to pitch-synchronous waveform modeling may also be used to correct speech clicks. In particular, given the location of the click event, it may be possible to estimate the left and right local periods. By comparing neighboring periods, the click can be replaced with a simple crossfade using "waveform slices" that match the relative click locations within those periods. To select the left or right period for correction, it may be possible to simply select one of the smaller waveform differences. If there are continuous clicks, the above method may not be very effective, in which case a more generative approach may be a better option.

[0123] Figure 5 is a schematic diagram illustrating an example of spectral attenuation according to one embodiment of the present disclosure. In Figure 5, the observed spectral waveform, the processed spectral waveform, the observed envelope, and the target envelope are illustrated, respectively. As can be seen from the example in Figure 5, the spectral region of the (detected) tongue click is attenuated. For completeness, the example shown here in Figure 5 may relate to "tongue click removal," but it should be noted that similar or equivalent attenuation concepts can also be applied to "plosive removal." For example, it will be understood by those skilled in the art that in some implementations this may include smoothing the envelope of the residual spectrum.

[0124] Secondly, a method related to "plosive sound removal" according to the embodiment of this disclosure will be described.

[0125] Similarly, in a broader sense, the methods for automatic detection and adaptive attenuation of speech plosives described herein mainly involve two important aspects. Firstly, the feature of the Zero Crossing Maximum (ZCM) index value is used. Compared to the Zero Crossing Ratio (ZCR) index value, ZCM can be considered to simply take the maximum zero crossing length. Therefore, ZCM can generally be considered robust to noisy crossing information, especially when used in an average manner similar to ZCR. Secondly, accurate detection of plosive event boundaries can be performed based on low-frequency energy (LFE) and ZCM. In particular, outliers from the observed low-frequency energy distribution (e.g., for all short frames across a file or recording) are selected as possible (unpleasant) plosive events, and then the event time location / boundary can be refined using ZCM. Finally, plosive attenuation may generally be performed based on high-pass filtering in either the time domain or the spectral domain, using a filter order adapted to the LFE and a filter frequency adapted to the ZCM of the detected plosive.

[0126] The “plosive deplosing” method will be described in more detail below with reference to Figures 9 and / or 10. Figures 9 and / or 10 provide a schematic functional overview of the (plosive deplosing) technique according to the embodiments of the present disclosure, respectively. In a broad sense, Figure 9 may be considered a more general example, while Figure 10 may be considered a more detailed example of a particular possible implementation. Accordingly, it will be understood and recognized by those skilled in the art that the examples shown in Figures 9 and 10 may simultaneously exhibit some similarities (e.g., in some blocks) and differences (e.g., in some other blocks).

[0127] More specifically, as shown in blocks 9010 or 10010, an input audio signal can be given and segmented / divided into multiple (short-duration) overlapping audio frames (e.g., having equal frame sizes). It will be understood and recognized by those skilled in the art that this can be achieved in any suitable way. For example, in some possible implementations, this segmentation of audio frames can be achieved by performing short-duration frame analysis using a Hamming window. In particular, in some possible implementations, the frame size can be set large enough to allow the extraction of a reliable numerical value of the zero-crossing maximum. Similarly, the size of the overlap can be set large enough to track short-duration features with fine temporal resolution.

[0128] Subsequently, two short-time features (sometimes also called feature parameters) can be calculated (obtained): namely, the low-frequency energy (LFE), exemplified in block 9020 or 10020, and the zero-crossing maximum value (ZCM), exemplified in block 9040 or 10050.

[0129] The Low-Frequency Energy (LFE) can be calculated in either the time domain or the spectral domain using any suitable means. In some possible implementations, in the time domain, the LFE can be calculated as the mean-squared (RMS) energy of the signal passed through a low-pass filter. In some possible implementations, the low-pass filter may be, for example, a fourth-order Butterworth filter with a predetermined cutoff frequency of 80 Hz. On the other hand, in some other possible implementations, in the spectral domain, the LFE can be calculated from the spectrum as the RMS energy below the cutoff frequency.

[0130] As described above, ZCM is generally the length of the maximum interval between consecutive zero crossings within a short-time frame. The ZCM is likely to be further normalized by the window size. In particular, the techniques proposed in this disclosure do not generally rely on ZCR, although ZCR is typically used in plosive detection mechanisms.

[0131] Since low-frequency abrupt pops are generally a major challenge, plosive detection can begin by identifying outliers in the observed LFE distribution (blocks 9030 or 10030). In some possible implementations, outliers can be identified based on the concept / principle of a standard score.

number

[0132] If an outlier is present, the process may proceed to the next threshold detection stage. Otherwise, it can be assumed that there are no potentially (unpleasant) plosives requiring further processing. In non-restrictive examples, an outlier may be indicated by z > 1 (or any other appropriate value).

[0133] In some possible implementations, the adaptive threshold th LFEThis can be used for detected outliers to select the dominant component, as follows:

number

number

number

[0134] For online (real-time) processing where low latency may be required, the above statistical threshold may not be reliably estimated. Therefore, in some cases, it may be possible to use the LFE ratio instead of the current frame n, as follows:

number

[0135] Otherwise, the above ratio can be calculated using the LFE that was valid last time.

[0136] Here, the detection function can be expressed as R>1+f(α), where f(α) is a customizable mapping function. Therefore, the detection function can also be simply written as R>1+α.

[0137] In some possible implementations, frames exceeding a detection threshold may be used to define the signal region that is considered a plosive event to be attenuated, which also implicitly defines the (initial) time position where the plosive event begins and / or ends (blocks 9030 or 10040). However, the event boundary may require further refinement (blocks 9050 or 10060) because, typically, actual plosives can begin and / or end at very low energies. Therefore, in some possible implementations, for example, a ZCM index value (blocks 9040 or 10050) may be used to extend the boundary to frames where ZCM < 0.1 (or any other suitable value).

[0138] Furthermore, as with "tongue click removal," in some cases where two plosive events may overlap or be very close, the two plosive events may be merged into a single plosive event (for example, for further "plosive removal" processing).

[0139] Figure 7 schematically illustrates an example of a comparison between ZCM and ZCR. In particular, as can be seen from the example in Figure 7, the ZCM diagram 7100 is generally noisier than the ZCR diagram 7200 and is therefore better suited to identifying latent plosive events.

[0140] After the speech plosive events within the audio frame and the range / location / boundary corresponding to these events (block 9080) are determined, the next step (block 9110) is to attenuate (or correct) these plosives (i.e., remove them). In some possible specific embodiments (e.g., shown in Figure 10), attenuation may be performed by using high-pass filtering (e.g., shown in block 10070).

[0141] In particular, as with "tongue click removal," attenuation of plosive sounds can also occur in either the time domain or the spectral domain.

[0142] Broadly speaking, in several possible implementations, time-domain attenuation can be performed using a Butterworth high-pass filter with adaptive order and frequency (or any other suitable means). On the other hand, spectral-domain attenuation can be performed using a superimposed summation short-time Fourier transform (STFT) with adaptive spectral slope and frequency (or any other suitable means).

[0143] In particular, for both time-domain and spectral-domain attenuation, the attenuation frequency (block 9070) or, in some possible implementations, the filter (cutoff) frequency freq C (For example, an example is shown in block 10072) can be set to be adaptive to the “velocity” of the plosive event (block 9070). This is generally 1-max (ZCM). plosive ) can be defined as. The ZCM used here is normalized between 0 and 1, and max(ZCM plosive ) is the maximum ZCM from the start frame to the end frame of the plosive event. Here, the mapping can be defined as follows:

number

[0144] In several possible implementations, the cutoff frequency freq C This can be further constrained to a predetermined range, for example, [minFreq=100Hz, maxFreq=150Hz]. Naturally, any other suitable range can also be adopted, depending on the specific implementation and / or requirements.

[0145] With respect to time-domain decay, the order of the Butterworth filter can be adaptive to the intensity of the plosive event (block 9060). In particular, the plosive intensity st can be defined as follows in several possible implementations.

number

[0146] Next, the attenuation gain (example shown in block 9090) or, in some possible cases, the filter order (example shown in block 10071) can be obtained by mapping as follows:

number

[0147] In some possible implementations, the order may be further constrained to a predetermined range, for example, [minOrder=2,maxOrder=12]. Naturally, any other suitable range may also be adopted, depending on the specific implementation and / or requirements.

[0148] Furthermore, in some possible implementations, an additional crossfade region, for example, 10 ms, may be used to provide a smooth transition from the input signal to the filtered signal.

[0149] On the other hand, in the case of spectral domain attenuation, in several possible implementations, the input short-time signal is processed using a Fast Fourier Transform (FFT), followed by the application of an attenuation gain with an adaptive cutoff frequency and slope, an inverse FFT, and finally, windowing and superimposition to produce the (attenuated) output. Naturally, as will be understood and recognized by those skilled in the art, any other suitable attenuation mechanism may also be applied, depending on the respective implementation.

[0150] Based on the plosive intensity, the spectral low-cut / high-pass gain slope can also be estimated. In some possible implementations, for each plosive event, the target reduction gain can be defined as follows:

number

[0151] When intensity is expressed using the LFE ratio, this ratio can be directly expressed as the target gain. In some cases, when targetGain is expressed in dB (negative value for reduction), the attenuation gain slope can be defined as follows:

number

number

[0152] In some possible implementations, noise spectrum estimation can be used to limit gain reduction so that attenuation does not affect the overall spectral profile in the low-frequency range.

[0153] Therefore, broadly speaking, the proposed method generally attenuates faster pops with a higher cutoff frequency, thus effectively adapting to the speaker's voice pitch. Furthermore, the proposed method attenuates stronger pops with a steeper cutoff frequency slope, thus effectively adapting to both weak and strong plosives.

[0154] In particular, when the above techniques are applied to content containing music or a combination of voice and music, the algorithm is susceptible to low-frequency transients such as those generated by a kick drum or bass. To address this problem, in some possible implementations, a content classifier (e.g., a voice / music activity detector) is used to calculate the probability p(n) that a given frame n contains (or does not contain) voice, and by modifying detection or attenuation parameters, it is possible to ensure that music content is not affected by the deplosive processing. In some possible implementations, p(n) > th p (Here, th p Frames that are at a predetermined threshold can be removed from the LFE and ZCM pool to ensure relevant plosive detection and attenuation. Alternatively, a logistic mapping function f(p(n)) can be used to assign p(n) to a logistic mapping function, for example, p(n) (where, for example, The amount parameter of JPEG0007887403000016.jpg542 can be dynamically changed by multiplying by (a continuous function that approaches 0 and 1 as x approaches 0 and 1, respectively). κ generally represents the steepness parameter of the mapping.

[0155] In some implementations, particularly where computational complexity allows, another embodiment may be employed for frequency / spectral domain attenuation. This embodiment is described in more detail below.

[0156] In particular, it may be proposed to first use an analysis filter to generate (approximately) equivalent rectangular band (ERB) spaced frequency bands over the plosive frequency range below a (predetermined) frequency threshold (e.g., approximately 500 Hz), and in addition, to generate one or more bands above this frequency threshold (e.g., 500 Hz) to cover the remaining frequency range. At each time instant t, the energy in each of these bands b (denoted as e(b,t)) is used to control the reduction process described above to generate a set of gains g(b,t) applied to each filtered signal. This result is then fed to a composite filter bank to generate an output signal with reduced plosive energy.

[0157] More specifically, in several possible implementations, the plosive reduction gain in each band g(b,t) can be calculated first by the output of a compression curve based on the energy of the band, as follows:

number

number

[0158] In some possible implementations, a compression curve with a threshold T, knee-width W, and compression ratio R (all quality values ​​are expressed in decibels) can be described as follows:

number

[0159] As will be understood and recognized by those skilled in the art, any appropriate values ​​can be used for the threshold T, knee width W, and compression ratio R. For example, T=-65, W=10, and R=6 may be used. In this case, the compression curve can be 0 dB at low energy and only attenuation as the energy increases. It is also understood that T can be dynamically constructed over time using a time-smoothed energy envelope for speech.

[0160] In some possible implementations, the gain is then calculated as follows: the power in each band is equal to the estimated noise floor value in the band ( It may be displayed as JPEG0007887403000020.jpg510, or in dB units. It may be further clipped to ensure that it is not reduced below the resolution shown (which is labeled as JPEG0007887403000021.jpg513).

number

[0161] This generally avoids audible dips in noise when plosives can be present in the presence of significant background noise.

[0162] One possible method for estimating noise is as follows:

number

[0163] In some cases, the challenge lies in distinguishing between unwanted low-frequency energy in plosive events and desired low-frequency energy in vowels, especially when the lowest frequency is, for example, around 80 Hz. Depending on the specific implementation, these conditions can generally be met using several tools. More specifically, in some possible implementations, a time-smoothed low-frequency energy estimate of signals above the noise threshold, attempting to maintain compression gain, and a tonality index (or, in some possible implementations, a (certain) periodicity index) may be used to detect recurring peakiness in vowels and reduce the gain. These can be implemented as follows:

number

number

[0164] Next, this can be further thresholded and scaled to a useful range to generate the above factor, for example, as follows:

number

[0165] In some possible implementations, the tonality (or, in some cases, the periodicity index value) can be (best) estimated before being transformed into the filter bank domain. In some possible implementations, the filter bank can compute the FFT values ​​of overlapping windowed audio signals. For ease of illustration, in some possible implementations, we can assume that the power at the FFT bin p(k) is available and that bins k=0 to k=K are used, where K corresponds, for example, to 500Hz at a given sample rate.

[0166] Next, periodicity index values ​​(e.g., cepstrum in some possible implementations) can be calculated on these bins.

number

number

[0167] In some possible implementations, instead of the above, the tonal index value may be calculated by, for example, searching for the maximum spectral peak at p(k) within the frequency range of 60Hz to 250Hz, and requiring that the peak be a reasonable sinusoidal peak (the main lobe should be sufficiently narrow and deep). For example, the tonal index value can take values ​​from 0 to 1 (e.g., linearly) if the depth range at ±60Hz from the peak center is 5 to 15dB.

[0168] Furthermore, this value can be smoothed over time, for example, by using an attack time of 75 ms and a release time of 300 ms to provide smoothing adjustment.

[0169] This (smoothed) tonal index value and the above-calculated f lf These can be further combined to form a gain scale factor, as follows:

number

[0170] Furthermore, the periodicity / tonality index values ​​exemplified above may also be referred to as "audio harmonic protection index values" in this disclosure. In addition, the periodicity and tonality index values ​​may be used interchangeably.

[0171] Next, the above gain can be further constrained, as follows, so that the gain cannot be attenuated more with respect to a certain (predetermined) threshold, for example, a frequency band above 70 Hz, than with respect to the frequency band immediately below it.

number

[0172] Broadly speaking, the proposed methods generally follow the physical reduction of plosive energy as frequency increases by implementing the reductions described above. In particular, if lower frequencies are significantly reduced in energy, and then higher frequencies have greater energy, this is more likely to be true speech energy rather than plosive-related energy. Generally speaking, in some possible implementations, the lowest frequency band (e.g., below 70 Hz) may not follow this trend, for example, a surplus 60 Hz mains hum may make certain frequencies louder, and a DC block filter may attenuate the lowest frequency band, but this should not limit the attenuation of plosive energy.

[0173] Finally, in some possible implementations, these gains g4(b,t) may be further smoothed over time, for example, with an attack time of 20ms and a release time of 50ms, to produce a final gain (b,t). This final gain will be applied to the filtered signal (e.g., a subband signal). In some implementations, the final gain may be applied, for example, in a bandwise manner.

[0174] Figure 8 is a schematic diagram showing an example of attenuation of a plosive sound according to one embodiment of the present disclosure. In particular, as can be seen from Figure 8, the plosive sound event (see the white area in the low-frequency portion of diagram 8200) is effectively attenuated in the corresponding attenuation diagram 8100.

[0175] Figure 11 is a schematic flowchart illustrating an example of a method 11000 for performing automatic audio enhancement on an input audio signal containing at least one speech articulation noise event, according to one embodiment of the present disclosure.

[0176] In particular, the method described herein 11000 can be adapted to perform automatic audio enhancement (e.g., detection, attenuation, etc.) for either speech plosive noise events or tongue-clicking noise events.

[0177] More specifically, Method 11000 may begin in step S11010 by segmenting the input audio signal into multiple audio frames (e.g., 100ms in size) (e.g., using one or more appropriate windows). Method 11000 may then continue in step S11020 by obtaining at least one feature parameter from the (segmented) audio frames (e.g., determination, calculation, extraction, etc.). In some possible implementations, the feature parameter thus obtained may be associated with a certain type of (detection-targeted) speech articulation noise event. That is, in some possible implementations, it may be necessary to obtain different feature parameters from the audio frames depending on the type of (detection-targeted) speech articulation noise event. Finally, Method 11000 may continue in step S11030 by determining (e.g., detection, calculation, etc.) each type of speech articulation noise event and their respective ranges (e.g., time and / or frequency ranges) associated with the speech articulation noise event in the input audio signal, at least in part, based on the obtained feature parameter.

[0178] Broadly speaking, the proposed method 11000 provides an efficient and flexible mechanism for determining (detecting) expected / possible speech articulation noise events (e.g., artifacts) contained within the input audio signal. This can facilitate appropriate further enhancement (post) processing (e.g., attenuation). This significantly avoids the manual editing / processing that was previously required to identify and attenuate noise events in the audio signal. At the same time, it can greatly improve the listening experience.

[0179] Figure 12 is a schematic flowchart illustrating an example of a method 12000 for automatic audio enhancement of an input audio signal to detect and / or attenuate at least one speech articulation noise event contained within the input audio signal, according to another embodiment of the present disclosure. The speech articulation noise event may, among other things, include at least one speech plosive event. Therefore, the method 12000 described herein may be particularly suitable for automatic audio enhancement (e.g., detection, attenuation, etc.) for speech plosive noise events.

[0180] In particular, Method 12000 may begin in step S12010 by generating multiple approximately equirectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and multiple bands above it (the predetermined frequency threshold is within the frequency range of speech plosive events) by using an analysis filter bank. Method 12000 may then continue in step S12020 by applying multiple attenuation gains to the audio signal in each frequency band, respectively, where the attenuation gains are calculated based on the energy calculated for the frequency band. Finally, Method 12000 may further continue in step S12030 by feeding the attenuated audio samples into a synthesis filter bank for generating an output audio signal.

[0181] Broadly speaking, the proposed method 12000 provides an efficient and flexible mechanism for determining (detecting) and attenuating expected / possible speech articulation noise events (e.g., speech plosive events) contained within the input audio signal. This significantly avoids the manual editing / processing that was previously required to identify and attenuate noise (e.g., plosive) events in the audio signal. At the same time, it can greatly improve the listening experience.

[0182] Incidentally, although the methods / techniques for removing tongue clicks and plosive sounds appear to be illustrated separately, it should be noted that those skilled in the art will understand and recognize that at least some of the illustrated techniques can be used interchangeably.

[0183] As an example of non-limiting illustration, in several possible implementations, the filter bank approach (as described above for plosive removal) can also be applied to click-clack removal. Here, the spectral envelope is defined by the ERB band energy, and a similar multiband compression scheme (where the compressor ratio is determined by the target decay gain with respective attack / release times) can be applied. Note that the effective ERB band can extend to the Nyquist limit for click-clack removal, but is limited to low frequencies (e.g., 500 Hz) for plosive removal. Furthermore, instead of a periodicity index value based on cepstrum, it may be possible to use "residue" (described above only for click-clack removal) for plosive removal as well. Note that the residue for plosive removal cannot use quadratic sample differences and requires some other suitable estimation.

[0184] Figure 13 illustrates an example of a combined technique for both tongue click removal and plosive removal, as an overview of the (individual) functions.

[0185] In particular, note that functional blocks 13010, 13020, and 13030 in Figure 13 are generally similar to or identical to functional blocks 6010, 6020, and 6030 in Figure 6, and therefore, for the sake of brevity, they may not be described again. Furthermore, note that the dashed blocks shown in Figure 13 generally indicate that each functional process is optional, as will be explained in more detail below.

[0186] As described above, for plosive deactivation, ERB banding analysis (dashed block 13050) may be applied to detect the corresponding speech artifact (in this case, a speech plosive event) (example shown in block 13060) and then attenuate such speech artifact (block 13070). On the other hand, in the case of tongue click deactivation, after the speech artifact (in this case, a tongue click event) has been detected, an ERB-related procedure (or, in some cases, also called a filter bank approach) may be performed (block 13060). In such cases, such an ERB-related procedure may also be called ERB banding synthesis (example shown in dashed block 13080). ERB banding synthesis is used to attenuate the detected tongue click (block 13070). As described above, when the filter bank approach (as stated above with respect to plosive deactivation) should be applied to tongue click deactivation, the spectral envelope may be defined by the ERB band energy, and similar multiband compression schemes (where the compressor ratio is determined by target decay gains with respective attack / release times, or by envelope interpolation) may be applied. As will be understood and recognized by those skilled in the art, any other or further suitable treatments may be employed depending on various implementation examples and / or requirements.

[0187] Furthermore, as shown above and also in Figure 13, the techniques described herein may further utilize "residue" (if necessary) for both tongue click deactivation and plosive deactivation (used in place of periodic / tonal index values) (e.g., by removing speech harmonic components (example shown in dashed block 13040)). However, in such cases (i.e., when residual is used), harmonics may ultimately need to be restored or re-added after, for example, envelope attenuation has been applied to the residual signal (example shown in dashed / optional block 13090).

[0188] This disclosure also relates to apparatus for performing the methods and techniques described throughout this disclosure. Figure 14 shows an example of such apparatus 14000, which comprises a processor 14010 and a memory 14020 connected to the processor 14010. The memory 14020 may store instructions for the processor 14010. The processor 14010 may receive audio data 14030 as input. The audio data 14030 may have the properties described above with respect to each method of performing automatic audio enhancement on the input audio signal to detect and / or attenuate at least one speech articulation noise event contained in the input audio signal. The processor 14010 may be configured to perform the methods / techniques described throughout this disclosure. Thus, the processor 14010 may output de-de-noised (e.g., de-clicking, de-plosive) audio data 14040. In some further possible implementations, the processor 14010 may also be enabled to receive additional inputs (e.g., control parameters (not shown in Figure 14)) to control, for example, audio enhancement processing behavior.

[0189] interpretation A computing device implementing the above technology may have the following exemplary architecture. Other architectures are also possible, such as architectures with more or fewer components. In some implementation examples, the architecture may include one or more processors (e.g., dual-core Intel® Xeon® processors), one or more output devices (e.g., LCDs), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disc, flash memory, etc.). These components can communicate and exchange data via one or more communication channels (e.g., buses). Such communication channels can utilize various hardware and software to facilitate the transfer of data and control signals between components.

[0190] The term "computer-readable medium" refers, but is not limited to, any medium involved in giving instructions to a processor for execution, including non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wires, and optical fibers.

[0191] Computer-readable media may further include an operating system (e.g., the Linux® operating system), a network communication module, an audio interface manager, an audio processing manager, and a live content distributor. The operating system enables multi-user, multi-processing, multi-tasking, multi-threading, real-time, etc. The operating system performs basic tasks such as recognizing input from network interfaces and / or devices and providing output to them, tracking and managing files and directories on computer-readable media (e.g., memory or storage devices), controlling peripheral devices, and managing traffic on one or more communication channels. The network communication module includes various components for establishing and maintaining network connectivity (e.g., software for implementing communication protocols such as TCP / IP and HTTP).

[0192] The architecture can be implemented in a parallel processing infrastructure or a peer-to-peer infrastructure, or on a single device with one or more processors. The software can include multiple software components or it can be a single body of code.

[0193] The above features have the advantage that they can be implemented in one or more computer programs executable on a programmable system, which includes a data storage system and at least one programmable processor connected to receive and transmit data and instructions from at least one input device and at least one output device. A computer program is a set of instructions that can be used directly or indirectly within a computer to perform a particular action or to produce a particular result. Computer programs can be written in any form of programming language, such as compiled or interpreted languages ​​(e.g., Objective-C, Java), and can be deployed as a standalone program or in any form such as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0194] Processors suitable for executing instruction programs include, for example, both general-purpose and dedicated microprocessors, as well as one of a single processor or multiple processors or cores in any type of computer. Generally, a processor can receive instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data files, or can be operablely connected to them. Such mass storage devices include magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for materially embodying computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and all forms of non-volatile memory such as CD-ROM and DVD-ROM disks. Processors and memory can be assisted by or incorporated into ASICs (Application-Specific Integrated Circuits).

[0195] To provide user interaction, the above features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or Retina display device for displaying information to the user. The computer may have a touch surface input device (e.g., a touchscreen) or a keyboard and a pointing device such as a mouse or trackball that allows the user to provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0196] The above features can be implemented in a computer system including backend components such as data servers, a computer system including middleware components such as application servers or internet servers, a computer system including frontend components such as client computers with a graphical user interface or internet browser, or any combination thereof. The components of the system can be connected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include, for example, LANs, WANs, and computers and networks that form the internet.

[0197] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from that user). Data generated on the client device (e.g., the results of user interaction) can be received from the client device by the server.

[0198] One or more computer systems can be configured to operate by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform specific actions while it is running. One or more computer programs can be configured to operate by including instructions that cause the data processing device to perform specific actions when executed by the device.

[0199] This specification includes many specific implementation details, which should not be interpreted as limitations on the scope of any invention or claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Specific features described herein in relation to separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in relation to a single embodiment can also be implemented separately or in any appropriate subcombination in multiple embodiments. Furthermore, features are described above as operating in a particular combination, and may even be initially described in the claims as such, but one or more features from a combination of claims can, in some cases, be extracted from that combination, and such combination of claims may be directed towards subcombinations or variations of subcombinations.

[0200] Similarly, although the drawings show operations in a specific order, this should not be understood as requiring that such operations be performed in a specific order or sequence shown in the drawings, or that all exemplary operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the above program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0201] Unless otherwise specified, as is evident from the following statements, the use of terms such as “process,” “calculate,” “calculate,” “determine,” and “analyze” throughout the disclosed statements is understood to refer to the operation and / or process of a computer or computing system or similar electronic computing device that manipulates and / or converts data expressed as physical quantities, such as electron quantities, into other data similarly expressed as physical quantities.

[0202] Throughout this disclosure, any reference to “one embodiment,” “several embodiments,” or “embodiment” means that any particular feature, structure, or property described in relation to such embodiment is included in at least one embodiment of this disclosure. Therefore, the phrases “in one embodiment,” “several embodiments,” or “in an embodiment” appearing in various places throughout this disclosure do not necessarily all refer to the same embodiment. Furthermore, any particular feature, structure, or property may be combined in one or more embodiments in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0203] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” and “third” to describe common objects merely indicates that different examples of similar objects are being referred to, and is not intended to imply that the objects described in this manner must be in a given order, whether temporally, spatially, in a ranking order, or in any other way.

[0204] Furthermore, it should be understood that the terms and descriptions used herein are for illustrative purposes only and should not be considered limiting. The use of “includes,” “equipped with,” or “possess,” and their variations, means to include the items listed thereafter and their equivalents, as well as any additional items. Unless otherwise specified, the terms “attached,” “connected,” “supported,” and “joined,” and their variations, are used in a broad sense and include both direct and indirect attachment, connection, support, and joining.

[0205] In the following claims and throughout this specification, both the terms “equipped with” and “inclusive” are open terms meaning that they include at least the elements / features that follow them, but do not exclude other elements / features. Therefore, the term “equipped with,” when used in the claims, should not be interpreted as being limited to the means, elements, or processes (processes) listed thereafter. For example, the expression “a device comprising A and B” should not be limited to a device consisting only of elements A and B. The term “inclusive,” as used herein, is also an open term meaning that it includes at least the elements / features that follow it, but does not exclude other elements / features. Therefore, “inclusive” is synonymous with and means the same as “equipped with.”

[0206] In the above description of the embodiments of this disclosure, please understand that various features of this disclosure may be summarized in a single embodiment, figure, or description thereof in order to simplify the disclosure and to aid in the understanding of one or more of the various inventive aspects. However, the method of this disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly described in each claim. Rather, as reflected in the following claims, the inventive aspects consist of fewer features than all the features of the single embodiment disclosed above. Accordingly, the claims following this specification are expressly incorporated herein, and each claim exists in itself as a separate embodiment of this disclosure.

[0207] Furthermore, some embodiments described herein include some features included in other embodiments and do not include other features, but combinations of features of different embodiments are, as will be understood by those skilled in the art, within the scope of the disclosure and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0208] Numerous specific details are provided in this specification. However, it should be understood that the embodiments of this disclosure may be carried out without these specific details. In other examples, well-known methods, structures, and techniques are not described in detail so as not to obscure the understanding of this description.

[0209] Therefore, while we have described what we believe to be the best form of this disclosure, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of this disclosure, and it is intended that all such variations and modifications will be within the scope of this disclosure. For example, any of the above formulas merely represent the procedures that may be used. Functions may be added to or removed from the block diagrams, and operations may be interchangeable between function blocks. Steps may be added to or removed from the methods described within the scope of this disclosure.

[0210] Furthermore, various aspects and implementation examples of this disclosure can be understood from the following enumerated example embodiments (EEEs) (not patent claims).

[0211] EEE1. A method for detecting and attenuating tongue click sounds in the recording of audio content, a. A step of dividing the audio into audio frames and non-audio frames, b. The process of calculating the quadratic waveform difference for the audio frame, c. A step of detecting a tongue click sound based on the kurtosis of each short-time waveform, d. calculating a target spectral gain based on interpolation of a spectral envelope between the start and end of a click sound; e. adapting the gain to each frame and performing overlap-and-add resynthesis; A method based on:

[0212] EEE2. The method of EEE1, where the identification of voice and non-voice frames is provided by an existing VAD (voice activity detector).

[0213] EEE3. The method of EEE1, where noise reduction can be applied to the input signal as necessary to better reveal potential click sounds.

[0214] EEE4. The method of EEE1a, where two window sizes are used to detect voice clicks (short) and non-voice clicks (long) respectively.

[0215] EEE5. For click sound detection, the kurtosis k W of the original waveform and the kurtosis of the second waveform difference are calculated, the method of EEE1c.

[0216] EEE6. The method of EEE1c, where click sounds are detected by a predetermined kurtosis threshold. The threshold may be different for k W and k D can be different.

[0217] EEE7. Non-voice click sounds are detected based on k W and voice click sounds are detected based on k D - α * k W where α is a weighting parameter, the method of EEE5.

[0218] EEE8. The method of EEE7, where voice transients can be further excluded from kurtosis-based detection. This needs to be corresponding only to voice click sounds.

[0219] EEE9. The method of EEE8, where voice transients can be detected based on the centroid (average time) of the short-time signal.

[0220] EEE10. The start of the tongue slap sound is defined as when the kurtosis exceeds the threshold, and the end of the tongue slap sound is defined as when the kurtosis falls below the threshold. Therefore, a tongue slap sound event is usually a method of EEE7 that covers several consecutive short-time frames.

[0221] EEE11. Since non-speech tongue slap sounds tend to have a long duration, it is preferable to merge nearby non-speech tongue slap sounds, which is a method of EEE7.

[0222] EEE12. A non-speech tongue slap sound immediately before the start of speech is regarded as a candidate for the lingual trill, which is a method of EEE7.

[0223] EEE13. The end position of the lingual trill event is extended based on the following features, namely, spectral slope, high / low peak ratio, and energy envelope, which is a method of EEE12.

[0224] EEE14. The high / low peak ratio is defined as the amplitude ratio between the maximum peak in the high-frequency band and the maximum peak in the low-frequency band, which is a method of EEE13.

[0225] EEE15. The high / low frequency bands are separated by a predetermined frequency, for example, 1.5 kHz, which is a method of EEE14.

[0226] EEE16. Since speech tongue slap sounds tend to be short, it is preferable to refine the start / end sample positions, which is a method of EEE7.

[0227] EEE17. A simple refinement method is to identify the position of the maximum second waveform difference (maxD) within the initial tongue slap sound range detected by kurtosis, which is a method of EEE16. At this time, using a predetermined speech tongue slap sound period (for example, 2 ms), the refined start / end positions near maxD can be determined.

[0228] EEE18. Another refinement method is the "maximum / minimum rate of change" method of EEE16. This is the zero-crossing rate of the transformed waveform (cZCR), which is -1 / +1 at the local minimum / maximum and 0 elsewhere. Frames with a cZCR above the threshold define the refined position.

[0229] EEE19. Spectral gain attenuation is calculated using the EEE1d method based on the observed spectral envelope and the target spectral envelope. Taken over by the spectral envelope, the spectral gain defines the frequency-dependent gain value in each spectral bin.

[0230] EEE20. The target spectral envelope can be estimated by the EEE19 method, which involves linear interpolation of the spectral envelope between two "clean" frames (without the clicking event) at each end of the clicking event.

[0231] EEE21. The spectral gain in each short-time frame is defined as the target envelope divided by the observed envelope, according to the method of EEE19.

[0232] EEE22. The spectral gain is limited only to attenuation, in the same manner as EEE19. The resulting amplification gain is forced to be set to 1.

[0233] EEE23. The spectral gain for an audio frame is applied to the spectral region above a given voiced frequency (e.g., 4 kHz) using the method of EEE19.

[0234] EEE24. A method for detecting and attenuating unwanted plosive events in the recording of audio content, a. The process of dividing the audio into overlapping frames, b. A process of analyzing the low-frequency energy (LFE) and zero-crossing maximum value (ZCM) of each frame, c. Detecting a plosive sound event having a correct start / end time position; d. Attenuating the plosive sound by high-pass filtering having an adaptive order and cut-off frequency; A method based on the above.

[0235] EEE25. LFE can be calculated in the time domain or in the spectral domain having a predetermined cut-off frequency, according to the method of EEE24b.

[0236] EEE26. Time-domain LFE can be calculated as the RMS energy of the input signal passing through a low-pass filter, according to the method of EEE25.

[0237] EEE27. Spectral-domain LFE can be calculated as the RMS energy of the short-time spectrum below the cut-off frequency, according to the method of EEE25.

[0238] EEE28. ZCM is the maximum interval of consecutive zero crossings normalized by the window size, according to the method of EEE24b.

[0239] EEE29. The frame size is set to be large enough to extract a reliable value of the zero-crossing maximum. The overlap size is set to be large enough to track short-time features with fine time resolution, according to the method of EEE24a.

[0240] EEE30. Plosive sound detection is based on the selection of outliers in the LFE distribution over all short-time frames of one file, according to the method of EEE24c.

[0241] EEE31. Outliers are detected by standard scores, and an adaptive threshold is used to select dominant outliers, according to the method of EEE30.

[0242] EEE32. The threshold is adaptive with respect to the difference between the maximum LFE and the standard score threshold multiplied by a scaling factor, according to the method of EEE31.

[0243] EEE33. The scale factor can be derived from the global plosive removal control [0,1] using the method of EEE32.

[0244] EEE34. Plosive detection for low latency use is based on the EEE24c method, which is based on the LFE ratio between two neighboring frames.

[0245] An EEE34 method in which a predetermined threshold, which can be defined as the EEE35.1+ detection sensitivity, is used for detection.

[0246] EEE36. Consecutive frames exceeding a threshold define the duration of a plosive event, as in the methods of EEE32 or EEE34.

[0247] The method of EEE24c, in which the boundary of the initial plosive event is defined by the methods of EEE37 and EEE36.

[0248] EEE38. The initial event boundary is further refined based on the ZCM method of EEE36.

[0249] EEE39. The method of EEE38, wherein the start and end positions are extended until the ZCM falls below a predetermined threshold.

[0250] EEE40. Attenuation processing can be performed in the time domain or the spectral domain, using the method of EEE24d.

[0251] EEE41. The filter frequency is adaptable to ZCM under a predetermined frequency constraint, according to the method of EEE40.

[0252] EEE42. Time-domain attenuation is performed using the method of EEE40, which employs a Butterworth filter having a filter order that is adaptive to the intensity of low-frequency energies under a given range of order constraints.

[0253] EEE43. The method of EEE42, wherein the filtered output crossfades with the original input signal at the event boundary for a predetermined transition period.

[0254] EEE44. Spectral domain attenuation is performed using the EEE40 method, which employs the standard STFT superimposed summation framework.

[0255] EEE45. The spectral attenuation gain slope is adaptive to the intensity of low-frequency energies, as in EEE44.

[0256] EEE46. The gain slope is expressed in dB / octave units below the cutoff frequency. (See EEE45 method.)

[0257] EEE47. To prevent excessive suppression, the attenuation gain can be limited by the estimated noise spectrum, as in EEE44.

[0258] EEE48. The scale factor can include the probability of speech obtained from the content classifier, as in the EEE32 method. Therefore, the resulting factor is used to weight the detection threshold to avoid processing non-voice frames.

[0259] EEE49. A method for detecting and attenuating tongue clicks in audio data, The process of receiving multiple audio frames representing audio data, A step of calculating one or more short-time waveforms based on the plurality of audio frames, A step of detecting one or more tongue-clicking sounds based on the kurtosis of one or more short-time waveforms, A step of calculating a set of target spectral gains based at least partially on interpolating the spectral envelope between the start and end of one or more detected tongue-clicking sounds, The process of applying the aforementioned set of target spectral gains to the plurality of audio frames and performing superimposed summation resynthesis to attenuate one or more tongue-clicking sounds, A method that includes

[0260] EEE50. Further comprising the step of classifying each of the plurality of audio frames as either an audio frame or a non-audio frame, The step of calculating one or more short-time waveforms based on the plurality of audio frames is: A step of calculating the original waveform derived from the audio content, A step of calculating the quadratic waveform difference for the aforementioned audio frame, Includes, The process of detecting one or more tongue clicks is: A step of detecting one or more tongue-clicking sounds in the non-speech frame using the original waveform derived from the audio content, A step of detecting one or more tongue-clicking sounds in the audio frame using the quadratic waveform difference of the audio frame, to include, The EEE49 method.

[0261] EEE51. A method of EEE49 or 50, further comprising the step of denoising the audio frame before calculating the one or more short-time waveforms.

[0262] EEE52. The step of classifying the plurality of audio frames into speech frames or non-speech frames is performed by an existing voice activity detector, using either of the methods of EEE50 and 51.

[0263] EEE53. One or more tongue clicks in the aforementioned audio frame are considered to be a first predetermined kurtosis threshold (K T1 ) Detected by one of the methods EEE49-52.

[0264] EEE54. One or more tongue clicks in the sound frame have a second predetermined kurtosis threshold (K) that is different from the first predetermined kurtosis threshold. T2 The EEE53 method, which is detected according to ).

[0265] EEE55. A speech transient is detected and excluded from the tongue click detection based on the kurtosis, by any of the methods described in EEE49-54.

[0266] EEE56. The method of EEE55, wherein the audio transient is detected at least in part on the centroid (average time) of the original waveform derived from the audio content (e.g., a short-time signal based on the audio content).

[0267] EEE57. The start of each tongue click sound is at kurtosis K. T It is defined as when the kurtosis exceeds K, and the end of each tongue click is defined as when the kurtosis is K T Any of the methods defined in EEE49-56 as when the value falls below a certain threshold.

[0268] EEE58. The step of detecting one or more tongue clicks in a non-voice frame includes any method from EEE50 to 57, wherein the step of merging non-voice tongue clicks separated by less than a first period is included.

[0269] EEE59. Any method of EEE50-58, wherein the step of detecting one or more tongue clicks in the audio frame further comprises the step of refining the start and end positions of each of the one or more tongue clicks in the audio frame.

[0270] EEE60. The process of refining the start and end positions is: For each tongue click sound, the process involves identifying the position of the maximum quadratic waveform difference (MD) within the coarse tongue click sound range detected by kurtosis, and A step of defining the refined start position or refined stop position of each of the aforementioned tongue-clicking sounds based on a predetermined tongue-clicking sound duration, including, The EEE59 method.

[0271] EEE61. The process of refining the start and end positions is: The process includes defining the refined start or refined stop position of each tongue click based on the zero-crossing ratio of the transformed waveform (cZCR) (for example, the transformed waveform maps the local minimum / maximum values ​​of the observed waveform to -1 / 1 and all other values ​​to 0), The EEE59 method.

[0272] EEE62. The method of EEE48, wherein the set of target spectral gains is calculated based at least in part on the observed spectral envelope and the target spectral envelope.

[0273] EEE63. The method of EEE62, wherein the target spectral envelope is estimated by linear interpolation of the spectral envelope between two "clean" frames (e.g., peripheral frames that do not contain the clicking event) at each end of the clicking event.

[0274] EEE64. The method of EEE62, in which the set of target spectral gains in each short-time frame is defined as the target envelope divided by the observed envelope.

[0275] EEE65. The method of EEE64, in which the set of target spectral gains is limited to attenuation only. (For example, the resulting amplification gain is forcibly set to 1).

[0276] EEE66. The method of EEE64, wherein a set of target spectral gains for the speech frame is applied to a spectral region above a predetermined voiced frequency.

[0277] EEE67. A method for detecting and attenuating unwanted plosive events in audio, including speech content, The process of dividing the aforementioned audio into multiple overlapping frames, A step of determining the low-frequency energy of each of the multiple overlapping frames, A step of determining the zero-crossing maximum value of at least one of the multiple overlapping frames, A process for detecting multiple plosive events having precise start / end time positions, A step of generating output audio by attenuating the plurality of plosive events using an adaptive high-pass filter, wherein the order and cutoff frequency of the adaptive high-pass filter are adapted to each of the plurality of plosive events. A method based on this.

[0278] EEE68. The low-frequency energy is the RMS energy of the input signal after passing through a low-pass filter, according to the method of EEE67.

[0279] EEE69. The maximum zero crossing value is the maximum interval between consecutive zero crossings, normalized by the window size, according to the EEE67 method.

[0280] EEE70. The frame size is set to be large enough to extract a reliable value of the zero-crossing maximum, as in EEE67. The overlap size is set to be large enough to track short-time features with fine temporal resolution.

[0281] EEE71. The method of EEE67, wherein the step of detecting a plurality of plosives includes the step of detecting outliers in the low-frequency energy distribution across all short frames of a file according to a first threshold.

[0282] EEE72. The step of detecting the plurality of plosive events is: A step of calculating a threshold for LFE outlier detection based on standard scores, A step of applying a second threshold different from the first threshold (for example, an adaptive threshold used to select the dominant component), Further encompassing, One of the methods described in EEE67-71.

[0283] EEE73. The method of EEE72, wherein the second threshold is adaptive to the difference between the largest outlier and the first threshold.

[0284] EEE74. The method of EEE73, in which consecutive frames exceeding the adaptive threshold define the duration of a plosive event.

[0285] EEE75. The method of EEE73, in which the global attenuation effect size [0,1] is mapped to the adaptive threshold scaled by the factor.

[0286] EEE76. The initial plosive event boundary (e.g., start / stop position) is defined by the method of EEE67, as defined by the method of EEE73.

[0287] EEE77. Any method from EEE67 to 74, further comprising the step of refining the location of a plosive event (e.g., an initial boundary) based on the zero-crossing maximum value.

[0288] EEE78. The method of EEE77, further comprising the step of extending the start and end positions of a plosive event until the zero-crossing maximum value falls below a predetermined threshold.

[0289] EEE79. The method of EEE67, wherein the step of generating output audio includes a step of crossfading at the boundaries of the plurality of plosive events using a predetermined transition period.

[0290] EEE80. The filter order is adaptive to the intensity of low-frequency energy within a predetermined order range, according to the method of EEE67.

[0291] EEE81. The method of EEE67, wherein the cutoff frequency is adaptive to the numerical zero crossing maximum value within a predetermined cutoff frequency range.

[0292] EEE82. A step of obtaining the probability of audio from a content classifier for one or more of the multiple overlapping frames, If each of the aforementioned probabilities is less than the first classification threshold, the process involves reducing the detection amount (for example, by changing the global attenuation effect), The EEE75 method further encompasses this.

[0293] EEE83. A step of obtaining the probability of audio from a content classifier for one or more of the multiple overlapping frames, If each of the aforementioned probabilities is less than the second classification threshold, the process of removing the frame from the detected plosive event is performed. The EEE75 method further encompasses this.

[0294] EEE84. The step of attenuating the multiple plosive events using an adaptive high-pass filter is: A step of filtering out the first plosive event of the plurality of plosive events using a first filter order and a first cutoff frequency, A step of filtering a second plosive event of a plurality of plosive events using a second filter order and a second cutoff frequency, wherein at least one of the second filter order and the second cutoff frequency is different from the first filter order and the first cutoff frequency, respectively. including, EEE67 method.

[0295] EEE85. The adaptive high-pass filter is a Butterworth filter, according to the method of EEE67.

[0296] EEE86. A non-temporary computer-readable storage medium that stores one or more programs containing instructions that perform any of the methods described in EEE67-85 when executed by one or more processors.

[0297] EEE87. An electronic device comprising one or more processors and memory, wherein the memory stores one or more programs that include instructions causing the device to perform any of the methods described in EEE67 to EEE85 when executed by the one or more processors.

[0298] EEE88. A method for automatically enhancing an input audio signal that includes at least one speech articulation noise event, The process of segmenting the input audio signal into multiple audio frames, A step of obtaining at least one feature parameter from the audio frame, A step of determining, at least in part, the type of speech articulation noise event and the respective time-frequency range related to the speech articulation noise event in the input audio signal, based on the obtained characteristic parameters, A method that includes

[0299] EEE89. The method of EEE88, wherein the determined range includes at least one boundary of the determined speech articulation noise event in the time and / or spectral domain.

[0300] EEE90. A step of attenuating the speech articulation noise event according to the determined type and range of the speech articulation noise event. A method of EEE88 or 89 that further encompasses this.

[0301] EEE91. The speech articulation noise event is any one of the preceding EEE methods, which includes at least one of a tongue-clicking event or a speech plosive event.

[0302] EEE92. The method of EEE91, wherein the speech articulation noise event includes one or more tongue-clicking events, and the one or more tongue-clicking events include at least one of a non-speech tongue-clicking event, a speech tongue-clicking event, or a tongue-tapping event.

[0303] EEE93. After the step of segmenting the input audio signal into multiple audio frames, The process of classifying the aforementioned audio frame into either an audio frame or a non-audio frame. The EEE92 method further encompasses this.

[0304] EEE94. The method of EEE93, wherein the input audio signal is identified and segmented into voice frames and non-voice frames by using a voice activity detector (VAD).

[0305] EEE95. The segmentation is performed by using two different window sizes, one of which is shorter than the other, in any one of the methods EEE92 to 94.

[0306] EEE96. The method of EEE95, when dependent on EEE93 or 94, wherein the shorter window size is used to detect speech click events in the speech frame, and the longer window size is used to detect non-speech click events in the non-speech frame.

[0307] EEE97. The step of obtaining at least one feature parameter from the audio frame is: For each audio frame, the process of obtaining at least one kurtosis index value based on the time-domain sample amplitude of that audio frame. It includes, and, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: The steps include comparing the obtained kurtosis index value with a predetermined kurtosis threshold, If the kurtosis index value exceeds the predetermined kurtosis threshold, it is determined that the audio frame contains a tongue-clicking sound event, and the start and end boundaries of the tongue-clicking sound event are determined based on the positions where the kurtosis index value is above and below the predetermined kurtosis threshold, respectively. to include, One of the methods from EEE91 to 96.

[0308] EEE98. The step of obtaining at least one feature parameter from the audio frame is: For each audio frame, the process involves obtaining an approximate value of the residual without audio harmonic components and a first kurtosis index value of the sample amplitude for the approximate value of the residual. Includes, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: The process involves comparing the obtained first kurtosis index value with a first predetermined kurtosis threshold, If the first kurtosis index value exceeds the first predetermined kurtosis threshold, the audio frame is determined to include an audio tongue-clicking event, and the start and end boundaries of the audio tongue-clicking event are determined based on the positions where the first kurtosis index value exceeds and falls below the first predetermined kurtosis threshold, respectively. including, One of the methods from EEE93 to 97.

[0309] EEE99. The approximate amount of residual without audio harmonic components is the second-order waveform difference, according to the method of EEE98.

[0310] EEE100. The above method is, A step to obtain a second kurtosis index value from the residual sample amplitude of the aforementioned audio frame. To further encompass, The type and range of the speech articulation noise event are determined based on the second kurtosis index value relative to the first kurtosis index value. EEE98 or 99 method.

[0311] EEE101. A step of identifying the sample position having the largest quadratic difference within the determined range of the audio tongue-clicking sound event, The process of determining the refined range of the audio tongue-clicking sound event by applying a predetermined duration of the audio tongue-clicking sound event to the area around the location-identified sample position, The process of refining the determined range of the aforementioned tongue-clicking sound event. One of the methods from EEE98 to 100, which further encompasses this.

[0312] EEE102. A step of determining the range of the speech tongue-clicking sound event based on the maximum / minimum change rate calculated from the local minimum and maximum values ​​in the speech frame. Any one of the methods from EEE98 to 101 further encompasses this.

[0313] EEE103. The step of obtaining at least one feature parameter from the audio frame is: For each non-speech frame, the process of obtaining the third kurtosis index value for each time-domain sample amplitude in that non-speech frame. It includes, and, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: A step of comparing the obtained third kurtosis index value with a second predetermined kurtosis threshold, If the third kurtosis index value exceeds the second predetermined kurtosis threshold, it is determined that the non-voiced frame includes a non-voiced tongue-clicking event, and the start and end boundaries of the non-voiced tongue-clicking event are determined based on the positions where the third kurtosis index value exceeds and falls below the second predetermined kurtosis threshold, respectively. to include, One of the methods from EEE93 to 102.

[0314] EEE104. A process of merging two neighboring non-audible tongue-clicking events into one audible tongue-clicking event when those events are within a predetermined gap threshold. The EEE103 method further encompasses this.

[0315] EEE105. Regarding non-audio tongue-clicking events determined in the non-audio frame immediately preceding the audio frame, A step of calculating the high / low bandwidth peak ratio as the amplitude ratio of the maximum peak above a predetermined frequency and the maximum peak below the predetermined frequency, If the calculated high / low bandwidth peak ratio is above a predetermined ratio threshold, the non-vocal tongue-clicking event is determined to be a tongue-clicking event. Method EEE103 or 104.

[0316] EEE106. The high / low bandwidth peak ratio is calculated as the amplitude ratio between the maximum peak above a predetermined frequency and the maximum peak below the predetermined frequency but above a further predetermined low frequency. The EEE105 method.

[0317] EEE107. A step of refining the determined range of the tongue click event based on the high / low band peak ratio, spectral slope, and energy envelope. The methods of EEE105 or 106, which further encompass the above.

[0318] EEE108. The step of refining the determined range of the tongue click event is: The process of extending the end position of the tongue-tapping event, determined by using the third kurtosis index value, provided that the high / low bandwidth peak ratio is above the predetermined ratio threshold, the spectral slope is below the predetermined slope threshold, and the energy in the energy envelope is reduced. to include, Method EEE107.

[0319] EEE109. A step to determine the speech articulation noise event according to a further predetermined threshold, based on the centroid COG calculated for the speech frame, in order to distinguish between tongue-clicking events and speech transients. One of the methods from EEE93 to 102 further encompasses this.

[0320] EEE110. A step of attenuating one or more determined tongue-clicking events based on the spectral envelope of the audio frame containing the detected tongue-clicking event, and the respective spectral gains derived from the target envelope calculated based on the respective reference frames. One of the methods from EEE98 to 109, which further encompasses this.

[0321] EEE111. The EEE110 method, wherein for each detected tongue-clicking event, the reference frame includes the audio frame preceding and following the audio frame containing the detected tongue-clicking event, and the target envelope is calculated by interpolating the spectral envelope of the reference frame.

[0322] EEE112. The attenuation is applied to a frequency band higher than a predetermined high-frequency threshold, according to the method of EEE110 or 111.

[0323] EEE113. A step of replacing one or more determined tongue-clicking events based on each neighboring audio frame. One of the methods from EEE98 to 109, which further encompasses this.

[0324] EEE114. The steps of obtaining at least one feature parameter from the audio frame include: The process of obtaining a low-frequency energy (LFE) index value for each of the audio frames in order to identify outliers in the low-frequency energy (LFE). to include, EEE91 method.

[0325] EEE115. The LFE index value is calculated in either the time domain or the spectral domain, using the method of EEE114.

[0326] EEE116. A step of determining the range of the audio plosive event according to the outliers identified from the LFE index value and the threshold calculated based on the LFE index value, or according to the LFE ratio calculated from the previous and current audio frames. Methods of EEE114 or 115, which further encompass the above.

[0327] EEE117. The above-mentioned length is In order to refine the range of the plosive sound event determined based on the LFE index value, the process of determining the respective zero-crossing maximum ZCM index value for each of the audio frames. To further encompass, The ZCM index value indicates the length of the maximum interval between consecutive zero crossings within the audio frame. The EEE116 method.

[0328] EEE118. A step of attenuating the determined plosive event, wherein the attenuation is performed in either the time domain or the spectral domain. The methods of EEE116 or 117, which further encompass the above.

[0329] EEE119. The method of EEE118, wherein the time-domain attenuation is performed by applying a high-pass filter, the cutoff frequency of the filter is determined based on the ZCM index value for the audio frames within the range of the determined speech plosive events, and the order of the filter is determined based on the LFE index value for the audio frames within the range of the determined speech plosive events.

[0330] EEE120. The spectral domain attenuation is performed by the method of EEE118, using adaptive spectral gradient and frequency and employing the superimposed summation short-time Fourier transform (STFT).

[0331] EEE121. The spectral domain attenuation method of EEE118 or 120, comprising the steps of processing the audio frame using a Fast Fourier Transform FFT to generate an attenuated output audio signal, applying an attenuation gain having an adaptive slope and frequency, and applying an inverse FFT, windowing and superimposition, wherein the frequency is determined based on the ZCM index value for the audio frame within the range of the determined speech plosive events, and the slope is determined based on the LFE index value for the audio frame within the range of the determined speech plosive events.

[0332] EEE122. A step of applying noise spectrum estimation in order to limit the attenuation gain and prevent excessive suppression. The EEE121 method further encompasses this.

[0333] EEE123. The process of applying a content classifier to the audio frame in order to distinguish between audio frames and non-audio frames in order to determine the audio plosive event. One of the methods from EEE114 to 122 further encompasses this.

[0334] EEE124. The spectral domain attenuation is, A step of generating a plurality of substantially equirectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and a plurality of bands above the predetermined frequency threshold, wherein the predetermined frequency threshold is within the frequency range of the determined speech plosive events, A step of applying multiple attenuation gains to the audio signal in each of the frequency bands, wherein the attenuation gains are calculated based on the energy calculated for the frequency band; To generate an output audio signal, the process involves providing the attenuated audio sample to a synthesis filter bank, including, The EEE118 method.

[0335] EEE125. The method of EEE124, wherein the attenuation gain in each frequency band is further constrained such that the energy in that frequency band is not reduced below the estimated noise threshold in that frequency band.

[0336] EEE126. A step to calculate time-smoothed low-frequency energy estimates of audio samples above the estimated noise threshold in order to distinguish between speech plosive events and higher-frequency content in the input audio signal. The EEE125 method further encompasses this.

[0337] EEE127. A step of calculating the audio harmonic protection index value in the spectrum of the input audio signal, A step of calculating the attenuation gain according to the aforementioned audio harmonic protection index value and the time-smoothed low-frequency energy estimate, The EEE126 method further encompasses this.

[0338] EEE128. The method of EEE127, wherein the audio harmonic protection index value is a periodic or tonal index value.

[0339] EEE129. The periodicity index value in the spectrum is calculated from the cepstrum of the audio sample before the final bandwidth calculation of the analysis filter bank, by the method of EEE128.

[0340] EEE130. The method of EEE128, wherein the tonal index value in the spectrum is calculated based on the principal lobe of the spectral peak compared to the principal lobe of the sinusoidal peak, prior to the final bandwidth calculation of the analysis filter bank.

[0341] EEE131. A step of further constraining the calculated attenuation gain based on the frequency band immediately below the frequency. One of the methods from EEE127 to 130, which further encompasses this.

[0342] EEE132. A method for automatically enhancing an input audio signal to detect and / or attenuate at least one speech articulation noise event contained in the input audio signal, wherein the speech articulation noise event includes at least one speech plosive event. A step of generating a plurality of substantially equal-rectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and a plurality of bands above the predetermined frequency threshold, wherein the predetermined frequency threshold is within the frequency range of the speech plosive event, A step of applying multiple attenuation gains to the audio signal in each of the frequency bands, wherein the attenuation gains are calculated based on the energy calculated for the frequency band; To generate an output audio signal, the process involves providing the attenuated audio sample to a synthesis filter bank, A method that includes

[0343] EEE133. The method of EEE132, wherein the attenuation gain in each frequency band is further constrained such that the energy in that frequency band is not reduced below the estimated noise threshold in that frequency band.

[0344] EEE134. A step to calculate time-smoothed low-frequency energy estimates of audio samples above the estimated noise threshold in order to distinguish between plosive events and higher-frequency content in the input audio signal. The EEE133 method further encompasses this.

[0345] EEE135. A step of calculating the audio harmonic protection index value in the spectrum of the input audio signal, A step of calculating the attenuation gain according to the aforementioned audio harmonic protection index value and the time-smoothed low-frequency energy estimate, A method of EEE132 or EEE134 that further encompasses the above.

[0346] EEE136. The method of EEE135, wherein the audio harmonic protection index value is a periodic or tonal index value.

[0347] EEE137. The periodicity index value in the spectrum is calculated from the audio input sample cepstrum before the final bandwidth calculation of the analysis filter bank, by the method of EEE136.

[0348] EEE138. The method of EEE136, wherein the tonal index value in the spectrum is calculated based on the principal lobe of the spectral peak compared to the principal lobe of the sinusoidal peak, prior to the final bandwidth calculation of the analysis filter bank.

[0349] EEE139. A step of further constraining the calculated attenuation gain based on the frequency band immediately below the frequency. Methods EEE132 to 138 further encompass this.

[0350] EEE140. The input audio signal is processed continuously using a predetermined look-ahead frame size, in one of the methods described in EEE132 to 139.

[0351] EEE141. A device comprising a processor and a memory connected to the processor, wherein the processor causes the device to perform any one of the preceding EEEs.

[0352] EEE142. A program that includes instructions that cause the processor to perform one of the methods described in EEE88 to 140 when executed by the processor.

[0353] A computer-readable storage medium for storing EEE143 and EEE142 programs.

Claims

1. A method for automatically enhancing an input audio signal that includes at least one speech articulation noise event, wherein the method is: The process of segmenting the input audio signal into multiple audio frames, A step of obtaining at least one feature parameter related to each of the speech articulation noise events from the audio frame, A step of determining, at least in part, the type of speech articulation noise event and the respective time-frequency range related to the speech articulation noise event in the input audio signal, based on the obtained characteristic parameters, A step of attenuating the speech articulation noise event according to the determined type and range of the speech articulation noise event, It includes, The aforementioned speech articulation noise event includes at least one of a tongue-clicking event or a speech plosive event. The step of obtaining at least one feature parameter from the audio frame is: For each audio frame, the process of obtaining at least one kurtosis index value based on the time-domain sample amplitude of the audio frame. It includes, and, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: The steps include comparing the obtained kurtosis index value with a predetermined kurtosis threshold, If the kurtosis index value exceeds the predetermined kurtosis threshold, it is determined that the audio frame contains a tongue-clicking sound event, and the start and end boundaries of the tongue-clicking sound event are determined based on the positions where the kurtosis index value is above and below the predetermined kurtosis threshold, respectively. to include, method.

2. The method according to claim 1, wherein the determined range includes at least one boundary of the determined speech articulation noise event in the time and / or spectral domain.

3. The method according to claim 1 or 2, wherein the speech articulation noise event includes one or more tongue-clicking events, and the one or more tongue-clicking events include at least one of a non-speech tongue-clicking event, a speech tongue-clicking event, or a tongue-tapping event.

4. After the step of segmenting the input audio signal into multiple audio frames, The method according to claim 3, further comprising the step of classifying the audio frame into either an audio frame or a non-audio frame.

5. The method according to claim 4, wherein the input audio signal is identified and segmented into voice frames and non-voice frames by using a voice activity detector (VAD).

6. The method according to claim 4 or 5, wherein the segmentation is performed by using two different window sizes, one of which is shorter than the other.

7. The method according to claim 6, wherein the shorter of the two window sizes is used to detect speech tongue-clicking events in the speech frame, and the longer of the two window sizes is used to detect non-speech tongue-clicking events in the non-speech frame.

8. The step of obtaining at least one feature parameter from the audio frame is: For each audio frame, the process involves obtaining an approximate value of the residual without audio harmonic components and a first kurtosis index value of the sample amplitude for the approximate value of the residual. It includes, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: The process involves comparing the obtained first kurtosis index value with a first predetermined kurtosis threshold, If the first kurtosis index value exceeds the first predetermined kurtosis threshold, the audio frame is determined to include an audio tongue-clicking event, and the start and end boundaries of the audio tongue-clicking event are determined based on the positions where the first kurtosis index value exceeds and falls below the first predetermined kurtosis threshold, respectively. to include, The method according to any one of claims 4 to 7.

9. The method according to claim 8, wherein the approximate amount of residual without audio harmonic components is a second-order waveform difference.

10. The aforementioned method, A step to obtain a second kurtosis index value from the time-domain sample amplitude in the non-speech frame. To further encompass, The type and range of the speech articulation noise event are determined based on the first kurtosis index value relative to the second kurtosis index value. The method according to claim 8 or 9.

11. A step of identifying the sample position having the largest quadratic difference within the determined range of the aforementioned audio tongue-clicking sound event, The process of determining the refined range of the audio tongue-clicking sound event by applying a predetermined duration of the audio tongue-clicking sound event to the area around the location-identified sample position, The method according to any one of claims 8 to 10, further comprising the step of refining the determined range of the speech tongue-clicking event.

12. The method according to any one of claims 8 to 11, further comprising the step of determining the range of the speech tongue-clicking sound event based on the maximum / minimum rate of change calculated from the local minimum and maximum values ​​in the speech frame.

13. The step of obtaining at least one feature parameter from the audio frame is: For each non-speech frame, the process of obtaining the third kurtosis index value for each time-domain sample amplitude in that non-speech frame. It includes, and, The step of determining the respective types and ranges of the speech articulation noise events in the input audio signal based on the characteristic parameters obtained is as follows: A step of comparing the obtained third kurtosis index value with a second predetermined kurtosis threshold, If the third kurtosis index value exceeds the second predetermined kurtosis threshold, it is determined that the non-voiced frame includes a non-voiced tongue-clicking event, and the start and end boundaries of the non-voiced tongue-clicking event are determined based on the positions where the third kurtosis index value exceeds and falls below the second predetermined kurtosis threshold, respectively. to include, The method according to any one of claims 4 to 12.

14. The method according to claim 13, further comprising the step of merging two neighboring non-audible tongue-clicking events into a single audible tongue-clicking event if the two neighboring non-audible tongue-clicking events are within a predetermined gap threshold.

15. Regarding non-audio tongue-clicking events determined in the non-audio frame immediately preceding the audio frame, A step of calculating the high / low bandwidth peak ratio as the amplitude ratio of the maximum peak above a predetermined frequency and the maximum peak below the predetermined frequency, If the calculated high / low bandwidth peak ratio is above a predetermined ratio threshold, the non-vocal tongue-clicking event is determined to be a tongue-clicking event. The method according to claim 13 or 14.

16. The aforementioned high / low bandwidth peak ratio is calculated as the amplitude ratio between the maximum peak above a predetermined frequency and the maximum peak below the predetermined frequency but above a further predetermined low frequency. The method according to claim 15.

17. The method according to claim 15 or 16, further comprising the step of refining the determined range of the tongue click event based on the high / low band peak ratio, spectral slope and energy envelope.

18. The step of refining the determined range of the tongue click event is as follows: The process of extending the end position of the tongue-tapping event, determined by using the third kurtosis index value, provided that the high / low bandwidth peak ratio is above the predetermined ratio threshold, the spectral slope is below the predetermined slope threshold, and the energy in the energy envelope is reduced. to include, The method according to claim 17.

19. The method according to any one of claims 4 to 12, further comprising the step of determining the speech articulation noise event according to a further predetermined threshold, based on the centroid COG calculated for the speech frame, in order to distinguish between tongue-clicking events and speech transients.

20. The method according to claim 7, further comprising the step of attenuating one or more determined tongue-clicking events based on the spectral envelope of the audio frame containing the detected tongue-clicking event, and the respective spectral gains derived from target envelopes calculated based on the respective reference frames.

21. The method according to claim 20, wherein for each detected tongue-clicking event, the reference frame includes the audio frame preceding and following the audio frame containing the detected tongue-clicking event, and the target envelope is calculated by interpolating the spectral envelope of the reference frame.

22. The method according to claim 20 or 21, wherein the attenuation is applied to a frequency band higher than a predetermined high-frequency threshold.

23. The method according to any one of claims 8 to 19, further comprising the step of replacing one or more determined tongue-clicking events based on each neighboring audio frame.

24. The steps include: the speech articulation noise event includes at least one speech plosive event, and obtaining at least one feature parameter from the audio frame; A step to obtain a low-frequency energy (LFE) index value for each of the audio frames in order to identify outliers in the low-frequency energy (LFE). to include, The method according to any one of claims 1 to 23.

25. The method according to claim 24, wherein the LFE index value is calculated in either the time domain or the spectral domain.

26. The method according to claim 24 or 25, further comprising the step of determining the range of the speech plosive event according to the outliers identified from the LFE index value and thresholds calculated based on the LFE index value, or according to LFE ratios calculated from previous and current audio frames.

27. The aforementioned method, In order to refine the range of the plosive sound event determined based on the LFE index value, the process of determining the respective zero-crossing maximum value ZCM index value for each of the audio frames. To further encompass, The ZCM index value indicates the length of the maximum interval between consecutive zero crossings within the audio frame. The method according to claim 26.

28. The method according to claim 27, further comprising the step of attenuating the determined speech plosive event, wherein the attenuation is performed in either the time domain or the spectral domain.

29. The method according to claim 28, wherein the time-domain attenuation is performed by applying a high-pass filter, the cutoff frequency of the filter is determined based on the ZCM index value for the audio frames within the range of the determined speech plosive events, and the order of the filter is determined based on the LFE index value for the audio frames within the range of the determined speech plosive events.

30. The method according to claim 28, wherein the spectral domain attenuation is performed using an adaptive spectral gradient and frequency, and by using a superimposed summation short-time Fourier transform (STFT).

31. The method according to claim 28 or 30, wherein the spectral domain attenuation includes the steps of processing the audio frame using a fast Fourier transform (FFT) to generate an attenuated output audio signal, applying an attenuation gain having an adaptive slope and frequency, and applying an inverse FFT, windowing and superimposition, wherein the frequency is determined based on the ZCM index value for the audio frame within the range of the determined plosive events, and the slope is determined based on the LFE index value for the audio frame within the range of the determined plosive events.

32. The method according to claim 31, further comprising the step of applying noise spectrum estimation to limit the attenuation gain and prevent excessive suppression.

33. The method according to any one of claims 24 to 32, further comprising the step of applying a content classifier to the audio frame to distinguish between audio frames and non-audio frames in order to determine the aforementioned audio plosive event.

34. The spectral domain attenuation described above is A step of generating a plurality of substantially equirectangular bandwidth (ERB) interval frequency bands below a predetermined frequency threshold and a plurality of bands above the predetermined frequency threshold, wherein the predetermined frequency threshold is within the frequency range of the determined speech plosive events. A step of applying multiple attenuation gains to the audio signal in each of the frequency bands, wherein the attenuation gains are calculated based on the energy calculated for the frequency band; To generate an output audio signal, the process involves providing the attenuated audio sample to a synthesis filter bank, including, The method according to claim 28.

35. The method according to claim 34, wherein the attenuation gain in each frequency band is further constrained such that the energy in that frequency band is not reduced below the estimated noise threshold in that frequency band.

36. The method of claim 35, further comprising the step of calculating time-smoothed low-frequency energy estimates of audio samples above the estimated noise threshold in order to distinguish between speech plosive events and higher-frequency content in the input audio signal.

37. A step of calculating the audio harmonic protection index value in the spectrum of the input audio signal, A step of calculating the attenuation gain according to the aforementioned audio harmonic protection index value and the time-smoothed low-frequency energy estimate, The method according to claim 36, further encompassing the present invention.

38. The method according to claim 37, wherein the audio harmonic protection index value is a periodic or tunic index value.

39. The method according to claim 38, wherein the periodicity index value in the spectrum is calculated from the cepstrum of the audio sample before the final bandwidth calculation of the analysis filter bank.

40. The method according to claim 38, wherein the tonal index value in the spectrum is calculated based on the principal lobe of the spectral peak compared to the principal lobe of the sinusoidal peak before the final bandwidth calculation of the analysis filter bank.

41. The method according to any one of claims 37 to 40, further comprising the step of further constraining the calculated attenuation gain based on a frequency band immediately below the frequency.

42. An apparatus comprising a processor and a memory connected to the processor, wherein the processor is configured to cause the apparatus to carry out the method according to any one of claims 1 to 41.

43. A program that, when executed by a processor, includes an instruction causing the processor to perform the method according to any one of claims 1 to 41.

44. A computer-readable storage medium for storing the program described in claim 43.

Images