Hum detection and removal for voice and music recordings

Abstract

Methods of processing audio data for hum detection and / or removal are described. The audio data includes a plurality of frames. One method includes classifying frames of the audio data as content frames or noise frames using one or more content activity detectors; determining a noise spectrum from one or more frames of the audio data that are classified as noise frames; determining one or more hum frequencies based on the determined noise spectrum; generating an estimated hum signal based on the one or more hum frequencies; and removing hum from at least one frame of the audio data based on the estimated hum signal. Apparatuses for performing the methods, as well as corresponding programs and computer-readable storage media, are also described.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to the following prior applications: Spanish application P202030814 (reference number: D20073ES), filed July 30, 2020; U.S. Provisional Application 63 / 088,827 (reference number: D20073USP1), filed October 7, 2020; and U.S. Provisional Application 63 / 223,252 (reference number: D20073USP2), filed July 19, 2021, all of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to methods and apparatus for processing audio data. The disclosure further describes techniques for dehumming processing (e.g., hum noise detection and / or removal) of audio recordings (including speech and music recordings). These techniques can be applied, for example, to (cloud-based) streaming services, online processing, and post-processing of music and speech recordings. Background Technology

[0004] Humming noise is frequently present in audio recordings. This noise can originate from ground loops, AC line noise, cables, RF interference, computer motherboards, microphone feedback, household appliances such as refrigerators, neon light buzzing, and more. Software solutions to handle humming noise are often necessary because the recording environment is not always guaranteed.

[0005] Humming noise typically behaves very much like a set of fixed-frequency "tones." Humming tones are usually spaced at fixed frequency intervals, resulting in harmonic sounds. However, "harmonics" may only occur in certain frequency bands, and the fundamental tone (e.g., the perceived dominant tone) may not correspond to its fundamental frequency.

[0006] To enhance speech / music recordings containing hum noise, it is crucial to identify the perceptually dominant hum and distinguish it from speech / music harmonics. Typically, improved techniques are needed for hum noise detection and / or removal. Summary of the Invention

[0007] In view of the above, this disclosure provides a method for processing audio data, as well as corresponding apparatus, computer programs, and computer-readable storage media, which have the features of the respective independent claims.

[0008] According to one aspect of this disclosure, a method for processing audio data is provided. The method may be a method for detecting and / or removing buzzing noise. For example, the audio data may involve an audio file, a video file including audio, an audio signal, or a video signal including audio. The audio data may include multiple frames. Frames may be overlapping frames. Thus, the audio data may include (or represent) a series of (overlapping) frames. The method may include classifying frames of the audio data into content frames or noise frames using one or more content activity detectors. A content frame may be a frame of the audio data containing content (such as music and / or speech). Thus, a content frame may be a frame perceptually dominated by content. A noise frame may be a frame of the audio data perceptually dominated by noise (e.g., a frame that does not contain content, a frame that may not contain content, or a frame that mainly contains noise). Frame classification may involve comparing one or more probabilities of a corresponding content type with a corresponding threshold. The probabilities may have already been determined by one or more content activity detectors. The content activity detector may also be referred to as a content classifier. Further, the content activity detector may be implemented by a appropriately trained deep neural network. The method may further include determining a noise spectrum based on one or more frames of the audio data classified as noise frames. The noise spectrum can be determined based on the spectrum of one or more frames classified as noise frames. The determined noise spectrum can be referred to as the clustered noise spectrum or the key noise spectrum. The method may further include determining one or more hum noise frequencies based on the determined noise spectrum. The method may further include generating an estimated hum noise signal based on one or more hum noise frequencies. The method may further include removing hum noise from at least one frame of audio data based on the estimated hum noise signal.

[0009] With the configuration described above, the proposed method distinguishes between noise frames and content frames. Only the noise frames are then used to determine the noise spectrum (e.g., the critical noise spectrum), and based on this, the hum noise frequency is determined. This allows for reliable and accurate evaluation of the hum noise frequency, and thus for efficient hum noise removal. The high accuracy of the determined hum noise frequency significantly reduces the likelihood of perceptible artifacts in the denoised output audio data.

[0010] In some embodiments, one or more hum noise frequencies can be identified as anomalous peaks in the noise spectrum. If the amplitude of a peak in the noise spectrum exceeds a frequency-dependent threshold, the peak can be identified / determined as being associated with an anomalous peak. This allows for efficient and automated detection of hum noise frequencies and further provides easily implementable control parameters (e.g., thresholds) to control the aggressiveness of hum noise removal. Furthermore, using such a frequency-dependent threshold results in easily implementable hum noise removal, but at the same time, by appropriately selecting the frequency-dependent threshold, more advanced removal processes tailored to specific applications can be automated.

[0011] In some embodiments, determining one or more humming noise frequencies may involve determining a smoothed envelope of the noise spectrum. For example, the smoothed envelope may be a cepstral envelope. Alternatively, the smoothed envelope may be determined based on a moving average across frequencies. Typically, the smoothed envelope can indicate the expected value of the noise spectrum. Determining one or more humming noise frequencies may further involve identifying one or more humming noise frequencies as anomalous peaks in the noise spectrum compared to the smoothed envelope.

[0012] In some embodiments, the smooth envelope can be determined on a perceptually curved scale. For example, a perceptually curved scale could be the Mel scale or the Bark scale. This allows for better handling of near-hum in low frequencies and compensates for potential overestimation that may occur when the envelope is computed on a linear scale.

[0013] In some embodiments, if the amplitude of a peak in the noise spectrum exceeds the smooth envelope by a threshold, the peak can be identified as an abnormal peak. For example, the threshold could be an amplitude threshold.

[0014] In some embodiments, the threshold can be a frequency-dependent threshold. For lower frequencies, the frequency-dependent (amplitude) threshold can be lower. For example, the frequency-dependent (amplitude) threshold can be defined as having a first value (e.g., 3 dB) for the low-frequency band and a second value (e.g., 6 dB) greater than the first value for the high-frequency band. Thus, the threshold adapts to the resolution limitations caused by envelope estimation bias and underlying sinusoidal components with similar frequencies.

[0015] In some embodiments, the noise spectrum may be determined based on the average of the spectra of one or more frames classified as noise frames. In this case, the noise spectrum will be the mean noise spectrum of one or more frames classified as noise frames.

[0016] In some embodiments, the noise spectrum can be determined based on the spectrum that includes the maximum energy among the spectra of one or more frames classified as noise frames. For example, the noise spectrum can be based on a weighted sum of the average spectrum (e.g., mean noise spectrum) and the spectrum that includes the maximum energy. Thus, a noise spectrum with fewer smooth frequency peaks can be obtained, and therefore the humming noise frequency can be detected more accurately.

[0017] In some embodiments, generating the estimated buzz noise signal may involve synthesizing a corresponding buzz tone for each of one or more buzz noise frequencies. For example, the synthesized buzz tone may be a sine wave. The estimated buzz noise signal may be the sum (superposition) of individual buzz tones.

[0018] In some embodiments, generating the estimated buzz noise signal may involve determining, for each buzz noise frequency, a corresponding buzz noise phase based on the corresponding buzz noise frequency and audio data in at least one frame. The buzz noise phase determined in this way may be referred to as the instantaneous buzz noise phase. For example, the buzz noise phase can be determined using the least squares method. Each buzz noise frequency may have its own associated buzz noise phase. Generating the estimated buzz noise signal may further involve synthesizing a corresponding buzz sound for each of one or more buzz noise frequencies based on the buzz noise frequency and the corresponding buzz noise phase.

[0019] In some embodiments, generating the estimated hum noise signal may involve determining, for each hum noise frequency, a corresponding (instantaneous) hum noise amplitude based on the corresponding hum noise frequency and audio data in at least one frame. Generating the estimated hum noise signal may further involve determining, for each hum noise frequency, a corresponding average hum noise amplitude based on the noise spectrum. Generating the estimated hum noise signal may further involve synthesizing a corresponding hum sound for each of one or more hum noise frequencies based on the corresponding hum noise frequency, the corresponding hum noise phase, and the smaller of the corresponding hum noise amplitude and the corresponding average hum noise amplitude. By selecting the smaller of the instantaneous hum noise amplitude and the average hum sound amplitude, overly aggressive hum noise removal that could lead to audible artifacts, such as the introduction of additional hum noise, can be avoided. Furthermore, the proposed technique can be similarly applied to all frames, whether they are content frames (e.g., speech, music) or noise frames.

[0020] In some embodiments, when at least one frame is classified as a noise frame, generating an estimated buzz noise signal may involve determining, for each buzz noise frequency, a corresponding buzz noise amplitude based on the corresponding buzz noise frequency and audio data in at least one frame. The buzz noise amplitude determined in this way may be referred to as the instantaneous buzz noise amplitude. For example, the buzz noise amplitude can be determined using the least squares method. Each buzz noise frequency may have its own associated buzz noise amplitude. In this case, generating the estimated buzz noise signal may further involve synthesizing a corresponding buzz sound for each of one or more buzz noise frequencies based on the corresponding buzz noise frequency, the corresponding (instantaneous) buzz noise phase, and the corresponding (instantaneous) buzz noise amplitude.

[0021] In some embodiments, when at least one frame is classified as a content frame, generating the estimated buzz noise signal may involve determining the corresponding average buzz noise amplitude based on the noise spectrum for each buzz noise frequency. Each buzz noise frequency may have its own associated average buzz noise amplitude. In this case, generating the estimated buzz noise signal may further involve synthesizing a corresponding buzz sound for each of one or more buzz noise frequencies based on the corresponding buzz noise frequency, the corresponding (instantaneous) buzz noise phase, and the corresponding average buzz noise amplitude. Alternatively, the instantaneous buzz noise amplitude of a previous (e.g., directly previous) noise frame may be used instead of the average buzz noise amplitude.

[0022] In some embodiments, generating the estimated buzz noise signal may involve determining a corresponding average buzz noise amplitude based on the noise spectrum for each buzz noise frequency. Each buzz noise frequency may have its own associated average buzz noise amplitude. Generating the estimated buzz noise signal may further involve synthesizing a corresponding buzz sound for each of one or more buzz noise frequencies based on the corresponding buzz noise frequency and the corresponding average buzz noise amplitude.

[0023] In some embodiments, removing buzz noise from at least one frame may involve subtracting the estimated buzz noise signal from at least one frame.

[0024] In some embodiments, the noise spectrum can be determined based on the spectra of all frames of the audio data that are classified as noise frames. This assumes that all frames of the audio data are available simultaneously and can be referred to as offline processing.

[0025] In some embodiments, the method may include sequentially receiving and processing frames of audio data. The method may further include, for a current frame, updating the noise spectrum based on the spectrum of the current frame if the current frame is classified as a noise frame. This scenario may be referred to as online processing. For online processing, the method may further include: determining one or more updated hum noise frequencies based on the updated noise spectrum; generating an updated estimated hum noise signal based on the one or more updated hum noise frequencies; and / or removing hum noise from the current frame based on the updated estimated hum noise signal.

[0026] In some embodiments, the noise spectrum can be determined based on multiple frames classified as noise frames. The method may further include determining the time-varying behavior of one or more hum noise frequencies based on the spectra of the multiple frames classified as noise frames. The method may further include applying a bandpass filter to frames of audio data based on the time-varying behavior. The bandpass filter can be designed such that the stopband includes one or more hum noise frequencies. For example, if the time-varying behavior indicates non-stationary hum noise, i.e., if the hum noise frequency is modulated at a rate exceeding a certain threshold, bandpass filtering can be applied. If the time-varying behavior exceeds a certain threshold, the presence of non-stationary hum noise can be determined, and bandpass filtering can be applied accordingly. This avoids audible artifacts, such as the introduction of additional hum noise, which may arise due to hum noise removal when applied to (highly) non-stationary hum noise.

[0027] In some embodiments, the width of the stopband can be determined based on the change of the corresponding hum noise frequency over time.

[0028] In some embodiments, the method may include determining, for at least one of one or more buzz noise frequencies, whether at least one buzz noise frequency exists as a peak in the spectrum of all frames of the audio data. The method may further include ignoring at least one buzz noise frequency when removing buzz noise if at least one buzz noise frequency does not exist as a peak in the spectrum of all frames of the audio data. In other words, if a buzz noise frequency determined from the noise spectrum exists throughout the entire audio data (e.g., from the first frame to the last frame), then only the buzz noise frequency can be considered for buzz noise removal. Therefore, assuming that only buzz noise exists throughout the entire audio recording, content-related harmonics (e.g., those in music) can be distinguished from buzz noise.

[0029] According to another aspect, a computer program is provided. The computer program may include instructions that, when executed by a processor (e.g., a computer processor, a server processor), cause the processor to perform all the steps of the methods described herein.

[0030] According to another aspect, a computer-readable storage medium is provided. This computer-readable storage medium can store the aforementioned computer program.

[0031] According to another aspect, an apparatus is provided, comprising a processor and memory coupled to the processor. The processor may be adapted to perform all steps of the methods described herein. For example, this apparatus may be associated with a server (e.g., a cloud-based server) or a server system (e.g., a cloud-based server system).

[0032] It will be understood that the features of the apparatus and the steps of the method can be interchanged in various ways. In particular, as will be understood by those skilled in the art, the details of the disclosed (multiple) methods can be implemented by the corresponding apparatus, and vice versa. Furthermore, any statement above regarding the (multiple) methods (and, for example, their steps) should be understood to apply equally to the corresponding apparatus (and, for example, its blocks, stages, units), and vice versa. Attached Figure Description

[0033] The following explanation of exemplary embodiments of this disclosure is based on the accompanying drawings, in which:

[0034] Figure 1 This is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.

[0035] Figure 2 This is a diagram illustrating an example of a noise spectrum according to an embodiment of the present disclosure.

[0036] Figure 3 The illustration shows an embodiment according to the present disclosure. Figure 1 A flowchart illustrating an example implementation of the method's steps.

[0037] Figure 4 This is a diagram illustrating an example of a smooth envelope for a noise spectrum according to an embodiment of the present disclosure.

[0038] Figures 5 to 9 The illustration shows an embodiment according to the present disclosure. Figure 1 A flowchart illustrating an example implementation of another step of the method,

[0039] Figure 10 This is a block diagram illustrating an example of a functional overview of the technology according to embodiments of the present disclosure, and

[0040] Figure 11 This is a block diagram of an apparatus for performing a method according to an embodiment of the present disclosure. Detailed Implementation

[0041] The accompanying drawings and the following description are for illustrative purposes only and relate to preferred embodiments. It should be noted that, based on the discussion below, alternative embodiments of the structures and methods disclosed herein will be readily recognized as feasible alternatives that can be employed without departing from the claimed principles.

[0042] Referring now to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that similar or identical reference numerals may be used in the drawings where feasible, and these reference numerals may denote similar or identical functions. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0043] One possible method for processing hum noise is to detect the hum based on the amount of power fluctuation over time in each frequency bin. The hum frequency is then refined using an adaptive notch filter algorithm. However, this method has difficulty, for example, safely preventing sustained low frequencies from being detected as hum noise.

[0044] Furthermore, while several common filters can be used to remove the hum, it has been found that there is room for improvement in the audio quality processed in this way. In addition, it is known that simple filters may introduce phase distortion and / or inevitably suppress content components, and thus cause artifacts, which can be particularly unpleasant when the hum noise interferes with speech harmonics and / or musical harmonics.

[0045] In another possible approach to processing hum noise, an FIR bandpass filter is designed to reduce the amplitude of the first five harmonics of a 50Hz hum by at least 40dB. Applying a fixed threshold of 40dB to the short-term amplitude of the FIR bandpass-filtered speech signal allows accumulation of both speech and non-speech signal channels. Based on the non-speech signal channels, an average spectral energy is obtained, and the hum is detected using simple peak picking or a fundamental frequency estimation. The detected hum is then removed from the original signal. Again, in this case, there is room for improvement in the processed audio quality because the fixed thresholding suppresses desired non-noise content (e.g., speech or musical content that is spectrally similar to the fundamental noise estimate).

[0046] This disclosure describes a method for automatically detecting and subsequently removing buzz noise from speech and music recordings, for example, by sinusoidal modeling of buzz noise.

[0047] The proposed method may have one or more of the following three key aspects:

[0048] • Content Activity Detection (CAD) was used to identify noisy (non-content, such as non-speech and non-music) frames, where buzzing noise was analyzed.

[0049] • Frequency-dependent and / or perception-related hum detection may not assume harmonic relationships.

[0050] • Adaptive use of global average hum amplitude and local instantaneous estimation of amplitude

[0051] Exemplary embodiments of this disclosure will now be described in more detail.

[0052] Figure 1 This is a flowchart illustrating an example of a method 100 for processing audio data according to an embodiment of the present disclosure. Method 100 may be a method for detecting and / or removing hum noise in an audio recording (or typically a file including audio) represented by audio data. Typically, for example, the audio data may relate to an audio file, a video file including audio, an audio signal, or a video signal including audio.

[0053] Audio data comprises multiple frames. For example, audio data may have been generated by performing short-time frame analysis. Short-time frame analysis can utilize windows (window functions) and / or the overlap between frames. Therefore, audio data may include (or represent) a series of (overlapping) frames. For example, a Hanning window (e.g., an 85ms Hanning window) can be used. Furthermore, 50% overlap can be used. Of course, other combinations of window functions, window lengths, and / or overlaps can be selected as needed, such as based on one or more minimum frequencies present or expected in the recording content.

[0054] In method 100 Step S110 At this stage, audio data frames are classified as content frames or noise frames. This can be done using one or more Content Activity Detectors (CAD) or content classifiers. Content frames are frames of audio data that contain content (such as music and / or speech). Noise frames are frames of audio data that do not contain content.

[0055] For example, existing content activity detectors can be used to estimate the instantaneous probabilities of different types of content, such as speech and music. If the probabilities of both music and speech are not higher than their respective thresholds, the frame can be classified as noise. Typically, frame classification may involve comparing one or more probabilities (likelihoods) of the corresponding content type with corresponding thresholds. The probabilities can be determined by one or more content activity detectors. It should be understood that, for example, content activity detectors can be implemented using appropriately trained deep neural networks.

[0056] exist Step S120At this point, a noise spectrum is determined based on one or more frames of audio data that have been classified as noise frames. Specifically, the noise spectrum can be determined (e.g., estimated) based on the spectra of one or more frames classified as noise frames. In other words, the spectra of noise frames can be accumulated to estimate the noise spectrum. Therefore, the noise spectrum can be referred to as a clustered noise spectrum or a key noise spectrum (KNS). In some embodiments, the noise spectrum (e.g., the key noise spectrum) can be determined based on a threshold number of frames classified as noise frames, in response to that threshold number of frames. For example, the method can first accumulate a threshold number of noise frames and determine the noise spectrum only after that threshold number of noise frames is available. In one embodiment, the noise spectrum (e.g., the key noise spectrum) can be determined (e.g., estimated) based on the average of the spectra of one or more frames classified as noise frames. Specifically, the noise spectrum can be determined as the average of all considered spectra (i.e., the spectra of all considered noise frames). The resulting noise spectrum can be the mean noise spectrum (MNS) of the considered noise frames (i.e., one or more frames classified as noise frames at step S110). MNS can be updated at each noise frame and therefore can be used in an online adaptive manner. For online scenarios, if the initial frames of the audio data are not noise, frequency-dependent CAD combined with steady-state tone tracking can be used until a noise frame becomes available. In this case, in some implementations, the mean noise spectrum can be determined based on a threshold number of frames classified as noise, in response to that threshold number of frames. For example, the method could first accumulate a threshold number of noise frames and determine the mean noise spectrum only after that threshold number of noise frames are available.

[0057] In another implementation, the noise spectrum (e.g., the critical noise spectrum) can be determined based on the spectrum that includes the maximum energy among the spectra of one or more frames classified as noise frames. For example, the noise spectrum can be based on (e.g., determined as) a weighted sum of the average spectrum (e.g., the mean noise spectrum) and the spectrum that includes the maximum energy. In other words, the noise spectrum (critical noise spectrum) can be determined as a weighted sum of the MNS and the strongest noise spectrum. This provides a spectrum with "more spikes" compared to the MNS, because the MNS tends to smooth out the hum peaks when the hum is slightly modulated. The resulting noise spectrum can be the weighted noise spectrum (WNS) of the noise frames under consideration (i.e., one or more frames classified as noise frames at step S110). The weights of the weighted sum can be selected as control parameters for the desired "spikiness" of the noise spectrum.

[0058] Figure 2 An example comparison between MNS (curve 210) and WNS (curve 220) is shown. As mentioned above, WNS is not as smooth as MNS.

[0059] In method 100 Step S130 At this point, one or more humming noise frequencies are determined based on the determined noise spectrum. For example, one or more humming noise frequencies can be identified as anomalous peaks in the noise spectrum. Peaks in the noise spectrum can be detected / identified based on counts at corresponding frequency bins (e.g., a corresponding indication of the (relative) energy at each frequency bin of the noise spectrum). For example, if the amplitude of a detected peak in the noise spectrum exceeds a threshold, such as a frequency-dependent threshold, the peak can be identified / determined as being associated with an anomalous peak.

[0060] As mentioned above, the detection of the humming sound is based on a given noise spectrum (e.g., KNS). One embodiment of step S130, 300, is... Figure 3 The flowchart is illustrated schematically. Therefore, determining one or more buzzing noise frequencies at step S130 may involve steps S310 and S320 as described below.

[0061] exist Step S310 At this point, a smooth envelope of the noise spectrum is determined. For example, the smooth envelope could be a cepstral envelope. The cepstral envelope can be said to represent the expected amplitude of the noise spectrum. It is a frequency-dependent smooth curve of the expected value across each frequency band. Alternatively, the smooth envelope can be determined based on a moving average across frequencies. Typically, the smooth envelope can indicate the expected value of the noise spectrum. Anomalies can then be selected as possible hums.

[0062] In one possible implementation, for example, the smooth envelope can be determined on a perceptually curved scale (such as the Mel scale or the Barker scale). Analysis on a perceptually curved scale (e.g., Mel, Barker, etc.) (e.g., cepstral analysis) can be used to adapt more quickly in the low-frequency region (and therefore more slowly in the high-frequency region). This allows for better handling of near-hum in low frequencies and compensates for potential overestimation that might occur when computed on a linear scale. Such an envelope also tends to be smooth at high frequencies, where the actual noise floor does not change very rapidly between frequency bins.

[0063] exist Step S320 At this point, one or more humming noise frequencies are compared to a smooth envelope to identify anomalous peaks in the noise spectrum. For example, if the amplitude of a peak in the noise spectrum exceeds a threshold beyond the smooth envelope, the peak can be identified as an anomalous peak. This threshold could be, for example, an amplitude threshold. In particular, outliers (e.g., KNS) in the noise spectrum can be selected for multiple peaks from the cepstral envelope that exceed the threshold.

[0064] The aforementioned threshold can be a frequency-dependent threshold. Therefore, in one implementation, different thresholds can be set for different frequency bands. Specifically, for lower frequencies (lower frequency bands), the frequency-dependent (amplitude) threshold may be lower. For example, the frequency-dependent (amplitude) threshold can be defined as having a first value (e.g., 3dB) for the low-frequency band (or multiple low-frequency bands) and a second value (e.g., 6dB) greater than the first value for the high-frequency band (or multiple high-frequency bands). For example, the frequency boundary between the low-frequency bands and the high-frequency bands can be set to 4kHz. In another example, the threshold can be defined as a smooth transfer function across frequencies.

[0065] Figure 4 An example comparing the noise spectrum 410 with the cepstral envelope 420 is shown. Peaks in the noise spectrum 410 that are sufficiently higher than the cepstral envelope 420 can be detected as anomalous peaks and therefore as humming. The humming frequency can be refined using a quadratic interpolation fast Fourier transform (QIFFT) method if needed or necessary.

[0066] Once the humming frequency is selected, its temporal amplitude (e.g., average humming amplitude MHA) can be obtained from the noise spectrum (e.g., KNS, MNS, WNS) at the detected frequency. For example, the average humming amplitude can be determined based on the noise spectrum (e.g., KNS), such as by determining the peak value of the corresponding humming frequency in the noise spectrum.

[0067] Back Figure 1 In method 100 Step S140 At this point, an estimated buzz noise signal is generated based on one or more buzz noise frequencies. This may involve synthesizing a corresponding buzz tone (e.g., a sine tone) for each of the one or more buzz noise frequencies. The estimated buzz noise signal can be the sum (superposition) of the buzz tones. In this sense, the buzz tone is modeled as an additive model: the sum of sine curves.

[0068] Given a humming frequency, the instantaneous amplitude and phase can be estimated at each short frame using the least squares method to synthesize the corresponding sine curve.

[0069] • If the short frame is noise, a sine curve can be synthesized using instantaneous amplitude and instantaneous phase.

[0070] • If the short frame contains mixed content (e.g., speech and music), sine wave synthesis can be performed using MHA and instantaneous phase. Alternatively, sine wave synthesis can be performed using instantaneous amplitude from an earlier frame and instantaneous phase for the current frame.

[0071] Alternatively, MHA can be used as a less aggressive hum removal option for all frame types (e.g., both noisy frames and frames containing content). As a further refinement, sine curve synthesis can be performed using the smaller of the instantaneous amplitude and MHA.

[0072] Figures 5 to 9 A non-limiting example of a possible implementation of step S140, consistent with the above, for generating an estimated buzz noise signal based on one or more buzz noise frequencies, is shown.

[0073] Figure 5 The method 500 illustrated in the figure includes steps S510 and S520, and can be applied to all frames regardless of the content type.

[0074] exist Step S510 For each hum noise frequency, a corresponding hum noise phase is determined based on the corresponding hum noise frequency and audio data in at least one frame. Therefore, each hum noise frequency can have its own associated hum noise phase. The hum noise phase can be determined using a least-squares method, for example, by fitting an audio signal to at least one frame. Further, as indicated above, the hum noise phase determined in this way can be referred to as the instantaneous hum noise phase. It should be understood that the instantaneous phase and instantaneous amplitude can be determined jointly in some implementations (e.g., by least-squares), but it is also feasible to determine the instantaneous phase and instantaneous amplitude separately (independently).

[0075] exist Step S520 At this point, based on the hum noise frequency and the corresponding (instantaneous) hum noise phase, a corresponding hum sound is synthesized for each of one or more hum noise frequencies. As will be described in more detail below, for each hum noise frequency, the synthesis can be further based on the corresponding hum noise amplitude. Possible hum noise amplitudes include instantaneous hum noise amplitude, MHA, or a function of one or both.

[0076] Figure 6 The method 600 illustrated in the figure includes steps S610, S620 and S630, and can be applied to all frames regardless of the content type.

[0077] exist Step S610For each buzzing noise frequency, the corresponding buzzing noise amplitude is determined based on the corresponding buzzing noise frequency and audio data in at least one frame. Therefore, each buzzing noise frequency can have its own associated buzzing noise amplitude. The buzzing noise amplitude can be determined using a least-squares method, for example, by fitting it to the audio signal in at least one frame. Further, as indicated above, the buzzing noise amplitude determined in this way can be referred to as the instantaneous buzzing noise amplitude. It should be understood that the instantaneous phase and instantaneous amplitude can be determined jointly in some implementations (e.g., by least-squares), but it is also feasible to determine the instantaneous phase and instantaneous amplitude separately (independently).

[0078] exist Step S620 For each humming noise frequency, the corresponding average humming noise amplitude is determined based on the noise spectrum. Therefore, each humming noise frequency can have its own associated average humming noise amplitude. For example, it can be determined in the manner described above. Importantly, the average humming amplitude is determined based on the noise spectrum (e.g., KNS) and is independent of the audio data in at least one frame. Therefore, the average humming amplitude is universal across all frames (except for possible adjustments or updates in online scenarios, as follows).

[0079] Then, in At step S630 Based on the corresponding hum noise frequency, the corresponding hum noise phase, and the smaller of the corresponding (instantaneous) hum noise amplitude and the corresponding average hum noise amplitude, a corresponding hum sound is synthesized for each of one or more hum noise frequencies. By selecting the smaller of the instantaneous hum noise amplitude and the MHA, overly aggressive hum noise removal that may lead to audible artifacts (e.g., the introduction of additional hum noise) can be avoided, and the proposed technique can be similarly applied to all frames, whether they are content frames (e.g., speech, music) or noise frames.

[0080] Alternatively, in some implementations, if MHA is unavailable, the instantaneous hum noise amplitude of a previous (e.g., directly previous) noise frame can be used instead of the average hum noise amplitude.

[0081] Figure 7 The method 700 illustrated in the figure includes steps S710 and S720, and is particularly suitable for noisy frames.

[0082] exist Step S710 At each humming noise frequency, the corresponding (instantaneous) humming noise amplitude is determined based on the corresponding humming noise frequency and audio data in at least one frame. This can be done in the same manner as described above, for example, relative to step S610 of method 600.

[0083] exist Step S720At each of the one or more buzzing noise frequencies, a corresponding buzzing sound is synthesized based on the corresponding buzzing noise frequency, the corresponding (instantaneous) buzzing noise phase, and the corresponding (instantaneous) buzzing noise amplitude.

[0084] Figure 8 The method 800 illustrated in the figure includes steps S810 and S820, and is particularly suitable for content frames (e.g., speech or music frames).

[0085] exist Step S810 For each humming noise frequency, the corresponding average humming noise amplitude is determined based on the noise spectrum. For example, this can be done in the manner described above.

[0086] exist Step S820 At each location, based on the corresponding buzz noise frequency, the corresponding buzz noise phase, and the corresponding average buzz noise amplitude, a corresponding buzz sound is synthesized for each of one or more buzz noise frequencies.

[0087] Figure 9 The method 900 illustrated in the figure includes steps S910 and S920, and can be applied to all frames regardless of the content type.

[0088] exist Step S910 At each humming noise frequency, the corresponding average humming noise amplitude is determined based on the noise spectrum.

[0089] exist Step S920 At each of the one or more buzzing noise frequencies, a corresponding buzzing sound is synthesized based on the corresponding buzzing noise frequency and the corresponding average buzzing noise amplitude. As above, for each buzzing noise frequency, the synthesis can be further based on the corresponding (instantaneous) buzzing noise phase.

[0090] Back Figure 1 In method 100 Step S150 At this point, based on the estimated buzz noise signal, buzz noise is removed from at least one frame of the audio data. This may involve subtracting the estimated buzz noise signal generated at step S140 from at least one frame. For example, for each short frame under consideration (e.g., each short frame of the audio data), a synthesized sine curve (e.g., one or more sine curves based on one or more identified buzz frequencies) may be subtracted from the input signal as the final buzz removal process.

[0091] Several techniques can be used to verify the removal of hum noise. Examples of these techniques will be described below.

[0092] To handle inaccurate estimations of the hum, a simple check can be performed based on a comparison of the energy before and after dehumming. If the energy increases by a predefined amount (or more) relative to the synthesized hum, then the time-domain subtraction is likely to have increased the hum due to inaccurate estimation. In this case, the algorithm will bypass the processed output (e.g., if for the corresponding portion of the audio data, the energy after dehumming exceeds a threshold amount compared to the energy before processing, then the dehumming processing for the corresponding portion of the audio data is omitted in the final output).

[0093] Furthermore, modulation of the humming noise frequency over time can be detected by considering the change in the detected humming noise frequency over time, which may affect the quality of humming noise removal. Assuming the noise spectrum (e.g., KNS) is determined based on multiple noise frames (i.e., frames classified as noise frames), method 100 may additionally include determining the change of one or more humming noise frequencies over time based on the spectra of the multiple noise frames. Instead of the humming noise removal in step S150, bandpass filtering can be applied to the frames of the audio data based on the change over time. For example, bandpass filtering can be applied to large changes over time (e.g., changes greater than a threshold), and the humming noise removal in step S150 can be applied to small changes over time (e.g., changes less than a threshold).

[0094] In other words, for example, if the change over time indicates non-stationary hum noise (or non-stationary noise exceeding an acceptable threshold of non-stationarity), i.e., if the hum noise frequency is modulated at a rate exceeding a certain threshold, then a bandpass filter can be applied. If the change over time exceeds a certain threshold, then the presence of non-stationary hum noise can be determined, and a bandpass filter can be applied accordingly.

[0095] Bandpass filters used for this purpose can be designed such that the stopband includes one or more hum noise frequencies. The width of the stopband can be determined based on the variation of the corresponding hum noise frequency over time.

[0096] It should be understood that the hum noise removal and bandpass filtering in step S150 can be applied in a mixed manner. That is, bandpass filtering can be applied to those hum noise frequencies that exhibit large changes over time, while the stopband includes these hum noise frequencies, and the hum noise removal according to step S150 can be applied to the remaining hum noise frequencies.

[0097] In particular (but not exclusively), music recordings may include intended sounds that could be confused with buzz noise, such as bass guitar. In this case, the actual buzz noise can be distinguished from the intended sound by checking whether the frequency under consideration is present throughout the recording process or at least for most of it. Therefore, method 100 may further include determining, for at least one of the detected buzz noise frequencies, whether at least one buzz noise frequency is present as a peak in the spectrum of most frames (or even all frames) of the audio data. If so, the corresponding buzz noise frequency can be assumed to be associated with the actual buzz noise. Otherwise, if at least one buzz noise frequency is not present as a peak in the spectrum of most frames of the audio data, then at least one buzz noise frequency can be ignored at step S150 when removing buzz noise. Most frames of the audio data can be associated with frames of a predefined share of the audio data, such as 90% of all frames, 95% of all frames, etc. Therefore, if the buzz noise frequency determined according to the noise spectrum is present in frames of a predefined share (or more) of the audio signal, then only the buzz noise frequency can be considered for buzz noise removal. In some implementations, if a buzzing noise frequency determined from the noise spectrum exists throughout the entire audio data (e.g., from the first frame to the last frame), it may be considered to use only the buzzing noise frequency for buzzing noise removal.

[0098] It should be further understood that the technology according to this disclosure can be used in both offline and online scenarios. In offline scenarios, it is assumed that the entire audio data is immediately (simultaneously) available, making it possible to analyze the buzzing noise based on all frames of the audio data. For offline processing, the noise spectrum can be determined based on the spectrum of all frames of the audio data that are classified as noise frames.

[0099] In an online scenario, frames of audio data are provided one by one for analysis. That is, for online processing, method 100 will involve sequentially receiving and processing frames of audio data. Then, for the current frame, if the current frame is classified as a noise frame at step S110, the noise spectrum will be updated based on the spectrum of the current frame at step S120. Steps S130 to S150 will continue substantially as described above. This may involve, for example, determining one or more updated buzz noise frequencies based on the updated noise spectrum at step S130, generating an updated estimated buzz noise signal based on one or more updated buzz noise frequencies at step S140, and removing buzz noise from the current frame based on the updated estimated buzz noise signal at step S150.

[0100] Figure 10 This is a block diagram illustrating a non-limiting example of a functional overview 1000 of an embodiment of the technology consistent with the present disclosure described above. It should be noted that the blocks shown in this figure and their corresponding functions can be implemented in software, hardware, or a combination of software and hardware.

[0101] Box 1010 receives audio input as (overlapping) frames. Box 1020 implements one or more content activity detectors for frame classification, for example, consistent with step S110 above. If a frame has no content activity, i.e., is a noise frame, it is provided to box 1030 for noise spectrum estimation (e.g., KNS). For example, this can be done consistent with step S120 above. Box 1035 determines a smooth envelope of the noise spectrum, such as a cepstral envelope. At box 1040, the noise spectrum and smooth envelope are used for hum detection, for example, by detecting anomalous peaks in the noise spectrum that exceed the smooth envelope. Then, box 1050 determines the hum noise frequency and average hum amplitude based on the hum detection results. For example, the operations of boxes 1035, 1040, and 1050 can continue consistent with step S130 above. The determined hum noise frequency and average hum amplitude are provided to box 1070 for hum synthesis. If the frame has no content activity, i.e., is a noisy frame, the instantaneous amplitude and phase are determined at box 1060. This can be further used using the hum noise frequency determined by box 1050. Hum sound synthesis is then performed at box 1070. The details may depend on the specific implementation and / or classification of the frames(s) for which hum noise is to be removed. For example, the operations of boxes 1060 and 1070 may continue in accordance with step S140 described above. The synthesized hum sound is then subtracted from the corresponding frame at adder / subtractor 1080. For example, this may be in accordance with step S150 described above. Finally, overlap and addition are performed at box 1090 to generate the output signal. Box 1090 may or may not be part of the actual hum noise removal process, depending on the specific implementation.

[0102] This disclosure also relates to an apparatus for performing the methods and techniques described throughout this disclosure. Figure 11 An example of such an apparatus 1100 is shown. Apparatus 1100 includes a processor 1110 and a memory 1120 coupled to the processor 1110. The memory 1120 may store instructions for the processor 1110. The processor 1110 may receive audio data 1130 as input. The audio data 1130 may have the aforementioned properties in the context of a corresponding method for buzz noise detection and / or buzz noise removal. The processor 1110 may be adapted to perform the methods / techniques described throughout this disclosure. Thus, the processor 1110 may output denoised audio data 1140. Further, the processor 1110 may receive inputs of one or more control parameters 1150. For example, these control parameters 1150 may include control parameters for controlling the aggressiveness of buzz noise removal.

[0103] explain

[0104] The aspects of the system described herein can be implemented in a suitable computer-based sound processing network environment (e.g., a server or cloud environment) to process digital or digitized audio files. Parts of the adaptive audio system may include one or more networks, comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0105] One or more of the components, blocks, processes, or other functional components may be implemented by one or more computer programs executed by one or more processor-based computing devices controlling the system. It should also be noted that the various functions disclosed herein may be described in terms of behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media such as optical, magnetic, or semiconductor storage media.

[0106] In particular, it should be understood that embodiments may include hardware, software, and electronic components or modules, which, for discussion purposes, may be illustrated and described as appearing to be implemented only in hardware. However, those skilled in the art, and based on a reading of this specific embodiment, will recognize that in at least one embodiment, the electronic aspects may be implemented in software (e.g., stored on a non-transitory computer-readable medium) executable by one or more electronic processors such as microprocessors and / or application-specific integrated circuits (“ASICs”). Therefore, it should be noted that embodiments may be implemented using multiple hardware and software-based devices and multiple different structural components. For example, the “content activity detector” herein may include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., system buses) connecting the various components.

[0107] While one or more embodiments have been described by way of example and specific examples, it should be understood that one or more embodiments are not limited to the disclosed embodiments. Rather, they are intended to cover various modifications and similar arrangements that will be obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation in order to cover all such modifications and similar arrangements.

[0108] Similarly, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” or “having,” and variations thereof, is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “coupled,” and variations thereof, are used extensively and cover direct and indirect installation, connection, support, and coupling.

[0109] The enumerated example embodiments

[0110] Various aspects and implementations of this disclosure can also be understood from the following enumerated exemplary embodiments (EEE), which are not claims.

[0111] EEE1. A method for automatically detecting and removing hum noise from audio data, the method comprising: dividing the audio into multiple overlapping frames; classifying each of the multiple overlapping frames as speech / music or noise using one or more content activity detectors (CAD); estimating a key noise spectrum (KNS) in a subset of the multiple overlapping frames; identifying a set of hum frequencies from the key noise spectrum; estimating a set of hum amplitudes associated with the set of hum frequencies from the mean noise spectrum (MNS; e.g., based on the average spectrum of the subset of frames classified as noise); estimating a set of instantaneous amplitudes and a set of instantaneous phases associated with the set of hum frequencies at each short-time frame; synthesizing a set of hum sounds based on the set of hum frequencies; and subtracting the synthesized set of hum sounds for one or more short-time frames of the audio.

[0112] EEE2. The method of EEE1, wherein dividing the received audio into multiple overlapping frames includes applying a window function and selecting a frame size based on one or more low-frequency tones associated with the audio (e.g., selected to adequately resolve the lowest audible frequencies present in the audio).

[0113] EEE3. Similar to EEE1 or EEE2, where one or more CADs include multiple parallel CADs specifically designed for detecting different content types.

[0114] EEE4. The method of any of EEE1 to EEE3, wherein KNS is estimated based on (e.g., on) the average spectrum (MNS) of frames classified as noise.

[0115] EEE5. The method of any of EEE1 to EEE3, wherein KNS is estimated based on (e.g., on) a noise spectrum including the maximum energy weighted by MNS.

[0116] EEE6. Similar to the method in EEE4, where, for offline scenarios, all noisy frames in the file are considered (e.g., all frames in the file classified as noise are used for KNS).

[0117] EEE7. Similar to the method in EEE4, where, for online scenarios, continuously received noisy frames are adaptively considered (e.g., KNS is updated as noisy frames in the file are analyzed).

[0118] EEE8. The method of any of EEE1 to EEE7, wherein one or more CADs determine the frequency-related probability.

[0119] EEE9. The method of any of EEE1 to EEE8, wherein a set of humming frequencies is identified as anomalous peaks by comparison with expected values ​​defined by the cepstral envelope of KNS.

[0120] EEE10. Similar to the method in EEE9, where the cepstral envelope is estimated on a perceptually curved scale (e.g., the Mel scale, the Buck scale, etc.).

[0121] EEE11, such as EEE9 or EEE10, wherein the detection is defined by an amplitude threshold exceeding the cepstral envelope.

[0122] EEE12. The method is similar to that of EEE11, where the amplitude threshold is an adaptive threshold (e.g., adaptive to different frequency bands).

[0123] EEE13. The method of any one of EEE1 to EEE12, wherein the instantaneous amplitude and instantaneous phase are estimated at the humming frequency.

[0124] EEE14. The method of EEE13, wherein estimating the instantaneous amplitude or estimating the instantaneous phase includes performing a least-squares estimation method in the time domain.

[0125] EEE15. The method of any one of EEE1 to EEE14, wherein synthesizing a set of humming sounds comprises summing multiple sine curves based on an identified set of humming frequencies and an estimated set of instantaneous phases.

[0126] EEE16. The method of EEE15, wherein a set of humming sounds is synthesized further based on amplitudes estimated from the MNS; and wherein one or more short frames of the audio are frames containing speech / music.

[0127] EEE17. The method of EEE15, wherein a set of humming sounds is synthesized further based on an estimated set of instantaneous amplitudes; and wherein one or more short frames of the audio are frames containing noise.

[0128] EEE18. The method of EEE15, wherein a set of hums is synthesized further based on amplitudes estimated from the MNS; and wherein one or more short frames of audio include frames containing speech / music and frames containing noise (e.g., amplitudes estimated from the MNS are used to synthesize and remove hums or other operations from all frames, regardless of how the frames are classified by one or more CADs).

[0129] Claims:

Claims

1. A method for processing audio data, wherein, The audio data includes multiple frames, and the method includes: The audio data frames are classified as content frames or noise frames using one or more content activity detectors. The noise spectrum is determined based on one or more frames of the audio data that are classified as noise frames; Determine the smooth envelope of the noise spectrum; One or more buzzing noise frequencies are determined based on abnormal peaks in the noise spectrum compared with the smooth envelope, wherein if the amplitude of a peak in the noise spectrum exceeds the smooth envelope by a greater than a threshold, the peak is determined to be an abnormal peak. An estimated buzz noise signal is generated based on the one or more buzz noise frequencies; and The buzz noise is removed from at least one frame of the audio data based on the estimated buzz noise signal.

2. The method according to claim 1, wherein, The smooth envelope is determined on a scale of perceived curvature.

3. The method according to claim 1, wherein, The threshold is a frequency-dependent threshold.

4. The method according to any one of claims 1-3, wherein, The noise spectrum is determined based on the average of the spectra of the one or more frames that are classified as noise frames.

5. The method according to any one of claims 1-3, wherein, The noise spectrum is determined based on the spectrum containing the maximum energy among the spectra of one or more frames classified as noise frames.

6. The method according to any one of claims 1-3, wherein, Generating the estimated buzz noise signal involves synthesizing a corresponding buzz sound for each of the one or more buzz noise frequencies.

7. The method according to any one of claims 1 to 3, wherein, Generating the estimated buzzing noise signal involves: For each buzzing noise frequency, the corresponding buzzing noise phase is determined based on the corresponding buzzing noise frequency and the audio data in the at least one frame; as well as Based on the buzzing noise frequency and the corresponding buzzing noise phase, a corresponding buzzing sound is synthesized for each of the one or more buzzing noise frequencies.

8. The method according to claim 7, wherein, Generating the estimated buzzing noise signal involves: For each buzzing noise frequency, the corresponding buzzing noise amplitude is determined based on the corresponding buzzing noise frequency and the audio data in the at least one frame; For each buzzing noise frequency, the corresponding average buzzing noise amplitude is determined based on the noise spectrum; as well as The corresponding buzzing sound is synthesized for each of the one or more buzzing noise frequencies based on the corresponding buzzing noise frequency, the corresponding buzzing noise phase, and the smaller of the corresponding buzzing noise amplitude and the corresponding average buzzing noise amplitude.

9. The method according to claim 7, wherein, Generating the estimated buzzing noise signal involves the following steps when the at least one frame is classified as a noisy frame: For each buzzing noise frequency, the corresponding buzzing noise amplitude is determined based on the corresponding buzzing noise frequency and the audio data in the at least one frame; as well as The corresponding buzzing sound is synthesized for each of the one or more buzzing noise frequencies based on the corresponding buzzing noise frequency, the corresponding buzzing noise phase, and the corresponding buzzing noise amplitude.

10. The method according to claim 7, wherein, Generating the estimated buzz noise signal involves the following steps when the at least one frame is classified as a content frame: For each buzzing noise frequency, the corresponding average buzzing noise amplitude is determined based on the noise spectrum; as well as The corresponding buzzing sound is synthesized for each of the one or more buzzing noise frequencies based on the corresponding buzzing noise frequency, the corresponding buzzing noise phase, and the corresponding average buzzing noise amplitude.

11. The method according to any one of claims 1 to 3, wherein, Generating the estimated buzzing noise signal involves: For each buzzing noise frequency, the corresponding average buzzing noise amplitude is determined based on the noise spectrum; as well as Based on the corresponding buzzing noise frequency and the corresponding average buzzing noise amplitude, the corresponding buzzing sound is synthesized for each of the one or more buzzing noise frequencies.

12. The method according to any one of claims 1-3, wherein, Removing the buzz noise from the at least one frame involves subtracting the estimated buzz noise signal from the at least one frame.

13. The method according to any one of claims 1-3, wherein, The noise spectrum is determined based on the spectrum of all frames of the audio data that are classified as noise frames.

14. The method according to any one of claims 1 to 3, comprising: Frames of the audio data are received and processed sequentially; as well as For the current frame, if the current frame is classified as a noisy frame, the noise spectrum is updated based on the spectrum of the current frame.

15. The method according to any one of claims 1-3, wherein, The noise spectrum is determined based on multiple frames classified as noise frames; and The method further includes: The frequency variation of the one or more buzzing noises over time is determined based on the spectrum of the plurality of frames classified as noise frames; and Based on the time-varying changes, a bandpass filter is applied to frames of the audio data, wherein the bandpass filter is designed such that the stopband includes the one or more hum noise frequencies.

16. The method according to claim 15, wherein, The width of the stopband is determined based on the change of the corresponding humming noise frequency over time.

17. The method according to any one of claims 1-3, further comprising: For at least one of the one or more buzzing noise frequencies, determine whether the at least one buzzing noise frequency exists as a peak in the spectrum of most frames of the audio data; as well as If the at least one buzzing noise frequency does not exist as a peak in the spectrum of most frames of the audio data, then the at least one buzzing noise frequency is ignored when removing the buzzing noise.

18. An apparatus for processing audio data, comprising a processor and a memory, the memory being coupled to the processor and storing instructions for the processor, wherein, The processor is configured to perform all the steps of the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform all the steps of the method according to any one of claims 1 to 17.

20. An apparatus for processing audio data, the apparatus comprising units for performing the steps of the method according to any one of claims 1 to 17.

21. A computer program product comprising computer instructions that, when executed by a processor, perform the steps of the method according to any one of claims 1 to 17.

Images