Method and audio processing system for wind noise suppression - Patents.com

Abstract

The present disclosure relates to a method and system (1) for suppressing wind noise. The method includes the steps of obtaining an input audio signal (100, 100') comprising a plurality of consecutive audio signal segments (101, 102, 103, 101', 102', 103') and suppressing the wind noise in the input audio signal using a wind noise suppression module (20) to generate a wind noise reduced audio signal. The method further includes the steps of using a neural network trained to predict a set of gains for reducing the noise in the input audio signal (100, 100') given samples of the input audio signal (100, 100'), the noise reduced audio signal being formed by applying the set of gains to the input audio signal (100, 100'), and mixing the wind noise reduced audio signal and the noise reduced audio signal using a mixer to obtain an output audio signal having suppressed wind noise.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 432,996, filed December 15, 2022, U.S. Provisional Application No. 63 / 327,030, filed April 4, 2022, and International Patent Application No. PCT / CN2022 / 080242, filed March 10, 2022, the entire contents of which are incorporated herein by reference.

[0002] [Technical field] The present invention relates to a method and an audio processing system for wind noise suppression. [Background technology]

[0003] There are many types of unwanted noise that can be present in an audio signal, including non-stationary noise (e.g., traffic noise or wind noise) and stationary noise (e.g., white or pink noise). Any noise reduces the signal-to-noise ratio of the audio signal, degrading the perceived quality of the audio signal. For example, high noise levels result in reduced intelligibility of speech content and / or less accurate rendering of spatial audio objects. Noise caused by wind, i.e., wind noise, is particularly disruptive to many types of audio content, including speech.

[0004] Wind noise is commonly present in audio signals recorded by headsets (e.g., wireless binaural headsets), external microphones, or mobile phones when a user moves rapidly through the air (e.g., when riding a bicycle) or experiences windy conditions outdoors. Wind noise is unpredictable and can suddenly appear and disappear in the audio content, causing an unpleasant listening experience for the listener and can also obscure the desired audio content in the audio signal. Generally, most of the spectral energy of wind noise is in the lower audible frequencies below 2 kHz, which unfortunately overlaps with part of the frequency band related to human speech, making wind noise particularly disruptive to speech and causing problems for, for example, telephone or teleconferencing applications.

[0005] To reduce wind noise, various solutions have been proposed. When recording audio signals with professional equipment (e.g. using dedicated microphones such as boom microphones), different types of physical filters are mounted on the microphone to protect it from strong wind gusts that reach the sensitive recording elements of the microphone. Examples of such physical filters are different windscreen filters, pop filters or mufflers. However, in many applications, the microphones are small and are, for example, integrated into earphones of earphone sets or mobile phones, making it difficult or impossible to use these generally large and bulky physical filters.

[0006] For this reason, various audio processing schemes have been proposed to reduce wind noise in audio signals. In some such processing schemes, a wind detector and a wind suppressor are used to form a noise suppression system operating on two audio signals. The wind noise detector has multiple analyzers, such as a spectral gradient analyzer, a ratio analyzer, a coherence analyzer, a phase variance analyzer, etc., and the detection results of each analyzer are weighted together to form an overall wind noise detection result for each of the two audio signals. The wind suppressor has a calculation unit that calculates a ratio based on the wind noise detection results for each of the two audio signals, and a mixer that mixes the two audio signals based on the wind noise detection result and the ratio of the calculation unit.

[0007] According to another proposed processing scheme, a neural network trained to predict the gain for removing noise in a mono audio signal is used. When processing two audio signals (e.g., a stereo audio signal pair), each audio signal is processed separately and the maximum gain predicted for either audio signal is applied to both audio signals to minimize distortion and maintain the perceived location of spatial audio objects. Since this noise reduction technique is aggressive, a remix module is also used that reintroduces the original (noisy) audio signal by mixing it with the noise-reduced audio signal. Summary of the Invention

[0008] A drawback of conventional audio processing solutions for wind noise reduction is that when wind noise is present in only one of the two audio signals, the output audio signal still contains a high level of residual noise. To avoid these problems, more aggressive noise processing techniques may be used in combination with a remixer that reintroduces part of the original audio signal to reduce acoustic distortion. However, when the wind noise is strong, the reintroduction of the original audio signal quickly reintroduces a noticeable level of wind noise into the audio signal. Therefore, there is a need for an improved method of suppressing wind noise that overcomes at least some of the above drawbacks.

[0009] A first aspect of the present invention relates to a method for suppressing wind noise, comprising obtaining an input audio signal comprising a plurality of consecutive audio signal segments, the method further comprising: suppressing wind noise in the input audio signal using a wind noise suppression module to generate a wind noise reduced audio signal, the wind noise suppression module including a high pass filter, and using a neural network trained to predict a set of gains for reducing noise in the input audio signal given samples of the input audio signal, the noise reduced audio signal being formed by applying the set of gains to the input audio signal. The method also comprises mixing the wind noise reduced audio signal and the noise reduced audio signal using a mixer to obtain an output audio signal having suppressed wind noise.

[0010] The wind noise suppression module may be any wind noise suppressor that performs some filtering or masking of the input audio signal with the purpose of removing wind noise. The resulting wind noise reduced audio signal is thus a processed version of the input audio signal with removed wind noise. The wind noise reduced audio signal may still feature one or more other types of noise, such as stationary white noise and dynamic traffic noise. The neural network is a noise suppression neural network or a source separation neural network trained to separate the desired audio content (e.g., speech or music) by suppressing all types of undesired noise or audio content. The resulting noise reduced audio signal is thus a processed version of the input audio signal with one or more types of reduced noise. The gain prediction neural network is envisaged to remove stationary and dynamic noise, e.g., wind noise.

[0011] The inventors have recognized that by mixing the wind noise suppressed audio signal with the noise suppressed audio signal, wind noise suppression is achieved without introducing undesirable distortion. Furthermore, the problem of remixing is also solved, since the original input audio signal is not reintroduced.

[0012] Using only the neural network may introduce undesirable artifacts, since aggressive noise reduction may cause noticeable distortions. Furthermore, the neural network may remove important environmental audio content that is important for immersion. On the other hand, the wind noise suppressor suppresses the wind noise and leaves all other types of noise that are undesirable and important for providing an immersive effect. By combining the two processed audio signals in a certain mixing ratio, the resulting output audio signal has reduced wind noise due to processing by both the wind noise suppressor and the neural network, the desired audio content is left by both processing schemes, and non-wind related noise is left by the wind noise suppressor and partially by the neural network. Thus, the resulting output audio signal essentially suppresses the wind noise and promotes the desired audio content, while other types of noise content are partially suppressed. Any distortion introduced by the aggressive neural network noise reduction is compensated by the output audio signal from the wind noise suppressor, which is less prone to introduce distortions.

[0013] In some implementations, the method further comprises determining a wind noise indicator for each segment of the input audio signal using a wind noise detector, the wind noise indicator indicating at least one of a probability and a magnitude of wind noise for each segment. Optionally, the method further comprises determining a wind noise condition based on the wind noise indicator. In some implementations, the wind noise indicator is equal to a wind noise condition. In some implementations, the wind noise condition is obtained by smoothing a wind noise metric to avoid rapidly varying steering due to a rapidly varying wind noise metric.

[0014] The wind noise indicator or wind noise conditions may be used to control at least one of the following processes: (A) suppressing the wind noise using a wind noise suppressor, (B) how a set of gains is applied to the input audio signal to form a noise reduced audio signal, and (C) mixing the noise reduced audio signal with the wind noise reduced audio signal, which means that the wind noise suppression method dynamically adapts to the current wind noise conditions.

[0015] In some implementations, the input audio signal, the wind noise reduced audio signal and the noise reduced audio signal each include two audio channels, and the method includes providing wind noise conditions to a gain steering module, determining a common gain set based on a predicted gain set of at least one of the two channels if the wind noise conditions for the two channels exceed a first threshold level or if a difference between the wind noise conditions of the two channels is below a second threshold level, and applying the common gain set to both channels, otherwise the method includes applying each individual gain set to the corresponding channel.

[0016] Thus, the method avoids applying different sets of gains when there is a small difference in wind noise between the channels or when the channels contain similar amounts of wind noise. This ensures that the spatial effect of either of the two channels is not distorted when similar or low levels of wind noise are present in both channels. On the other hand, when one of the channels contains strong wind noise or there is a large difference in the amount of noise content between the two channels, the method applies different sets of gains to the different channels to suppress wind noise when it is most needed, potentially at the expense of causing spatial distortion.

[0017] In some implementations, determining the wind noise state includes providing the wind noise indicator to a state machine having at least two states: a no-wind-noise state (NWN) and a wind-noise-hold state (WNH). The state machine transitions to the WNH state in response to detecting a first number of subsequent segments associated with the wind noise indicator exceeding a high threshold and outputs a high wind noise state at least until the next state change, and the state machine transitions to the NWN state in response to detecting a second number of subsequent segments associated with the wind noise indicator below a first low threshold and outputs a low wind noise state at least until the next state change. Optionally, the state machine includes four states.

[0018] A state machine is used to obtain stable wind noise conditions, which ensures that control of system components based on wind noise conditions does not suffer from oversteering, which can cause noticeable acoustic artifacts due to rapid process changes.

[0019] In some implementations, the input audio signal, the wind noise reduced audio signal and the noise reduced audio signal include two audio channels, and the method further includes determining a difference measure for the two audio channels using a dual mono detector, the difference measure indicating a difference in spectral energy in at least one frequency band of the two audio channels. If the difference measure is less than a difference threshold, the method includes determining a wind noise indicator based only on mono features of each individual audio channel. Otherwise, the method includes determining a wind noise indicator based on both the mono features of each audio channel and difference features associated with both audio channels.

[0020] Examples of mono features determined for a single channel are the spectral slope of one or more frequency bands and the power density centroid of one or more frequency bands. Examples of difference features are measures of difference in spectral power, coherence and phase for one or more corresponding frequency bands of the two audio channels.

[0021] Therefore, by determining how different the channels are and adapting the method for determining the wind noise indicator accordingly, a robust and accurate wind noise indicator is extracted regardless of the level of difference between the channels of the input audio signal.For example, if there is no distinction between very similar and very different audio channels, the difference features may result in false positives indicating strong wind noise due to channels containing very similar content, even if the channels contain very low or no wind noise.Furthermore, since only mono features are determined for very similar audio signals, the computational complexity is reduced.

[0022] According to a second aspect of the invention there is provided a wind noise suppression system including a processor and a memory coupled to the processor, the processor adapted to perform the method of the first aspect of the invention.

[0023] According to a third aspect of the invention there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out a method according to the first aspect of the invention.

[0024] According to a fourth aspect of the present invention there is provided a computer readable storage medium storing a computer program according to the third aspect of the present invention.

[0025] The invention according to the second, third and fourth aspects feature the same or equivalent advantages as the method according to the first aspect. Any feature described with respect to the wind noise suppression method may have a corresponding feature in the wind noise suppression system and vice versa. [Brief description of the drawings]

[0026] The present invention will now be described in more detail with reference to the accompanying drawings, which show a presently preferred embodiment of the invention. [Figure 1] 1 illustrates a system for suppressing wind noise, according to some implementations. [Figure 2A] 2 illustrates a schematic of an input audio signal having a single channel in some implementations; [Figure 2B] 2A-2C illustrate schematic diagrams of different input audio signals having two channels in some implementations; [Diagram 3] A system for suppressing wind noise having a dual-mono detector, a wind noise detector and a smoother as well as a smoothing module according to some implementations is shown. [Figure 4] 1 is a flowchart illustrating a method for suppressing wind noise, according to some implementations. [Diagram 5] 13 illustrates a schematic of a state machine having four states used in the smoothing module, according to some implementations. [Figure 6] 1 illustrates a system for suppressing wind noise using a context analysis module for extended non-real-time processing, according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The systems and methods disclosed in this application may be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of a task does not necessarily correspond to a division into physical units, but rather one physical component may have multiple functions and one task may be performed by several physical components working together. The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, a wearable device (XR or AR or VR or MR headset, an audio headset, etc.), or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the computer hardware. Furthermore, the present disclosure relates to any collection of computer hardware that individually or jointly executes instructions to perform any one or more of the concepts discussed herein.

[0028] Certain or all of the components may be implemented by one or more processors that accept computer-readable (also referred to as machine-readable) code that includes a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify an action to be taken is included. Thus, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, SSD, RAM, and / or ROM. A bus subsystem may be included for communication between the components. Software may reside in the memory subsystem and / or in the processor during its execution by the computer system.

[0029] One or more of the processors may operate as stand-alone devices or may be connected to other processors, e.g., networked, such as over a variety of different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0030] Software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes various forms of physical (non-transitory) storage media, such as, but not limited to, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Additionally, communication media (transitory) typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and is well known to those skilled in the art to include any information delivery media.

[0031] FIG. 1 illustrates a system 1 for suppressing wind noise, according to some embodiments. The components of the system 1 may be implemented by a computer including a processor and a memory coupled to the processor. An input audio signal is received and provided to a wind noise suppression module 20 and a neural network 10 that is trained to predict a set of gains for reducing noise given samples of the input audio signal. The predicted set of gains is provided to a gain applicator 11, which applies the gains to the input audio signal and provides a resulting noise-reduced audio signal Y to a mixer 30. The mixer 30 also receives a wind noise-reduced audio signal X from the wind noise suppression module 20 and mixes the two audio signals in a mixing ratio indicated by a mixing coefficient μ to generate an output audio signal Z.

[0032] The input audio signal includes one or more audio channels. In some implementations, the input audio signal includes wind noise mixed with the desired audio content, and optionally other types of noise, such as stationary white noise, stationary pink noise, and / or non-stationary traffic noise. The neural network 10 is any type of neural network trained to predict a set of gains for suppressing all types of noise other than the target source (e.g., speech or music). The neural network may be implemented as an RNN, a CNN, etc.

[0033] The set of predicted gains includes multiple gains, each gain associated with an individual frequency band of the segment. It is understood that as the term "gain" is used herein, "gain" may involve either actual gain (i.e., amplification) or attenuation (i.e., amplitude reduction). Most noise reduction neural networks 10 are trained to predict gains for all audio content that is different from the desired audio content (e.g., speech or music). For example, if a particular frequency band in a segment of the input audio signal contains noise and little or no desired audio content, the neural network 10 outputs a gain that suppresses this frequency band for this segment. It is understood that this process is dynamic, and as the spectral distribution of the desired audio content and noise change over time, the frequency bands that are suppressed and the degree to which they are suppressed will change from segment to segment.

[0034] The wind noise suppressor 20 is configured to suppress wind noise in the input audio signal. Because wind noise often appears in the low frequency range (below 2 kHz), the wind noise suppressor includes a high pass filter with a cutoff frequency and, optionally, a passband variable gain. In some implementations, the cutoff frequency and passband gain are dynamically adjustable (based on the wind noise indicator / condition), as described below.

[0035] The wind noise suppressor 20 is configured to remove wind noise from an audio signal. Since wind noise is typically present at low frequencies, the wind noise suppressor generally does not remove high frequency noise, such as the high frequency components of stationary white or pink noise. On the other hand, the neural network 10 typically performs more aggressive noise suppression, where all noise content that is not recognized as the desired audio content is suppressed. This may require the neural network 10 to generate gains to suppress stationary noise (e.g., white noise) and dynamic noise, such as wind noise or traffic noise. The neural network 10 can be very effective at isolating the desired audio content, but can cause undesirable distortions or misclassify the desired audio content as noise, whereas the analytical wind noise suppressor 20 operates in a more controlled and predictable manner.

[0036] 2a shows a schematic representation of an input audio signal 100. The input audio signal includes a number of consecutive segments 101, 102, 103, each of which includes a portion of the audio signal 100. The multiple segments 101, 102, 103 may form a complete audio file of any duration (e.g. ranging from a few minutes if the audio file is a music track to several hours if the audio file is a telephone call audio content or a movie soundtrack). Each segment may be of any suitable length. In some implementations, each segment 101, 102, 103 is between 2 ms and 50 ms long, such as 5 ms or 10 ms long.

[0037] The input audio signal 100 can be represented in the time domain or in the frequency domain. The input audio signal 100 may be represented in time-frequency tiles or may be represented in a filter bank (e.g., a QMF filter bank) as known to those skilled in the art. The input audio signal may have one channel or two or more channels.

[0038] In Fig. 2b, an input audio signal 100' is shown that includes two audio channels. As can be seen, the two audio channels are divided into corresponding segments 101, 102, 103, 101', 102', 103'. The two audio channels are processed separately or together in the wind noise suppression system of Fig. 1. For example, the wind noise suppressor may determine and apply different high-pass filters or a common filter to the two channels. Similarly, the neural network may predict a set of individual gains for each channel, whereby the gain applicator applies the individual gain sets to the corresponding channels or combines the individual gain sets into a common gain set, as described below.

[0039] The two audio channels may be any type of audio channel, such as a stereo, a binaural audio channel, or any selection of two arbitrary channels. The stereo audio channel may be, for example, a left and right audio channel. It is also envisaged that the stereo audio channel is a different stereo representation, for example formed by a mid and a side audio channel. The two audio channels may contain the same or at least very similar audio content (such as in the case of center-panned stereo music content), or audio content with no or little audio content in common (for example audio content targeted to a center loudspeaker and a left rear loudspeaker in a 5.1 presentation of a sound file associated with a movie).

[0040] FIG. 3 shows the system for wind noise suppression 1 of FIG. 1 with the addition of a dual mono detector 40 and a wind noise detector 50.

[0041] The wind noise detector 50 analyzes the input audio signal and determines a wind noise indicator for each segment of the input audio signal. The wind noise indicator is a measure (e.g., one or more numerical values) indicating whether wind noise is present in the input audio signal. Thus, the wind noise indicator may be a measure of the magnitude or probability of wind noise. The wind noise indicator may be a binary value (i.e., indicating the presence or absence of wind noise) or a scalar value (soft score) indicating the magnitude / probability of wind noise in each segment. For example, the scalar value (soft score) may range from 0 to 1, with smaller values ​​indicating lower wind noise magnitude / probability and higher values ​​indicating higher wind noise magnitude / probability.

[0042] The wind noise detector 50 extracts several features from the input audio signal and determines a wind noise indicator based on these features. The wind noise indicator may determine a mono feature for each individual channel of the input audio signal and / or a difference feature based on at least two channels of the input audio signal. The wind noise detector 50 may determine at least one of the following features: a spectral slope in one or more frequency bands of a channel, a power centroid in one or more frequency bands of a channel, a power ratio between two channels in at least one frequency band, a coherence between two channels in at least one frequency band, a phase difference between two channels in at least one frequency band, and determine a wind noise indicator based on the features determined for each channel and segment of the input audio signal.

[0043] The wind noise detector 50 may also be implemented by conventional machine learning algorithms such as support vector machines, AdaBoost or deep neural networks, depending on the available computing resources.

[0044] At least one of the wind noise indicator and the wind noise condition (derived from the wind noise indicator) is provided to at least one of the mixer 30, the wind noise suppressor 20 and the gain applicator 11 to dynamically control the operation of at least one of the components.

[0045] 3, the wind noise indicator is provided to a smoother 60, which extracts the wind noise condition by smoothing the wind noise indicator, as described in more detail below, whereby the wind noise condition is provided to at least one of the mixer 30, the wind noise suppressor 20, and the gain applicator 11 to dynamically control the operation of at least one of the components. In some implementations, the wind noise indicator is instead provided to at least one of the mixer 30, the wind noise suppressor 20, and the gain applicator 11 (e.g., when the smoother 60 is not present) to dynamically control the operation of at least one of the components. It is also envisioned that both the wind noise indicator and the wind noise condition may be provided to at least one of the mixer 30, the wind noise suppressor 20, and the gain applicator 11.

[0046] Mixer 30 may receive the wind noise indicator and / or wind noise conditions and modify the mix coefficient μ (mix ratio) accordingly. In Figure 3, the wind noise reduced audio signal from wind noise suppressor 20 is labeled X and the noise reduced audio signal from gain applicator 11 is labeled Y. Mixer 30 mixes outputs X and Y to form an output audio signal Z based on the mix coefficient μ according to the following equation: Z = μY + (1-μ)X (1) That is, the mixing coefficient μ determines the mixing ratio for X and Y.

[0047] The wind noise indicator and / or wind noise condition affect the value of the blend coefficient μ, thereby adding more or less emphasis to X or Y in the output Z. In some implementations, μ is toggled between two values, such as μ=0.9 and μ=0.6, with the higher value assigned when the wind noise indicator and / or wind noise condition indicates a high level of wind noise and the lower value assigned when the wind noise indicator and / or wind noise condition indicates a low level of wind noise. It is also envisioned that with a scalar value wind noise indicator and / or wind noise condition, the blend coefficient μ is interpolated between two values ​​based on the scalar value.

[0048] Alternatively, the mix coefficient μ may be static or specified by the user: if cleaner audio is desired, μ is set to 0.75, if more ambient sound is desired, μ is set to 0.5, etc. The value of μ may also be selected according to the environmental context.

[0049] The wind noise suppressor 20 may receive the wind noise indicator and / or the wind noise conditions and perform wind noise suppression based on the wind noise indicator. For example, the characteristics of a high pass filter may be adjusted based on the wind noise indicator and / or the wind noise conditions. For a scalar value (soft score) wind noise indicator, the high pass filter gain (HPF) for a given frequency band b may be determined as follows:

number

[0050] If only binary wind noise indicators and / or wind noise states are available, the wind noise power spectral density N(n,f) is determined via recursive smoothing of the squared wind noise spectral magnitude as follows: Φ(n,f)=a(n)Φ(n-1,f)+(1-a(n))|N(n,f)| 2 ,f <F (3) where a(n) and F are assigned values ​​such as a(n) = 0.8 and F = 1.6 kHz. Wind noise power |N(n,f)| 2 is determined only for wind noise segments associated with high binary wind noise indicators and / or wind noise conditions. Once the wind noise power spectral density is determined, classical noise reduction algorithms such as spectral subtraction may be used by the wind noise suppressor 20 to reduce the wind noise. That is, applying a high pass filter may include performing a spectral subtraction of the estimated wind noise power spectral density.

[0051] The gain applicator 11 may receive the wind noise indicator and / or wind noise condition from the wind noise detector 50 and modify the gain application based on the wind noise indicator and / or wind noise condition. This is particularly useful when the input audio signal includes two audio channels, the wind noise detector 50 has determined the wind noise indicator and / or wind noise condition for each segment of each channel, and the neural network has predicted an individual set of gains for each channel. The gain applicator 11 analyzes the wind noise indicator and / or wind noise condition for each channel and decides based on the wind noise indicator and / or wind noise condition from each channel whether to apply a respective set of gains to each channel (Mode A) or to determine a common set of gains to be applied to both channels (Mode B), the common set of gains being based on the individual set of gains for each channel. The common set of gains may be, for example, a maximum or average value for the elements of the two gain sets.

[0052] If the gain applicator 11 determines that the wind noise indicator and / or wind noise conditions indicate that only one channel is characterized by wind noise (exceeds a first threshold level γ1) or if the difference in the wind noise indicator and / or wind noise conditions between the channels exceeds a second threshold level γ2, the gain applicator 11 operates in mode A.

[0053] If the gain applicator 11 determines that the wind noise indicator and / or wind noise conditions indicate that both channels are characterized by wind noise (exceeding a first threshold level γ1), or if the difference in the wind noise indicator and / or wind noise conditions between the channels falls below a second threshold level γ2, the gain applicator 11 operates in mode B.

[0054] For example, the second threshold level γ2 is selected between 0.1 and 0.3, such as γ=0.2. Also, there are many suitable values ​​for the first threshold level γ1. For example, γ1 may be selected in the interval 0.1-0.6, such as 0.5 or 0.25. However, these values ​​of γ2 and γ1 are merely exemplary, and other values ​​of γ2 and γ1 are also envisaged depending on the characteristics (e.g., sensitivity) of the wind noise detector 50.

[0055] Mode A ensures sufficient wind noise attenuation for channels with very strong wind noise, while Mode B ensures not to introduce differential filtering of the two channels for small amounts of wind noise that could alter the spatial balance of the two channels in an undesirable and distracting way.

[0056] As described above, the wind noise detector 50 determines the wind noise indicator based on the mono channel feature and / or based on the difference feature between the two channels. If the input audio signal is a mono audio signal, only the mono channel feature can be used. However, if the input audio signal includes more than one channel, the wind noise detector may use either or both of the mono channel feature and the difference feature.

[0057] In order to increase accuracy when determining the wind noise indicator for the two input channels, it is important to detect how similar the two channels of the input audio signal are in terms of audio content. If the channels contain similar or equal audio content according to the difference measure, the wind noise detector 50 uses only mono features when determining the wind noise indicator. Since highly similar channels may resemble a duality of the same mono audio signal, such audio channels associated with a low difference measure are referred to herein as dual-mono audio channel pairs. On the other hand, if the channels are less similar (i.e. associated with a high difference measure), the wind noise detector 50 uses both mono features and channel difference features for each audio channel.

[0058] It will be appreciated that pairs of audio channels exhibiting a low difference measure have a lot of common audio content between the channels, while pairs of audio channels having a higher difference measure have little or no common audio content.

[0059] To this end, the system 1 includes a dual mono detector 40 configured to determine whether two channels of an input audio signal are sufficiently different from one another to enable the wind noise detector 50 to use both channel difference features and mono features when determining the wind noise indicator. The dual mono detector 40 analyzes the audio channels and determines a difference measure. The difference measure may, for example, be indicative of at least one of a difference in spectral energy in one or more frequency bands.

[0060] In one implementation, the dual-mono detector 40 operates in the frequency domain (e.g., the Short-Time-Fourier-Transform (STFT) domain) and calculates the sum S of the absolute values ​​of the spectral differences between the first and second channels as follows:

number

number

[0061] Once the sum S and the individual channel energies E1, E2 have been determined, the dual-mono detector 40 calculates the normalized sum (ratio) S N is determined as follows:

number

[0062] Japanese S, S N and the relationship between the channel energies E1, E2 are all examples of difference measures that can be used to determine whether the channels are similar enough to use only mono features.

[0063] In some implementations, the dual mono detector 40 includes a dual mono counter that counts the number of encountered segments that have audio content similar enough to be labeled as dual mono segments. decorr When the counter reaches N, the entire input audio signal is classified as a dual mono audio signal and all subsequent segments are treated as dual mono segments. decorr When it reaches this value, it may be deactivated at least until the next audio signal is input to the system 1. The counter of the dual mono detector 40 may be reset each time it detects a segment that does not contain channels that are similar enough to be classified as a dual mono segment.

[0064] As an example, the low band index b1=0, the high band index b2=512, and the first predetermined threshold α=10 -4 and the second predetermined threshold β=10 -8 and N decorr =40.

[0065] Optionally, the system 1 for wind noise suppression cooperates with a smoothing module 60 that smoothes the wind noise indicator. Based on the smoothed wind noise indicator, a wind noise condition is determined that is used to control a component of the system 1. That is, the wind noise condition is a processed version of the wind noise indicator, and either or both of the wind noise indicator and the wind noise condition can be used to control a component as described above. The wind noise condition may, for example, be equal to a smoothed version of the wind noise indicator. With further reference to the flow chart of FIG. 4, a method for suppressing wind noise will now also be described in detail.

[0066] In step S1, an input audio signal is obtained and provided to the wind noise suppressor 20, the neural network 10 and the gain applicator 11. The input audio signal is further provided to the dual mono detector 40 which in step S2a determines whether the input audio signal contains similar (dual mono) audio channels (e.g. according to equations 4-7 above). If the audio channels are similar, i.e. dual mono channel, the method proceeds to S2b and determines a wind noise indicator for each channel based only on the mono features. On the other hand, if the audio channels are determined to be dissimilar, i.e. non-dual mono channel, the method proceeds to S2c and determines a wind noise indicator for each audio channel based on both the mono features and the channel difference features.

[0067] Optionally, the wind noise indicator is provided to a smoothing module 60 which smoothes the wind noise indicator in step S3 to obtain a more stable wind noise condition.

[0068] A smoothing module 60 smoothes the wind noise indicator over multiple adjacent segments to obtain a more stable wind noise state from the wind noise indicator. The wind noise state then replaces the wind noise indicator and is provided to at least one of the mixer 30, the wind noise suppressor 20 and the gain applicator 11 to enable dynamic control of these components in a manner similar to the wind noise indicator steering highlighted in connection with FIG. 3 above.

[0069] If the wind noise indicator is a scalar value for each segment, the smoothing performed by the smoothing module 60 may involve averaging the wind noise indicator over multiple adjacent segments. Furthermore, smoothing can also be performed if the wind noise indicator is a binary wind noise indicator. For example, a high binary level is represented by a value of 1 and a low binary level is represented by a value of 0, whereby smoothing over multiple frames allows the wind noise state to assume a fractional value between 0 and 1. Although smoothing using averaging over adjacent segments removes the most rapid changes in steering of different components in the wind noise suppression system, there are still cases where even averaging over many segments can cause, for example, rapid toggling between applying a common set of gains or applying individual sets of gains. For this reason, the smoothing module may use a state machine, as described in more detail below.

[0070] In step S4, the input audio signal is provided to a wind noise suppressor 20 which suppresses the wind noise and generates a wind noise reduced audio signal X. The wind noise suppressor 40 may be controlled by the wind noise indicator to apply a high pass filter that dynamically varies with the wind noise indicator or wind noise conditions as described in Equation 2. The wind noise suppressor 40 may determine and apply separate high pass filters for each channel or a common high pass filter for both channels. The common high pass filter may be, for example, an average filter gain for each frequency band.

[0071] In step S5a the input audio signal is provided to a neural network 10 which predicts a set of gains for each channel, and in step S5b applying the sets of gains using a wind noise applicator 11 reduces noise in the input audio signal resulting in a noise reduced audio signal Y. As above, the gain applicator may be controlled by a wind noise indicator or wind noise condition to determine whether a set of gains should be applied to the channels individually or may be used to determine a common set of gains to apply to both channels.

[0072] In step S6, the wind noise reduced audio signal X and the noise reduced audio signal Y are combined in a mixing ratio by the mixer 30. The mixing ratio is determined by the mixing coefficient μ and may be dynamically steered using the wind noise indicator or wind noise conditions as described in relation to Equation 1 above.

[0073] The smoothing module 60 may include a state machine 61 as shown in FIG. 5 for smoothing the wind noise indicator to a more stable wind noise state. The state machine includes four states: a no wind noise (NWN) state, a wind noise attack (WNA) state, a wind noise hold (WNH) state, and a wind noise release (WNR) state. In each state, the state machine outputs either a high (e.g., 1) wind noise state or a low (e.g., 0) wind noise state, and the state machine continues to output the same wind noise state until the state is changed. Some states, such as the NWN and WNH states, are always associated with the same wind noise state (low state for NWN and high state for WNH), while some states, such as the WNA state, may be associated with either a high wind noise state or a low wind noise state depending on which state the state machine transitions from.

[0074] The state machine 61 is evaluated once per segment (i.e., once for each updated value of the wind noise indicator), and for embodiments having multiple channels, one state machine 61 is used to smooth the wind noise indicator for each channel.

[0075] The state machine 61 is configured to start in an initial state of the NWN or WNH.

[0076] Without loss of generality, the state machine 61 of Figure 5 starts by transitioning to the NWN state at L1, outputting a low wind noise state. For each new segment and associated wind noise indicator, the state machine 61 checks whether the wind noise indicator exceeds a high threshold T high Check whether the wind noise indicator is greater than the high threshold T high If it remains below , the NWN state is held and the state machine continues to output the low wind noise state.

[0077] When the state machine 61 is in the NWN state, T high If the state machine 61 detects a wind noise indicator exceeding the high threshold T, the state machine 61 transitions to the WNA state at L2 and continues to output the low wind noise state. When the state machine 61 enters the WNA state, it also detects a wind noise indicator exceeding the high threshold T high 5. Start an attack counter 501 which counts the number of segments having a wind noise indicator exceeding a first predetermined number N acc The WNA state is maintained as long as the wind noise indicator is below the first low threshold T low1 If it falls below this, the attack counter 501 is reset to zero.

[0078] Attack counter 501 is N acc When the wind noise indicator reaches a second low threshold T, the state machine 61 transitions to the WNH state at H1 and the output wind noise state changes from a low wind noise state to a high wind noise state. In the WNH state, the state machine 61 transitions to the WNH state when the wind noise indicator reaches a second low threshold T low2, and checking whether the wind noise indicator exceeds a second lower threshold T low2 As long as the second low threshold T low2 is the first low threshold T low1 Greater than.

[0079] When the state machine 61 is in the WNH state, the wind noise indicator exceeds a second low threshold T low2 If the state machine 61 detects that the wind noise level falls below a second low threshold T, the state machine 61 transitions to the WNR state at H2 while continuing to output the high wind noise state. low2 Trigger a release counter 502 that counts the number of segments associated with the wind noise indicator falling below a second predetermined number N low As long as the counter continues to run, the state machine 61 remains in the WNR state and outputs the low wind noise state.

[0080] When the release counter 502 reaches a second predetermined number N low When this is reached, the state machine 61 transitions to the NWN state at L3 and begins outputting a low wind noise state.

[0081] If the state machine 61 in the WNR state detects that the wind noise indicator is greater than the high threshold, the state machine 61 transitions to the WNA state at H3 while keeping the output in the high wind noise state. This is in contrast to when the state machine 61 entered the WNA state from the NWN state along L2 where the output was fixed at the low wind noise indicator. In some embodiments, the first predetermined number N acc is a second predetermined number N low , which biases the state machine 61 towards the NWN state. low = 20, and N acc =40.

[0082] In some embodiments, the second lower threshold T low2is the average of the first low threshold and the second low threshold. For example, T low1 = 0.04, and T high = 0.2, and T low2 =0.12.

[0083] Furthermore, in some embodiments, it is beneficial to use more than two smoothing techniques. For example, the wind noise indicator is first smoothed by averaging over adjacent segments, performing a smoothing window over the wind noise indicator, or determining a history weighted sum, where the current wind noise indicator is given the maximum weight, older wind noise indicators are given less weight, and the most recently considered wind noise indicator is given the minimum weight. This conventional smoothing process produces a smoothed wind noise indicator. The smoothed wind noise indicator is then provided to a state machine 61 which determines a more stable wind noise condition from the smoothed wind noise indicator.

[0084] The above system 1 for wind noise suppression is suitable for implementation in real-time (causal) and non-real-time (non-causal) processing. However, in some implementations, a context analysis module 70 is added to the wind noise suppression system to further improve non-real-time processing.

[0085] The context analysis module 70 aggregates the per-segment outputs of the wind noise detector 50 (i.e., the wind noise indicator) and the smoother 60 (i.e., the wind noise state extracted from the wind noise indicator, e.g., by a state machine) across many segments (e.g., all segments in the audio file) and generates a global wind noise metric W for all segments. pThe context analysis module 70 may, for example, determine a weighted average of the wind noise metrics and / or wind noise conditions, with the global wind noise metric being based on (or equal to) the weighted average. In some implementations, the global wind noise metric W p is a scalar value between 0 and 1, with lower values ​​indicating lower wind noise loudness / confidence for the analysis segment and higher values ​​indicating higher wind noise loudness / confidence for the analysis segment.

[0086] In some implementations, a wind noise suppression system 1' is used that is configured to utilize an auxiliary sensor and / or an auxiliary classifier unit. The auxiliary sensor may be, for example, an acoustic sensor (e.g., a microphone), an environmental sensor, a GPS receiver, a motion sensor, a vibration sensor, or any other type of sensor typically available on a user device such as a smartphone, an earphone, a smartwatch, a tablet, all examples of devices that may implement the wind noise suppression method. The auxiliary classifier unit may be, for example, one of a music classifier, a voice classifier, and an acoustic scene classifier. The classifier unit may be implemented in software and / or hardware and is typically available on the exemplary devices described above.

[0087] In Fig. 6 this is illustrated by a context analysis module 70 accepting auxiliary data, which may be the output of the auxiliary sensors and / or auxiliary classifier units mentioned above. For example, the auxiliary data may come from a voice activity detection (VAD) classifier unit, a music classifier unit and / or an acoustic scene classifier unit. The context analysis module 70 may then generate a more accurate global wind noise metric W pFor example, if the auxiliary data indicates that a segment contains speech and / or music (by receiving a high VAD value or a high music classifier confidence in the auxiliary data) and may contain wind noise (due to a high wind noise indicator), then context analysis module 70 may determine W since it is unclear whether the segment is music, speech, or wind. p On the other hand, if the segment does not contain speech and / or music (by receiving a low VAD value and / or a low music confidence in the auxiliary data) but does contain wind noise (due to a high wind noise indicator), the context analysis module 70 may provide a lower weight for this segment when calculating W since the confidence for the segment containing wind noise is higher. p This segment may be given a higher weight when calculating

[0088] Then, the global wind noise metric W p is provided to at least one of a context-modified wind noise suppressor 20' and a mixer 30' that uses a modified type of processing compared to the processing described above.

[0089] The context-modified wind noise suppressor 20' uses the global wind noise metric W p and determines a high pass filter according to:

number

[0090] The context correction mixer 30' uses the global wind noise indicator W p Receive the W p Based on the mixing coefficient μ p Determine. μ p =W p *μ1+(1-W p )*μ0(9) where μ1 is greater than μ0. Typically, μ1 is between 0.6 and 0.9, such as 0.75, and μ0 is between 0.4 and 0.6, such as 0.5. The mixing performed by mixer 30 is determined by the following equation: Z=μ p Y+(1-μ p )X (10) where Z is the output audio signal, X is the wind noise suppressed output audio signal from the context modified wind noise suppressor 20', and Y is the audio signal output by the gain applicator module 11 that has applied the set of gains predicted by the neural network 10. Thus, the mix coefficient μ p is kept constant at an interpolated value between μ1 and μ0 for the multiple segments that form the audio file. W is used to process the current (previous) segment as a later (future) segment. p , it will be appreciated that the processing performed using the context analysis module 70 is no longer causal.

[0091] Unless otherwise indicated, and as will be apparent from the discussion that follows, it will be appreciated that throughout this disclosure, discussions using terms such as "processing," "calculating," "computing," "determining," "analyzing," and the like refer to operations and / or processes of computer hardware or computing systems or similar electronic computing devices that manipulate and / or transform data represented as physical quantities, such as electronic quantities, into other data also represented as physical quantities.

[0092] In the above description of exemplary embodiments of the present invention, it should be recognized that various features of the invention may be grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in fewer than all features of a single above-disclosed embodiment. Thus, the claims following the detailed description are expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention. Furthermore, as will be understood by those skilled in the art, some embodiments described herein include some features but not other features included in other embodiments, and combinations of features of different embodiments are meant to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0093] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means for performing the function. Thus, a processor having instructions for performing such a method or elements of a method forms a means for performing the method or elements of a method. It is noted that when a method includes several elements, e.g., several steps, no order of such elements is implied unless specifically stated. Furthermore, the elements described herein of an apparatus embodiment are examples of means for performing the functions performed by the elements for performing an embodiment of the present invention. In the description provided herein, numerous specific details are described. However, it is understood that an embodiment of the present invention can be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0094] Those skilled in the art will recognize that the present invention is in no way limited to the above preferred embodiment. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, any dynamic control based on wind noise conditions (which are smoothed versions of the wind noise indicator) may be replaced with a wind noise indicator, regardless of whether the wind noise indicator is a scalar value or a binary metric. A smoother and more stable wind noise condition over time features a beneficial effect in that it does not result in rapid and noticeable changes in audio processing, but it is understood that for some audio signals, the use of a wind noise indicator directly provides sufficient performance. Furthermore, while most exemplary implementations are described using one or two channel audio signals, the method and system can be applied in a similar manner to three or more channels. For example, processing may be based on pairwise selection between multiple channels.

Claims

1. 1. A method for suppressing wind noise, comprising: obtaining an input audio signal comprising a plurality of consecutive audio signal segments; suppressing wind noise in the input audio signal using a wind noise suppression module to generate a wind noise reduced audio signal, the wind noise suppression module including a high pass filter; using a neural network trained to predict a set of gains for reducing noise in the input audio signal given samples of the input audio signal, a noise reduced audio signal being formed by applying the set of gains to the input audio signal; mixing the wind noise reduced audio signal and the noise reduced audio signal using a mixer to obtain an output audio signal having suppressed wind noise; The method includes:

2. 2. The method of claim 1, further comprising: determining, using a wind noise detector, a wind noise indicator for each segment of the input audio signal, the wind noise indicator indicating at least one of a probability and a magnitude of wind noise for each segment.

3. The method of claim 2 further comprising determining wind noise conditions based on the wind noise indicator.

4. The method of claim 3 , wherein filter coefficients of the high pass filter are based on the wind noise conditions.

5. Each of the input audio signal, the wind noise reduced audio signal and the noise reduced audio signal includes two audio channels, the method comprising: providing said wind noise conditions to a gain steering module; if the wind noise conditions for the two channels exceed a first threshold level, or if the difference between the wind noise conditions of the two channels is below a second threshold level; determining a common set of gains based on the predicted set of gains for at least one of the two channels; applying said common set of gains to both channels; If not, applying each set of gains to a corresponding channel; The method of claim 3 further comprising:

6. The method of claim 5 , wherein the common set of gains is an average, maximum or minimum gain across the set of gains.

7. The step of determining wind noise conditions includes: providing said wind noise indicator to a state machine having at least two states: a no wind noise (NWN) state and a wind noise hold (WNH) state; the state machine transitions to the WNH state in response to detecting a first number of subsequent segments associated with a wind noise indicator exceeding a high threshold and outputs a high wind noise state until at least a next state change; 4. The method of claim 3, wherein the state machine transitions to the NWN state in response to detecting a second number of subsequent segments associated with a wind noise indicator below a first low threshold and outputs a low wind noise state until at least a next state change.

8. The state machine (61) has four states, namely, the NWN state, the WNH state, a Wind Noise Attack (WNA) state, and a Wind Noise Release (WNR) state; the state machine transitions from the NWN state to the WNA state in response to detecting a segment having a wind noise indicator above the high threshold and outputs the low wind noise state until at least a next state change; the state machine transitions from the WNA state to the WNH state in response to detecting the first number of subsequent segments associated with a wind noise indicator exceeding a high threshold and outputs a high wind noise state until at least a next state change; the state machine transitions from the WNH state to the WNR state in response to detecting a segment having a wind noise indicator below the first low threshold and outputs the high wind noise state until at least a next state change; 8. The method of claim 7, wherein the state machine transitions from the WNR state to the NWN state in response to detecting the second number of subsequent segments below the first low threshold and outputs the low wind noise state until at least a next state change.

9. 9. The method of claim 8, wherein the state machine transitions from the WNR state to the WNA state in response to detecting a segment associated with a wind noise indicator that exceeds the high threshold, and outputs the high wind noise state until at least a next state change.

10. Detecting the first number of subsequent segments associated with a wind noise indicator that exceeds the high threshold includes: counting the number of segments that exceed the high threshold using an attack counter; resetting the attack counter when a segment associated with the wind noise indicator falls below a second low threshold; 9. The method of claim 8, wherein the second low threshold is lower than the first low threshold.

11. The step of determining wind noise conditions comprises: The method of claim 3 , comprising smoothing the wind noise indicator over at least two segments, the wind noise condition being based on the smoothed wind noise indicator.

12. The method of claim 1 , wherein each set of gains includes a plurality of gains associated with respective frequency bands of an audio channel.

13. The method of claim 1 , wherein the mixing of the wind noise reduced audio signal and the noise reduced audio signal is performed in a fixed mixing ratio.

14. The method of claim 3 , further comprising the step of controlling, for each segment of the input audio signal, a mixing ratio of the mixer based on the wind noise condition.

15. the input audio signal, the wind noise reduced audio signal and the noise reduced audio signal each include two audio channels; The method of claim 2 , wherein the step of determining the wind noise indicator is based on a mono feature of each individual audio channel and a difference feature associated with both audio channels.

16. The input audio signal, the wind noise reduced audio signal and the noise reduced audio signal include two audio channels, the method comprising: determining, with a dual mono detector, whether corresponding segments of each of the two audio channels are similar dual mono segments or dissimilar non-dual mono segments based on the spectral power distributions of the segments in at least one frequency band; if the segment is a non-dual mono segment, determining the wind noise indicator based on a mono feature and a difference feature associated with both segments; otherwise, determining said wind noise indicator based only on the mono characteristics of each individual segment; The method of claim 2 further comprising:

17. determining whether the two audio segments are similar dual mono segments or dissimilar non-dual mono segments, determining a first sum as a sum of absolute values ​​of spectral differences between the two audio segments of the input audio signal; determining a second sum as a sum of total spectral energy for each of the segments; calculating a ratio between the first sum and the second sum; if the ratio is below a predetermined ratio threshold and the second sum is above a predetermined sum threshold, the segments are determined to be similar dual mono segments; If not, the segment is determined to be a dissimilar non-dual mono segment.

18. forming a non-real-time global wind noise indicator by aggregating the wind noise indicators for a plurality of segments of the input audio signal; The method of claim 2 , wherein the filter coefficients of the high-pass filter are based on a global wind noise metric.

19. forming a non-real-time global wind noise indicator by aggregating the wind noise indicators for a plurality of segments of the input audio signal; controlling a mixing ratio of the mixer for each segment of the input audio signal based on a global wind noise metric; The method of claim 2 further comprising:

20. 1. A wind noise suppression system comprising: a processor; and a memory coupled to the processor, A wind noise suppression system, wherein the processor is adapted to perform the steps of the method according to any one of claims 1 to 19.

21. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 19.

22. 22. A computer readable storage medium storing the computer program of claim 21.

Images