Content Recognition Audio Noise Management

Abstract

Methods and apparatus for managing noise in mixed audio content types. For example, a noise management method includes performing audio event segmentation on an input audio signal to generate content-aware segmentation information. The method also includes estimating a noise floor level associated with the input audio signal based on the content-aware segmentation information and further sorting frequency bin data corresponding to fixed-length portions of the input audio signal. The method further includes applying noise suppression to the input audio signal to generate an output audio signal, the noise suppression being performed in frequency bins having selected frequency resolution and at least in part based on the content-aware segmentation information and the estimated noise floor level.

Description

Technical Field

[0001] Various exemplary embodiments relate to audio signal processing and enhancement.

Background Art

[0002] Unless otherwise specified, the content described in this chapter is not prior art to the claims of this application and is not admitted to be prior art by virtue of being included in this chapter.

[0003] Noise reduction processing typically aims to significantly reduce background and / or broadband noise with minimal reduction to the audio signal quality. Such processing can be configured to remove various noise components including, but not limited to, tape hiss, microphone background noise, hum noise, wind noise, etc. An appropriate amount of noise reduction typically depends on the type of the useful audio signal, the type of the noise, and the acceptable loss to the useful audio signal. In some cases, the noise reduction processing can increase the signal-to-noise ratio (SNR) by 5 - 20 dB while maintaining a high audio quality for the remaining useful audio signal.

[0004] The disclosure made in this specification is presented with respect to these and other considerations.

Summary of the Invention

[0005] Techniques for processing audio signals are described. Examples found in this specification provide systems, devices, apparatuses, and methods for performing content recognition audio noise management. Some embodiments can be beneficially used to automatically manage the noise floor in mixed content types, including, for example, voice, music, and noise sounds. At least some embodiments can be adapted for applications such as real-time communication, content creation, and audio capture by user devices.

[0006] According to an exemplary embodiment, a noise management method includes: performing audio event segmentation on an input audio signal to generate content recognition segmentation information; estimating a noise floor level associated with the input audio signal based on sorting frequency bin data corresponding to a fixed-length portion of the input audio signal further based on the content recognition segmentation information; applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having a selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level; <正确的文本应该是0000096,你提供的文本有误,我按照正确文本翻译>A method is provided that includes the above.

[0007] In some embodiments of the above method, the audio event segmentation is performed in real time.

[0008] In some embodiments of any of the above methods, the performing includes processing frame-level speech, music, and noise reliability to identify a continuous sequence of audio events corresponding to the input audio signal, the audio events being selected from a plurality of event clusters, and each of the plurality of event clusters being associated with one or more classes selected from a group consisting of a vocal class, a music class, and a noise class.

[0009] In some embodiments of any of the above methods, the performing includes a moving average convergence / divergence process of frame-by-frame speech, music, and noise reliability.

[0010] In some embodiments of any of the above methods, the estimating includes updating the fixed-length portion using a first-in-first-out buffer configured to receive the frequency bin data.

[0011] In some embodiments of any of the methods described above, the estimation includes truncating the sorted frequency bin data to a size determined using an adaptive percentile estimator based on the content recognition segmentation information.

[0012] In some embodiments of any of the methods described above, the adaptive percentile estimator is configured to select different percentiles for speech events, music events, and noise events, and the determined size is based on one of the selected different percentiles.

[0013] In some embodiment of any of the above methods, the application includes calculating a suppression gain value based on the calculated signal-to-noise ratio value for each bin, and further based on the estimated noise floor level.

[0014] In some embodiment of the above method, for each of the frequency bins, the calculation is as follows: The first suppression gain value for each bin is calculated based on each of the signal-to-noise ratio values ​​for each bin, Each second suppression gain value is calculated based on each of the first suppression gain values ​​and further based on each of the estimated noise floor levels. This includes calculating the product of each of the first suppression gain values ​​and each of the second suppression gain values.

[0015] In some embodiments of any of the above methods, the application includes applying different aggressiveness settings of the noise suppression to different types of audio content based on the content recognition segmentation information.

[0016] In some embodiment of the above method, the method involves using a Fourier transform to convert the input audio signal into frequency bin data, The method further includes applying an inverse Fourier transform to the frequency bins after applying the noise suppression in order to generate an output audio signal.

[0017] According to another exemplary embodiment, a non-temporary computer-readable medium is provided that, when executed by an electronic processor, stores instructions causing the electronic processor to perform an operation including one of the methods described above.

[0018] According to yet another exemplary embodiment, a noise control device, At least one processor, At least one memory location containing program code, The at least one memory and the program code are provided to the device by the at least one processor, Performing audio event segmentation on the input audio signal to generate content recognition segmentation information, Based on the content recognition segmentation information, and further based on sorting the frequency bin data corresponding to the fixed-length portion of the input audio signal, the noise floor level associated with the input audio signal is estimated. The process involves applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level. A noise control device is provided that is configured to perform the following actions.

[0019] Various aspects of this disclosure relate to audio signal processing and provide improvements in at least the technical fields of audio processing, audio event classification and segmentation, noise estimation, and noise suppression.

[0020] Some embodiments disclosed in this specification may be described in general terms as "technical," and the term "technical" may refer to systems, apparatus, methods, computer-readable instructions, modules, components, hardware logic, and / or operations, as suggested in the context presented below.

[0021] Features and technical benefits other than those explicitly stated above will become apparent upon reading the following detailed description and related drawings. This summary is provided to introduce the selection of technology in a simplified form and is not intended to identify key or essential features of the claimed subject matter as defined by the attached claims. [Brief explanation of the drawing]

[0022] These and other more detailed and specific features of various embodiments are further fully disclosed in the following description and with reference to the attached drawings.

[0023] [Figure 1] This block diagram shows an exemplary content recognition noise management system in which various aspects of this disclosure may be implemented.

[0024] [Figure 2] A block diagram showing an exemplary circuit implementation of the content recognition noise management system of Figure 1, according to several aspects of this disclosure.

[0025] [Figure 3] This block diagram shows the workflow performed in the audio event classification and segmentation component of the content recognition noise management system shown in Figure 1, according to several aspects of this disclosure.

[0026] [Figure 4] This is a schematic diagram illustrating the audio event cluster used in the workflow of Figure 3 according to several aspects of this disclosure.

[0027] [Figure 5] This flowchart shows noise management methods according to several aspects of the present disclosure.

[0028] [Figure 6A] A schematic block diagram of an exemplary device architecture that may be used to implement various aspects of this disclosure is shown.

[0029] [Figure 6B] Figure 6A shows a schematic block diagram of an exemplary CPU implemented in the device architecture which can be used to carry out various aspects of the present invention. [Modes for carrying out the invention]

[0030] The following description includes numerous details, such as audio device configuration, timing, and operation, to provide an understanding of one or more aspects of the present disclosure. It will be immediately apparent to those skilled in the art that these specific details are merely examples and are not intended to limit the scope of the application.

[0031] When used in this specification, the term “include” and its variations shall be interpreted as a broad term meaning “include, but not limited to.” The term “or” shall be interpreted as “and / or” unless explicitly indicated in the context. Such terms should be interpreted as having an inclusive meaning. For example, “A and B” may mean at least: “both A and B” or “at least both A and B.” For example, “A or B” may mean at least: “at least A” or “at least B” or “both A and B” or “at least both A and B.” For example, “A and / or B” may mean at least: “A and B” or “A or B.” Particular care should be taken when an exclusive OR is intended (e.g., “either A or B” or “at most one of A and B”). The term “based on” shall be interpreted as “at least partially based.” The terms “one exemplary implementation” and “exemplary implementation” should be interpreted as “at least one exemplary implementation.” The term “another implementation” should be interpreted as “at least one other implementation.” The terms “determined,” “determine,” or “to determine” should be interpreted as “to obtain,” “to receive,” “to calculate,” “to calculate,” “to estimate,” “to predict,” or “to derive.” Furthermore, in the following descriptions and claims, unless otherwise noted, all technical and scientific terms used in this specification have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs.

[0032] Throughout this disclosure and in the relevant claims and / or drawings, various acronyms may appear, which are listed below. Other commonly used acronyms and technical terms may be omitted from this list for brevity. A short list of acronyms is provided below for the reader's easy reference. MACD (Moving Average Convergence and Divergence): Moving average convergence and divergence. FIFO (First In First Out): First-in, first-out SNR (Signal-to-Noise Ratio): Signal-to-noise ratio FFT (Fast Fourier Transform): Fast Fourier Transform IFFT (Inverse FFT): Inverse FFT

[0033] Noise suppression is an audio processing technique that removes background noise from captured signals. Millions of internet-connected devices are used in an increasingly diverse range of everyday scenarios involving audio capture, such as phone calls, capturing / creating user content, virtual music lessons, and live streaming of new or evolving forms of media content (e.g., podcasts, audiobooks, augmented reality content, virtual worlds, etc.). In these scenarios, it is desirable that noise management systems implementing noise suppression not only provide clear speech but also contribute to richer sound experiences such as high-fidelity music, virtual meetings, live streaming, and user-generated content.

[0034] Speech-centric noise management algorithms designed specifically for speech are disadvantageous when applied to music or other audio signals that contain useful non-speech components, as they may result in over-suppression. Therefore, the exemplary embodiments disclosed in this specification provide content-aware audio noise management systems and methods that can be usefully used, for example, to manage the noise floor in mixed content types. At least some embodiments can be adapted to applications such as real-time communication, content creation, and audio capture by user equipment.

[0035] Figure 1 is a block diagram showing an exemplary content-aware noise management system 100 in which various aspects of the present disclosure may be implemented. The content-aware noise management system 100 includes an audio event classification and segmentation component 130. The audio event classification and segmentation component 130 receives an input audio signal 102 and processes the input audio signal 102 to generate content-aware segmentation information. The content-aware noise management system 100 also includes a content-aware noise estimation component 110 and a content-aware noise suppression component 120. The audio event classification and segmentation component 130 is connected to the content-aware noise estimation component 110 and the content-aware noise suppression component 120 via an information path 128, through which content-aware segmentation information is provided to them. The content-aware noise estimation component 110 and the content-aware noise suppression component 120 are also connected to each other via an intermediate path 112. In some examples, the audio event classification and segmentation component 130 generates content-aware segmentation information in real time.

[0036] As used in this specification, the term “real-time” refers to a computer-based process that controls or monitors a corresponding environment by receiving data, processing the received data, and generating a response fast enough to affect or characterize the environment without significant delay. In the context of control or processing software, a real-time response is often understood to be on the order of milliseconds, or sometimes microseconds.

[0037] The content recognition noise estimation component 110 receives the input audio signal 102, processes the received input audio signal 102 based on the content recognition segmentation information to generate an intermediate output signal, which is then provided to the content recognition noise suppression component 120 via path 112. The content recognition noise suppression component 120 receives the intermediate output signal and processes the received intermediate output signal based on the content recognition segmentation information to generate an output audio signal 122. In some examples, the processing implemented in the content recognition noise estimation component 110 and the content recognition noise suppression component 120 is configured to generate an output audio signal 122, and noise suppression is performed in a frequency subband (bin) with a selected frequency resolution and is at least partially based on the content recognition segmentation information. The estimated noise data is provided in the intermediate output signal transmitted via path 112.

[0038] Figure 2 is a block diagram showing exemplary circuit implementations of the content-aware noise management system 100 according to several aspects of the present disclosure. In the illustrated example, the content-aware noise management system 100 includes a fast Fourier transform (FFT) module 202. The FFT module 202 is a shared module for the content-aware noise estimation component 110 and the content-aware noise suppression component 120. In other implementations, the FFT module 202 may be replaced by another suitable time-domain-frequency-domain converter, such as a filter bank. In the illustrated example, the FFT module 202 is connected via an input bus 204 to the downstream modules of the content-aware noise estimation component 110 and the audio event classification and segmentation component 130.

[0039] During operation, the FFT module 202 receives an input audio signal 102, converts the received input audio signal 102 into a corresponding frequency-domain signal, and transmits this signal downstream via bus 204. In some examples, the frequency-domain signal is represented by multiple frequency bins. The exemplary transformation operation performed in the FFT module 202 includes (i) partitioning the audio signal waveform x[t] provided via the input signal 102 into windowed semi-overlapping frames, and (ii) applying a Fourier transform to each frame to generate a spectral block for each bin. The amplitude of the contents of the spectral block is collectively denoted as X[t,k], where k represents the frequency index and t represents the time index. The quantity X[t,k] for a fixed time index t is called the frame of the frequency bin corresponding to time t.

[0040] In the illustrated example, the audio event classification and segmentation component 130 also includes a multi-classification module 240 and an energy module 244, both of which receive frequency bins via bus 204. The audio event classification and segmentation component 130 further includes an audio event segmentation module 248. The multi-classification module 240 and the energy module 244 are connected to the audio event segmentation module 248 via a first path 242 and a second path 246, respectively. During operation, the multi-classification module 240 processes frequency domain signals (on bus 204) to generate a corresponding stream of classification information, which is supplied to the audio event segmentation module 248 via the first path 242. The energy module 244 processes frequency domain signals (on bus 204) to generate a corresponding stream of energy information, which is supplied to the audio event segmentation module 248 via the second path 246. The audio event segmentation module 248 processes the stream of classification information received via the first pass 242 and the stream of energy information received via the second pass 246 to generate the content recognition segmentation information described above.

[0041] In one example, the frequency bins corresponding to the current audio frame and adjacent historical audio frame sequences are processed in the multi-classification module 240 to determine the speech confidence C of the current frame. S [t], Music reliability C M [t], and noise reliability C N The corresponding classification information including [t] is generated as follows:

number

[0042] In the illustrated example, the content recognition noise estimation component 110 also includes a first-in-first-out (FIFO) buffer 212, a sorting module 216, and a percentile estimator 220 connected in series. The FIFO buffer 212 includes, in addition to the current frame of the bin, the X of the bin TbufIt has a fixed size selected to contain 100 history frames. During operation, the FIFO buffer 212 receives frequency domain signals (on bus 204) and continuously refreshes the frequency bin data stored therein by replacing the oldest frame in a bin with the most recent frame bin provided via the frequency domain signals (on bus 204). The FIFO buffer 212 is connected to the sort module 216 via a third pass 214. The sort module 216 is further connected to the percentile estimator 220 via a fourth pass 218. The percentile estimator 220 further receives the content recognition segmentation information described above from the audio event classification and segmentation component 130.

[0043] In one example, the sort module 216 uses the sort function f to access the current contents of the FIFO buffer 212, which is made available via the third pass 214. sort Apply this. The formula for this sorting operation is as follows:

number

[0044] In the noise segment, the energy in each frequency bin represents the noise level, but the minimum and maximum levels can vary significantly. The minimum level typically underestimates the noise floor and is quite sensitive to outliers. The maximum level can represent transient impulses and therefore can lead to an overestimation of the noise floor. Furthermore, the maximum level may also be susceptible to outlier influences. Therefore, a percentile of a sorted FIFO buffer can typically better represent the noise floor in the noise segment. For example, in the speech segment, the FIFO buffer is not permanently occupied by speech, for example, due to the silence in sentences that are inherently present in any speech. Therefore, for some time (e.g., quantified as a percentage), the energy at each bin frequency is on the noise floor level in the speech segment. In the music segment, since harmonics of certain instruments such as the didgeridoo can persist for relatively long periods, a single energy in each frequency bin is not necessarily at the noise floor level. In such cases, even the minimum level can be an overestimation of the noise floor. Therefore, in some cases, noise estimation can be stopped in the music segment to avoid excessive suppression of subsequent useful sounds.

[0045] Different content types typically have different silence ratios. Therefore, in some examples, the percentile estimator 220 is configured to adaptively estimate the noise floor percentile p[t] of a sorted FIFO buffer based on segment classifications such as speech, music, or noise. The corresponding mathematical representation of the percentile estimation operation is as follows:

number

[0046] The percentile estimator 220 uses the sorted FIFO buffer, with a time buffer length T. buf It is further configured to truncate to a duration index[t] size smaller than that. The truncation operation can be expressed mathematically as follows:

number

number

[0047] The content recognition noise estimation component 110 further includes a noise floor update module 224 and a noise update controller 228. The noise floor update module 224 is connected to a percentile estimator 220 via a fifth path 222 and also to the noise update controller 228 via a control path 226. The noise update controller 228 also receives content recognition segmentation information. During operation, the noise floor update module 224 performs a recursive noise floor update based on the noise floor estimate calculated by the percentile estimator 220 and received from there via the fifth path 222. In one example, the recursive noise floor update is performed according to equation (7):

number

[0048] In one example, the noise update controller 228 calculates the dynamic smoothing coefficient as follows:

number

number

number

[0049] The content recognition noise suppression component 120 includes an SNR calculation module 260, a first suppression gain calculation module 264, a second suppression gain calculation module 270, an aggressiveness controller 274, a multiplication module 278, a gain application module 282, and an inverse FFT (IFFT) module 290. The input signal received by the content recognition noise suppression component 120 includes the intermediate output signal and content recognition segmentation information described above. The first signal component of the intermediate output signal is applied to the SNR calculation module 260 and the gain application module 282. The second signal component of the intermediate output signal is applied to the SNR calculation module 260 and the second suppression gain calculation module 270. The output signal generated by the IFFT module 290 is the output audio signal 122.

[0050] The SNR calculation module 260 is connected to the first suppression gain calculation module 264 via the sixth pass 262. The first suppression gain calculation module 264 is further connected to the second suppression gain calculation module 270 and the multiplication unit 278 via the seventh pass 266. The second suppression gain calculation module 270 is further connected to the multiplication module 278 via the eighth pass 272. The aggressiveness controller 274 is connected to the multiplication module 278 via the ninth pass 276. The multiplication module 278 is further connected to the gain application module 282 via the tenth pass 280. The gain application module 282 is further connected to the IFFT module 290 via the eleventh pass 284.

[0051] In one example, the SNR calculation module 260 is configured to calculate the SNR value based on the following formula:

number

[0052] The first suppression gain calculation module 264 calculates the first suppression gain G based on the following formula. sup It is configured to compute [t,k].

number

[0053] The second suppression gain calculation module 270 calculates the residual signal X[t,k]G based on the following formula. sup It is configured to remove residual noise from [t,k].

number

[0054] The aggression controller 274 is configured to apply different aggression settings to different types of content in order to mitigate the harmful effects of potential over-suppression of relevant content. For example, the aggression controller 274 is configured to apply different aggression settings to different types of content. ca [t,k] is derived as follows:

number

[0055] The multiplication module 278 is configured to calculate the final noise suppression gain G[t,k] by multiplying the respective gain values ​​received through paths 266, 272, and 276, as follows:

number

number

[0056] Figure 3 is a block diagram showing a workflow 300 performed in the audio event classification and segmentation component 130 of the content recognition noise management system 100 according to several aspects of the present disclosure. The input 302 to the workflow 300 includes the input audio signal 102 and / or a frequency domain signal (on bus 204) as described above. The output 332 of the workflow 300 includes the content recognition segmentation information as described above.

[0057] Workflow 300 includes an audio analysis and classification block 310, a moving average convergence / divergence (MACD) block 320, and an audio event segmentation block 330. In the audio analysis and classification block 310, the audio signal of each frame of input 302 is analyzed to obtain energy and / or classified to generate raw confidence. In the MACD block 320, the energy or confidence is post-processed using an appropriate MACD algorithm. In the audio event segmentation block 330, the audio events of input 302 are segmented based on the real-time MACD processing results to generate output 332.

[0058] The operation of MACD block 320 involves calculating the difference between the short-term and long-term smoothing of a given value in order to predict trend momentum. In one example, the MACD processing implemented in MACD block 320 recursively generates four outputs based on the value v[t] as follows:

number

[0059] In some examples, an audio signal waveform x[t] is first partitioned into windowed semi-overlapping frames, and then the frame data is converted to the frequency domain using either a filter bank or a time-frequency converter such as the FFT module 202. As previously described, the amplitudes of the contents of the spectral blocks are collectively represented as X[t,k], where k represents the frequency index and t represents the time index.

[0060] In some examples of workflow 300, energy module 244 is used to calculate the total energy (formatted in dB) for each frame according to the following formula:

number

[0061] Based on the above considerations, E is performed in MACD block 320. dB The MACD processing of [t] can be expressed as follows:

number

[0062] In some examples of workflow 300, the operation of the audio event segmentation block 330 includes segmenting common audio events based on the following conditions:

number

[0063] Figure 4 is a schematic diagram illustrating audio event clusters used in workflow 300 according to several aspects of the present disclosure. In the illustrated example, various audio events are assigned to multiple clusters distributed across several audio classes, including vocal class 410, music class 420, and noise class 430. Vocal class 410 includes non-speech cluster 412 and speech cluster 414. Music class 420 includes vocal cluster 422, instrument cluster 424, and mixed cluster 426. Noise class 430 includes selective noise cluster 432, transient noise cluster 434, and “other noise” cluster 436. Several exemplary members of clusters are listed as examples in Figure 4. Note that some clusters may span multiple classes. For example, vocal cluster 422 spans both vocal class 410 and music class 420. Other appropriate clustering schemes may also be used in other examples.

[0064] In practice, due to the aforementioned overlap between clusters and classes, it is difficult to identify the cluster to which an audio signal belongs with 100% accuracy. In the case of live communication systems (such as systems that carry audio or video calls, live streaming, etc.), static room background noise, equipment fan noise, and electrical equipment noise are noise types that can be identified and suppressed with relatively high reliability. However, with regard to transient noise, it is usually more difficult to clearly determine the user's preference, whether to suppress or share it. Therefore, when designing a noise classifier, a subset smaller than the entire noise class 430 can be selected as training data for noise classification. In this way, any overlap between noise and speech or music can be reduced, resulting in a more robust selective noise classifier.

[0065] In some examples of framework 300, the current frame and adjacent historical frame sequences are supplied to the multi-classification module 240, and the speech confidence C is calculated as described above with reference to Figure 2 and equation (1).S [t], Music reliability C M [t], and noise reliability C N This generates a raw confidence score like [t]. In such an example, the raw speech confidence score C is implemented in MACD block 320. S The MACD processing for [t] can be formatted as follows:

number

number

[0066] In some examples, the audio event segmentation block 330 is configured to perform segmentation of musical and noise events using a similar method, as illustrated by equations (23) to (24), for example. In such examples, the smoothing coefficient α short , α long , α diff , and threshold th mma_onset , th mma_offset , th short_onset , th short_offset The selection of values ​​typically differs for each of the 410, 420, and 430 classes in order to appropriately balance the sensitivity and latency aspects of the framework 300.

[0067] Figure 5 is a flowchart illustrating exemplary noise management methods 500 according to several aspects of the present disclosure. Method 500 may be implemented using a content recognition noise management system 100. Method 500 may be implemented by a processor which may be configured to implement Method 500 via machine-executable instructions. Method 500 may be divided into various blocks or partitions, such as blocks 502, 504, 506, 508, and 510. The various processing blocks shown in Figure 5 provide examples of the various methods disclosed in this specification, and it is understood that some blocks may be removed, added, combined, or modified without departing from the spirit of the present disclosure. In some examples, the processing of various blocks, which may be described as processes, methods, steps, blocks, operations, or functions, may begin in block 502.

[0068] In block 502, "Converting an input audio signal to frequency bin data," method 500 includes converting the input audio signal 102 to frequency bin data. In some examples, the operation of block 502 includes performing a Fourier transform, which is implemented, for example, using the FFT module 202. The process can proceed from block 502 to block 504.

[0069] In block 504, “Performing audio event segmentation on the converted input audio signal,” method 500 also includes performing audio event segmentation on the converted input audio signal. The operation of block 504 includes generating content-aware segmentation information (on path 128) corresponding to the input audio signal 102. In some examples, the audio event segmentation is performed in real time. In some examples, the content-aware segmentation information (on path 128) includes frame-by-frame speech, music, and noise confidence. The operation of block 504 also includes processing frame-by-frame speech, music, and noise confidence to identify a sequence of audio events corresponding to the input audio signal 102. Audio events are selected from multiple event clusters, each associated with one or more classes selected from a group consisting of vocal class 410, music class 420, and noise class 430. In some examples, the processing operation is configured to mitigate the occurrence of incorrect selections in the identified sequence of events. In some examples, the operation of block 504 also includes moving average convergence and divergence processing 320 for frame-by-frame speech, music, and noise confidence.

[0070] In block 506, “Estimating the Noise Floor Level Based on Segmentation Information,” method 500 also includes estimating the noise floor level. The noise floor level (on path 230) is associated with the input audio signal 102 and is calculated based on content-aware segmentation information (on path 128) and further based on sorting the frequency bin data (on bus 204) corresponding to the fixed-length portion of the input audio signal 102. The operation of block 506 includes updating the fixed-length portion using a FIFO buffer 212 configured to receive the frequency bin data (on bus 204). The operation of block 506 also includes truncating the sorted frequency bin data 218 to a size determined using an adaptive percentile estimator 220 based on the content-aware segmentation information (on path 128). In some examples, the adaptive percentile estimator 220 is configured to select different percentiles for speech events, music events, and noise events, and the determined size is based on one of the selected percentiles. The operation of block 506 also includes updating the noise floor level (on path 230) using the update controller 224. In some examples, the update controller 224 is configured to start and stop updates based on audio event data included in the content recognition segmentation information (on path 128).

[0071] In block 508, “Applying Noise Suppression to Frequency Bin Data,” method 500 also includes applying noise suppression to the input signal 102 represented by frequency bin data. In some examples, noise suppression is performed on frequency bins having a selected frequency resolution and is at least partially based on content recognition segmentation information (on path 128) and the estimated noise floor level (on path 230). The operation of block 508 includes calculating a per-bin SNR value based on the amplitude of the frequency bins and further based on the estimated noise floor level (on path 230). The operation of block 508 also includes calculating a suppression gain value based on the calculated per-bin SNR value and further based on the estimated noise floor level (on path 230). In some examples, such calculations include (i) calculating each first suppression gain value based on each of the signal-to-noise ratio values ​​for each bin, (ii) calculating each second suppression gain value based on each first suppression gain value and further based on each of the estimated noise floor levels, and (iii) calculating the product of each first suppression gain value and each second suppression gain value. The operation of block 508 also includes applying different aggressive settings of noise suppression to different types of audio content based on content-aware segmentation information (on path 128).

[0072] In some examples, at least some of the operations of block 506 are performed using frequency bins having a first-choice frequency resolution. In contrast, at least some of the operations of block 508 are performed using frequency bins having a second-choice frequency resolution different from the first-choice frequency resolution. The second-choice frequency resolution may be higher than the first-choice frequency resolution. In such examples, the operations of block 508 may also include converting the estimated noise floor level (on path 230) from the first-choice frequency resolution to the second-choice frequency resolution.

[0073] In block 510, “Applying an inverse Fourier transform to noise-suppressed frequency bins to generate an output audio signal,” method 500 also includes applying an inverse Fourier transform to the output frequency bins. The operation of block 510 also includes concatenating the time-domain signal portions generated via the inverse Fourier transform to generate an output audio signal 122.

[0074] Figure 6A shows a schematic block diagram of an exemplary device architecture 600 (e.g., device 600) that may be used to implement various aspects of the present disclosure. The architecture 600 includes, but is not limited to, server and client devices, systems, and methods, as described with reference to Figures 1-5. As shown, the architecture 600 includes a central processing unit (CPU) 601 that can perform various operations according to a program stored, for example, in read-only memory (ROM) 602 or a program loaded, for example, from a storage unit 608 into random access memory (RAM) 603. The CPU 601 may be, for example, an electronic processor 601. The RAM 603 also stores data necessary for the CPU 601 to perform various operations, as needed. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0075] The following components are connected to the I / O interface 605: an input unit 606 which may include a keyboard, mouse, etc.; an output unit 607 which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 608 which includes a hard disk or another suitable storage device; and a communication unit 609 which includes a network interface card such as a network card (e.g., wired or wireless).

[0076] In some implementations, the input unit 606 includes one or more microphones located at different positions (depending on the host device) that enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, or other appropriate formats).

[0077] In some implementations, the output unit 607 includes a system with varying numbers of speakers. The output unit 607 can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, or other appropriate formats) depending on the capabilities of the host device.

[0078] In some embodiments, the communication unit 609 is configured to communicate with other devices (e.g., via a network). The drive 610 is also connected to the I / O interface 605, if necessary. A removable medium 611, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or another suitable removable medium, is mounted on the drive 610, and as a result, computer programs read from it are installed in the storage unit 608, if necessary. Those skilled in the art will understand that although the device 600 is described as including the components described above, it is possible to add, remove, and / or replace some of these components in various applications, and that all such changes or variations are included within the scope of this disclosure.

[0079] According to exemplary embodiments of the present disclosure, the processing described above may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product which includes a computer program tangibly embodied on a machine-readable medium. The computer program includes program code for performing the method. In such embodiments, the computer program may be downloaded and implemented from a network via a communication unit 609 and / or installed from a removable medium 611, as shown in Figure 6A.

[0080] Figure 6B shows a schematic block diagram of a CPU 601 implemented in the architecture 600 of Figure 6A, as an example. The CPU 601 includes an electronic processor 620 and a memory 621. The electronic processor 620 is electrically and / or communicatively connected to the memory 621 for bidirectional communication. The memory 621 stores software 622. In some examples, the memory 621 may be located inside the electronic processor 620, such as an internal cache memory or other internally located ROM, RAM, or flash memory. In other examples, the memory 621 may be located outside the electronic processor 620, such as in ROM 602, RAM 603, flash memory, or removable media 611, or another non-temporary computer-readable medium intended for architecture 600. In some cases, the electronic processor 620 executes the software 622 stored in the memory 621 to perform, among other things, various methods and operations related to content recognition noise management as described above with reference to Figures 1 to 5.

[0081] Those skilled in the art will readily recognize that the present invention is by no means limited to the exemplary embodiments described above. Rather, many modifications and variations are possible and should be considered within the scope of the appended claims. Various aspects and implementations of this disclosure can also be understood from the enumerated example embodiments (EEEs) listed below. These may represent systems, methods, and apparatus arranged according to various aspects of this disclosure, rather than being claims themselves.

[0082] (EEE1) A noise management method, Performing audio event segmentation on the input audio signal to generate content recognition segmentation information, Based on the content recognition segmentation information, and further based on sorting the frequency bin data corresponding to the fixed-length portion of the input audio signal, the noise floor level associated with the input audio signal is estimated. The process involves applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level. A method that includes this.

[0083] (EEE2) The aforementioned audio event segmentation is performed in real time according to the method described in EEE1.

[0084] (EEE3) The content recognition segmentation information is the method according to EEE1 or 2, including frame-by-frame speech, music, and noise confidence.

[0085] (EEE4) The method according to EEE3, wherein the execution described above includes processing the frame-by-frame speech, music, and noise confidence to identify a sequence of audio events corresponding to the input audio signal, the audio events being selected from a plurality of event clusters, each of which is associated with one or more classes selected from a group consisting of vocal classes, music classes, and noise classes.

[0086] (EEE5) The method according to EEE4, wherein the processing is configured to mitigate the occurrence of incorrect selections in the identified sequence.

[0087] (EEE6) The method described above, which includes a moving average convergence and divergence process for the frame-by-frame speech, music, and noise confidence levels, is described above.

[0088] (EEE7) The method according to any one of EEE1 to 6, wherein the estimation includes updating the fixed-length portion using a first-in, first-out buffer configured to receive the frequency bin data.

[0089] (EEE8) The method according to any one of EEE1 to 6, wherein the estimation includes truncating the sorted frequency bin data to a size determined using an adaptive percentile estimator based on the content recognition segmentation information.

[0090] (EEE9) The adaptive percentile estimator is configured to select different percentiles for speech events, music events, and noise events, and the determined size is based on one of the selected different percentiles, according to the method of EEE8.

[0091] (EEE10) The method according to any one of EEE1 to 6, wherein the estimation is performed by updating the noise floor level using an update controller, the update controller being configured to start and stop the update based on audio event data included in the content recognition segmentation information.

[0092] (EEE11) The method according to any one of EEE1 to 10, wherein the application includes calculating a signal-to-noise ratio value for each bin based on the amplitude of the frequency bin and the estimated noise floor level.

[0093] (EEE12) The method according to EEE11, further comprising calculating a suppression gain value based on the calculated signal-to-noise ratio value for each bin and the estimated noise floor level.

[0094] (EEE13) For each of the aforementioned frequency bins, the calculation is as follows: The first suppression gain value for each bin is calculated based on each of the signal-to-noise ratio values ​​for each bin, Each second suppression gain value is calculated based on each of the first suppression gain values ​​and further based on each of the estimated noise floor levels. The method according to EEE12, comprising calculating the product of each of the first suppression gain values ​​and each of the second suppression gain values.

[0095] (EEE14) The method according to any one of EEE1 to 13, wherein the application includes applying different aggressiveness settings of the noise suppression to different types of audio content based on the content recognition segmentation information.

[0096] (EEE15) The estimation described above is performed using a frequency bin having a first-choice frequency resolution. The above application is performed using frequency bins having a second selected frequency resolution different from the first selected frequency resolution. The method described in any one of the EEE1-14 items.

[0097] (EEE16) The method described in EEE15, wherein the second select frequency resolution is higher than the first select frequency resolution.

[0098] (EEE17) The method according to EEE16, wherein the estimation further comprises converting the estimated noise floor level from the first selected frequency resolution to the second selected frequency resolution.

[0099] (EEE18) Converting the input audio signal into frequency bin data using a Fourier transform, To generate the output audio signal, an inverse Fourier transform is applied to the frequency bins after applying the noise suppression, The method described in any one of the EEE1 to 17, further including the method described in any one of the EEE1 to 17.

[0100] (EEE19) A non-temporary computer-readable medium storing instructions, wherein, when executed by an electronic processor, the instructions cause the electronic processor to perform an operation including the method described in any one of EEE1 to EEE18.

[0101] (EEE20) It is a noise management device, At least one processor, At least one memory location containing program code, The at least one memory and the program code are provided to the device by the at least one processor, Performing audio event segmentation on the input audio signal to generate content recognition segmentation information, Based on the content recognition segmentation information, and further based on sorting the frequency bin data corresponding to the fixed-length portion of the input audio signal, the noise floor level associated with the input audio signal is estimated. The process involves applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level. A noise control device configured to perform the following action.

[0102] While processes, systems, methods, heuristics, etc., are described in this specification, it should be understood that such steps, etc., are described as occurring in a specific ordered sequence, but such processes may be performed in conjunction with described steps that are performed in an order different from that described in this specification. It should be further understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described in this specification may be omitted. In other words, the descriptions of processes in this specification are provided for the purpose of illustrating specific embodiments and should not be considered as limiting the claims.

[0103] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Reading the above description will reveal many embodiments and applications beyond those provided. The scope should be determined without reference to the above description, but instead by reference to the attached claims, along with the entire equivalent scope of the claims granted. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that this application is subject to modification and alteration.

[0104] All terms used in the claims are intended to give their broadest, most reasonable form and ordinary meaning so that they may be understood by those familiar with the art described herein. In particular, the use of singular articles such as “a,” “the,” and “said” should be read to refer to one or more of the elements shown, unless the claim expressly states the opposite limitation.

[0105] This summary of the disclosure is provided to enable readers to quickly assess the characteristics of the technical disclosure. It is understood that it is not to be used to interpret or limit the scope or meaning of the claims. Furthermore, it is found that in the prior detailed description, various features are grouped together into various embodiments for the purpose of streamlining the disclosure. This method of the disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than those expressly described in each claim. Rather, as reflected in the following claims, the subject matter of the invention lies in fewer features than all of a single disclosed embodiment combined. Accordingly, the following claims are incorporated herein into the detailed description, and each claim stands alone as separately claimed subject matter.

[0106] This disclosure includes references to exemplary embodiments, but this specification is not intended to be constrained. Various modifications of the embodiments described, and other embodiments within the scope of the disclosure that are apparent to those skilled in the art to whom the disclosure relates, are deemed to be within the principles and scope of the disclosure, as set forth, for example, in the following claims.

[0107] Some embodiments may be implemented as circuit-based processing, including possible implementations on a single integrated circuit.

[0108] Some embodiments may be implemented in the form of methods and apparatus for carrying out these methods. Some embodiments may also be implemented in the form of program code recorded on a tangible medium such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy diskette, a CD-ROM, a hard drive, or any other non-temporary machine-readable storage medium, where the program code is loaded into and executed by a machine such as a computer, which then becomes an apparatus for carrying out the patented invention. Some embodiments may also be implemented, for example, in the form of program code stored in a non-temporary machine-readable storage medium that is loaded into and / or executed by a machine, where the program code is loaded into and executed by a machine such as a computer or processor, which then becomes an apparatus for carrying out the patented invention. When implemented on a general-purpose processor, the program code segment is combined with the processor to provide a unique device that operates similarly to a particular logic circuit.

[0109] Unless otherwise explicitly stated, each number and range should be interpreted as an approximation, as if preceded by the words "approximately" or "about."

[0110] The use of figure numbers and / or figure reference labels in a claim is intended to identify one or more possible embodiments of the subject matter to be claimed, in order to facilitate the interpretation of the claim. Such use is not necessarily construed as limiting the scope of these claims to the embodiments shown in the corresponding figures.

[0111] Where present, the elements in the claims of the following methods are described in a specific order by corresponding labels; however, unless the reference to the claim specifically indicates a particular order for the implementation of some or all of these elements, these elements are not necessarily intended to be limited to being implemented in that particular order.

[0112] In this specification, any reference to “one embodiment” or “embodiment” means that certain features, structures, or characteristics described in relation to an embodiment may be included in at least one embodiment of this disclosure. The occurrence of the phrase “in one embodiment” in various places in this specification does not necessarily all refer to the same embodiment, nor does a separate or alternative embodiment necessarily exclude other embodiments. The same applies to the term “implementation.”

[0113] Unless otherwise provided in this specification, the use of ordinal adjectives such as “first,” “second,” and “third” to refer to multiple similar objects simply indicates that different instances of such similar objects are being referred to, and does not imply that the similar objects referred to in this manner must exist in a corresponding order or sequence in time, space, rank, or otherwise.

[0114] Unless otherwise provided in this specification, the conjunction "if," in addition to its simple meaning, may be interpreted, depending on the corresponding specific context, as meaning "when," "upon," "in response to determining," or "in response to detecting." For example, the phrases "if it is determined" or "if [a stated condition] is detected" may be interpreted as meaning "upon determining," "in response to determining [the stated condition or event]," or "in response to determining [the stated condition or event]."

[0115] Furthermore, for the purposes of this specification, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any method known or subsequently developed in the art in which energy can be transferred between two or more elements, and the intervention of one or more additional elements is intended, though not essential. Conversely, terms such as “directly coupled” or “directly connected” mean that no such additional elements exist.

[0116] When used in this specification with respect to elements and standards, the term "interchangeable" means that an element communicates with other elements in a manner fully or partially defined by the standard, and is recognized by other elements as being sufficiently capable of communicating with other elements in a manner specified by the standard. Interchangeable elements are not required to operate internally in a manner defined by the standard.

[0117] The functionality of the various elements shown in the diagram, including any functional blocks labeled as “processor” and / or “controller,” may be provided through the use of dedicated hardware, as well as hardware capable of executing software in conjunction with appropriate software. Where provided by a processor, functionality may be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, the explicit use of the terms “processor” or “controller” should not be interpreted as referring only to hardware capable of executing software, but implicitly including, but not limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM), random-access memory (RAM), and non-volatile storage for storing software. Other conventional and / or custom hardware may also be included. Similarly, all switches shown in the diagram are conceptual. Their functionality may be performed through the operation of programmable logic, dedicated logic, the interaction of programmable control and dedicated logic, or manually. Specific techniques are at the implementer's discretion, as they are better understood from the context.

[0118] As used in this application, the term “circuit” or “circuitry” can mean one or more of the following: (a) a hardware-only circuit implementation (such as an implementation of analog and / or digital circuits only); (b) a combination of hardware circuits and software (where appropriate): (i) a combination of analog and / or digital hardware circuits and software / firmware; (ii) any part of a hardware processor, software, and memory having software (including a digital signal processor) that works together to enable a device such as a mobile phone or server to perform various functions; and (c) a hardware circuit and / or processor, such as a microprocessor or part of a microprocessor, which requires software (such as firmware) to operate but may not exist if software is not required to operate. This definition of circuit applies to all use of the term in this application, including in the claims. As a further example, as used in this application, the term “circuit” also includes a mere hardware circuit or processor (or more processors), or a part of a hardware circuit or processor, and its (or their) accompanying software and / or firmware implementation. The term “circuit” also includes, for example, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device, where applicable to the components of a particular claim.

[0119] Those skilled in the art will understand that the block diagrams in this specification represent conceptual diagrams of exemplary circuits embodying the principles of disclosure. Similarly, flowcharts, flow charts, state transition diagrams, pseudocode, etc., whether or not a computer or processor is explicitly indicated, will be understood to represent various processes performed by a computer or processor, substantially represented in a computer-readable medium.

[0120] The “Summary of the Invention” in this specification is intended to present several exemplary embodiments, and additional embodiments are described with reference to the “Modes for Carrying Out the Invention” and / or one or more drawings. The “Summary of the Invention” is not intended to identify or limit the scope of the claimed subject matter.

Claims

1. A noise management method, Performing audio event segmentation on the input audio signal to generate content recognition segmentation information, Based on the content recognition segmentation information, and further based on sorting the frequency bin data corresponding to the fixed-length portion of the input audio signal, the noise floor level associated with the input audio signal is estimated. The process involves applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level. A method that includes this.

2. The method according to claim 1, wherein the audio event segmentation is performed in real time.

3. The method according to claim 1, wherein the content recognition segmentation information includes frame-by-frame speech, music, and noise confidence.

4. The method according to claim 3, wherein performing the audio event segmentation includes processing the frame-by-frame speech, music, and noise confidence to identify a sequence of audio events corresponding to the input audio signal, the audio events being selected from a plurality of event clusters, each of which is associated with one or more classes selected from a group consisting of vocal classes, music classes, and noise classes.

5. The method according to claim 4, wherein the processing is configured to mitigate the occurrence of incorrect selections in the identified sequence.

6. The method according to claim 3, wherein performing the audio event segmentation includes moving average convergence and divergence processing of the frame-by-frame speech, music, and noise confidence.

7. The method according to claim 1, wherein the estimation includes updating the fixed-length portion using a first-in, first-out buffer configured to receive the frequency bin data.

8. The method according to claim 1, wherein the estimation comprises truncating the sorted frequency bin data to a size determined using an adaptive percentile estimator based on the content recognition segmentation information.

9. The method according to claim 8, wherein the adaptive percentile estimator is configured to select different percentiles for speech events, music events, and noise events, and the determined size is based on one of the selected different percentiles.

10. The method according to claim 1, wherein the estimation is performed by updating the noise floor level using an update controller, the update controller being configured to start and stop the update based on audio event data included in the content recognition segmentation information.

11. The method according to claim 1, wherein the application includes calculating a signal-to-noise ratio value for each bin based on the amplitude of the frequency bin and the estimated noise floor level.

12. The method according to claim 11, wherein the application further comprises calculating a suppression gain value based on the calculated signal-to-noise ratio value for each bin and the estimated noise floor level.

13. For each of the aforementioned frequency bins, the calculation is as follows: The first suppression gain value for each bin is calculated based on one of the signal-to-noise ratio values ​​for each bin, Each second suppression gain value is calculated based on each of the first suppression gain values ​​and further based on each of the estimated noise floor levels. The method according to claim 12, comprising calculating the product of each of the first suppression gain values ​​and each of the second suppression gain values.

14. The method according to claim 1, wherein the application includes applying different aggressiveness settings of the noise suppression to different types of audio content based on the content recognition segmentation information.

15. The estimation described above is performed using a frequency bin having a first-choice frequency resolution. The above application is performed using frequency bins having a second selected frequency resolution different from the first selected frequency resolution. The method according to claim 1.

16. The method according to claim 15, wherein the second select frequency resolution is higher than the first select frequency resolution.

17. The method according to claim 16, wherein the estimation further comprises converting the estimated noise floor level from the first selected frequency resolution to the second selected frequency resolution.

18. Converting the input audio signal into frequency bin data using a Fourier transform, To generate the output audio signal, an inverse Fourier transform is applied to the frequency bins after applying the noise suppression, The method according to claim 1, further comprising:

19. A non-temporary computer-readable medium storing instructions, wherein, when executed by an electronic processor, the instructions cause the electronic processor to perform an operation including the method according to any one of claims 1 to 18.

20. It is a noise management device, At least one processor, At least one memory location containing program code, The at least one memory and the program code are provided to the device by the at least one processor, Performing audio event segmentation on the input audio signal to generate content recognition segmentation information, Based on the content recognition segmentation information, and further based on sorting the frequency bin data corresponding to the fixed-length portion of the input audio signal, the noise floor level associated with the input audio signal is estimated. The process involves applying noise suppression to the input audio signal to generate an output audio signal, wherein the noise suppression is performed in frequency bins having selected frequency resolution and is at least partially based on the content recognition segmentation information and the estimated noise floor level. A noise control device configured to perform the following action.

Images