Enhancing audio content

Abstract

Techniques for enhancing an audio signal are provided herein. In some embodiments, a method for enhancing an audio segment may include receiving an input audio segment to be enhanced. The method may also include generating a feature indicative of an audio content type of the input audio segment. The method may also include generating a feature indicative of the presence of noise in the input audio segment by providing at least the feature indicative of the audio content type to the trained de-noising feature extraction model. The method may also include generating a de-noising mask based on a feature indicating the presence of noise in the input audio segment. The method may also include generating an enhanced audio segment at least by applying a de-noising mask to the input audio segment.

Description

Cross Reference to Related Applications

[0001] This application claims priority to PCT Application No. PCT / CN2023 / 102113 filed June 25, 2023, and U.S. Provisional Application No. 63 / 625,184 filed January 25, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to systems, methods, and media for enhancing audio content. BACKGROUND

[0003] Enhancement of audio content can be performed by an audio content creator, who can capture audio content that includes speech and / or music as well as noise. The noise can include environmental noise (e.g., bird sounds, wind sounds, etc.), background conversation, etc. For example, to enhance speech content or other primary content in an audio signal, it can be desirable to remove such noise. However, removing noise, especially while not degrading the quality of the primary content of the audio signal, can be difficult.

[0004] Symbols and Terminology Throughout this disclosure, including in the claims, the terms “speaker,” “loudspeaker,” and “audio reproduction transducer” are used as synonyms to mean any sound-emitting transducer (or set of transducers). A typical pair of headphones includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) can be subject to different processing in different circuit branches that are coupled to different transducers.

[0005] Throughout this disclosure, including in the claims, the expression performing an operation “on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used in a broad sense to mean performing the operation directly on the signal or data, or performing the operation on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or pre-processing prior to performing the operation thereon).

[0006] Throughout this disclosure, including in the claims, the expression “system” is used in a broad sense to mean a device, a system, or a subsystem. For example, a subsystem that implements a decoder can be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other X-M inputs are received from an external source) can also be referred to as a decoder system.

[0007] Throughout this disclosure, including in the claims, the term "processor" is used broadly to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) for performing operations on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the Invention

[0008] According to some embodiments, this document provides techniques for enhancing audio signals. In some embodiments, a method for enhancing an audio signal may include: receiving an input audio segment to be enhanced. The method may further include: generating features indicating the audio content type of the input audio segment. The method may further include: generating features indicating the presence of noise in the input audio segment by providing at least the features indicating the audio content type to a trained denoising feature extraction model. The method may further include: generating a denoising mask based on the features indicating the presence of noise in the input audio segment. The method may further include: generating an enhanced audio segment by at least applying the denoising mask to the input audio segment.

[0009] In some examples, the method may further include: obtaining initial features associated with the input audio segment by providing the input audio segment to a trained initial feature extraction model, wherein features indicating the audio content type of the input audio segment are generated using the initial features. In some examples, the trained denoising feature extraction model receives the initial features as input.

[0010] In some examples, the features indicating the audio content type are extracted by a trained discriminative feature model. In some examples, the trained discriminative feature model and the trained denoising feature extraction model are trained in a multi-stage training process, which includes: training both the discriminative feature model and the denoising feature extraction model in a first stage; freezing the weights of the denoising feature extraction model in a second stage while fine-tuning the weights of the discriminative feature model in a third stage; and freezing the weights of the discriminative feature model in a third stage while fine-tuning the weights of the denoising feature extraction model in a third stage. In some examples, at least the trained discriminative feature model is trained using a training set comprising audio content samples labeled as including or excluding music content, wherein the labels are automatically generated based on energy rules. In some examples, the discriminative feature model and the denoising feature extraction model are trained using training samples comprising at least noisy speech audio segments and noisy music audio segments, wherein the lowest signal-to-noise ratio (SNR) of the noisy speech audio segments is lower than the lowest SNR of the noisy music audio segments. In some examples, the trained discriminative feature model includes a gated recurrent unit (GRU).

[0011] In some examples, the trained denoising feature extraction model includes a gated recurrent unit (GRU).

[0012] In some examples, the feature indicating the type of audio content includes an indication of whether the input audio segment includes musical content. In some examples, the method further includes smoothing the indication of whether the input audio segment includes musical content across multiple frames of the input audio segment.

[0013] In some examples, the method further includes determining the degree to which the harmonics of the input audio segment are amplified in the enhanced audio segment by providing at least the features indicating the type of the audio content to a trained pitch intensity feature extraction block. In some examples, the degree of harmonic amplification of the input audio segment is inversely proportional to the confidence that the input audio segment includes musical content.

[0014] In some examples, the denoising mask is generated such that the denoising strength is inversely proportional to the confidence that the input audio segment includes musical content.

[0015] In some examples, at least the denoising feature extraction model is trained using a training set that includes labeled ambient sounds.

[0016] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more non-transitory media on which software is stored.

[0017] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be capable of performing at least partially the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0018] Details of one or more embodiments of the subject matter described herein are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0019] Figure 1 This is a simplified diagram illustrating an example system for enhancing audio content based on content type, according to some embodiments; Figure 2 Illustrations are shown according to some embodiments Figure 1 Example implementations of the system shown in the figure; Figure 3 An example implementation of a system for enhancing audio content according to some embodiments is shown, wherein the audio content includes pitch enhancement based on content type; Figure 4 It is a graph that correlates the gain parameter of noise attenuation based on the possibility that the audio content includes music, according to some embodiments; Figure 5 This is another example graph that correlates the gain parameter of noise attenuation based on the possibility that the audio content includes music, according to some embodiments; Figure 6 This is an example of a graph that correlates pitch enhancement parameters based on the possibility that the audio content includes music, according to some embodiments; Figure 7 This is a flowchart illustrating an example process of using a trained model to perform audio content enhancement according to some embodiments; Figure 8 This is a flowchart of an example process for training a model using a multi-stage process according to some embodiments; Figure 9 This is a flowchart illustrating an example process of using a trained model to perform audio content enhancement according to some embodiments; Figure 10 A block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0020] In the various figures, similar reference numerals and names indicate similar elements. Detailed Implementation

[0021] Audio content enhancement can be performed by audio content creators, who can capture audio content that includes speech and / or music, as well as noise. Noise may include ambient noise (e.g., birdsong, wind, etc.), background conversations, etc. For example, to enhance the speech content or other key content of interest in an audio signal, it might be desirable to remove such noise. However, noise removal, especially without degrading the quality of the key content of the audio signal, can be difficult. For example, when noise is attenuated, ambient sounds and music may also be attenuated, which could alter the audio signal in undesirable ways.

[0022] This article discloses techniques for enhancing audio signals. For example, an enhanced audio signal can attenuate noise in a manner that takes into account the type of audio content in the audio signal. As a more specific example, noise may be attenuated more strongly for audio signals that do not contain music or are less likely to contain music. As another example, an enhanced audio signal may have enhanced pitch intensity, sometimes referred to as enhanced harmonics. Harmonics can be enhanced in a manner that depends on the likelihood that the audio signal contains music. For example, harmonics may be enhanced more strongly for audio signals that are unlikely to contain music, while harmonic enhancement may generally be suppressed for audio signals that may contain music.

[0023] Audio signal enhancement based on audio content type can be achieved by generating a denoising mask and / or pitch intensity index based on features of the audio signal indicating the content type. For example, a discriminative feature extraction model can extract features that can be used to determine the likelihood that an audio signal contains music. These features can be provided to a trained model (e.g., a trained denoising feature extraction model) configured to generate features indicating noise, which can then be used to generate a denoising mask. Similarly, these features can be provided to a trained model configured to generate a pitch intensity index indicating the degree of harmonic enhancement. By utilizing features indicating the audio content type to generate a denoising mask and / or pitch intensity index, the enhancement of the audio signal can be dependent on the audio content type, such that the enhancement of an audio signal that may contain music will differ from that of an audio signal that is unlikely to contain music. Furthermore, in some embodiments, the denoising gain and / or pitch intensity gain can be modified based on the likelihood that the audio signal contains music, thereby providing an additional avenue for audio signal enhancement that may depend on the content type. By enhancing audio signals based on content type, specific types of noise (e.g., "annoying noise") can be attenuated or removed, while allowing the preservation of some types of ambient sounds (e.g., birdsong) to provide contextual background. Furthermore, by enhancing audio signals based on content type—for example, whether the audio signal includes music (or the probability that the audio signal includes music)—audio signals containing music can be enhanced in a way that does not typically alter the music.

[0024] Figure 1 An example system for enhancing audio signals according to some embodiments is illustrated. As shown, a shared feature extraction block 102 can be configured to receive an audio signal or a representation of the audio signal (e.g., a frequency domain representation of the audio signal) as input. The shared feature extraction block 102 can be configured to generate or extract features associated with the input audio signal. In some embodiments, the extracted features can be considered as coarse features that can be used by multiple other downstream models or feature extraction blocks.

[0025] The extracted features can be provided to both the discriminative feature extraction block 104 and the denoising feature extraction block 106 simultaneously. The discriminative feature extraction block 104 can be configured to generate or extract features that can be used to determine content type information associated with the input audio signal. For example, these features can be used to determine the probability or confidence that the audio signal includes music. The denoising feature extraction block 106 can be configured to generate or extract features that can be used to generate a denoising mask, which, when applied to the input audio signal (or its frequency domain representation), enhances the audio signal by removing or attenuating noise. Note that, as... Figure 1As shown, the denoising feature extraction block 106 is configured to receive features generated by the discriminative feature extraction block 104 as input. The denoising features can be content-type dependent by considering features that can be used to determine content type information. For example, the denoising features can allow for the generation of different denoising masks for audio signals that include music than for audio signals that do not include music; for example, weaker attenuation of noise in an audio signal that includes music compared to an audio signal that does not include music.

[0026] like Figure 1 As illustrated, the features generated by the discriminative feature extraction block 104 can be provided to a classification layer 108, which can be configured to generate a music flag. The music flag can indicate whether an audio signal includes music. In some embodiments, the music flag can be a predicted Boolean value indicating whether an audio signal includes music. Alternatively, in some embodiments, the music flag can be a continuous value (e.g., from 0 to 1) indicating the probability that an audio signal includes music.

[0027] like Figure 1 As illustrated, the features generated by the denoising feature extraction block 106 can be provided to the denoising block 110, which is configured to generate a denoising mask based on the denoising features. As described above, since the denoising features are generated based on features indicating the content type, the denoising mask is also generated based on the content type, specifically possibly based on the possibility that the audio signal includes music.

[0028] In some embodiments, feature extraction blocks (e.g., shared feature extraction blocks, discriminative feature extraction blocks, and / or denoising feature extraction blocks) can be implemented using one or more trained machine learning model architectures. Various feature extraction blocks can utilize the same type of architecture or different architectures. In some embodiments, feature extraction blocks can utilize a gated recurrent unit (GRU) architecture. A GRU architecture can include a gating mechanism that allows the exploitation or non-exploitation (e.g., forgetting) of specific features or parameters. In some embodiments, other architectures such as Long Short-Term Memory (LSTM) networks, transformer networks, etc., can be utilized.

[0029] Figure 2 The diagram shows Figure 1 The example implementation of the system shown in the figure utilizes a GRU network to implement the discriminative feature extraction block 104 and the denoising feature extraction block 106. Note that at 202, at the initial concatenation layer of the denoising feature extraction block 106, the output from the discriminative feature extraction block 104 is provided as input to the denoising feature extraction block 106. This initial concatenation layer allows the discriminative features to be concatenated with the shared extracted features used by the denoising feature extraction block 106.

[0030] In some embodiments, enhancing an audio signal may include enhancing a harmonic structure. Harmonic structure enhancement can be achieved using a comb filter. Note that the degree of comb filtering can affect the perception of the enhanced audio signal; for example, insufficient filtering may cause the enhanced audio signal to sound rough, while excessive filtering may cause the enhanced audio signal to sound harsh. Therefore, the filtering strength used to enhance the harmonic structure can be determined and represented by a pitch intensity index, which can serve as a value indicating the degree to which the comb filter is applied. In some embodiments, the pitch intensity can be determined based on the output of an intensity feature extraction block. In some embodiments, the intensity feature extraction block can take features generated or extracted by the discriminative feature extraction block as input, which can be used to determine the content type associated with the input signal. Thus, the pitch intensity index that confirms the degree to which harmonic enhancement is applied may, in turn, depend on the content type of the audio signal. For example, for an audio signal that is highly likely to contain music, harmonic enhancement may be suppressed or its application reduced.

[0031] Figure 3 An example implementation of generating a pitch intensity index using pitch intensity feature extraction block 302 and intensity layer 304 is illustrated. As described above, the pitch intensity index can be a value (e.g., between 0 and 1) indicating the degree to which harmonic enhancement (e.g., comb filtering) is applied to an audio signal. Note that at 308, shared features (e.g., coarse features) extracted by the shared feature extraction block can be provided to pitch intensity feature extraction block 302, and at 310, discriminative features extracted by discriminative feature extraction block 104 can be provided to intensity feature extraction block 302. The use of discriminative features can allow the pitch intensity index to be determined based on the content type of the audio signal. For example, in some embodiments, the pitch intensity index can be determined based on the likelihood that the audio signal includes music.

[0032] As described above, the features extracted by the discriminative feature extraction block can be used to generate a music flag. The music flag can indicate the likelihood that an audio signal contains musical content. For example, in some embodiments, the music flag can be a continuous value indicating the probability that an audio signal contains music. In some embodiments, the music flag can be used to modify the denoising gain and / or the pitch enhancement gain so that the gain is applied based on the likelihood that the content contains music. For example, denoising may be attenuated for content that may contain music to avoid altering the musical content. As another example, pitch enhancement (e.g., harmonic enhancement) may be attenuated or suppressed for content that may contain music to avoid altering the harmonic structure of the music.

[0033] In some embodiments, the initial denoising gain can be modified based on the probability that the audio signal includes music. For example, the initial denoising gain can be modified using a power-law function, such that the initial denoising gain is increased to a power of the probability that the audio signal includes music. In some embodiments, the modified denoising gain can be determined by the following formula:

[0034] In the equation given above, "gain" represents the initial value of the denoising gain, and γ is an indicator based on the probability that the audio signal includes music. For example, in some embodiments, γ can be determined by the following formula:

[0035] In the equation given above, a It is a scalar value. In some embodiments, a It can be 10, 15, 20, 25, etc. As described above, " music flag " is a value indicating the likelihood that the audio signal includes music, and can be a continuous value between 0 and 1. In some embodiments, " music TH This is the threshold used to distinguish between speech and music. Example values ​​include 0.5, 0.7, etc. Note that... music TH The value of γ can be tuned based on the false alarm rate. In some embodiments, γ can be limited to have a maximum value of 1.

[0036] Figure 4 The diagram illustrates an example of how γ varies with the music flag value. Note that for music flag values ​​less than approximately 0.4, γ is 1. Therefore, the initial denoising gain is not modified. However, as the confidence that the audio signal includes music increases (e.g., for music flag values ​​greater than 0.4), the value of γ decreases. The lower γ value results in a reduced initial denoising gain due to the increased likelihood that the audio signal contains music content.

[0037] Figure 5 A two-dimensional graph depicts the change in post-processing gain as a function of the initial gain and the confidence level that the audio signal contains music. Note that for the case where the confidence level of the audio signal containing music is 0, the initial denoising gain is not modified and remains constant. However, to increase the confidence level that the audio signal contains music, the initial denoising gain is continuously modified, causing the gain to decrease and thus attenuating the denoising effect. In other words, the denoising strength may be inversely proportional to the probability that the audio signal contains music.

[0038] In some embodiments, the pitch intensity index indicating the degree of harmonic enhancement can be modified based on musical signatures. This may allow harmonic enhancement of audio signals that may include musical content to be suppressed, thus avoiding alteration of the harmonic structure of the music. In some embodiments, the initial pitch intensity can be modified by multiplying the initial pitch intensity by a parameter θ. The parameter θ can be a function of the probability musical signature (e.g., a nonlinear function), or in other words, a function of the probability that the audio signal includes music. In one example, the initial pitch intensity can be modified by the following formula:

[0039] The parameter θ can be a non-linear function of the musical notation, such as an exponential function. In some embodiments, the parameter θ can be based on a polynomial function of the musical notation. In one example, θ can be determined by the following formula:

[0040] As described above, music TH It is a predetermined threshold. In the equation given above, λ can be an even number greater than or equal to 2 (e.g., 2, 4, 6, etc.).

[0041] Figure 6 This is a two-dimensional graph illustrating the modified pitch gain (or modified pitch intensity) as a function of the initial pitch gain (or initial pitch intensity) and the confidence or probability that the audio signal includes music. Note that when it is determined that the audio signal does not contain music (e.g., a music confidence value of 0), the initial pitch intensity is not modified. However, when the confidence level that the audio signal includes music is high, the initial pitch intensity decreases, which allows for the suppression of harmonic enhancement of the audio signal that may include music. In other words, the degree of harmonic enhancement may be inversely proportional to the probability that the audio signal includes music.

[0042] To avoid discontinuous jumps in the audio signal during frame-to-frame enhancement due to the possibility of music being included in the frame, the music flag value can be smoothed. As described above, the smoothed music flag value can then be used to determine the denoising gain and / or pitch enhancement gain. The music flag can be smoothed using start-up and release rules, allowing the flag value to increase or decrease at a rate controlled by release and / or start-up factors. For example, in some embodiments, the music flag can be smoothed using the following formula:

[0043] In the above equations, α and β are tunable parameters. In some embodiments, α can be 0.02, 0.05, 0.08, etc. In some embodiments, β can be 0.85, 0.9, 0.95, 0.98, etc.

[0044] Figure 7 This is a flowchart of an example process for enhancing audio signals based on content type, according to some embodiments. In some implementations, blocks of process 700 may be executed by one or more control systems or processors of computing devices such as servers, laptops, desktop computers, tablets, and mobile phones. An example of such a control system is described below. Figure 10 The control system 1010 is shown and described herein. In some embodiments, the blocks of process 700 may be arranged differently from those in the control system 1010. Figure 7 The processes are executed in the order shown. In some embodiments, two or more blocks of process 700 may be executed substantially in parallel. In some embodiments, one or more blocks of process 700 may be omitted.

[0045] Process 700 can begin at 702, which involves receiving the input audio segment to be enhanced. The input audio segment can be, for example, a frame of an audio signal. The input audio segment can correspond to audio content captured by the user, such as audio content captured via one or more microphones of a user device like a mobile phone or tablet.

[0046] At point 704, process 700 can provide a representation of the input audio segment to a trained feature extraction model configured to extract an initial feature set. The representation of the input audio segment can be a frequency domain representation of the input audio segment. The trained feature extraction model can be a combination of the above... Figure 1 The shared feature extraction model is shown and described in the figure. In some embodiments, the initial feature set can be regarded as a coarse feature set of the input audio segment, which can be used by the downstream feature extractor to extract more refined features.

[0047] At point 706, the initial feature set can be provided to a trained discriminative feature extraction model, which is configured to extract features indicating the type of audio content. An example of a trained discriminative feature extraction model is... Figure 1 The discriminative feature extraction block 104 is shown in the figure. In some embodiments, the trained discriminative feature extraction model may utilize a GRU architecture. The extracted features can be used to classify an input audio segment as including music, and / or can be used to predict the probability that an input audio segment includes music.

[0048] At point 708, the initial feature set and features indicating the audio content type can be fed into a trained denoising feature extraction model configured to extract features of the indicator noise. In other words, the features of the indicator noise can be generated at least in part based on features indicating the audio content type. An example of a trained denoising feature extraction model is... Figure 1 The denoising feature extraction block 106 is shown in the figure. The trained denoising feature extraction model can utilize the GRU architecture.

[0049] At 710, process 700 may optionally generate a prediction of whether the input audio segment includes music content based on features indicating the type of audio content. For example, at 710, process 700 may generate a music flag, as described above, which may indicate the probability or confidence level that the input audio segment includes music. Note that, as described above, in some embodiments, the music flag may be smoothed between frames to allow the processing performed using the music flag to have stability (as described above in conjunction with...). Figure 4 to Figure 6 (As described).

[0050] At position 712, process 700 can generate a denoising mask based on the characteristics of the indicator noise. For example, as described above... Figure 1 As described, the characteristics of the noise can be provided to a denoising block configured to generate a denoising mask. In some embodiments, the denoising mask can also be generated based on a musical flag as determined above at block 710. For example, in some embodiments, the initial denoising gain can be modified based on the musical flag such that the gain decreases as the likelihood of an audio segment including music increases.

[0051] At 714, process 700 may apply a denoising mask to the representation of the input audio segment. For example, process 700 may multiply the denoising mask by the frequency domain representation of the input audio segment to generate a denoised version of the input audio segment. Note that since the generation of the denoising mask takes into account the audio content type of the input audio segment, denoising can be performed in a way that controls the denoising intensity based on the content type.

[0052] At 716, process 700 may optionally perform harmonic enhancement based on features indicating the type of audio content. For example, in some embodiments, process 700 may determine a pitch intensity index representing the degree of comb filtering performance based on features indicating the type of audio content. In some embodiments, the pitch intensity index may be modified based on a music flag value (e.g., as determined at block 710) such that harmonic enhancement is attenuated or suppressed for audio segments that may include music content.

[0053] In some embodiments, process 700 may cyclically execute blocks 702 to 716 on multiple frames of an audio signal to generate an enhanced audio signal. In some embodiments, the enhanced audio signal may be stored (e.g., stored in the memory of a user device or a server device) and / or may be played back.

[0054] In some embodiments, the shared feature extraction block, the discriminative feature extraction block and the classification layer, and the denoising feature extraction block and the denoising layer can be trained using a training set. In some embodiments, the training set may include training samples that include noisy speech audio clips and noisy music audio clips. In some embodiments, the lowest SNR of the noisy speech audio clip may be lower than the lowest SNR of the noisy music audio clip. For example, in some embodiments, the lowest SNR of the noisy speech audio clip may be -10 dB, -5 dB, -3 dB, or 0 dB, while the lowest SNR of the noisy music audio clip may be 0 dB, 5 dB, 10 dB, etc. In one example, the lowest SNR of the noisy speech audio clip may be 15 dB lower than the lowest SNR of the noisy music audio clip. In some embodiments, the audio clips included in the training samples may be labeled as including or excluding music. In some embodiments, labels may be automatically generated based on, for example, energy-based rules. In some embodiments, the audio clips in the training set may be labeled as including or excluding ambient sounds, such as birdsong, car horns, etc. For example, specific audio objects in the audio clips may be labeled. In some embodiments, at least the denoising feature extraction model or block (optionally, other models or blocks, such as shared feature extraction blocks) can be trained using audio segments marked as including or excluding ambient sound.

[0055] In some embodiments, the shared feature extraction block, the discriminative feature extraction block and the classification layer, and the denoised feature extraction block and the denoising layer can be trained in multiple stages. For example, in some embodiments, all blocks and layers can be trained simultaneously in an initial training stage. Continuing with this example, in some embodiments, a first fine-tuning stage can be performed by freezing the weights associated with the shared feature extraction model, the denoised feature extraction block, and the denoising layer, while continuing to update the weights associated with the discriminative feature extraction block and the classification layer. Furthermore, in a second fine-tuning stage, the weights associated with the shared feature extraction model, the discriminative feature extraction block, and the classification layer can be frozen, while continuing to train and update the weights associated with the denoised feature extraction block and the denoising layer. Each fine-tuning stage can allow the training focus to be placed on poorly performing layers using a new training dataset that has not yet been used to train any part of the model. Note that in some embodiments, the weights associated with each layer can be tracked in the dataset or through metadata associated with that layer, which allows freezing the weights associated with a subset of each layer while continuing to update the weights associated with other layers.

[0056] Figure 8This is a flowchart illustrating example processes for training blocks of an audio signal enhancement system according to some embodiments. In some implementations, the blocks of process 800 may be executed by one or more control systems or processors of a computing device, such as a server device or other computing device. An example of such a control system is described below. Figure 10 The control system 1010 is shown and described herein. In some embodiments, the blocks of process 800 may be arranged differently from those in the control system 1010. Figure 8 The processes are executed in the order shown. In some embodiments, two or more blocks of process 800 may be executed substantially in parallel. In some embodiments, one or more blocks of process 800 may be omitted.

[0057] Process 800 can begin at 802, i.e., obtaining a training set, where each training sample includes an audio segment, a music content indicator, and a target denoising mask. The music content indicator may be the result of manual annotation. The target denoising mask can be a denoising mask used to generate audio segments including noisy content. For example, each audio segment can be an audio segment generated by applying the target denoising mask to a clean audio segment to generate the audio segments included in the training samples. Note that these audio segments can include a range of signal-to-noise ratios (SNR). In some embodiments, the training set can include training samples that include noisy speech audio segments and noisy music audio segments. In some implementations, the lowest SNR of the noisy speech audio segment may be lower than the lowest SNR of the noisy music audio segment. For example, in some embodiments, the lowest SNR of the noisy speech audio segment may be -10 dB, -5 dB, -3 dB, or 0 dB, while the lowest SNR of the noisy music audio segment may be 0 dB, 5 dB, 10 dB, etc. In one example, the lowest SNR of the noisy speech audio segment may be 15 dB lower than the lowest SNR of the noisy music audio segment.

[0058] At point 804, process 800 can perform the first phase of training by using a first subset of the training set to train shared feature extraction blocks, discriminative feature extraction blocks and classification layers, as well as denoising feature extraction blocks and denoising layers. During the first phase of training, the weights associated with each block and / or layer may be updated.

[0059] At point 806, procedure 800 can perform the second phase of training by freezing the weights associated with the shared feature extraction block and the weights associated with the denoised feature extraction block and the denoising layer, while continuing to train the discriminative feature extraction block and the classification layer using a second subset of the training set. Note that the second phase of training may continue until the target classification performance is achieved.

[0060] At point 808, process 800 can perform the third stage of training by freezing the weights associated with the shared feature extraction blocks and the weights associated with the discriminative feature extraction blocks and the classification layer, while using a third subset of the training set to train the denoised feature extraction blocks and the denoising layer. Note that the third stage of training can continue until the target error associated with the generated denoised mask is reached.

[0061] Figure 9 This is a flowchart of an example process for enhancing audio signals based on content type, according to some embodiments. In some implementations, the blocks of process 900 may be executed by one or more control systems or processors of computing devices such as servers, laptops, desktop computers, tablets, and mobile phones. An example of such a control system is described below. Figure 10 The control system 1010 is shown and described herein. In some embodiments, the blocks of process 900 may be arranged differently from those in the control system 1010. Figure 9 The process is executed in the order shown. In some embodiments, two or more blocks of process 900 may be executed substantially in parallel. In some embodiments, one or more blocks of process 900 may be omitted.

[0062] Process 900 can begin at 902, which involves receiving the input audio segment to be enhanced. The input audio segment can be, for example, a frame of an audio signal. The input audio segment can correspond to audio content captured by the user, such as audio content captured via one or more microphones of a user device such as a mobile phone or tablet.

[0063] At step 904, process 900 can generate features indicating the audio content type of the input audio segment. For example, the audio content type could be music, speech, etc. These features indicating the audio content type can be generated by a discriminative feature extraction model, for example, as described above. Figure 1 The discriminative feature extraction model is shown and described herein. In some embodiments, the discriminative feature extraction model can take coarse features as input, for example, from the combination of the above. Figure 1 The shared feature extraction model shown and described in the figure generates coarse features. Alternatively, in some embodiments, the discriminative feature extraction model may take a representation of an audio segment (e.g., a frequency domain representation of the audio segment) or the audio segment itself as input.

[0064] At point 906, process 900 can generate features indicating the presence of noise in the input audio segment by providing at least the features indicating the type of audio content to a trained denoising feature extraction model. An example of a trained denoising feature extraction model is... Figure 1 The denoising feature extraction block 106 is described above. In some embodiments, the trained denoising feature extraction model can also take coarse features as input, for example, from the combination of the above. Figure 1The coarse features generated by the shared feature extraction model are shown and described.

[0065] At 908, process 900 can generate a denoising mask based on features indicating the presence of noise in the input audio segment. For example, the output of a trained denoising feature extraction model can be provided to a denoising layer configured to generate a denoising mask based on features indicating noise.

[0066] At 910, process 900 can generate an enhanced audio segment at least by applying a denoising mask to the input audio segment. For example, in some embodiments, process 900 can multiply the denoising mask by the frequency domain representation of the input audio segment to generate the enhanced audio segment. Note that in some embodiments, as described above... Figure 4 to Figure 6 As described, the denoising gain can be modified based on a music flag indicating the probability that the input audio segment contains music. Therefore, denoising can be performed based on the content of the audio segment using features indicating the type of audio content (e.g., as described above in conjunction with block 906), and in some embodiments, denoising can also be performed based on the music flag by modifying the initial denoising gain.

[0067] Note that in some embodiments, additional enhancement to the input audio segment can be performed by performing pitch enhancement or enhancing the harmonics of the audio segment. (As described above...) Figure 3 and Figure 6 As described, pitch intensity indicators can be determined based on features that indicate the type of audio content, which can optionally be modified based on musical markers. Therefore, harmonic enhancement can be performed based on the audio content type, such that the degree of harmonic enhancement depends on the likelihood that the audio segment includes music.

[0068] In some embodiments, blocks 902 to 910 may be executed cyclically over multiple frames of, for example, an audio signal. The enhanced audio segments may be stored (e.g., stored in local memory and / or on a server device or other remote device) and / or may be played back.

[0069] Figure 10 This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure. (Compared to other methods provided herein...) Figure 1 Sample, Figure 10 The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, device 1000 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 1000 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

[0070] According to some alternative implementations, device 1000 may be or may include a server. In some such examples, device 1000 may be or may include an encoder. Thus, in some instances, device 1000 may be a device configured for use in an audio environment such as a home audio environment; however, in other instances, device 1000 may be a device configured for use in the “cloud,” such as a server.

[0071] In this example, device 1000 includes an interface system 1005 and a control system 1010. In some embodiments, the interface system 1005 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 1005 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may be related to one or more software applications being executed by device 1000.

[0072] In some implementations, the interface system 1005 may be configured to receive or provide a content stream. The content stream may include audio data. Audio data may include, but is not limited to, audio signals. In some instances, audio data may include spatial data such as channel data and / or spatial metadata. In some examples, the content stream may include video data and corresponding audio data.

[0073] Interface system 1005 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, interface system 1005 may include one or more wireless interfaces. Interface system 1005 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 1005 may include a control system 1010 and a memory system (such as... Figure 10 The system 1005 may have one or more interfaces between the optional memory system 1015 shown in the diagram. However, in some instances, the control system 1010 may include a memory system. In some embodiments, the interface system 1005 may be configured to receive input from one or more microphones in the environment.

[0074] For example, the control system 1010 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0075] In some implementations, the control system 1010 may reside in more than one device. For example, in some implementations, a portion of the control system 1010 may reside in a device within one of the environments described herein, while another portion of the control system 1010 may reside in a device outside that environment, such as a server, mobile device (e.g., a smartphone or tablet). In other examples, a portion of the control system 1010 may reside in a device within one environment, while another portion of the control system 1010 may reside in one or more other devices within that environment. For example, a portion of the control system 1010 may reside in a device implementing a cloud-based service (e.g., a server), while another portion of the control system 1010 may reside in another device implementing the cloud-based service (e.g., another server, a storage device, etc.). In some examples, the interface system 1005 may also reside in multiple devices. In some implementations, a portion of the control system may reside in or on an earpiece.

[0076] In some implementations, the control system 1010 may be configured to perform at least partially the methods disclosed herein. According to some examples, the control system 1010 may be configured to train one or more models, thereby using one or more models to generate enhanced audio content, etc.

[0077] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. For example, one or more non-transitory media may reside on... Figure 10 The optional memory system 1015 and / or control system 1010 shown herein. Therefore, various innovative aspects of the subject matter described herein can be implemented in one or more non-transitory media on which software is stored. For example, the software can be provided by a control system (such as...) Figure 10 The control system 1010 is executed by one or more components.

[0078] In some examples, device 1000 may include Figure 10The optional microphone system 920 is shown. The optional microphone system 1020 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, device 1000 may not include microphone system 1020. However, in some such embodiments, device 1000 may still be configured to receive microphone data from one or more microphones in an audio environment via interface system 1010. In some such embodiments, cloud-based implementations of device 1000 may be configured to receive microphone data or at least partially corresponding noise metrics from one or more microphones in an audio environment via interface system 1010.

[0079] According to some embodiments, the device 1000 may include Figure 10 The optional loudspeaker system 1025 is shown in the diagram. The optional loudspeaker system 1025 may include one or more loudspeakers, which may also be referred to herein as “loudspeakers” or more generally as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), device 1000 may not include the loudspeaker system 1025. In some implementations, device 1000 may include headphones. The headphones may be connected or coupled to device 1000 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0080] Some aspects of this disclosure include a system or apparatus configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including input devices, memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0081] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform desired processing on one or more audio signals, including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that may include input devices and memory) programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the invention may be implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system may also include other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or keyboard), memory, and a display device.

[0082] Another aspect of this disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code for performing (e.g., executable by an encoder) one or more examples of the disclosed methods or steps thereof.

[0083] While specific embodiments and applications of this disclosure have been described herein, it will be apparent to those skilled in the art that many changes can be made to the described embodiments and applications without departing from the scope of this disclosure as described and claimed herein. It should be understood that although certain forms of this disclosure have been shown and described, this disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method for enhancing an audio signal, the method comprising: Receive the input audio segment to be enhanced; Generate features that indicate the audio content type of the input audio segment; Features indicating the presence of noise in the input audio segment are generated by providing at least the features indicating the type of the audio content to a trained denoising feature extraction model; A denoising mask is generated based on the features indicating the presence of noise in the input audio segment; as well as An enhanced audio segment is generated by applying the denoising mask to the input audio segment at least once.

2. The method according to claim 1, further comprising: Initial features associated with the input audio segment are obtained by feeding the input audio segment to a trained initial feature extraction model, wherein features indicating the audio content type of the input audio segment are generated using the initial features.

3. The method according to claim 2, wherein, The trained denoising feature extraction model receives the initial features as input.

4. The method according to any one of claims 1 to 3, wherein, The features indicating the type of audio content are extracted by a trained discriminative feature model.

5. The method according to claim 4, wherein, The trained discriminative feature model and the trained denoising feature extraction model are trained in a multi-stage training process, which includes: in the first stage, training both the discriminative feature model and the denoising feature extraction model concurrently; in the second stage, freezing the weights of the denoising feature extraction model while fine-tuning the weights of the discriminative feature model; and in the third stage, freezing the weights of the discriminative feature model while fine-tuning the weights of the denoising feature extraction model.

6. The method according to claim 4 or 5, wherein, At least the trained discriminative feature model is trained using a training set that includes audio content samples labeled as including or excluding music content, wherein the labels are automatically generated based on energy-based rules.

7. The method according to any one of claims 4 to 6, wherein, The discrimination feature model and the denoising feature extraction model are trained using training samples that include at least noisy speech audio segments and noisy music audio segments, wherein the lowest signal-to-noise ratio (SNR) of the noisy speech audio segments is lower than the lowest SNR of the noisy music audio segments.

8. The method according to any one of claims 4 to 7, wherein, The trained discriminative feature model includes a gated recurrent unit (GRU).

9. The method according to any one of claims 1 to 8, wherein, The trained denoising feature extraction model includes a gated recurrent unit (GRU).

10. The method according to any one of claims 1 to 9, wherein, The feature indicating the type of audio content includes an indication of whether the input audio segment includes musical content.

11. The method of claim 10, further comprising: The indication of whether the input audio segment includes musical content is smoothed across multiple frames of the input audio segment.

12. The method according to any one of claims 1 to 11, further comprising: The degree to which the harmonics of the input audio segment are enhanced in the enhanced audio segment is determined by providing at least the features indicating the type of the audio content to a trained pitch intensity feature extraction block.

13. The method according to claim 12, wherein, The degree of harmonic enhancement of the input audio segment is inversely proportional to the confidence level that the input audio segment includes musical content.

14. The method according to any one of claims 1 to 13, wherein, The denoising mask is generated such that the denoising strength is inversely proportional to the confidence level of the input audio segment including musical content.

15. The method according to any one of claims 1 to 14, wherein, At least the denoising feature extraction model is trained using a training set that includes labeled ambient sounds.

16. A system comprising: One or more processors; and A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations according to claims 1 to 15.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to claims 1 to 15.

Images