Audio de-reverberation

Abstract

Methods and systems for generating a set of resultant AIR from a real acoustic impulse response AIR (t), and training a machine learning model using the set of resultant AIR, such that the machine learning model after training is configured to generate a dereverberation audio signal given an input audio signal. A synthetic AIR is generated by selecting a random separation point in time s and a random crossing gradient duration d to form an early portion AIRe (t) and a later portion AIRl (t) of the real AIR. By means of the proposed method, the real AIR is "soft" separated into early AIR and later AIR. In particular, the early AIR decays to zero during the transfer period d, while the late AIR gradually increases from zero during the transfer period. The sum of the early AIR and the later AIR is still equal to the real AIR.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to International Application No. PCT / CN2022 / 102984 filed on June 30, 2022, U.S. Provisional Application No. 63 / 434,093 filed on December 21, 2022, and U.S. Provisional Application No. 63 / 490,063 filed on March 14, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to dereverberation of audio signals. Background Art

[0004] With audio recording devices readily available in many environments, audio content is being generated in a variety of situations and with varying qualities. Audio content such as podcasts, radio shows, TV shows, music videos, user-generated content, short videos, video conferences, teleconferences, panel discussions, and interviews may include various types of distortion, including reverberation.

[0005] Reverberation occurs when an audio signal is distorted by reflections from various surfaces (such as walls, ceilings, floors, and furniture) before being received by a receiver. Reverberation may have a significant impact on sound quality and speech intelligibility. More specifically, the sound that reaches a receiver (e.g., a human listener and a microphone, etc.) consists of direct sound and reverberant sound, where the direct sound includes the sound emitted directly from the source without any reflections, and the reverberant sound includes the sound reflected from various surfaces in the environment. Reverberant sound includes early reflections and late reflections. Early reflections may arrive at the receiver shortly after the direct sound or at the same time as the direct sound, and therefore may be partially integrated into the direct sound. The integration of early reflections with direct sound produces a spectral coloring effect that contributes to the perceived sound quality. Late reflections arrive at the receiver after the early reflections (e.g., more than 50-80 milliseconds after the direct sound). Late reflections may have an adverse effect on speech intelligibility. Therefore, the audio signal may be de-reverberated to reduce the impact of late reflections present in the audio signal, thereby improving speech intelligibility and clarity.

[0006] De-reverberation of audio signals is an area where machine learning has been found to be very useful. For example, machine learning models, such as deep neural networks, can be used to predict a dereverberation mask, which when applied to a reverberant audio signal produces a dereverberated audio signal.

[0007] The machine learning model for dereverberation audio signals can be trained using a training set comprising an appropriate number of training samples, where each training sample comprises a clean audio signal (e.g., without reverberation) and a corresponding reverberant audio signal. In order for the machine learning model to be robust, the training set may need to capture reverberation from a large number of different room types (e.g., rooms with different sizes, layouts, and furniture, etc.) and a large number of different speakers, etc.

[0008] The training set can be generated by obtaining various acoustic impulse responses (AIR), each of which characterizes room reverberation. The training samples can then be formed from a clean audio signal and a corresponding reverberant audio signal generated by convolving the AIR with the clean audio signal. However, the number of available real AIRs may be limited, and the available real AIRs may not fully characterize the underlying reverberation effects (e.g., may not adequately capture rooms of different dimensions and layouts, etc.).

[0009] Document WO 2023 / 287782 discloses a technique for generating an enhanced training set that can be used to train a robust machine learning model for dereverberation audio signals. In WO 2023 / 287782, a set of synthetic AIR is generated using (measured or modeled) real AIR. Synthetic AIR can be generated by changing and / or modifying various characteristics of early reflections and / or late reflections of real AIR. Summary of the invention

[0010] While the methods disclosed in WO 2023 / 287782 have been found to be very useful, it would be advantageous to further improve the process for synthesizing AIR.

[0011] The purpose of the present invention is to further improve the process of AIR synthesis so that enhanced training data can be generated for dereverberation machine learning models.

[0012] According to the first aspect of the present invention, this object and other objects are achieved by a method for training a machine learning model, the method comprising generating a set of synthetic AIRs from a real acoustic impulse response AIR(t), generating multiple training samples using the set of synthetic AIRs, each training sample comprising a non-reverberant audio signal and a reverberant audio signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal, and training the machine learning model using the multiple training samples so that the machine learning model is configured after training to generate a dereverberated audio signal given an input audio signal.

[0013] Synthetic AIR is the early part of the real AIR that corresponds to the early reflections of the direct sound. e(t) and the later part AIR of the real AIR corresponding to the later reflection of the direct sound l (t), and generated by modifying at least one of the early part and the late part and recombining the possibly modified early part and the possibly modified late part to generate each synthetic AIR.

[0014] The early and late parts are formed by selecting a random separation time point s, selecting a random crossfade duration d, defining a transfer function of time f(t) that describes a continuous decrease from 1 to 0 as t goes from s to s+d, and defining the early crossfade function as:

[0015]

[0016] Define the late cross-fade function as:

[0017] f l (t) = 1-f e (t), and

[0018] Calculate the early part of AIR e (t) and later part AIR l (t), respectively:

[0019] AIR e (t)=AIR(t)*f e (i) and AIR l (t)=AIR(t)*f l (t).

[0020] By the proposed method, the real AIR is "softly" separated into early AIR and late AIR. Specifically, the early AIR decays to zero during the transition period d, while the late AIR gradually increases from zero during the transition period. The sum of the early AIR and the late AIR is still equal to the real AIR.

[0021] By such a "soft" transfer, discontinuities in early and late AIR are avoided, thereby improving training. In addition, the transfer period introduces another variable that can contribute to diversity. For a single separation time point, there can be several different transfer periods.

[0022] Therefore, with this approach, the diversity of synthesized AIR patterns for deep learning-based speech enhancement algorithms can be increased, thereby improving the performance of the trained models.

[0023] The enhanced AIR training data can improve the robustness of deep noise suppression models under reverberant conditions and improve the dereverberation performance of deep speech dereverberation models. It can also be used to improve the robustness of speech / audio processing, such as echo reduction under adverse reverberant conditions for real use cases.

[0024] In some embodiments, the modification of the late part is accomplished by applying a random decay function g(t) to the late part. By using a random decay function, the diversity is further increased. For a given separation point and a given transfer period, there can be several different modified late parts by using different random decay functions.

[0025] According to a second aspect of the invention, this object and other objects are achieved by a computer-implemented system for training a machine learning model, the system comprising a computer-implemented process for generating a set of synthetic AIRs from a real acoustic impulse response AIR(t); a computer-implemented process for generating a plurality of training samples, each training sample comprising a non-reverberant audio signal and a reverberant audio signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal; and a computer-implemented training process for training a machine learning model using a training set comprising a plurality of training samples, each training sample comprising a non-reverberant signal and a reverberant signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal.

[0026] A computer-implemented process for generating a set of synthetic AIRs comprises a separation block configured to receive a real acoustic impulse response AIR(t) and to form an early portion AIRe(t) of the real AIR corresponding to early reflections of the direct sound and a late portion AIRe(t) of the real AIR corresponding to late reflections of the direct sound. l (t); at least one processing block for modifying at least one of the early part and the late part; and a combination block for recombining the possibly modified early part and the possibly modified late part to form a synthetic AIR.

[0027] The separation block is configured to select a random separation time point s, select a random crossfade duration d, define a transition function for time f(t), the transition function describes the continuous decrease from 1 to 0 as t goes from s to s+d, and define the early crossfade function as:

[0028]

[0029] Define the late cross-fade function as:

[0030] f l (t) = 1-f e (t), and

[0031] Calculate the early part of AIR e (t) and later part AIR l (t), respectively:

[0032] AHr e (t)=AIR(t)*f e (t) and AIR l (t)=AIR(t)*f l (t). BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Various aspects of the invention will now be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments of the invention.

[0034] Figure 1 An audio signal with reverberation is shown.

[0035] Figure 2 A dereverberation system according to an embodiment of the present invention is shown.

[0036] Figure 3 Shown for training Figure 2 The process of machine learning models in .

[0037] Figure 4 An example of a measured acoustic impulse response (AIR) is shown.

[0038] Figure 5 The process of generating synthetic AIR from real AIR according to an embodiment of the present invention is schematically shown.

[0039] Figure 6 Examples of early and late crossover functions for separating the real AIR into early and late parts are shown.

[0040] Fig. 7A The general separation of AIR into early and late fractions is shown.

[0041] Figure 7B Shows the use of Figure 5 The crossover function in separates AIR into early and late parts.

[0042] Fig. 8A The original (untruncated) late part is shown.

[0043] FIG. 8B to FIG. 8D Shows Fig. 8A The late part of is truncated by an exponential decay function with a different exponent.

[0044] Fig. 9A process for generating a set of training samples according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed collaboratively by several physical components.

[0046] For example, the computer hardware can be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a network device, a network router, a switch or a bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the computer hardware. In addition, the present disclosure shall relate to any collection of computer hardware that executes instructions, either individually or in combination, to perform any one or more of the concepts discussed herein.

[0047] Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken is included. Therefore, a typical processing system (i.e., computer hardware) including one or more processors is an example. Each processor may include one or more CPUs, a graphics processing unit, and a programmable DSP unit. The processing system may also include a memory subsystem, which includes a hard drive, an SSD, RAM, and / or ROM. A bus subsystem for communicating between components may also be included. During the execution of software by the computer system, the software may remain in the memory subsystem and / or the processor.

[0048] One or more processors may operate as standalone devices or may be connected, for example, to other processors via a network. Such a network may be based on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0049] The software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile media, removable media and non-removable media, which are implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to the technician that the communication medium (transitory) generally embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information transfer medium.

[0050] In the following description, the focus is on how to obtain a training set. Other parts of the process, such as how to train a machine learning model and how to use the trained machine learning model, are only briefly discussed. More details can be found in PCT Patent Publication No. WO 2023 / 287773, entitled "SPEECH ENHANCEMENT", which is incorporated herein by reference.

[0051] As briefly discussed above, the sound that arrives at a receiver (human listener, microphone, etc.) consists of direct sound (directly from the source, without any reflections) and reverberant sound. The total energy of the reverberant sound can be decomposed into two parts: early reflections and late reflections. Early reflections arrive at the receiver fairly shortly after the direct sound and are partially integrated into it, producing a spectral coloring effect on the speech. Late reflections, including all reflections that arrive after the early reflections, primarily have an adverse effect on speech perception. For example, late reflections can be considered to arrive 50ms to 80ms after the direct sound.

[0052] Figure 1 Shown is an example of a time domain input audio signal 100 and a corresponding spectrogram 102. As illustrated by the spectrogram 102, early reflections may produce changes to the spectrogram as depicted by spectral coloration 106. The spectrogram 102 also illustrates late reflections 108 that may adversely affect the intelligibility of speech.

[0053] A machine learning model, such as a deep neural network, can be used to predict a dereverberation mask, which, when applied to a reverberant audio signal, generates a dereverberated audio signal. Figure 2An example of a dereverberation system 200 for dereverberation of an audio signal 202 is shown. Note that Figure 2 The system may also be applied to other types of signal enhancement, such as noise suppression, a combination of noise suppression and dereverberation, etc. In other words, in some embodiments, instead of generating a predicted dereverberation mask and a predicted dereverberation audio signal, a predicted enhancement mask may be generated and the predicted enhancement mask may be used to generate a predicted enhanced audio signal, wherein the predicted enhanced audio signal is a denoised and / or dereverberated version of the distorted input audio signal.

[0054] In some embodiments, components of system 200 may be implemented via user devices such as mobile phones, tablet computers, laptop computers, wearable computers (e.g., smart watches, etc.), desktop computers, game consoles, smart televisions, etc.

[0055] Figure 2 The dereverberation system 200 in the embodiment of the present invention takes an input audio signal 202 as input and generates a dereverberation audio signal 204 as output. The input audio signal 202 can be an audio signal captured in real time, such as live streaming content, an audio signal corresponding to an ongoing video conference or audio conference, etc. In some embodiments, the input audio signal can be a pre-recorded audio signal, such as an audio signal associated with a pre-recorded audio content (e.g., television content, video, movie, podcast, etc.). In some embodiments, the input audio signal can be received by a microphone of a user device. In some embodiments, the input audio signal can be transmitted to the user device, such as from a server device, another user device, etc.

[0056] In the illustrated embodiment, the system 200 includes a feature extractor 208 for generating a frequency domain representation of an input audio signal 202, which can be viewed as an input signal spectrum. The frequency domain representation of the input audio signal can be generated using a transform, such as a short-time Fourier transform (STFT), a modified discrete cosine transform (MDCT), etc. In some embodiments, the frequency domain representation of the input audio signal is referred to herein as a "striped feature" of the input audio signal. In some embodiments, the frequency domain representation of the input audio signal can be modified by applying a perceptual-based transform that mimics the filtering of the human cochlea. Examples of perceptual-based transforms include gammatone filters, equivalent rectangular bandwidth filters, Mel-scale filters, etc. The modified frequency domain transform is sometimes referred to herein as a "banded feature" of the input audio signal.

[0057] The input signal spectrum is then provided to a trained machine learning model 210. The machine learning model is trained to generate a dereverberation mask that, when applied to a frequency domain representation of the input audio signal, generates a frequency domain representation of the dereverberation audio signal. In some embodiments, the logarithm of the extracted features may be provided to the trained machine learning model.

[0058] The machine learning model 210 can have any suitable architecture or topology. For example, in some embodiments, the machine learning model can be or can include a deep neural network, a convolutional neural network (CNN), a long short-term memory (LSTM) network, a recurrent neural network (RNN), etc. In some embodiments, the machine learning model can combine two or more types of networks. For example, in some embodiments, the machine learning model can combine a CNN with a recurrent element. Examples of recurrent elements that can be used include GRU, LSTM networks, Elman RNN, etc.

[0059] The predicted dereverberation mask generated by the trained machine learning model 210 is provided to a dereverberation signal spectrum generator 212. In some embodiments, the predicted dereverberation mask is modified by applying an inverse perceptually based transform, such as an inverse gammatone filter, an inverse equivalent rectangular bandwidth filter, or the like.

[0060] The dereverberation signal spectrum generator 212 applies the predicted dereverberation mask to the input signal spectrum to generate a dereverberation signal spectrum (e.g., a frequency domain representation of the dereverberation audio signal). In some embodiments, the predicted dereverberation mask is multiplied by the frequency domain representation of the input audio signal. In an instance where the logarithm of the frequency domain representation of the input audio signal is provided to the trained machine learning model, the logarithm of the predicted dereverberation mask is subtracted from the logarithm of the frequency domain representation of the input audio signal, and the difference is exponentiated to obtain the frequency domain representation of the dereverberation audio signal.

[0061] Finally, the dereverberated signal spectrum is provided to a time domain transform component 214, which generates the dereverberated audio signal 204. For example, the time domain representation of the dereverberated audio signal may be generated by applying an inverse transform (e.g., inverse STFT, inverse MDCT, etc.) to the frequency domain representation of the dereverberated audio signal.

[0062] The time domain representation of the dereverberated audio signal may be played or presented (e.g., by one or more speaker devices of the user device). In some embodiments, the dereverberated audio signal may be stored, such as in a local memory of the user device. In some embodiments, the dereverberated audio signal may be transmitted, such as to another user device for presentation by the other user device, to a server for storage, etc.

[0063] Figure 3 A process 300 for training a machine learning model 210 is shown, such that the trained machine learning model 210 is configured to generate a dereverberated audio signal given an input audio signal.

[0064] First, a training set is obtained in step 302. The training set includes training samples, each of which includes a clean audio signal (e.g., without reverberation) and a corresponding reverberant audio signal. The clean audio signal can be considered as a "ground-truth" signal that the machine learning model will be trained to predict or generate. The set can include any number of samples, such as 100 training samples, 1000 training samples, 10,000 training samples, etc. As mentioned in the background technology section, the training samples can be obtained by generating a pair of clean audio signals and corresponding reverberant audio signals generated by convolving an acoustic impulse response (AIR) with the clean audio signal.

[0065] In step 304, for a given training sample, process 300 provides the reverberant audio signal to the machine learning model to obtain a predicted dereverberation mask. In some implementations, a frequency domain representation of the reverberant audio signal may be provided to the machine learning model. The frequency domain representation of the reverberant audio signal may be filtered using a filter that approximates human cochlear filtering or otherwise transformed.

[0066] In step 306, the process 300 obtains a predicted dereverberation audio signal using the predicted dereverberation mask. For example, the process 300 may apply the predicted dereverberation mask to a frequency domain representation of the reverberant audio signal to obtain a frequency domain representation of the dereverberation audio signal. Continuing with this example, in some implementations, the process 300 may then generate a time domain representation of the dereverberation audio signal.

[0067] In step 308, the process 300 determines a reverberation metric value associated with the predicted dereverberated audio signal. The reverberation metric may be a speech-to-dereverberation modulation energy of one or more frames of the predicted dereverberated audio signal. In step 310, the process 300 determines a loss term based on the clean audio signal, the predicted dereverberated audio signal, and the reverberation metric value. The loss term may be a combination of the difference between the clean audio signal and the predicted dereverberated audio signal and the reverberation metric value. In some embodiments, the combination is a weighted sum, where the reverberation metric value is weighted by the importance of minimizing reverberation in the output generated using the machine learning model.

[0068] In step 312, process 300 updates the weights of the machine learning model based at least in part on the loss term. Process 300 may use gradient descent and / or any other suitable technique to calculate updated weight values ​​associated with the machine learning model. The weights may be updated based on other factors, such as a learning rate, a random dropout rate, etc. The weights may be associated with various nodes, layers, etc. of the machine learning model.

[0069] In step 314, process 300 determines whether the machine learning model requires more training. Step 314 may involve determining that an error associated with the machine learning model has dropped below a predetermined error threshold, determining that a change in weights associated with the machine learning model from one iteration to the next is less than a predetermined change threshold, etc.

[0070] If, at step 314, the process 300 determines that the machine learning model 210 does not require additional training ("No" at block 314), the process 300 will end 316. Conversely, if, at step 314, the process 300 determines that the machine learning model 210 requires additional training ("Yes" at block 314), the process 300 will loop back to step 304 and repeat steps 304-314 using different training samples.

[0071] In order to generate a larger set of training samples, available, real AIR is used to generate a set of synthetic AIR. It should be noted that the real AIR can be measured AIR measured in a room environment (e.g., using one or more microphones placed in the room). Alternatively, the real AIR can be modeled AIR, for example, generated using a room acoustic model that incorporates the room shape, the materials in the room, the layout of the room, the objects in the room (e.g., furniture), and / or any combination thereof. In contrast, synthetic AIR is AIR generated based on real AIR (e.g., by modifying the composition and / or characteristics of the real AIR), regardless of whether the real AIR is measured or modeled.

[0072] Figure 4An example of measuring AIR in a reverberant environment is shown. In the figure, time zero (t=0) represents the arrival of direct sound. As illustrated, early reflections 401 arrive at the receiver at the same time or shortly after time zero. In contrast, late reflections 402 arrive at the receiver after early reflections 401. Early reflections include multiple spikes 403. Late reflections 402 are associated with duration 404, which can be on the order of 100 milliseconds, 0.5 seconds, 1 second, 1.5 seconds, etc. Late reflections 403 are also associated with attenuation 405, which characterizes how the amplitude of late reflections 403 decays or decreases over time. The boundary 406 between early reflections and late reflections can be in the range of approximately 50 milliseconds and 80 milliseconds. Although the boundary 406 illustrated here is a sharp boundary (one time point), the boundary can be considered as a gradual transition.

[0073] Figure 5 The process of generating synthetic AIR from real AIR is shown. In the first separation block 501, the real AIR 502 is randomly separated into an early part 503 AIR corresponding to the early reflection of the direct sound. e (t) and the later part 504AIR corresponding to the later reflection of the direct sound l (t). The early part 503 is processed (enhanced) in an early part processing block 505 to form a set of randomly modified early AIR 506. The late part 504 is processed (enhanced) in a late part processing block 507 to form a set of randomly modified late AIR 508. Then, the modified early AIR and the modified late AIR are combined in a combination block 509 to form a set of synthesized AIR 510.

[0074] Block 502 implements a pair of crossfade functions to provide a continuous transition between early AIR and late AIR. e (t) is defined as:

[0075]

[0076] And the later cross-fade function is defined as:

[0077] f l (t) = 1-f e (t),

[0078] where f(t) is the transfer function of time, describing the continuous decrease from 1 to 0 as t goes from s to s+d, where s is the randomly chosen separation time point s and d is the randomly chosen crossfade duration.

[0079] The separation time s is typically between 5 ms and 100 ms, preferably between 20 ms and 80 ms. The crossfade duration d is typically between 1 ms and 10 ms.

[0080] Various continuous transfer functions f(t) can be considered, including simple linear ramps and step-wise continuous functions. Figure 6 shows the cross-fade function f e (t) and f l (t), where the transfer function f(t) is chosen as exist Figure 6 In FIG. 1 , the x-axis is shown in samples. The sampling frequency in this example is 32 kHz, indicating that s is approximately 5 ms and d is approximately 5 ms.

[0081] Early part AIR e (t) and later part AIR l (t) can now be calculated as AIR e (t)=AIR(t)*f e (t) and AIR l (t)=AIR(t)*f l (t), where AIR(t) is the true AIR.

[0082] Fig. 7A The output shows how the early and late parts of the real AIR are formed based on the sharp cutoff, and Figure 7B shows how the early and late parts of the same real AIR are formed using the method discussed above. Fig. 7A and 7B In FIG. 1 , the x-axis is shown in samples. The sampling frequency is 32 kHz.

[0083] The enhancement of the early portion in block 505 may be performed in a conventional manner, for example involving randomly reordering the spikes 403 in time.

[0084] The enhancement of the late part in block 507 can be a simple truncation, as proposed in the prior art, or it can involve a random attenuation (decay) function. The decay function can be an exponential decay, a linear function, a part of a polynomial function, etc.

[0085] In one embodiment, the decay function is an exponential decay function:

[0086]

[0087] Where a is a random attenuation parameter, 0<a<1.

[0088] Fig. 8Ashows an example of an original post-AIR section consisting of Figure 4 The process in is obtained. FIG. 8B to FIG. 8D Shows Fig. 8A The late AIR part in , which obeys the exponential decay function described above. Figure 8B In , the decay parameter is 0.5. Figure 8C In , the decay parameter is 0.1. Fig.8D In these examples, the attenuation parameter is 0.01. The sampling frequency in these examples is 32kHz.

[0089] Fig. 9 An example of a process 800 for generating an enhanced training set using real and / or synthetic AIR is shown. The enhanced training set can be used to train a machine learning model for dereverberation of an audio signal. Processing 900 begins at 901 by obtaining a set of clean input audio signals (e.g., input audio signals without any reverberation and / or noise). The clean input audio signals in the set of clean input audio signals may have been recorded by one or more recording devices in one or more different room environments. Each clean input audio signal may include any combination of audible sound types, such as speech, music, sound effects, etc. However, each clean input audio signal is preferably free of reverberation, echo, and / or noise.

[0090] At block 902, process 900 obtains a set of AIRs, the set of AIRs comprising Figure 4 At least one real (measured or modeled) AIR and / or multiple synthetic AIRs obtained by the process in . A set of AIRs may include any appropriate number of AIRs (e.g., 100 AIRs, 200 AIRs, 500 AIRs, etc.). A set of AIRs may include any appropriate ratio of real AIR to synthetic AIR, such as 90% synthetic AIR and 10% real AIR, 80% synthetic AIR and 20% real AIR, etc.

[0091] At block 903, for each pairwise combination of a clean input audio signal in a set of clean input audio signals and an AIR in a set of AIRs, the process 900 may generate a reverberant audio signal based on the clean input audio signal and the AIR. For example, in some embodiments, the process 900 may convolve the AIR with the clean input audio signal to generate the reverberant audio signal. In principle, given N clean input audio signals and M AIRs, the process 900 may generate N×M reverberant audio signals.

[0092] At block 904, the process 900 adds noise to one or more reverberant audio signals to generate a noise reverberant audio signal. Examples of noise that can be added include white noise, pink noise, brown noise, multi-talker speech noise, etc. The process 900 can add different types of noise to different reverberant audio signals. For example, in some embodiments, the process 900 can add white noise to a first reverberant audio signal to generate a first noise reverberant audio signal. Continuing with this example, in some embodiments, the process 900 can add a multi-talker speech noise type noise to the first reverberant audio signal to generate a second noise reverberant audio signal. Further continuing with this example, in some embodiments, the process 900 can add brown noise to the second reverberant audio signal to generate a third noise reverberant audio signal. In other words, in some embodiments, different versions of noise reverberant audio signals can be generated by adding different types of noise to the reverberant audio signals. It should be noted that in some embodiments, block 900 can be omitted, and the training set can be generated without adding noise to any reverberant audio signal.

[0093] At the end of block 904, process 900 has generated a training set including multiple training samples. Each training sample includes a clean audio signal and a corresponding reverberant audio signal with or without added noise. It is important to note that a single clean audio signal can be used to generate multiple reverberant audio signals by convolving the clean audio signal with multiple different AIRs. And similarly, a single reverberant audio signal can be used (e.g., generated by convolving a single clean audio signal with a single AIR) to generate multiple noise-reverberant audio signals, each of which corresponds to a different type of noise added to a single reverberant audio signal. Accordingly, a single clean audio signal can be associated with 20, 30, 100, etc. training samples, each of which includes a different corresponding reverberant audio signal (or noise-reverberant audio signal).

[0094] Unless otherwise expressly stated, as will be apparent from the following discussion, it should be understood that throughout the public discussion, use of terms such as "process," "compute," "determine," "analyze," etc. refers to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices that operate on and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0095] It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of simplifying the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly listed in each claim. On the contrary, as reflected in the following claims, the inventive aspects exist in fewer features than all the features of a single preceding disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into this specific embodiment, with each claim independently as a separate embodiment of the present invention. In addition, although some embodiments described herein include some features included in other embodiments but not other features, as understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the present invention and to form different embodiments. For example, in the following claims, any of the embodiments claimed for protection may be used in any combination.

[0096] In addition, some embodiments are described herein as a combination of methods or elements of methods that can be implemented by a processor of a computer system or by other means of performing the function. Therefore, a processor with instructions for executing the method or elements of the method forms a component for executing the method or elements of the method. Please note that when a method includes several elements, such as several steps, the order of these elements is not implied unless otherwise specified. In addition, the elements of the device embodiments described herein are examples of components for functions performed by the elements for the purpose of performing embodiments of the present invention. In the description provided herein, many specific details are listed. However, it should be understood that embodiments of the present invention can be implemented without these specific details. In other examples, in order not to obscure the understanding of this specification, well-known methods, structures and techniques are not shown in detail.

[0097] It will be appreciated by those skilled in the art that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations may be made within the scope of the appended claims. For example, other methods of synthesizing the early and late parts of the synthetic AIR may be considered.

Claims

1. A method for training a machine learning model, the method comprising: Generate a set of synthetic AIR from the real acoustic impulse response AIR(t); generating a plurality of training samples using the set of synthetic AIRs, each training sample comprising a non-reverberant audio signal and a reverberant audio signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal; as well as training the machine learning model using the plurality of training samples such that the machine learning model, after training, is configured to generate a dereverberated audio signal given an input audio signal; The set of synthetic AIRs is generated by the following operations: The early part AIR of the real AIR corresponding to the early reflection of the direct sound is formed e (t) and the later part AIR of the real AIR corresponding to the later reflection of the direct sound l (t), and generating each synthesized AIR by modifying at least one of the early portion and the late portion and recombining the possibly modified early portion and the possibly modified late portion; The early part and the late part are formed by the following operations: Choose a random separation time point s, Choose a random crossfade duration d, Define a transfer function f(t) of time that describes a continuous decrease from 1 to 0 as t increases from s to s+d, Define the early cross-fade function as: Define the late cross-fade function as: f l (t) = 1-f e (t), and Calculate the early part AIR e (t) and the latter part AIR l (t), respectively: AIR e (t) = AIR(t) * f e (t) and AIR l (t) = AIR(t) * f l (t).

2. The method according to claim 1, wherein: The transfer function f(t) is defined as:

3. The method according to claim 1 or 2, wherein: The separation time s is between 5 ms and 100 ms, preferably between 20 ms and 80 ms.

4. A method according to any one of the preceding claims, wherein: The cross-fade duration d is between 1 ms and 10 ms.

5. A method according to any one of the preceding claims, wherein: Modifying the late portion involves applying a random decay function g(t) to the late portion.

6. The method according to claim 5, wherein: The random decay function is an exponential decay function: Among them, a is a random attenuation parameter.

7. The method according to claim 6, wherein: 0<a<1。 8. A method for dereverberation of an input audio signal, comprising: providing the input audio signal to a machine learning model trained according to the method of any preceding claim, and A dereverberated output audio signal is generated using the machine learning model.

9. A computer-implemented system for training a machine learning model, the system comprising: A computer-implemented process (500) for generating a set of synthetic AIR from real acoustic impulse responses AIR(t); A computer-implemented process (900) for generating a plurality of training samples, each training sample comprising a non-reverberant audio signal and a reverberant audio signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal; as well as a computer-implemented training process (300) for training the machine learning model using a training set comprising a plurality of training samples, each training sample comprising a non-reverberant signal and a reverberant signal formed by applying one of the synthetic AIRs to the non-reverberant audio signal; The computer-implemented process for generating a set of synthetic AIRs includes: A separation block configured to receive a real acoustic impulse response AIR(t) and to form an early portion AIR of the real AIR corresponding to an early reflection of the direct sound e (t) and the later part AIR of the real AIR corresponding to the later reflection of the direct sound l (t); at least one processing block for modifying at least one of the early portion and the late portion; a combining block for recombining the possibly modified earlier portion and the possibly modified later portion to form a composite AIR; Wherein, the separation block is configured as follows: Choose a random separation time point s, Choose a random crossfade duration d, Define a transfer function f(t) of time that describes the continuous decrease from 1 to 0 as t goes from s to s+d, Define the early cross-fade function as: Define the late cross-fade function as: f l (t) = 1-f e (t), and Calculate the early part AIR e (t) and the latter part AIR l (t), respectively: AIR e (t) = AIR(t) * f e (t) and AIR l (t) = AIR(t) * f l (t).

10. The system of claim 9, wherein the transfer function f(t) is defined as:

11. The system according to claim 9 or 10, wherein: The separation time s is between 5 ms and 100 ms, preferably between 20 ms and 80 ms.

12. A system according to any one of claims 9 to 11, wherein: The cross-fade duration d is between 1 ms and 10 ms.

13. A system according to any one of claims 9 to 12, wherein: One of the at least one processing block is configured to modify the late portion by applying a random decay function g(t) to the late portion.

14. The system according to claim 13, wherein: The random decay function is an exponential decay function: Among them, a is a random attenuation parameter.

15. The system of claim 14, wherein: 0<a<1。 16. A computer program product comprising computer program code portions configured to perform the method according to any one of claims 1 to 8 when executed on a computer processor.

Images