Machine learning for microphone style transfer

Abstract

Example implementations of the present disclosure relate to machine learning for microphone style transfer, for example, to facilitate augmentation of audio data such as speech data to improve robustness of machine learning models trained on audio data. Systems and methods for microphone style transfer can include one or more machine-learned microphone models trained to augment and augment signal data to mimic characteristics of signal data obtained from a target microphone. The systems and methods can include a speech enhancement network for enhancing samples prior to style transfer. The augmented output can then be used for various downstream tasks.

Description

[0001] Related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 092,760, filed October 16, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to machine learning for microphone style transfer. More specifically, this disclosure relates to machine learning for microphone style transfer to, for example, facilitate the augmentation of audio data such as speech data to improve the robustness of machine learning models. Background Technology

[0004] Recent advances in audio recognition models have yielded significant performance improvements across multiple tasks. However, these systems suffer severe performance degradation when encountering domain shifts (e.g., variations in the characteristics of the input data, such as those resulting from using different microphones to collect the input audio data). This situation frequently arises in practice: deployed models are faced with audio data collected by heterogeneous devices in various environments, each characterized by different hardware and signal processing pipelines.

[0005] One important approach to addressing domain offset is to develop models that demonstrate improved robustness against domain offsets such as microphone variability. Model robustness has been a long-standing topic in the speech recognition community.

[0006] Regarding robustness to microphone variability, it has been noted that performance is significantly impacted when inference is performed on audio captured by microphones different from those used to collect the training set. To mitigate this issue, some existing methods employ additive correction in the cepstral domain, while others use a set of microphones to perform multi-style training. A limitation of the former approach is that it is only compatible with applications operating on cepstral inputs, while the latter assumes the availability of data from multiple microphones during training time.

[0007] In another example approach, CycleGAN is used to learn the mapping between microphones. While this method requires neither paired data nor imposes constraints on the type of microphone transformations it models, it relies on training a separate CycleGAN from scratch for each microphone type encountered during testing time, which introduces significant computational overhead during deployment. Furthermore, even for a simple keyword detection task, each microphone requires several minutes of unpaired training data. This method also inherits the weaknesses of CycleGAN, such as mapping completely unrelated samples in the source and target domains. Summary of the Invention

[0008] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0009] One example aspect of this disclosure relates to a computer-implemented method for microphone grid transfer (which can be used to modify a set of audio data that accurately represents a sound sample into audio data representing audio data output by a microphone after receiving a sound sample). The method may include obtaining input audio data collected by a first microphone by a computing system including one or more computing devices. The method may include processing the input audio data by the computing system using a machine learning microphone model to generate predicted target audio data for a target microphone different from the first microphone. Processing the input audio data by the computing system using a machine learning microphone model may include: determining first signal data by the computing system based at least in part on the audio data and machine learning impulse response data of the machine learning microphone model; and processing the first signal data by the computing system using a machine learning power-frequency model of the machine learning microphone model to generate second signal data, wherein the machine learning power-frequency model models one or more frequency band cutoffs.

[0010] A microphone model using machine learning can be employed, for example, to generate training data for training an adaptive system (machine learning system), such as for recognizing speech (e.g., recognizing at least one predefined word or phrase within speech). The method may include: performing the training operation, and optionally performing speech recognition on sound samples collected by the microphone using the trained speech recognition system. This aspect of the disclosure may alternatively be expressed as one or more non-transitory computer-readable media that share instructions that, when executed by one or more processors, cause the processor to perform the method. It may also be expressed as a computer system comprising one or more processors and one or more non-transitory computer-readable media. In some embodiments, processing input audio data by the computing system using a microphone model using machine learning may include: processing a noise signal by the computing system using a filter using machine learning to generate filtered noise data; and combining the filtered noise data with second signal data by the computing system to generate third signal data. Processing input audio data by the microphone model using machine learning may include processing the second or third signal by the computing system using a clipping model using machine learning to generate clipped data, wherein the clipped data may include predicted target audio data. The input audio data may include unamplified training examples, and the predicted target audio data may include augmented training examples. In some implementations, the method may include training an audio processing model using augmented training examples.

[0011] In some implementations, the impulse response data may include room reverberation impulse response data. The room reverberation impulse response data may be at least partially based on the room in which the microphone is located. The machine learning power-frequency model may apply one or more short-term Fourier transforms (which may be different Fourier transforms for different segments of the input audio data, wherein the segments can be obtained by dividing the input audio data in the time domain; the machine learning power-frequency model may multiply each frequency component of the Fourier transform with the corresponding component of the frequency domain data of the corresponding segment of the input audio data). In some implementations, one or more frequency band cutoffs may be at least partially based on power spectral density. The noise signal may include a white Gaussian noise signal. The machine learning amplitude limiting model may apply the learned maximum absolute value of the waveform. In some implementations, the machine learning amplitude limiting model may include a smoothed minimum function and a smoothed maximum function. The machine learning power-frequency model may include a sigmoid activation function.

[0012] Another exemplary aspect of this disclosure relates to a computer system including one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. These operations may include obtaining a training target audio signal. These operations may include processing the training target audio signal with a speech enhancement model to generate a source audio signal, wherein the source audio signal may include refined audio data. These operations may include processing the source audio signal with a machine learning microphone model to generate a predicted target audio signal. These operations may include evaluating a loss function that compares the predicted target audio signal with the training target audio signal. These operations may include modifying one or more values ​​of one or more parameters of the machine learning microphone model based on the loss function. The machine learning microphone model may be used, for example, to generate training data for training an adaptive system, such as for speech recognition. The recording medium may include instructions that cause the processor to perform the training operations, and optionally, to perform speech recognition operations on sound samples collected by the microphone using the trained speech recognition system. This aspect of the disclosure may alternatively be expressed as one or more non-transitory computer-readable media, or as a method including operations.

[0013] In some implementations, these operations may include using a microphone model employing machine learning to perform augmentation of a training dataset. These operations may include using the augmented training dataset to train a keyword recognition model. Obtaining the training target audio signal may include amplifying audio data at least in part based on simulated microphone augmentation to generate simulated target data. In some implementations, the simulated target data may include the training target audio signal. Evaluating the loss function may include: generating a predicted target spectrogram based on the predicted target audio signal, generating a training target spectrogram based on the training target audio signal, and comparing the predicted target spectrogram with the training target spectrogram.

[0014] Another example aspect of this disclosure relates to one or more non-transitory computer-readable media that commonly store instructions that, when executed by one or more processors, cause a computing system to perform operations. These operations may include: obtaining style transfer training data comprising a source audio signal and a training target audio signal associated with a target microphone style (e.g., an audio signal representing the signal output by the target microphone when it receives a sound signal; this may include any artifacts applied to the sound signal by the target microphone). These operations may include processing the source audio signal with a machine learning microphone model to generate a predicted target audio signal. The machine learning microphone model may include one or more of the following: a machine learning impulse response; a machine learning power-frequency model; a machine learning noise input filter; or a machine learning clipping function. These operations may include evaluating a loss function that compares the predicted target audio signal with the training target audio signal. These operations may include modifying one or more values ​​of one or more parameters of the machine learning microphone model based on the loss function. The machine learning microphone model may be used, for example, to generate training data for training an adaptive system to recognize speech. The recording medium may include instructions that cause a processor to perform the training operations, and optionally, operations to perform speech recognition on sound samples collected by the microphone using the trained speech recognition system. This aspect of the disclosure may alternatively be expressed as one or more non-transitory computer-readable media, or as a method including operation.

[0015] In some implementations, evaluating the loss function may include: generating a predicted target spectrogram based on the predicted target audio signal; generating a training target spectrogram based on the training target audio signal; and comparing the predicted target spectrogram with the training target spectrogram. These operations may include using a microphone model employing machine learning to augment the training dataset. In some implementations, the operations may include using the augmented training dataset to train a keyword recognition model.

[0016] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0017] These and other features, aspects, and advantages of the various embodiments of this disclosure will become better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and form part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0018] The specification provides a detailed discussion of embodiments for those skilled in the art, and references the accompanying drawings, in which:

[0019] Figure 1A A block diagram of an example computing system for performing microphone style transfer according to an example embodiment of the present disclosure is depicted.

[0020] Figure 1B A block diagram of an example computing device performing microphone style transfer according to an example embodiment of the present disclosure is depicted.

[0021] Figure 1C A block diagram of an example computing device performing microphone style transfer according to an example embodiment of the present disclosure is depicted.

[0022] Figure 2 A block diagram depicts an example generation of synthetic data for model training using a microphone model, according to an example embodiment of the present disclosure.

[0023] Figure 3 A block diagram depicts an example microphone model according to an example embodiment of the present disclosure.

[0024] Figure 4 A block diagram depicting example training of a microphone model according to an example embodiment of the present disclosure is provided.

[0025] Figure 5 A block diagram depicts an example microphone model according to an example embodiment of the present disclosure.

[0026] Figure 6 A flowchart is depicted for an example method for performing target audio data prediction according to an example embodiment of the present disclosure.

[0027] Figure 7 A flowchart depicts an example method for training a microphone model to perform machine learning according to an example embodiment of the present disclosure.

[0028] Figure 8 A flowchart depicts an example method for training a microphone model to perform machine learning according to an example embodiment of the present disclosure.

[0029] Figure 9 A block diagram depicts the parameters learned in an example according to an example embodiment of the present disclosure.

[0030] Figure 10 A block diagram depicts the model results of an example machine learning according to an example embodiment of the present disclosure.

[0031] The repeated reference numerals across multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0032] Overview

[0033] Exemplary embodiments of this disclosure relate to machine learning for microphone style transfer, to facilitate, for example, augmentation of audio data such as speech data, to improve the robustness of machine learning models trained on audio data. In particular, a key aspect of successfully deploying audio-based models "in the field" can be robustness to transformations introduced by heterogeneous microphones. Aspects of this disclosure achieve improved model robustness (e.g., to microphone variability) by generating augmented training data, wherein the training data can be augmented to reflect different potential microphone transformations. Specifically, in some embodiments, systems and methods are capable of performing one-shot microphone style transfer to learn microphone transformations only from a single data example from the microphone. Specifically, given only a few seconds of audio recorded by a target device, exemplary embodiments of this disclosure can identify transformations associated with the microphone and can use the learned transformations to synthesize audio as if the audio were recorded by that device. These systems and methods can successfully apply these transformations to real or other audio contained in the training data, and when used for data augmentation in downstream tasks, the transformations can significantly improve model robustness to microphone variability.

[0034] In particular, some exemplary implementations of this disclosure may be referred to as "MicAugment". MicAugment systems and methods can perform few-shot (e.g., single-shot) microphone style transfer. MicAugment systems and methods may include a machine learning microphone model, and in some instances, a speech enhancer model. The microphone model can learn a transformation that approximates microphone variability. For example, a microphone model can be trained to modify a source audio signal to produce a predicted target audio signal that sounds as if it were captured or recorded by a target microphone.

[0035] In some implementations, the microphone model can acquire and / or receive audio input data and can process and / or transform the audio input data using impulse response data, a power-frequency model, a filtering model, and / or a clipping model. Some or all of the impulse response data, power-frequency model, filtering model, and / or clipping model may include and / or apply machine learning parameter values ​​learned according to the training process described herein.

[0036] For example, these systems and methods may include acquiring input audio data collected and / or generated by a first microphone (e.g., a source). The input audio data may include speech data or another form of audio data. The input audio data can then be processed using a machine learning microphone model to generate predicted target audio data. The predicted target audio data may include audio data derived from the input audio data, which has been transformed to mimic the style of audio data generated by a second microphone (e.g., a target microphone). The first microphone and the target microphone may be different microphones. For example, the first microphone may be a dynamic microphone, and the target microphone may be a large-diaphragm condenser microphone, or vice versa. Additionally and / or alternatively, the first microphone and the target microphone may be the same type of microphone (e.g., a dynamic microphone, a large-diaphragm condenser microphone, a small-diaphragm condenser microphone, a ribbon microphone, etc.).

[0037] Processing input audio data using a machine learning microphone model can include determining first signal data based at least in part on the audio data and the machine learning impulse response data of the microphone model. The first signal data can be processed using a machine learning power-frequency model of the microphone model to generate or produce second signal data. The machine learning power-frequency model can be trained to model one or more frequency band cutouts.

[0038] In some implementations, the first step of a microphone model may include comparing (e.g., convolving) audio input and impulse response data. The impulse response data may be data collected on the impulse responses of a particular microphone or a set of microphones and / or may be learned as part of a learning process performed to learn the microphone model. This comparison may be used to generate first signal data. The first signal data can then be processed with a power-frequency model that models one or more frequency band cutoffs to generate second signal data. The power-frequency model may use power spectral density data and a Fourier transform (e.g., which includes or applies learned values) to generate the second signal data. For example, the Fourier transform may include a short-time Fourier transform.

[0039] White noise signals can be processed using a filtering model (e.g., using a learned filter) to generate filtered noise data. The white noise signal may include white Gaussian noise signals (e.g., with learned Gaussian parameters). The filtering model may include convolving the white noise signal with one or more truncated frequency bands and / or performing addition operations to add noise to the signal. For example, filtered noise data may be added to second signal data to generate third signal data. The final step in the microphone model may include processing the filtered data with a clipping model to generate clipped data. For example, the clipping model may apply a learned clipping threshold. The output of the microphone model may be a predicted target audio signal. For example, clipped data may be output as the predicted target audio signal.

[0040] In some implementations, training the microphone model may include processing paired audio samples, wherein the paired audio samples include source data and training target data. Specifically, the source data may be processed by the microphone model to predict the target data. The predicted target data may be used to generate a predicted target spectrogram. The training target data may be used to generate a training target spectrogram. In some implementations, the spectrogram may include a Mel spectrogram. The predicted target spectrogram and the training target spectrogram may be compared to evaluate a loss function. As an example, the loss function may include the mean absolute error of the comparison and / or other distance or difference measures. In response to the evaluation of the loss function, the parameters of the microphone model (e.g., impulse response, power-frequency model, filtering model, and / or clipping model) may be modified (e.g., via backpropagation of the loss function).

[0041] In some implementations, which may be referred to as operating in a “paired” setup, training versions of the source and target audio signals can be generated separately and both are available. For example, two demos of the same audio can be captured separately by two different microphones (e.g., a higher-quality microphone creating a “clean” source sample and a lower-quality microphone creating the training target audio signal). As described above, a microphone model can be trained to transform the source audio signal into a predicted target audio signal such that the predicted target audio signal matches the training target audio signal.

[0042] In other implementations, which may be referred to as operating in an “unpaired” setting (which is more common), only one of the source audio signal or the training audio signal may be available. In some of these cases, such as when only the training target signal is available, a speech enhancer model can be used to enhance the training audio signal to produce a “clean” source audio signal. For example, the speech enhancer model can invert or remove microphone transformations encountered in the target domain, thereby producing samples containing clean speech. After using the speech enhancer to generate the source audio signal based on the training target audio signal, the microphone model can be trained as described above—effectively learning to invert the speech enhancement. Thus, the microphone model and the speech enhancer model can operate in cascade to allow systems and methods to train the microphone model to output or perform transformations that modify source samples to sound as if the samples were recorded by the target microphone.

[0043] In some implementations, the speech enhancer model can use microphone impulse response, room reverberation impulse response, and noise segmentation to help perform blind deconvolution and denoising.

[0044] In other implementations, training audio signals can be synthesized by selecting different parameter values ​​of a microphone model from a set of candidate values ​​(e.g., values ​​observed from different real-world microphones) (e.g., randomly or pseudo-randomly). This creates a set of training target signals that mimic audio signals captured by different microphones. These training target signals can then be used as described above.

[0045] Therefore, in some implementations, microphone impulse response, room reverberation impulse response, and noise segments can be used to process or generate a synthetic dataset. The synthetic dataset can be split into mini-batches. Each mini-batch can be convolved with a randomly selected reverberation impulse response to generate a first output. The first output can then be convolved with a randomly selected microphone impulse response to generate a signal. The signal can then be divided by a frequency range, and the segmented data can be sampled relative to a threshold to generate a second output. Randomly selected noise samples can be added to the second output to generate a third output. The third output can then be clipped to generate final enhanced audio data. The final enhanced audio data can be used as source data for a microphone model.

[0046] Therefore, the example aspects of this disclosure provide a solution to the problem of transforming other audio samples, given a few seconds of audio recorded by a target microphone, into sounds as if they were recorded by the target microphone. To this end, focusing on speech data, this disclosure provides MicAugment, a machine learning method for few-shot (e.g., single-shot) microphone style transfer. Operating in the time domain, this method can be independent of downstream tasks and therefore has broad applicability. MicAugment can be used as a data augmentation method to improve the robustness of models to domain shifts due to microphone variability. Two key elements in achieving this are likely recent advances in speech enhancement and novel nonlinear microphone models that enable fast model identification through strong priors incorporating a pipeline for capturing microphone signals.

[0047] The systems and methods disclosed herein offer numerous technical effects and benefits. As an example, this disclosure provides a lightweight microphone model based on strong priors. Due to the lightweight nature of the model, it can be learned using only a few seconds of audio from the target device. As another example of technical effect, the systems and methods of this disclosure can significantly increase the robustness of the model to microphone variability when used to perform data augmentation. As yet another example of technical effect, the microphone model can be used in adversarial training to improve model robustness. A more robust model can provide consistent model performance when exposed to distributional shifts (e.g., microphone variability), thus avoiding the need to train a new model for each different setting (e.g., different microphones). Avoiding the need to train many different models can save computational resources such as processor usage, memory usage, and network bandwidth usage.

[0048] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0049] Example devices and systems

[0050] Figure 1A A block diagram of an example computing system 100 performing microphone style transfer according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0051] User computing device 102 can be any type of computing device, such as, for example, personal computing devices (e.g., laptops or desktop computers), mobile computing devices (e.g., smartphones or tablets), game consoles or controllers, wearable computing devices, embedded computing devices, or any other type of computing device. In some implementations, some of these devices may have limited processing and memory capacity, so it is important that they can operate with reduced requirements for processing operations or data storage.

[0052] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0053] In some implementations, the user computing device 102 may store or include one or more microphone models 120. For example, the microphone model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Reference Figures 2 to 5 Example microphone model 120 was discussed.

[0054] In some implementations, one or more microphone models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single microphone model 120 (e.g., performing parallel synthesis data generation across multiple instances of microphone styles).

[0055] More specifically, microphone models can be used to transfer microphone style to audio data to generate synthetic data that can be used to train other models, such as voice command models.

[0056] Additionally or alternatively, one or more microphone models 140 may be included in, stored in, or implemented by, the server computing system 130, which communicates with the user computing device 102 according to a client-server relationship. For example, the microphone model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a synthesized audio data generation service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0057] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components may include a microphone, a conventional keyboard, or other means that a user can use to provide user input.

[0058] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 executed by processor 132 to cause server computing system 130 to perform operations.

[0059] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0060] As described above, the server computing system 130 may store or otherwise include one or more machine learning microphone models 140. For example, model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. (Reference) Figures 2 to 4 Discuss example model 140.

[0061] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0062] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0063] The training computing system 150 may include a model trainer 160 that trains machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, error backpropagation. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., gradients based on the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update the parameters over multiple training iterations.

[0064] In some implementations, backpropagation of the execution error may include backpropagation through time truncation. The model trainer 160 may perform various generalization techniques (e.g., weight decay, loss of weight, etc.) to improve the generalization ability of the model being trained.

[0065] Specifically, model trainer 160 can train microphone models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, processing source data with the microphone model to generate processed source data. The processed source data can then be used to generate a predicted target spectrogram. The predicted target spectrogram can then be compared with the target spectrogram to evaluate a loss function. The loss function can then be used to modify the parameters of the microphone model.

[0066] In some implementations, training examples may be provided by the user computing device 102 if the user has already provided consent. Therefore, in such an implementation, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process may be referred to as a personalized model.

[0067] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or optical or magnetic media.

[0068] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication over network 180 can be carried over any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0069] The machine learning models described in this specification can be used in a variety of tasks, applications, and / or use cases.

[0070] In some implementations, the input to the machine learning model of this disclosure can be speech data. The machine learning model can process the speech data to generate an output. As another example, the machine learning model can process the speech data to generate a latent embedding output. As another example, the machine learning model can process the speech data to generate an encoded speech output (e.g., encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model can process the speech data to generate a higher quality speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model can process the speech data to generate a predicted output.

[0071] In some implementations, the input to the machine learning model of this disclosure can be latent encoded data. The machine learning model can process the latent encoded data to generate an output. As another example, the machine learning model can process the latent encoded data to generate a reconstructed output.

[0072] In some implementations, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate output. As another example, the machine learning model can process the sensor data to generate segmented output. As yet another example, the machine learning model can process the sensor data to generate visualization output.

[0073] Figure 1A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems may also be used. For example, in some embodiments, user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, model 120 may be trained and used locally on user computing device 102. In some such embodiments, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0074] Figure 1B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0075] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning models. For example, each application may include machine learning models. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0076] like Figure 1B As shown, each application can communicate with many other components of the computing device, such as, for example, one or more sensors, a scene manager, a device status component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0077] Figure 1C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0078] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0079] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 1C As shown, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by a central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of computing device 50.

[0080] The central intelligence layer can communicate with the central device data layer. The central device data layer can be the central data repository of computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a field manager, a device status component, and / or other components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0081] Example model layout

[0082] Figure 2 A block diagram of an example system 200 for generating augmented training data using a microphone model 204, according to an example embodiment of the present disclosure, is depicted. In some embodiments, the microphone model 204 is trained using initial training data 202. Once trained, the microphone model 204 can be used to augment data samples 206 to generate augmented training data, which can then be used to train another model 208.

[0083] therefore, Figure 2 The illustration depicts one implementation of generating amplified audio data for training a model. Microphone model 204 can be used to generate synthetic data that can be used to train other models 208. An example model that could benefit from synthetic data would be a speech recognition model, for example, that processes audio data to determine whether a keyword or command is present in the audio data. The microphone model can allow the generation of new amplified data with added noise or other microphone transformations corresponding to microphone variability.

[0084] exist Figure 2 In this illustration, an initial training dataset 202 exists. The dataset can be source data and target data from paired datasets. In some implementations, the dataset may include "unpaired" target data and source data generated by the speech enhancer model. In some examples, training data 202 may be real-world data collected "in the wild."

[0085] In this implementation 200, microphone model 204 obtains or acquires source data and target data from initial data source 202. Microphone model 204 processes the source data and target data to learn and generate microphone style transfer data. The generated microphone style transfer data may include the microphone style transferred to the target data from the source data.

[0086] Once trained, microphone model 204 can be used to generate augmented data based on one or more data samples from audio library 206 or other training datasets. The augmented data can then be used as training data for model 208. Model 208 can be any form of audio processing model, such as, in one example, a speech recognition model that processes audio data to determine whether keywords or phrases are present in the audio data. The augmented data can include keywords or phrases from a microphone-style data file containing audio augmented data transferred from the output of microphone model 204 or performed by microphone model 204. Training of model 208 can further employ content data indicating whether, for each item in the augmented data, the augmented data includes one or more words (e.g., keywords or phrases). This content data can be obtained from corresponding content data included in initial data 202, which indicates whether each item in the source or target data includes one or more words. Therefore, model 208 can be trained using a supervised learning algorithm.

[0087] Figure 3 An illustration depicts an example implementation of a microphone model. Microphone model 300 receives audio data x as input and outputs microphone type-transfer data y. In this implementation, microphone model 300 receives the audio data and convolves the input time-domain waveform of the audio data with microphone impulse response data 302. The microphone impulse response may, in some instances, also include the room reverberation impulse response. The resulting output y1, or first signal, can then be processed by component 304 of the microphone model, which models the microphone's bandwidth truncation based on power spectral density to generate a second signal y2. Component 304 may be a power-frequency model that may include a short-term Fourier transform.

[0088] In this embodiment, the microphone model includes a filter model 306, wherein a learned filter is used to convolve the white Gaussian noise signal. The resulting convolutional data can then be appended 308 to the second signal y2 to generate a third signal y3. The third signal can then be processed by a clipping model 310 to generate clipped data. The clipped data may be or may include microphone style transfer data y.

[0089] Figure 4An illustration depicts one implementation of a system for training a microphone model. The microphone model 404 can be trained using training data 402, which includes source data and target data. In this implementation, the source data is processed by the microphone model 404 to generate microphone style transfer data. In some implementations, one or more spectrograms 406 are generated for the target data. In some implementations, one or more spectrograms 408 are generated for the microphone style transfer data. The predicted target spectrogram generated based on the microphone style transfer data can be compared with the target spectrogram. This comparison can be used to evaluate a loss function 410.

[0090] The source data from training data 402 can be clean waveform data (e.g., data generated by a high-quality microphone and substantially free of microphone artifacts, or data in which artifacts generated by a particular microphone are removed or minimized), while the target data from training data 402 can be data collected by a microphone with microphone variability. Microphone model 404 can be configured to transfer the microphone style of the target data to the source data. Microphone model 404 can convolve the source waveform data with microphone impulse response data, and process the resulting data using a power-frequency model, a filtering model, and a clipping model to generate clipped data with the desired microphone variability. The clipped data can be used to generate a predicted target spectrogram at 408. In some implementations, a target spectrogram at 406 can be generated for comparison with a predicted target spectrogram. The comparison used to evaluate the loss function 410 can include determining the mean absolute error between the predicted target spectrogram and the target spectrogram.

[0091] In this implementation, one or more parameters of the microphone model 404 may be modified in response to the evaluation of the loss function. The parameters of the impulse response, power-frequency model, filter model, and / or limiting model may be modified in response to the comparison (e.g., to iteratively reduce the loss function).

[0092] Figure 5 An overview of an example machine learning microphone model system is provided. The trained microphone model 508 can generate learned augmentations 510 based on the target sample 502 (i.e., a term of sound data generated by the target microphone (e.g., a microphone with microphone variability)) and an augmented version 506 of the target sample generated by the speech augmentation network 504.

[0093] More specifically, Figure 5A block diagram of an example machine learning microphone model 508 according to an exemplary embodiment of the present disclosure is depicted. In some embodiments, the machine learning microphone model 508 is trained to receive a set of input data 502 describing a target sample from a target microphone, and as a result of receiving the input data 502, provides output data 510 including amplified audio data (e.g., audio data amplified to transform audio data to mimic the output from the target microphone). Thus, in some embodiments, the machine learning microphone model 508 may include an enhancer model 504 operable to perform speech enhancement techniques on the target sample 502 to generate enhanced samples.

[0094] In some embodiments, the systems and methods disclosed herein can obtain a target sample 502 describing a target microphone. The target sample 502 can be processed by an enhancer model 504 to generate enhanced samples 506. In some embodiments, the enhancer model may include a speech enhancer network. The target sample 502 and enhanced samples 506 can then be processed by a microphone model 508 to generate a learned amplified output 510, which may include audio data amplified to mimic audio data generated by the target microphone.

[0095] Figure 9 The diagram depicts the estimated power spectral density based on pairing data from the mobile device and parameters of a machine learning-based microphone model. More specifically, the first plot 902 depicts a power spectral density map, including corresponding lines for each of the source and target, which can be used to compare the differences between the source and target. The microphone model can then be trained to amplify the source audio data to transform it into data describing the target microphone (which can represent the audio data output by the target microphone). The second through fifth plots depict different learning parameters, including microphone frequency response parameter 904, frequency threshold parameter 906, noise frequency response parameter 908, and clipping threshold parameter 910.

[0096] Figure 10 The impact of the target audio length on the success rate of deceiving a source device identification model is depicted. The source device identification model is configured to receive audio samples of sound with the target audio length from a target microphone and output an indication of whether the audio sample was indeed generated by the target microphone. Specifically, Figure 10 A graph 1000 depicts paired setting data 106 and unpaired setting data 108. As conveyed in graph 1000, both paired setting data 106 and unpaired setting data 108 experience an increased success rate 102 in response to an increase in sequence length 104 up to approximately nine seconds. Therefore, the system and method can provide performance that increases with increasing sequence length, with a diminishing return around nine seconds of sequence length.

[0097] Example Method

[0098] Figure 6 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 6 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specific illustrated order or arrangement. The steps of method 600 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0099] At point 602, the computing system can obtain the input audio data collected by the first microphone.

[0100] At 604, the computational system can process the input audio data using a machine learning microphone model to generate predicted target audio data for a target microphone that differs from the first microphone. In some implementations, the input audio data may include unamplified training examples (i.e., audio data output by a microphone different from the target microphone (e.g., a high-quality microphone that outputs a substantially accurate representation of the received sound signal), and the predicted target audio data may include amplified training examples (i.e., modified audio data identifying the audio output by the target microphone when it receives the sound signal). Furthermore, the system can use the amplified training examples to train the audio processing model.

[0101] In some implementations, processing the input audio data with a machine learning microphone model to generate predicted target audio data for a target microphone that is different from the first microphone may include computational system executions 606 and 608.

[0102] In 606, the computing system may determine the first signal data based at least in part on the audio data and machine learning impulse response data of a microphone model. In some embodiments, the impulse response data may include room reverberation impulse response data. The room reverberation impulse response data may be based at least in part on the room in which the target microphone is located.

[0103] At 608, the computing system can process the first signal data using a machine learning power-frequency model of the microphone model to generate second signal data. In some implementations, the machine learning power-frequency model can model one or more frequency band cutoffs. The one or more frequency band cutoffs can be at least partially based on the power spectral density. Additionally and / or alternatively, the machine learning power-frequency model can apply one or more short-term Fourier transforms. The machine learning power-frequency model can include a sigmoid activation function.

[0104] In some implementations, processing input audio data using a machine learning microphone model may include: processing a noise signal with a machine learning filter to generate filtered noise data; and combining the filtered noise data with second signal data to generate third signal data. In some implementations, the noise signal may include a white Gaussian noise signal. The machine learning amplitude limiting model may apply the learned maximum absolute value of the waveform. In some implementations, the machine learning amplitude limiting model may include a smoothed minimum function and a smoothed maximum function.

[0105] Alternatively and / or additionally, processing the input audio data with a machine learning microphone model may include processing a second or third signal with a machine learning amplitude limiting model to generate amplitude limiting data, wherein the amplitude limiting data may include predicted target audio data.

[0106] Figure 7 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 7 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specific illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0107] At 702, the computing system can acquire the training target audio signal. Acquiring the training target audio signal may include acquiring the "raw audio" signal output by the target microphone, and / or may include amplifying the audio data at least partially based on analog microphone amplification to generate analog target data (which may be audio data representing the audio data output by the target microphone). The analog target data may be included in the training target audio signal.

[0108] At 704, the computational system can process the training target audio signal using a speech enhancement model to generate a source audio signal, which represents the audio signal output by a high-quality microphone. In some implementations, the source audio signal may include refined audio data (e.g., artifacts representing shortcomings of the target microphone can be removed).

[0109] At point 706, the computing system can use a microphone model developed through machine learning to process the source audio signal to generate a predicted target audio signal.

[0110] At point 708, the computational system can evaluate a loss function that compares the predicted target audio signal with the training target audio signal. In some implementations, evaluating the loss function may include generating a predicted target spectrogram based on the predicted target audio signal, generating a training target spectrogram based on the training target audio signal, and comparing the predicted target spectrogram with the training target spectrogram.

[0111] At 710, the computational system can modify one or more values ​​of one or more parameters of the microphone model of the machine learning based on the loss function (e.g., to reduce the loss function, for example, through a backpropagation algorithm).

[0112] In some implementations, the computing system may employ a machine learning microphone model to augment the training dataset. Alternatively and / or additionally, the computing system may use the augmented training dataset to train a keyword recognition model.

[0113] Figure 8 A flowchart depicts an example method performed according to an example embodiment of this disclosure. Although Figure 8 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specific illustrated order or arrangement. The steps of method 800 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0114] At point 802, the computational system can obtain style transfer training data, including the source audio signal and a training target audio signal associated with the target microphone style. The style transfer training data can instruct the target microphone how to distort the received audio signal when generating audio data based on the sound signal.

[0115] At 804, the computational system can process the source audio signal using a machine learning microphone model to generate a predicted target audio signal (e.g., audio data representing audio data output by the target microphone). The machine learning microphone model can include one or more of the following: a machine learning impulse response, a machine learning power-frequency model, a machine learning noise input filter, or a machine learning limiting function.

[0116] At 806, the computational system can evaluate a loss function that compares the predicted target audio signal with the training target audio signal. In some implementations, evaluating the loss function may include: generating a predicted target spectrogram based on the predicted target audio signal, generating a training target spectrogram based on the training target audio signal, and comparing the predicted target spectrogram with the training target spectrogram.

[0117] At 808, the computational system can modify one or more values ​​of one or more parameters of the microphone model of the machine learning based on the loss function (e.g., to reduce the loss function, for example, through backpropagation).

[0118] Additionally and / or alternatively, the computational system may employ a microphone model of machine learning to augment the training dataset. The augmented training dataset can then be stored in a training database for use in training other machine learning models or for retraining the current machine learning model.

[0119] In some implementations, the computing system can use an expanded training dataset to train a keyword recognition model.

[0120] Example Implementation

[0121] Deploying audio-based models “in the field” can be hampered by the robustness of transformations introduced by heterogeneous microphones. In some implementations, the systems and methods disclosed herein can perform one-shot microphone style transfer. Given only a few seconds of audio recorded by a target device, the system and methods can identify transformations associated with the microphone and can use the learned transformations to synthesize audio as if it were recorded by that device. The system and methods can include a machine learning-based microphone model that can apply transformations to real audio and can significantly improve the model’s robustness to microphone variability when used for data augmentation in downstream tasks.

[0122] In some implementations, these systems and methods can be used to address domain shifts in the development of robust models. Regarding robustness to microphone variability, performance is significantly impacted when inference is performed on audio captured by microphones different from those used to collect the training set. To mitigate the model's performance issues when receiving audio data from microphones different from the training microphone, systems and methods can use additive correction in the cepstral domain and multi-style training using a set of microphones. However, this correction can be limited if training data from multiple microphones is not available.

[0123] Therefore, systems and methods can address the problem of transforming other audio samples, given a few seconds of audio recorded by a target microphone, into sounds as if they were recorded by the target microphone, using a machine learning microphone model. These systems and methods are largely applicable to speech data and other audio data. In some implementations, systems and methods may include a machine learning microphone model for single-shot microphone style transfer. Operating in the time domain, systems and methods can be task-independent and therefore have broad applicability. Furthermore, systems and methods can leverage microphone models and speech enhancement techniques to achieve rapid model identification. Additionally and / or alternatively, the machine learning microphone model can be trained using strong priors from a captured microphone signal processing pipeline.

[0124] In some implementations, the system and method may include a lightweight microphone model based on strong priors, which can be trained using only a few seconds of audio from a target device. The computational operations and the data storage requirements for performing this operation are likely within the capabilities of a moderately powerful mobile computing device. The machine learning microphone model can output samples transformed by the learned microphone model, which can fool the source device into identifying the model. Furthermore, the system and method can be used for data augmentation to significantly improve the model's robustness to microphone variability. Alternatively and / or additionally, the microphone model can be used in adversarial training to further enhance model robustness.

[0125] The system and method may acquire training data including: i) a collection of speech samples (e.g., output from a high-quality microphone that substantially does not add artifacts to the received sound signal), which may be referred to as the source, and ii) short speech segments recorded by the microphone of interest, which may be referred to as the target. The system and method may assume that the source samples contain clean speech samples collected with a high-quality microphone having a flat frequency response and only slightly affected by background noise and room reverberation. In some implementations, the output of the system and method may include a transformation that modifies the source samples to sound as if they were recorded by the target microphone.

[0126] exist Figure 5 The document outlines an example system with a microphone model. A first building block could be a speech enhancement network 504. In some implementations, the enhancement network could include a wave-to-wave fully convolutional U-Net. The speech enhancement network 504 can successfully augment frequencies lost due to microphone filtering in noise removal and even in removing early room reverberation. Therefore, robustness to microphone transformations can be achieved with the speech enhancement network 504 by enhancing samples before feeding them to a model serving downstream tasks. However, running the speech enhancement network at inference time can introduce latency.

[0127] In some implementations, the output of the speech enhancer model may include one or more artifacts. Therefore, the system and method may further include a microphone model 508 that leverages strong priors from the microphone's signal processing pipeline. The task of the machine learning microphone model may be to identify transformations introduced by the microphone from sample pairs. Strong priors can be used to allow model identification given only a few seconds of audio, even in the presence of artifacts that may be introduced by the enhancement network. Once the transformations have been identified, they can be used for data augmentation during training to achieve robustness in the target domain.

[0128] Alternatively and / or additionally, the system and method may include learning transformations directly from target samples without relying on a speech enhancement network. In some implementations, the system and method may be trained using time-aligned source-target sample pairs.

[0129] In some implementations, the system and method can be trained to transform source samples to sound as if they were recorded by a target microphone. In some implementations, the microphone model can be evaluated via a series of downstream tasks (i.e., a supervised learning problem that receives the microphone style-transfer output as input).

[0130] The task of the microphone model can be in x 目标 =T(x 源 Under the assumption that ), based on a single pair (x) containing only a few seconds of speech. 源 x 目标 The transformation T is approximated by the microphone variability. To achieve this from a single, potentially short sample, the system and method can incorporate strong priors into modeling the microphone acquisition pipeline, while still being flexible enough to achieve convincing experimental results.

[0131] exist Figure 3 An example microphone model is depicted. As the first step in the pipeline, the input time-domain waveform is... Can be compared with microphone impulse response (IR) f m Perform convolution:

[0132] y1=f m *x. (1)

[0133] Therefore, with a slight misuse of the nomenclature, the synthesis of the microphone impulse response and the room impulse response can be expressed as f m Since the microphone model can handle variable-sized inputs on the order of thousands of samples, the system and method can perform all convolutions as multiplications in the frequency domain, with a time complexity of O(n log n).

[0134] As a second step, signal y1 can be passed through a component that models the microphone's frequency band truncation based on power spectral density (PSD): when the signal power in a frequency band falls below a learned threshold, the corresponding power may be further reduced in that band. Formally,

[0135] y2=STFT-1 STFT(y1)·σ(|STFT(y1)| 2 -t), (2)

[0136] Here, σ(·) is a sigmoid function, and t propagates over time, having a size corresponding to the STFT window length. Next, the system can sample a white Gaussian noise signal, which is then compared with the learned filter f. n Convolve and add to y2 to produce y3.

[0137]

[0138] The final component can model the microphone clipping effect by learning a threshold for the maximum permissible absolute value of the waveform. The system and method can achieve this via smoothed minimum and maximum functions, i.e.:

[0139] y=smoothmin(smoothmax(y3,-τ),τ), (4)

[0140] Where, smoothmax(a, b) = (ae a +be b ) / (e a +e b And smoothmax(a, b) = (ae) -a +be -b ) / (e -a +e -b Furthermore, the set of learnable parameters can be represented as θ = {f} m f n The resulting microphone transformation can be expressed as mic θ (·)

[0141] After obtaining only a few seconds of paired audio samples (x) 源 x 目标 After that, the microphone model parameters can be optimized. For training, the system and method can perform gradient descent on the target.

[0142] min θ l1(MEL(mic θ (x 源 MEL(x) 目标 (5)

[0143] Where l1(·,·) represents the mean absolute error, and MFL(·) computes the Mel spectrogram. It can be assumed that all signals are sampled at 16 kHz. The Mel spectrogram can be computed with a window length of 1024 samples (64 ms) and a jump length of 160 (10 ms) using 128 divisions. Optimization can be performed on the training dataset using a step size of 0.005 and 1000 iterations; due to the small number of parameters, this can be performed on a single GPU in less than a minute. For the microphone model, the system and method can include fixing the STFT window length to 2048 (128 ms) and the jump length to 160 in experiments. Both the speech enhancement network and the microphone model can operate on gain-normalized signals. For illustration, Figure 9 Example parameters of a microphone model learned from a single audio sample in the MOBIPHONE dataset are shown.

[0144] The estimation of the microphone model can assume the input audio x 源 This represents the clean, original waveform. Alternatively and / or additionally, the speech enhancement network can generate an approximation in the following manner. Enhance sample x by reversing the microphone transformations encountered in the target domain. 目标 This results in samples containing clean speech. Because x 目标 It may be affected by noise, microphone filtering and room reverberation, so the network's task may be to perform blind deconvolution and denoising.

[0145] Therefore, the training data for speech enhancement networks can encompass a wide range of possible transformations. In some implementations, the system and methods can create synthetic datasets. For example, clean reference audio might contain sentences from 56 native English speakers, broken into 1-second sequences. Each sequence can then be modified by applying transformations represented by our microphone model through random sampling. While the microphone model can capture complex transformations, its modular structure based on strong priors allows the system to individually adapt to different transformations. Figure 3 Each component presented in the sample is sampled. The task of the speech enhancement network can be to learn the inversion so that a transformation can be applied based on the transformed and clean reference sample.

[0146] In some implementations, the system and method may obtain a set of 70 microphone impulse responses from a training dataset, supplemented by a set of 200 bandpass filters with cutoff frequencies uniformly sampled from [50Hz, 150Hz] and [3kHz, 8kHz]. Next, the system and method may obtain a set of 1000 room reverberation impulse responses by running an image-source-based room simulator on rooms with diverse geometries. Finally, the system and method may involve extracting 10,000 noise segments from a speech command dataset by applying an extraction method (i.e., obtaining 1-second samples by finding short, low-energy segments of 100ms length and copying them by overlapping and adding (randomizing the phase), thus matching the length used during training).

[0147] Equipped with these sets, the system and method can create a parallel corpus by applying the following operations to each VCTK mini-batch: i) convolving clean samples with randomly selected room reverberation impulse responses; ii) convolving the result with randomly selected microphone impulse responses; iii) performing thresholding of the resulting signal using Equation (2) by dividing the frequency range into 8 equal buckets and randomly and uniformly sampling thresholds from the range [2,3] for each bucket; iv) adding randomly selected noise samples and adjusting the gain so that the resulting SNR is in the range [5dB, 30dB]; and finally, v) performing clipping using Equation (4), where τ is randomly selected between half and the maximum absolute time-domain signal value. Each of the preceding operations can be performed independently with probabilities of 0.8, 0.9, 0.6, 0.9, and 0.1, where these values ​​have been selected to ensure diversity of the resulting transform.

[0148] The architecture of the augmented network can include the same fully convolutional wave-to-wave U-Net as the pure audio version of the media training dataset. Additionally and / or alternatively, minimizing the loss function can involve optimizing a combination of adversarial loss and feature matching loss, where the features are extracted from the intermediate activations of the multi-scale discriminator.

[0149] This system and method can present a novel nonlinear microphone model that can learn from several seconds of paired audio. Based on the microphone model and speech enhancement network, the system and method can include a machine learning-based microphone model that can identify microphone transitions in unpaired settings. The microphone model coupled with adversarial training can be used to improve robustness without relying on data from the target device. The system and method can be evaluated on the quality of style transfer through downstream tasks, and the results show that the proposed method significantly outperforms existing baselines.

[0150] Example Experiment

[0151] To evaluate the quality of samples generated by example microphone models (e.g., MicAugment), the experimental system can utilize two distinct downstream tasks: i) spoofing the source device identification model and ii) evaluating the robustness of fully supervised models trained with different amplification strategies. While this evaluation approach may inherently depend on the model used in the downstream tasks, the evaluation reveals that the relative ranking of competing methods remains constant across different tasks.

[0152] The system could consider the problem of identifying mobile devices by the footprints left by their microphones, a topic of interest in audio forensics. In this experiment, the system could first train a fully supervised source device identification model. Then, given only a few seconds of audio from the target mobile device, the system could train MicAugment and apply it to clean samples, aiming to fool the source device identification model into believing they were taken from the target device.

[0153] For this task, the system can rely on a training dataset that records sentences from 12 male and 12 female speakers randomly selected from the TIMIT database with 21 mobile devices, resulting in 30 seconds of audio for each speaker and device. The dataset can specify the original sentences from TIMIT used in the recordings, and the system can create paired and aligned TIMIT-MOBIPHONE datasets based on these sentences. The joint dataset allows the system to evaluate our microphone model independently on real-world datasets.

[0154] In some implementations, the system can begin by training a source device identification network on a mobile phone. The system can split the dataset into training and test sets with non-overlapping sets of 16 and 8 speakers, respectively. Furthermore, there may be no overlap between sentences spoken in the training and test splits. For training, the audio can be split into one-second blocks. For testing, the system can report both the average accuracy on a one-second block and the accuracy when a majority vote is applied to each speaker and device. The model architecture can include a CNN (window length 25ms, jump length: 10ms, 64 mel channels) running on a log-mel spectrogram input. Additionally and / or alternatively, the network can include a sequence of six convolutional blocks, each block comprising separable convolutions along the time and frequency axes with a kernel size equal to 3, followed by ReLU nonlinearities interleaved with batch normalization. The number of channels in each block can be equal to [64, 128, 256, 256, 512, 512]. As the number of channels increases, max pooling with a stride of 2 can be applied along both time and frequency axes. The convolutional layer can be followed by a two-layer header: a fully connected layer with 256 outputs having ReLU nonlinearity; and a final linear layer with 20 outputs. The results in Table I convey a slight improvement over other techniques and confirm that the model generalizes well, thus producing perfect classification accuracy with a majority vote for unseen speakers and content.

[0155] More specifically, Table I presents the experimental results of device identification using CNN on the MOBIPHONE dataset.

[0156]

[0157] To establish a baseline, the system can sample a new set of 100 random speakers from TIMIT. It retrieves random sentences with an average length of 3 seconds for each speaker. The goal of the attack method is to propose transformations for each target device, such that when applied... Sometimes, transformed samples may be misclassified by the CNN as belonging to the target device. The system can report this. The average success rate across both devices and samples. Furthermore, the system can be designed as a baseline for both the lower and upper bounds of the MicAugment. The strongest baseline for establishing the upper bound of the success rate could be a white-box attack (i.e., an attack that has access to the trained source device identifier model and can backpropagate gradients through it). However, unlike common methods that perturb the input samples in the feature space, the system might adversarially optimize the microphone model to deceive the device detector. More formally, for the target device class c, the attacker's goal could be:

[0158]

[0159] Here, g(·) is the device detector CNN, and l(·,·) is the cross-entropy loss. The success of this method can also be used to validate the flexibility of the microphone model. By listening to adversarial examples, the system is able to verify that attacks do not alter the semantic content of the speech.

[0160] In some implementations, the system and method may include two baselines dependent on the TIMIT-MOBIPHONE dataset for pairing and alignment. To train these, the system may select 15 seconds of paired audio for each device, containing a sentence from five randomly selected speakers. The first baseline dependent on the paired data may be a spectral equalization (i.e., a calibration method based on the power spectral density (PSD) ratio estimated from the TIMIT and MOBIPHONE batches: this method adjusts...) The PSD is matched to the PSD of the samples from the target device, thus capturing only linear transformations. The second paired baseline can be a microphone transformation model learned from only 15 seconds of available paired data for each device. Furthermore, the evaluation can include a demonstration of MicAugment, which works in an unpaired setting and has access only to the target device's recording.

[0161] Table II presents the success rates of various style transfer methods in deceiving device identifier networks.

[0162] method Success rate White-box attacks 100.0±0.0 Spectrum equalization 34.4±1.3 Microphone model (supervised) 88.3±0.6 MicAugment 68.3±2.4

[0163] Additionally and / or alternatively, the system can obtain several insights into the quality of the transferred samples from the results in Table II. This table can convey information from five different randomly selected samples. The mean and standard deviation of the rates on batches and initializations with different random seeds. The success of the white-box attack demonstrates the flexibility of our model. This is further reinforced by the fact that in the paired setting, the microphone model learned a transformation to fool the device detector with a success rate of 88.3%, although no classification task was noticed at hand. Finally, when provided with only 15 seconds of audio samples from each target device, MicAugment achieved a success rate of 68.3%. The results of this study suggest that the 20% drop in accuracy compared to the paired setting may be due to the speech augmentation network, which faces real-world data transformations not encountered during training. Sets of source and target samples, as well as samples transformed with MicAugment, can be provided. The evaluation may include performing an ablation study on the impact of the length of audio data available from the target device on the success rate of the device identification model. Results are as follows: Figure 10As shown, peak performance can be achieved with 9 seconds of audio from the target device. It can be noted that, due to the voice enhancement network, the 20% difference between paired and unpaired settings may be constant across different target audio lengths.

[0164] Accuracy degradation in keyword detection systems due to microphone variability can be mitigated by using a microphone model (e.g., a MicAugment system). In some implementations, the evaluation of a MicAugment system may include: (1) a synthesis setting, wherein the test set is synthesized via a microphone model learned in the TIMIT-MOBIPHONE pairing setting in the previous section (e.g., the model may be used as a proxy for real microphone transformation); and (2) a real setting, wherein the test set is recaptured, audio clips are played through a computer speaker, and they are recorded using the built-in microphone.

[0165] Table III conveys the performance of different methods used to address microphone variability related to keyboard detection.

[0166]

[0167] Table III (first column) shows that accuracy drops from 95.7% to 91.8% or 88.3% depending on the test scenario. Results can be reported as averages over 5 random seeds. To obtain the upper limit of accuracy achievable during testing, the system can learn an oracle model that, during training, jointly utilizes the same amplifications applied at test time: (i) all transformations in the synthesis setting and (ii) a microphone model learned from 10 seconds of paired audio from a re-recorded test set. Table III (last column) shows that the model achieves a similar level of accuracy on a clean test set (95.4% vs. 95.7%), but is significantly more robust on the two modified test sets (94.5% vs. 91.8%, 91.0% vs. 88.3%).

[0168] The results show that the accuracy loss can be significantly recovered by augmenting the training data with MicAugment, which can learn from just 15 seconds of data from the target devices. In the synthetic setting, the system can use the MicAugment model learned from unpaired settings from each target device in previous experiments. In the real-world setting, the system can train MicAugment using 15-second clips from the training dataset recaptured by the computer. In this case, the system can use the dataset instead of voice commands to learn the microphone model, as the latter already contains significant degradation due to microphone recording. The MicAugment columns in Table III show the average accuracy achieved during training using the augmentation method, which is close to the accuracy level achievable with the oracle model.

[0169] Furthermore, this evaluation can be used to compare MicAugment with other methods that can operate in unpaired settings, namely spectral equalization and SpecAugment. For spectral equalization, the system can estimate... The power spectral density ratio is calculated between clean samples and 15 seconds of audio from each target device in the synthesis setup, or between 100 seconds of random samples from the training dataset and 15 seconds of re-recorded portions of the training dataset. For SpecAugment, the system can be configured to mask two slices in both the time and frequency domains, with each slice having up to 10 elements.

[0170] In summary, Table III conveys the following results: Spectral equalization recovers 25% of the accuracy loss in the synthetic setting and 35% in the real setting. SpecAugment may not notice the specific distortion applied to the test samples, but it still manages to recover 20% and 26% of the accuracy loss, respectively, while achieving quality accuracy on the original clean test set. In contrast, MicAugment recovers over 70% of the accuracy loss with only minor degradation on the clean test set.

[0171] Finally, the evaluation results show that the microphone model can be used to achieve robustness to a wide range of possible microphone variations without using any samples from the target device. This can include knowledge from supervised downstream tasks and corresponding labeled datasets.

[0172] While adversarial attacks can be performed in the image domain by perturbing pixel values, here the system can instead perform the attack by optimizing our microphone model, similar to a white-box attack on the source device identifier. Downstream tasks might involve working with the dataset... Learning classifier g φ The system can include the following optimization objectives for a K-class classification task:

[0173]

[0174] in,

[0175] Where l(·,·) is the cross-entropy loss, and C is the set of constraints on the microphone model parameters. In simpler terms, for each input sample, the system can detect when a cheat classifier misclassifies a sample as a randomly uniformly sampled class. Parameterization of the microphone model.

[0176] Set C can define constraints on the microphone model parameters, and this set can be carefully chosen. In the image domain, the adversarial constraint set can typically be defined as l in pixel space. p Norm constraint. Instead, the system can initialize θ0 in the internal optimization problem of equation (8) with an equivalent identity transformation and can perform a finite number of gradient descent steps to find θ. * (We used 50 steps in our experiments). This evaluation can be used to examine such adversarial microphone transformations and can convey that the speech content in the resulting transformed samples remains semantically unchanged and can be easily and correctly categorized by human listeners.

[0177] The systems and methods can be evaluated on synthetic and real-world settings from previous experiments, and results are reported in the adversarial amplification column of Table III.

[0178] Additional Publication

[0179] This article discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed in this article can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can run sequentially or in parallel.

[0180] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not limitation. Modifications, variations, and equivalents to such embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, as will be apparent to those skilled in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for microphone style transfer, the method comprising: The input audio data collected by the first microphone is obtained by a computing system including one or more computing devices; as well as The computing system processes the input audio data using a machine learning microphone model to generate target audio data for a target microphone that is different from the first microphone. The processing of the input audio data by the computing system using the machine learning microphone model includes: The first signal data is determined by the computing system based at least in part on the input audio data and the machine learning impulse response data of the microphone model of the machine learning; The computing system processes the first signal data using a machine learning power-frequency model of the microphone model to generate second signal data, wherein the machine learning power-frequency model adjusts the signal power of one or more frequency bands of the first signal data. The input audio data includes unamplified training examples, and the predicted target audio data includes amplified training examples, wherein the amplified training examples are generated as follows: Obtain the target audio signal associated with the target microphone; The target audio signal is processed using a speech enhancement model to generate enhanced samples; and The learned amplified output is generated by processing the target audio signal and the augmented sample using the microphone model of the machine learning method, and serves as a training example for the amplification.

2. The computer-implemented method according to claim 1, wherein, Processing the input audio data by the computing system using the machine learning microphone model further includes: The computing system processes the noise signal using machine learning filters to generate filtered noise data; and The computing system combines the filtered noise data with the second signal data to generate the third signal data.

3. The computer-implemented method according to claim 2, wherein, The noise signal includes white Gaussian noise.

4. The computer-implemented method according to claim 1, wherein, Processing the input audio data by the computing system using the machine learning microphone model further includes: The computing system processes the second signal using a machine learning-based amplitude limiting model to generate amplitude limiting data, wherein the amplitude limiting data includes the predicted target audio data.

5. The computer-implemented method according to claim 4, wherein, The machine learning amplitude-limiting model applies the maximum absolute value of waveform learning.

6. The computer-implemented method according to claim 4, wherein, The machine learning-based amplitude-limiting model includes a smoothed minimum function and a smoothed maximum function.

7. The computer-implemented method according to claim 1, wherein, The method further includes using the amplified training examples to train an audio processing model.

8. The computer-implemented method according to claim 1, wherein, The impulse response data includes room reverberation impulse response data, wherein the room reverberation impulse response data is at least partially based on the room in which the target microphone is located.

9. The computer-implemented method according to claim 1, wherein, The power-frequency model of the machine learning method applies one or more short-term Fourier transforms.

10. The computer-implemented method according to claim 1, wherein, The one or more frequency band cutoffs are based at least in part on power spectral density.

11. The computer-implemented method according to claim 1, wherein, The power-frequency model of the machine learning includes a sigmoid activation function.

12. A computing system, comprising: One or more processors; One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media jointly storing instructions, the instructions, when executed by the one or more processors, causing the computing system to perform operations, the operations including: Obtain the input audio data collected by the first microphone; and The input audio data is processed using a microphone model based on machine learning to generate target audio data that is different from the target microphone predicted by the first microphone. The processing of the input audio data using the microphone model derived from the machine learning method includes: The first signal data is determined by the computing system based at least in part on the input audio data and the machine learning impulse response data of the microphone model of the machine learning; The computing system processes the first signal data using a machine learning power-frequency model of the microphone model to generate second signal data, wherein the machine learning power-frequency model adjusts the signal power of one or more frequency bands of the first signal data. The input audio data includes unamplified training examples, and the predicted target audio data includes amplified training examples, wherein the amplified training examples are generated as follows: Obtain the target audio signal associated with the target microphone; The target audio signal is processed using a speech enhancement model to generate enhanced samples; and The learned amplified output is generated by processing the target audio signal and the augmented sample using the microphone model of the machine learning method, and serves as a training example for the amplification.

13. The system according to claim 12, wherein, Adjusting the signal power of the one or more frequency band cutoffs in the first signal data includes: The nonlinearity of the device audio processing pipeline, which is truncated by the frequency band, is modeled using the power frequency model of the microphone model of the machine learning.

14. The system according to claim 13, wherein, The modeling is performed based on power spectral density.

15. The system according to claim 12, wherein, Adjusting the signal power of the one or more frequency band cutoffs in the first signal data includes: Determine that the signal power in the frequency band is below the learned threshold; and The signal power in the frequency band is reduced because the signal power in the frequency band is lower than a learned threshold.

16. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions, the instructions, when executed by one or more processors, causing a computing system to perform operations, the operations including: Obtain the input audio data collected by the first microphone; as well as The input audio data is processed using a microphone model based on machine learning to generate target audio data that is different from the target microphone predicted by the first microphone. The processing of the input audio data using the microphone model derived from the machine learning method includes: The first signal data is determined by the computing system based at least in part on the input audio data and the machine learning impulse response data of the microphone model of the machine learning; The computing system processes the first signal data using a machine learning power-frequency model of the microphone model to generate second signal data, wherein the machine learning power-frequency model adjusts the signal power of one or more frequency bands of the first signal data. The input audio data includes unamplified training examples, and the predicted target audio data includes amplified training examples, wherein the amplified training examples are generated as follows: Obtain the target audio signal associated with the target microphone; The target audio signal is processed using a speech enhancement model to generate enhanced samples; and The learned amplified output is generated by processing the target audio signal and the augmented sample using the microphone model of the machine learning method, and serves as a training example for the amplification.

17. One or more non-transitory computer-readable media according to claim 16, wherein, The second signal data is further processed using a machine learning-based amplitude limiting model that models the microphone amplitude limiting effect.

18. One or more non-transitory computer-readable media according to claim 17, wherein, The microphone clipping effect is modeled by learning a threshold for the maximum permissible absolute value of the waveform.

19. One or more non-transitory computer-readable media according to claim 16, wherein, The microphone model includes the machine learning power frequency model, the machine learning impulse response model, the machine learning noise input filter, and the machine learning amplitude limiting model.

Images