Representation learning using notification masking for voice and other audio applications

Abstract

Some disclosed methods include receiving, by a control system configured to implement at least one neural network, input audio data and feature weights, and generating, by the control system and based at least in part on the input audio data and the feature weights, a potential spatial embedding. In some examples, the input audio data corresponds to an input mathematical space, and the potential spatial embedding may correspond to an unmasked portion of the input audio data. According to some examples, the potential space embedding may be a mathematical representation of the input audio data indicated by feature weights in a potential space that is a different mathematical space than the input mathematical space. In some examples, the feature weights may or may be based on masking data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 325,127 filed on March 29, 2022 and U.S. Provisional Application No. 63 / 490,212 filed on March 14, 2023, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to devices, systems and methods for representation learning, particularly speech representation learning (SRL). Background Art

[0004] Some methods, devices, and systems for SRLs are known. Although existing devices, systems, and methods for SRLs may provide benefits in certain circumstances, improved devices, systems, and methods would be desirable.

[0005] Symbols and nomenclature

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or group of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. Speakers can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, speaker signals can undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, statements about performing an operation on a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to indicate that the operation is performed directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the term "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and receives other XM inputs from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0010] Throughout this disclosure, including in the claims, the term "couple" is used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections.

[0011] As used herein, a "smart device" is an electronic device that is typically configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, near field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc., which can operate interactively and / or autonomously to some extent. Several notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, tablets and tablets, smart watches, smart wristbands, smart key chains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some attributes of ubiquitous computing, such as artificial intelligence.

[0012] Here, the expression "smart audio device" is used to refer to a smart device that is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of the virtual assistant functionality). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is designed to achieve a single purpose to a great extent or primarily. For example, although televisions can generally play (and are considered to be able to play) audio from program materials, in most cases, modern televisions run some operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device with one or more speakers and one or more microphones is typically configured to run local applications and / or services to directly use the one or more speakers and one or more microphones. Some single-purpose audio devices can be configured to be grouped together to enable audio to be played in a zone or user-configured zone.

[0013] A common type of multi-purpose audio device is an intelligent audio device, such as a "smart speaker", which can be configured to implement at least some aspects of the virtual assistant functionality, while other aspects of the virtual assistant functionality can be implemented by one or more other devices, such as one or more servers with which the multi-purpose audio device is configured to communicate. Such a multi-purpose audio device may be referred to as a "virtual assistant" here. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (other than a virtual assistant) for applications that are cloud-enabled in a sense or are not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, such as speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate via a network such as the Internet. Virtual assistants can sometimes work together, such as in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in a sense, with one virtual assistant (e.g., the virtual assistant that is most confident that it has heard the wake-up word) responding to the wake-up word. In some implementations, the connected virtual assistants can form a constellation, which can be managed by a master application, which can be (or implement) the virtual assistant.

[0014] Here, "wake-up word" is used in a broad sense to refer to any sound (e.g., speech uttered by a person, or some other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this case, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some cases, the word referred to as a "wake-up word" herein may include more than one word, such as a phrase.

[0015] Here, the expression "wake-up word detector" refers to a device configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model, or software including instructions for configuring the device to perform such operations. Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0016] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some cases a collection of one or more video signals, at least a portion of which are meant to be heard together. Examples include a selection of music selections, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from intelligent assistants, etc. In some cases, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same dialogue in more than one language. In this case, only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is intended to be reproduced at a time. Summary of the invention

[0017] At least some aspects of the present disclosure may be implemented by one or more methods. In some cases, the method may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some disclosed methods include receiving input audio data and feature weights by a control system configured to implement at least one neural network. Some such methods may include generating a latent space embedding by a control system and at least in part based on the input audio data and feature weights. In some examples, the input audio data may correspond to an input mathematical space, and the latent space embedding may be or may include a mathematical representation of the input audio data indicated by the feature weights in a latent space, the latent space being a mathematical space different from the input mathematical space. According to some examples, the input audio data may include an audio signal corresponding to speech.

[0018] In some examples, the feature weights may be or may include masking data. According to some examples, the masking data may be derived from estimates of noise and signal in the input audio data. In some examples, the latent space embedding may correspond to an unmasked portion of the input audio data.

[0019] In some examples, the control system can be configured to implement a convolutional neural network configured to perform weighted convolution. In some such examples, the weighted convolution can be based at least in part on feature weights.

[0020] According to some examples, generating the latent space embedding may include applying, by the control system, a context encoding process. In some examples, at least one neural network may have been trained to implement the context encoding process.

[0021] Some methods may involve applying a hidden representation process to a latent space embedding by a control system. The hidden representation process can produce a representation of the input audio data in the latent space. Some such methods may include applying a context decoding process to the representation of the input audio data in the latent space by the control system to produce a modified audio signal. Some methods may include generating a residual signal based at least in part on the modified audio signal and a version of the input audio data. In some examples, the version of the input audio data may include frequency partitioned audio data. According to some examples, the modified audio signal may be in the frequency domain, and wherein generating the residual signal may include transforming the frequency domain version of the residual signal to the time domain. In some examples, the input audio data and the feature weights may correspond to frequency bands.

[0022] According to some examples, the input audio data may have been pre-processed according to one or more audio data processing methods. In some such examples, the input audio data may have been pre-processed according to at least one of an echo cancellation process, an echo suppression process, a noise suppression process, or a beamforming process.

[0023] In some examples, at least one neural network may also have been trained to implement an attention-based masking process for generating the embedding. In some such examples, at least one of the attention-based masking process or the context encoding process may have been trained to identify and compensate for one or more errors in the masking process. In some examples, at least one neural network may have been trained based on the masking data and based on a contaminated audio signal output from the audio enhancement process. According to some examples, the audio enhancement process may include adding noise, adding reverberation, adding an audio signal corresponding to speech or other interfering audio sources, or a combination thereof. In some examples, training at least one neural network may include modulating masking data parameters. According to some examples, training at least one neural network may include maintaining a constant target audio signal during one or more time intervals of modulating the masking data parameters.

[0024] In some examples, the control system can be configured for speech representation learning (SRL). In some such examples, at least one neural network can include an SRL encoder. According to some examples, the SRL encoder can be or can include a convolutional encoder. In some examples, the convolutional encoder can include a partial convolutional layer.

[0025] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include storage devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented by one or more non-transitory media having software stored thereon.

[0026] At least some aspects of the present disclosure can be implemented by a device. For example, one or more devices (e.g., a system including one or more devices) can at least partially perform the method disclosed herein. In some embodiments, the device is or includes an audio processing system with an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components or combinations thereof. The control system may be configured to implement some or all of the methods disclosed herein.

[0027] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The same reference numbers and names in different drawings represent the same elements.

[0029] Figure 1A is a block diagram illustrating an example of components of a device capable of implementing various aspects of the present invention.

[0030] Figure 1B An example of an audio signal enhancement chain is shown.

[0031] Figure 2 Alternative examples of audio signal enhancement chains according to some disclosed implementations are shown.

[0032] Figure 3 Example blocks of a masquerade module according to some disclosed implementations are shown.

[0033] Figure 4 Example blocks of a camouflage module according to some alternative implementations are shown.

[0034] Figure 5 Example blocks of a process for training a camouflage module according to some implementations are shown.

[0035] Fig. 6A and 6B Example blocks of a camouflage module according to some alternative implementations are shown.

[0036] Figure 7 is a flow chart outlining one example of the disclosed method. DETAILED DESCRIPTION

[0037] Using voice as a user interface has become a convenient medium for human-computer communication. However, the technology that can provide users with a seamless experience in this human-computer interaction seems to be in its early stages of development.

[0038] The advent of machine learning (ML) and its advancements in areas such as audio processing (including but not limited to speech processing) provides the opportunity to represent audio data in a so-called "latent space" where high-level properties or unique features of the audio signal can be automatically derived from the data. Representations in latent spaces can be used to enable or improve a variety of use case applications, including but not limited to sound event classification, speaker identification, and automatic speech recognition.

[0039] Speech and audio representation learning that relies on self-supervised training has been shown to be effective in generating high-level information from input audio data, thereby capturing different properties of the input audio data. Some of these properties may be domain-independent.

[0040] Contextual awareness has proven to be important for inferential analysis of devices implementing automatic speech recognition and other speech-related functions during the speech capture and decision-making stages. Contextual speech representation learning (SRL)—where high-level information and unique properties of audio data are captured in an embedding space—can be used to infer information from the context learned holistically. This can be particularly important in situations where the input audio data corresponding to speech is masked or highly contaminated by environmental artifacts such as noise, echo, other interfering speech, etc.

[0041] Generative SRL is a method that learns this unique property of input audio data by finding a high-level representation that can be used to regenerate the signal. In this method, the input audio data or input features extracted from the input audio data are randomly masked, and a neural network is trained to predict high-level representations of these masked regions using neighboring frames of the context.

[0042] The present disclosure provides examples of how to use context-generated SRLs and feature weights to recover a target audio signal (e.g., an audio signal corresponding to a user's voice) from a contaminated input audio signal. In some examples, the feature weights may be or may include masking data provided by using an echo canceller, a noise suppressor, or a priori scene analysis. The masking data is derived from an estimate of the signal and noise in the input audio data. Thus, the masking data may indicate which portions of the target signal may be masked due to environmental artifacts, and which portions of the target signal may be intact and therefore include relatively more useful information. According to some examples, the masking data may be used to distort the signal, and the context-generated SRL may predict the audio content, with the goal of recovering the target signal.

[0043] Some disclosed implementations involve self-supervised learning techniques that combine prior knowledge of masking data with SRL contextual capabilities to create a desired target signal. The masking data may indicate which parts of the audio data—such as which intervals or bands—may be signals, and which parts may be noise.

[0044] Thus, some disclosed implementations may be directed to improving the quality of the generated audio signal. Alternatively or additionally, some disclosed implementations may create SRL embeddings that do not represent undesirable artifacts present in the input audio signal. Some such SRL embeddings may represent high-level properties of the input audio data that are not affected by environmental and acoustic artifacts. SRL embeddings generated according to some disclosed methods may improve various learning tasks and downstream audio processing tasks, including but not limited to source separation, speech enhancement, speaker binarization, speech recognition, noise suppression, echo suppression, and speaker recognition.

[0045] Figure 1A is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present invention. As with other figures provided herein, Figure 1A The types, quantities and arrangements of the elements shown in are provided as examples only. Other implementations may include more, less and / or different types, quantities and arrangements of elements. According to some examples, device 100 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 100 may be or may include one or more components of an office workstation, one or more components of a home entertainment system, etc. For example, device 100 may be a laptop computer, a tablet device, a mobile device (e.g., a cellular phone), an intelligent home hub, a television, or other types of equipment.

[0046] According to some alternative implementations, the apparatus 100 may be or include a server. In some such examples, the apparatus 100 may be or include an encoder. In some examples, the apparatus 100 may be or include a decoder. Thus, in some cases, the apparatus 100 may be a device configured for use in an environment such as a home environment, while in other cases, the apparatus 100 may be a device configured for use in the "cloud", such as a server.

[0047] In this example, the device 100 includes an interface system 105 and a control system 110. In some implementations, the interface system 105 can be configured to communicate with one or more other devices in the environment. In some examples, the environment can be a home environment. In other examples, the environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some implementations, the interface system 105 can be configured to exchange control information and associated data with other devices in the environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 100.

[0048] In some implementations, the interface system 105 can be configured to receive or provide a content stream. In some examples, the content stream can include video data and audio data corresponding to the video data. The audio data can include, but is not limited to, an audio signal. In some cases, the audio data can include spatial data, such as channel data and / or spatial metadata. For example, metadata can be provided by a device referred to herein as an "encoder".

[0049] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, a gesture sensor system, or a combination thereof. Thus, although in Figure 1A Some of these devices are shown separately in , but in some examples, these devices may correspond to aspects of interface system 105.

[0050] In some examples, the interface system 105 may include a control system 110 and a memory system (eg, Figure 1A Alternatively or additionally, in some cases, the control system 110 may include a storage system. In some implementations, the interface system 105 may be configured to receive input from one or more microphones in the environment.

[0051] The control system 110 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0052] In some implementations, the control system 110 may reside in more than one device. For example, in some implementations, a portion of the control system 110 may reside in a device within one of the environments mentioned here, while another portion of the control system 110 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet), etc. In other examples, a portion of the control system 110 may reside in a device within one of the environments described here, while another portion of the control system 110 may reside in one or more other devices in the environment. For example, control system functionality may be shared by a coordination device (e.g., a device that may be referred to as a smart home hub here) and one or more other devices in the environment. In other examples, a portion of the control system 110 may reside in a device (e.g., a server) that implements a cloud-based service, while another portion of the control system 110 may reside in another device (e.g., another server, a storage device, etc.) that implements a cloud-based service. In some examples, the interface system 105 may also reside in more than one device.

[0053] In some implementations, the control system 110 may be configured to at least partially perform the method disclosed herein. According to some examples, the control system 110 may be configured to receive input audio data and feature weights. In some examples, the control system 110 may be configured to generate embedding based at least in part on the input audio data and feature weights. According to some examples, the control system 110 may be configured to apply a context encoding process to embedding to generate a latent space embedding in a latent space. In some examples, the control system 110 may be configured to apply a hidden representation process to a latent space embedding to generate a representation of the input audio data in the latent space. In some examples, the feature weights may be or may include masking data derived from an estimate of the signal and noise in the input audio data.

[0054] As described elsewhere herein, the control system 110 may be located in a single device or in multiple devices, depending on the particular implementation. In some examples, all of the foregoing processes may be performed by the same device. In some alternative examples, the foregoing processes may be performed by two or more devices. For example, the embedding may be generated by one device, and the context encoding process may be performed by one or more other devices, such as by one or more servers configured to implement a cloud-based service.

[0055] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 1A The optional memory system 115 shown and / or the control system 110. Thus, various innovative aspects of the subject matter described in the present disclosure can be implemented in one or more non-transitory media having software stored thereon. The software can, for example, include instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software can, for example, be executed by a control system (e.g. Figure 1A The system may be executed by one or more components of the control system 110).

[0056] In some examples, apparatus 100 may include Figure 1A Optional microphone system 120 shown. Optional microphone system 120 may include one or more microphones. According to some examples, optional microphone system 120 may include a microphone array. In some examples, the microphone array may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, for example, according to instructions from control system 110. In some cases, the microphone array may be configured for receiving side beamforming, for example, according to instructions from control system 110. In some implementations, one or more microphones may be part of or associated with another device, such as a speaker of a speaker system, an intelligent audio device, etc. In some examples, device 100 may not include microphone system 120. However, in some such implementations, device 100 may still be configured to receive microphone data corresponding to one or more microphones in one environment or corresponding to one or more microphones in another environment via interface system 110. In some such implementations, a cloud-based implementation of device 100 may be configured to receive microphone data obtained in one or more microphones in an environment or metadata corresponding to microphone data via interface system 110.

[0057] According to some implementations, the apparatus 100 may include Figure 1A Optional speaker system 125 is shown. Optional speaker system 125 may include one or more loudspeakers, which may also be referred to herein as “speakers” or, more generally, “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), device 100 may not include speaker system 125.

[0058] In some implementations, the apparatus 100 may include Figure 1AOptional sensor system 130 is shown. Optional sensor system 130 may include one or more touch sensors, gesture sensors, motion detectors, cameras, eye tracking devices, or combinations thereof. In some implementations, one or more cameras may include one or more stand-alone cameras. In some examples, one or more cameras, eye trackers, etc. of optional sensor system 130 may be located in a television, a mobile phone, a smart speaker, or a combination thereof. In some examples, device 100 may not include sensor system 130. However, in some such implementations, device 100 may still be configured to receive sensor data from one or more sensors (e.g., cameras, eye trackers, etc.) residing in or on other devices in an environment via interface system 110.

[0059] In some implementations, the apparatus 100 may include Figure 1A Optional display system 135 shown. Optional display system 135 may include one or more displays, such as one or more light emitting diode (LED) displays. In some cases, optional display system 135 may include one or more organic light emitting diode (OLED) displays. In some examples, optional display system 135 may include one or more displays of a television, a laptop, a mobile device, a smart audio device, or another type of device. In some examples where apparatus 100 includes display system 135, sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 135. According to some such implementations, control system 110 may be configured to control display system 135 to present one or more graphical user interfaces (GUIs).

[0060] According to some such examples, the apparatus 100 may be or may include an intelligent audio device, such as a smart speaker. In some such implementations, the apparatus 100 may be or may include a wake-up word detector. For example, the apparatus 100 may be configured to (at least in part) implement a virtual assistant.

[0061] Figure 1B An example of an audio signal enhancement chain is shown. As with other figures provided herein, Figure 1B The types, quantities, and arrangements of the elements shown in the drawings are provided only as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements.

[0062] According to this example, Figure 1B As shown by reference Figure 1A A block diagram of an audio signal enhancement chain 150 is depicted for an example implementation of the control system 110. As described elsewhere herein, in some cases, the control system 110 may reside in more than one device.

[0063] In this example, the input audio data 101 is or may be contaminated by ambient noise, one or more interfering noise sources, or a combination thereof. In some examples, the input audio data 101 may be microphone data, in other words, a microphone signal corresponding to sound captured by one or more microphones, including an audio signal corresponding to a user's voice.

[0064] According to this example, the transform block 102 is configured to transform the time domain signal into the frequency domain.The transform block 102 may, for example, be configured to transform the time domain signal of the input audio data 101 into a frequency interval, for example by means of a Fast Fourier Transform.

[0065] For various use cases where the received audio data is or may be contaminated by noise or echo, the pre-processing of the input audio signal is an important process. The pre-processing block 104 can be configured to improve the signal-to-noise ratio (SNR) between the target audio signal (e.g., the audio signal of the input audio data 101 corresponding to the user's voice) and the interfering audio signal (e.g., echo or noise). As used herein, the term "echo" refers to the sound played back by the speaker in the environment. Therefore, depending on the specific implementation, the pre-processing block 104 can provide various types of functions. In this example, the pre-processing block 104 is configured as an acoustic echo canceller (AEC). In other examples, the pre-processing block 104 can be configured for echo suppression, noise suppression or other audio pre-processing functions.

[0066] In this example, the reference signal 145 is an echo or interference signal corresponding to audio being played back by a nearby speaker. According to this example, the reference signal 145 is input to the pre-processing block 104 and the masking estimation block 109. In this instance, the pre-processing block 104 generates pre-processed audio data 106 based on the transformed input audio signal 103 and the reference signal 145. According to this example, the pre-processing block 104 is configured to generate the pre-processed audio data 106, which improves the SNR between the target audio signal and the echo corresponding to the reference signal 145.

[0067] According to this example, the transform block 107 is configured to transform the frequency intervals of the preprocessed audio data 106 into a smaller number of frequency bands of the preprocessed audio data 108. In some examples, the transform block 107 may be configured to transform the frequency intervals of the preprocessed audio data 106 into frequency bands of the preprocessed audio data 108 that take into account human auditory characteristics, such as Mel-spaced frequency bands. In other examples, the transform block 107 may be configured to transform the frequency intervals of the preprocessed audio data 106 into other types of frequency bands, such as logarithmically spaced frequency bands.

[0068] In this example, the masking estimator 109 is configured to output masking data 151 based on the preprocessed audio data 108 and the reference signal 145. In some examples, the reference signal 145 input to the masking estimator 109 can be transformed by the masking estimator 109 itself or by another block of the audio signal enhancement chain 150 into a frequency band corresponding to the frequency band of the preprocessed audio data 108. The masking data 151 can be derived, for example, from an estimate of the signal and noise in the input audio data. For example, the masking estimator 109 can be configured to determine the masking data 151 by assigning a value to each of a plurality of frequency bands corresponding to the frequency band of the preprocessed audio data 108. These values ​​can indicate which frequency bands of the preprocessed audio data 108 are relatively more likely or relatively less likely to correspond to or include a target audio signal such as a user's voice. In other words, these values ​​can indicate which frequency bands of the preprocessed audio data 108 are relatively more trustworthy, and which frequency bands have been masked by interference signals. In this example, the known interference signal is the reference signal 145. In some such examples, these values ​​may range from 0 to 1, where 0 indicates an estimate of 100% noise and 1 indicates an estimate of 100% signal.

[0069] According to this example, the suppressor block 160 is configured to attenuate the portion (in this example, the frequency band) of the pre-processed audio data 108 indicated by the masking data 151 to be contaminated by the interfering signal. In this example, the suppressor block 160 is configured to output the corresponding frequency band gain 111 to achieve the attenuation determined by the suppressor block 160.

[0070] In this example, the inverse transform block 112 is configured to transform the frequency band gains 111 into frequency bin gains 113. According to this example, the frequency bins of the frequency bin gains 113 correspond to the frequency bins of the pre-processed audio data 106.

[0071] According to this example, the multiplication block 114 is configured to apply the frequency bin gain 113 to the preprocessed audio data 106 output by the preprocessing block 104 to generate frequency domain output audio data 155. In this example, the inverse transform block 116 is configured to transform the frequency domain output audio data 155 into time domain output audio data 117. According to this example, the time domain output audio data 117 has a relatively higher SNR than the input audio data 101. Therefore, the target audio signal (e.g., an audio signal corresponding to the voice of a specific person) can be enhanced in the time domain output audio data 117 compared to the input audio data 101. (As used herein, the terms "audio data", "audio signal" and "multiple audio signals" can be used interchangeably. For example, interfering audio data from a single source (e.g., corresponding to a second person speaking in an environment) can be referred to as an "interfering audio signal" in the singular, or as "multiple interfering audio signals" in the plural)

[0072] However, even after the above-described operations of the audio signal enhancement chain 150 have been performed, a certain level of interfering audio signals will often remain in the time domain output audio data 117. Due in part to the temporal variations of the target audio signal and the interfering audio signals, some portions (e.g., some frequency bands) of the target audio signal may be obscured by high-energy interfering audio signals, while other portions may not be obscured. In some cases, the interfering audio signals may be so loud that suppressing these interfering audio signals may result in significant distortion of the target audio signal in the masked portion of the target signal. Some aspects of the present disclosure address the above-described problems of previously deployed methods by implementing what may be referred to herein as a "camouflage" module.

[0073] Figure 2 An alternative example of an audio signal enhancement chain according to some disclosed implementations is shown. As with other figures provided herein, Figure 2 The types, quantities, and arrangements of the elements shown in the drawings are provided only as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements.

[0074] According to this example, Figure 2 As shown by reference Figure 1A 1 is a block diagram of an audio signal enhancement chain 200 that depicts an example implementation of the control system 110. As described elsewhere herein, in some cases, the control system 110 may reside in more than one device.

[0075] In this example, except for Figure 2 In addition to those noted in the description of , elements 101, 102, 103, 104, 106, 107, 108, 109, 111, 112, 113, 114, 116, 117, 145, 150, and 155 may be those referred to above. Figure 1B Therefore, the description of elements 101, 102, 103, 104, 106, 107, 108, 109, 111, 112, 113, 114, 116, 117, 145, 150, and 155 will not be repeated here.

[0076] However, Figure 2 The illustrated example includes the newly disclosed camouflage module 201. In some examples, the camouflage module 201 can be configured to use the SRL and masking information to extract information about the target signal from those portions that are not masked or only lightly masked to ensure that after processing, the interfering signal is suppressed by only the correct amount, thereby producing a less distorted result.

[0077] In this example, the process of estimating the interfering signal energy (eg, by masking estimator 109 or a similar block) may be substantially as described with reference to Figure 1B When determining a representation of a partially masked underlying target signal, the corresponding feature weights (in this example, masking data) can inform the disguise module 201 which portions of the pre-processed audio data 108 most likely correspond to the target signal.

[0078] More generally, in some examples, the camouflage module 201 can be configured to generate embeddings based at least in part on the input audio data and feature weights. In some examples, the input audio data can be preprocessed audio data, such as the preprocessed audio data 108 of Figures 1 and 2. However, in some examples, the input audio data may not be preprocessed. In some examples, the feature weights can be masked data output by a masking estimator 109 or similar block. The masked data can be derived from an estimate of the signal and noise in the input audio data. The embedding can correspond to an unmasked portion of the input audio data.

[0079] According to some examples, the disguise module 201 can be configured to apply a context encoding process to the embedding to generate a latent space embedding in the latent space. In some examples, the context encoding process can be Figure 2 The camouflage module 201 may be configured to apply a hidden representation process to a latent space embedding to generate a representation of the input audio data in the latent space.

[0080] In some examples, the disguise module 201 can be configured to apply the context decoding process to the representation of the input audio data in the latent space to produce a modified audio signal, or to produce an output from which the modified audio signal can be produced. An example of an output from which the modified audio signal can be produced is Figure 2The frequency band gain 111 generated by the disguise module 201 in the example of FIG. 1 is shown in FIG. 1 . Thus, in this example, the output of the disguise module 201 is or includes the following output, from which a modified audio signal can be generated. In an alternative example, the output of the disguise module 201 can be or can include a modified audio signal.

[0081] In some examples, the camouflage module 201 can be trained on an audio signal that has been masked. In some cases, the makeup module 201 can be trained on a synthesized mask. According to some examples, the mask may have been synthesized independently of the target application (e.g., echo suppression or noise suppression). In some examples, the camouflage module 201 can be trained to be robust to various errors in the masking estimate. After such training, the camouflage module 201 can be used to de-mask the underlying target signal based on information from various classical signal processing or neural network-based methods for interference estimation, and for applications such as but not limited to echo suppression, noise suppression, beamforming, and source separation. In some examples, the camouflage module 201 can be used in a signal processing chain that derives a masking estimate from a combination of two or more such use cases.

[0082] Figure 3 An example block of a camouflage module according to some disclosed implementations is shown. As with other figures provided herein, Figure 3 The types, quantities, and arrangements of the elements shown in are provided merely as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements. For example, in some alternative implementations, camouflage module 201 may include blocks 301 and 303, but not block 305. In some such implementations, another block of the audio signal processing chain may include block 305.

[0083] According to this example, the camouflage module 201 includes the following elements:

[0084] 301: a speech representation learning (SRL) encoder block configured to map contaminated input audio data 308 and masking data 151 to a high-level representation of the audio that is domain-independent and robust to acoustic perturbations;

[0085] 302: Outputs of different layers of the SRL encoder block 301, which may be referred to as latent space embeddings 302. In some examples, the SRL encoder block 301 may be implemented by a neural network. In some such examples, different latent space embeddings 302 may be output from different layers of the neural network;

[0086] 303: a hidden representation block configured to construct a high-level representation 304 of the input audio data 308 in a latent space using the latent space embedding 302;

[0087] 304: high-level representation or code of input audio data 308;

[0088] 305: a speech representation learning (SRL) decoder block configured to transform the high-level representation 304 to the domain of the contaminated input audio data 308 so as to enhance the contaminated input audio data 308;

[0089] 111: Enhanced audio signal, or data from which an enhanced audio signal can be constructed from the contaminated input audio data 308, depending on the specific implementation.

[0090] In some examples, the hidden representation block 303 may apply a merging process, such as a pooling process, to the multiple latent space embeddings 302 to produce a single high-level representation 304. In some examples, the hidden representation block 303 may produce a high-level representation 304 that is a lower dimensional representation of the multiple latent space embeddings 302. According to some examples, the hidden representation block 303 may produce a single high-level representation 304 according to one or more averaging processes. In some examples, the hidden representation block 303 may implement one or more attention-based averaging processes. In some such examples, the hidden representation block 303 may produce a single high-level representation 304 according to at least one time-based averaging process, which is, for example, a process that operates on the latent space embeddings 302 corresponding to multiple frames of input data.

[0091] Figure 4 An example block diagram of a camouflage module according to some alternative implementations is shown. As with other figures provided herein, Figure 4 The types, quantities, and arrangements of the elements shown in are provided merely as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements. For example, in some alternative implementations, camouflage module 201 may include blocks 301 and 303, but not block 305. In some such implementations, another block of the audio signal processing chain may include block 305. According to some alternative implementations, camouflage module 201 may not include optional attention-based masking block 401.

[0092] In this example, the disguise module 201 is composed of Figure 1A According to this example, the disguise module 201 includes a reference Figure 3 An example of an SRL encoder block 301, a hidden representation block 303, and an SRL decoder block 305 is described. Figure 4 Otherwise, the hidden representation block 303 and the SRL decoder block 305 can be as described in reference Figure 3 Works as described. Figure 4Additional details of the SRL encoder block 301 are shown according to one example.

[0093] According to this example, the disguise module 201 is shown as outputting target audio data 411, which is a "clean" version of the contaminated input audio data 308. Figure 4 In the example shown, the y-axis of the graph associated with masking data 151, polluted input audio data 308, and target audio data 411 represents frequency, and the x-axis represents time. The graph associated with target audio data 411 indicates a "clean", unpolluted audio signal, which corresponds to the voice of a specific person in this example. The graph associated with polluted input audio data 308 indicates a combination of an unpolluted audio signal and an interfering audio signal, which corresponds to the voice of another person in this example. In other examples, the interfering audio signal may correspond to audio data played back by a nearby speaker, or to some other types of interfering audio signals. In the graph associated with masking data 151, the white area represents noise. According to this example, masking data 151 is derived from an estimate of a signal (unpolluted audio signal) and noise (interfering audio signal) in polluted input audio data 308. Therefore, the white area of ​​masking data 151 corresponds to the interfering audio signal of polluted input audio data 308.

[0094] In this example, the SRL encoder block 301 includes an optional attention-based masking block 401 and a context encoder 403. In some examples, the control system 110 includes a neural network that has been trained to implement the context encoder 403. The neural network can be, for example, a transformer neural network or a conformer neural network. Alternatively or additionally, the control system 110 can include a neural network that has been trained to implement the attention-based masking block 401.

[0095] According to this example, the attention-based masking block 401 is configured to generate an embedding 402 based at least in part on the contaminated input audio data 308 and feature weights. In this example, the feature weights are the masking data 151.

[0096] In this example, the attention-based masking block 401 is configured to generate an embedding 402 according to an attention-based masking process informed by the masking data 151. In some such examples, the attention-based masking block 401 can be configured to generate the embedding 402 by giving relatively less attention to the area of ​​the contaminated input audio data 308 that the masking data 151 indicates includes noise and giving relatively more attention to the area of ​​the contaminated input audio data 308 that the masking data 151 indicates includes a signal. In some such examples, the attention-based masking block 401 can be or include a convolutional neural network (CNN) that calculates weighted convolutions. In some examples, the weighted convolutions can be weighted by the input mask. In some examples, the weighted convolutions can generate weights associated with the output of each layer of the neural network that implements the attention-based masking block 401. For example, the masking data 151 can indicate a value assigned to each of a plurality of frequency bands corresponding to the frequency band of the contaminated input audio data 308. These values ​​may indicate which frequency bands of the contaminated input audio data 308 are relatively more likely or relatively less likely to correspond to the target audio signal 411. In some such examples, these values ​​may range from 0 to 1, where 0 indicates an estimate of 100% noise and 1 indicates an estimate of 100% signal.

[0097] In some examples, embedding 402 may be or may include a mathematical representation of a portion of the input audio data in an embedding space, for example, the portion being estimated to correspond to those portions of the target audio data 411, the embedding space being different from the mathematical space of the input audio data. In this example, the embedding space may be different from the time / frequency space of the contaminated input audio data 308 and the target audio data 411. In some examples, the embedding space may include more dimensions than the mathematical space of the input audio data or the target audio data. In some examples, the input audio data may be represented by energy in multiple frequency bands, for example 16, 30, 32, 40, 48, 50, 64, 80, 100, or 128 frequency bands. In some examples, the frequency bands may be Mel or logarithmically spaced frequency bands. According to some such examples, the embedding space may be configured to generate embeddings in a potential space including 256 dimensions, 512 dimensions, 768 dimensions, etc.

[0098] In this example, context encoder 403 is configured to generate latent space embedding 302. In some examples, different latent space embeddings 302 can be output from different layers of the neural network implementing context encoder 403.

[0099] In this example, the input audio data (contaminated input audio data 308) is not pre-processed. However, in some alternative examples, the input audio data may have been pre-processed according to one or more audio data processing methods, such as an echo cancellation process, an echo suppression process, a noise suppression process, a beamforming process, or a combination thereof.

[0100] Figure 5 An example block diagram of a process for training a camouflage module according to some implementations is shown. As with other figures provided herein, Figure 5 The types, quantities, and arrangements of the elements shown in are provided merely as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements. For example, in some alternative implementations, camouflage module 201 may include blocks 301 and 303, but not block 305. According to some embodiments, camouflage module 201 may not include optional attention-based masking block 401.

[0101] In this example, the transformation blocks 102 and 107, the augmentation block 503, the loss function module 505 and the disguise module 201 are composed of Figure 1A According to this example, the disguise module 201 includes a reference Figure 3 and 4 An example of an SRL encoder block 301, a hidden representation block 303, and an SRL decoder block 305 is described.

[0102] According to this example, the clean time domain signal 501 is transformed into real and imaginary frequency components by the transform block 527. The transform block 527 converts these frequency components into the frequency band power spectrum domain to produce the transformed clean input audio signal 502. In this example, the transform block 527 includes Figure 1B The functions of the transform blocks 102 and 107.

[0103] In this example, the SRL encoder block 301 includes an optional attention-based masking block 401 and a context encoder 403. In this example, the control system 110 includes a neural network trained to implement the context encoder 403. The neural network can be, for example, a transformer neural network or a conformer neural network. According to this example, the control system 110 includes a neural network trained to implement the attention-based masking block 401.

[0104] According to this example, the SRL encoder block 301 is trained according to the masking data 551 and according to the contaminated audio signal 508 output from the audio augmentation process implemented by the augmentation block 503. The audio augmentation process can, for example, involve adding noise, adding reverberation, adding an audio signal corresponding to speech, adding an audio signal corresponding to other interfering audio sources, or a combination thereof to the transformed clean input audio signal 502. The graph associated with the contaminated audio signal 508 indicates a clean signal corresponding to one person's speech, and an interfering audio signal, which in this example is an audio signal corresponding to another person's speech. The white portion of the masking data 551 is an estimate of the interfering audio signal portion of the contaminated audio signal 508.

[0105] In this example, the contaminated audio signal 508 and the masking data 551 are provided to the optional attention-based masking block 401. Here, the optional attention-based masking block 401 is configured to generate the embedding 402 according to an attention-based masking process, which may be described in reference to Figure 4 In alternative examples that do not include the optional attention-based masking block 401 , the contaminated audio signal 508 and the masking data 551 may be provided directly to the context encoder 403 , or may be provided to an intermediate block that produces an embedding that is input to the context encoder 403 .

[0106] According to this example, the disguise module 201 generates a predicted target signal 511 during the training process, which is provided to the loss function module 505. In this example, the loss function module 505 is configured to determine a loss function gradient 555 based on the predicted target signal 511 and the transformed clean input audio signal 502. According to some examples, the loss function module 505 can be configured to implement a loss function based on the negative value of the similarity measure between the predicted target signal 511 and the transformed clean input audio signal 502. According to this example, the control system 110 is configured to update the parameters of the disguise module 201 according to the loss function gradient 555 until one or more convergence metrics are obtained. In some examples, the control system 110 can be configured to determine that convergence has been reached when the training process of the disguise module 201 reaches the following state, which is a state in which the loss determined by the loss function is stable within an error range around the final value, or a state in which the difference between the predicted target signal 511 and the transformed clean input audio signal 502 is no longer reduced, or a state in which the difference is not reduced for a predetermined number of steps or periods.

[0107] According to some examples, training the SRL encoder block 301 can include modulating masking data parameters of the masking data 551. In some such examples, one or more other types of data used in the training process or aspects of the training process can remain constant when modulating the masking data parameters of the masking data 551. For example, when modulating the masking data parameters of the masking data 551, the transformed clean input audio signal 502 can remain constant. Alternatively or additionally, when modulating the masking data parameters of the masking data 551, the augmentation process implemented by the augmentation block 503 can remain unchanged. In some examples, the SRL encoder block 301 can be trained to identify and compensate for one or more errors in the masking process.

[0108] Fig. 6A and 6B An example block diagram of a camouflage module according to some alternative implementations is shown. As with other figures provided herein, Fig. 6A and 6B The types, quantities, and arrangements of the elements shown in the examples are provided only as examples. Other implementations may include more, fewer, and / or different types, quantities, and arrangements of elements. In these examples, each camouflage module 201 is composed of Figure 1A According to these examples, the disguise module 201 includes a reference Figure 3 An example of an SRL encoder block 301, a hidden representation block 303, and an SRL decoder block 305 is described. Figure 4 Otherwise, the hidden representation block 303 and the SRL decoder block 305 can be as described in reference Figure 3 Works as described.

[0109] In these examples, the input audio data (contaminated input audio data 308) is not pre-processed. However, in some alternative examples, the input audio data may have been pre-processed according to one or more audio data processing methods, such as an echo cancellation process, an echo suppression process, a noise suppression process, a beamforming process, or a combination thereof.

[0110] According to these examples, each disguise module 201 is shown as outputting target audio data 411, which is a "clean" version of the contaminated input audio data 308. In these examples, the y-axis of the graphs associated with the masking data 151, the contaminated input audio data 308, and the target audio data 411 indicates frequency, while the x-axis indicates time. In the graphs associated with the masking data 151, the contaminated input audio data 308, and the target audio data 411, the blue areas correspond to "clean," uncontaminated audio signals, which in these examples correspond to the speech of a particular person. The graphs associated with the contaminated input audio data 308 indicate both uncontaminated audio signals and interfering audio signals, the latter being audio signals corresponding to the speech of another person. In other examples, the interfering audio signals may correspond to some other type of interfering audio signals. In the graphs associated with the masking data 151, the white areas indicate estimates of the interfering audio signals. According to these examples, the masking data 151 is derived from estimates of the signal and noise in the contaminated input audio data 308. As Figure 4 In the example shown, white areas of the masking data 151 correspond to interfering audio signals of the contaminated input audio data 308 .

[0111] exist Fig. 6A In the example shown, the SRL encoder block 301 is configured to generate an embedding 402 based on the contaminated input audio data 308 and based on a feature weight, which in this example is the masking data 151. According to the example, the SRL encoder block 301 is configured to calculate the dot product of the contaminated input audio data 308 and the masking data 151 to generate the embedding 402. In this example, the embedding 402 is a new audio signal in which the energy in the portion to be ignored has been muted according to the masking data 151. Therefore, in this example, the embedding 402 includes a mathematical representation of the portion of the input audio data in an embedding space that is the same as the mathematical space of the contaminated input audio data 308.

[0112] In this example, control system 110 includes a neural network that has been trained to implement convolutional encoder 603. According to this example, convolutional encoder 603 is configured to produce latent space embedding 302. In this example, convolutional encoder 603 has been trained to identify masked or silent portions of embedding 402 and generate a representation of new audio data corresponding to the masked portions - latent space embedding 302. In some examples, different latent space embeddings 302 can be output from different layers of the neural network that implements convolutional encoder 603.

[0113] In some examples, convolutional encoder 603 is configured to generate latent space embedding 302 in an embedding space that is different from the mathematical space of embedding 402. In some examples, the embedding space can be a higher dimensional space than the mathematical space of embedding 402.

[0114] exist Figure 6B In the example shown, the SRL encoder block 301 includes a version of the convolutional encoder 603 that includes a partial convolutional layer. According to this example, audio data masked to zero is not included in each convolution step implemented by the convolutional encoder 603, and the convolution calculation is scaled according to the number of inputs. In this example, the mask passed to subsequent convolutional layers is updated so that the "holes" corresponding to the masked audio data shrink at each subsequent convolutional layer.

[0115] Figure 7 700 is a flow chart summarizing an example of the disclosed method. Like other methods described herein, the blocks of method 700 are not necessarily executed in the order indicated. According to some examples, one or more blocks may be executed in parallel. In addition, some similar methods may include more or fewer blocks than those shown and / or described.

[0116] Method 700 may be performed by an apparatus or system (e.g., Figure 1A 1 and described above). In some examples, the apparatus 100 includes at least the control system 110 disclosed herein. In some examples, at least some aspects of the method 700 may be performed by one or more devices in the audio environment, for example, by an audio system controller (such as a device that may be referred to herein as a smart home hub) or by another component of the audio system (such as a television, a television control module, a laptop computer, a mobile device (such as a cellular phone), etc.). However, in some implementations, at least some blocks of the method 700 may be performed by one or more devices (such as one or more servers) configured to implement a cloud-based service.

[0117] In this example, block 705 includes receiving, by a control system configured to implement at least one neural network, input audio data and feature weights. In some examples, the input audio data may include an audio signal corresponding to speech. In some examples, the feature weights may be or may include masking data. For example, the masking data may be derived from an estimate of the signal and noise in the input audio data.

[0118] According to the example, block 710 includes generating a latent space embedding by the control system and based at least in part on the input audio data and the feature weights. In this example, the input audio data may correspond to an input mathematical space, such as a time / frequency space. In this example, the latent space embedding is or includes a mathematical representation of the input audio data in a latent space, which is a mathematical space different from the input mathematical space, such as a higher dimensional mathematical space. According to the example, the latent space embedding corresponds to an unmasked portion of the input audio data.

[0119] In some examples, the control system can be configured to implement a convolutional neural network configured to perform weighted convolution. In some such examples, method 700 can include performing a weighted convolution based at least in part on feature weights.

[0120] According to some examples, the input audio data and feature weights may correspond to frequency bands.

[0121] In some examples, the input audio data may have been pre-processed according to one or more audio data processing methods, such as an echo cancellation process, an echo suppression process, a noise suppression process, a beamforming process, or a combination thereof.

[0122] According to some examples, generating the latent space embedding may include applying, by the control system, a context encoding process. In some such examples, at least one neural network may have been trained to implement the context encoding process.

[0123] In some examples, method 700 may include applying, by the control system, a context decoding process to a representation of input audio data in a latent space to produce a modified audio signal. Alternatively or additionally, method 700 may include applying the context decoding process to a representation of input audio data in a latent space to produce output data from which a modified audio signal may be produced, such as with reference to Figure 2 The frequency band gain 111 described. In some examples, method 700 may include generating a residual signal based at least in part on the modified audio signal and a version of the input audio data. In some cases, the version of the input audio data may be or may include frequency partitioned audio data. According to some examples, the modified audio signal may be located in the frequency domain. Generating the residual signal may include transforming the frequency domain version of the residual signal into the time domain.

[0124] In some examples, method 700 can include applying, by the control system, a hidden representation process to the latent space embedding. The hidden representation process can produce a representation of the input audio data in the latent space.

[0125] According to some examples, the control system may be configured to implement at least one neural network that has been trained to implement an attention-based masking process. In some examples, method 700 may include generating an embedding according to an attention-based masking process.

[0126] In some examples, at least one neural network may have been trained based on the masking data and based on the contaminated audio signal output from an audio augmentation process. The audio augmentation process may include adding noise, adding reverberation, adding an audio signal corresponding to speech or other interfering audio sources, or a combination thereof.

[0127] According to some examples, training the at least one neural network may include modulating the masking data parameters. In some examples, training the at least one neural network may include maintaining a constant target audio signal during one or more time intervals of modulating the masking data parameters. In some examples, the attention-based masking process, the context encoding process, or both may have been trained to identify and compensate for one or more errors in the masking process.

[0128] In some examples, the control system can be configured for speech representation learning (SRL). In some such examples, at least one neural network can include an SRL encoder. In some cases, the SRL encoder can include or be a convolutional encoder. According to some examples, the convolutional encoder can include a partial convolutional layer.

[0129] Some aspects of the present disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0130] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed or otherwise configured) to perform the desired processing on the audio signal, including performing one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations including one or more examples of the disclosed method. Alternatively, elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP that is configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0131] Another aspect of the disclosure is a computer-readable medium (eg, a disk or other tangible storage medium) storing code (eg, code executable to implement the example) for implementing one or more examples of the disclosed method or steps thereof.

[0132] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method comprising: receiving, by a control system configured to implement at least one neural network, input audio data and feature weights; and A latent space embedding is generated by controlling the system and based at least in part on input audio data and feature weights, wherein the input audio data corresponds to an input mathematical space, and wherein the latent space embedding includes a mathematical representation of the input audio data indicated by the feature weights in a latent space, the latent space being a mathematical space different from the input mathematical space, and wherein the latent space embedding corresponds to unmasked data of the input audio data. The method of claim 1 , wherein the feature weights include masking data.

3. The method according to claim 1 or 2, wherein: The masking data is derived from estimates of the noise and signal in the input audio data.

4. The method of any one of claims 1-3, wherein the control system is configured to implement a convolutional neural network configured to perform weighted convolutions, and wherein the weighted convolutions are based at least in part on the feature weights.

5. The method of any one of claims 1-4, wherein generating a latent space embedding comprises applying a context encoding process by a control system.

6. The method according to claim 5, wherein: The at least one neural network has been trained to implement the context encoding process.

7. The method of any one of claims 1-6, further comprising applying, by a control system, a hidden representation process to a latent space embedding to produce a representation of the input audio data in a latent space.

8. The method of claim 7, further comprising applying, by the control system, a contextual decoding process to the representation of the input audio data in the latent space to produce a modified audio signal.

9. The method of claim 8, further comprising generating a residual signal based at least in part on a version of the input audio data and the modified audio signal.

10. The method of claim 9, wherein the version of the input audio data comprises frequency partitioned audio data.

11. The method according to claim 9 or 10, wherein the modified audio signal is in the frequency domain, and wherein generating the residual signal comprises transforming a frequency domain version of the residual signal into the time domain.

12. The method according to any one of claims 1 to 11, wherein: The input audio data and the feature weights correspond to frequency bands.

13. The method according to any one of claims 1 to 12, wherein: The input audio data has been pre-processed according to one or more audio data processing methods.

14. The method according to claim 13, wherein: The input audio data has been pre-processed according to at least one of an echo cancellation process, an echo suppression process, a noise suppression process or a beamforming process.

15. The method according to any one of claims 1 to 14, wherein: The at least one neural network has also been trained to implement an attention-based masking process for generating the embedding.

16. The method of claim 15, wherein at least one of the attention-based masking process or the context encoding process has been trained to recognize and compensate for one or more errors in the masking process.

17. The method according to claim 15 or 16, wherein: The at least one neural network has been trained based on the masking data and based on a contaminated audio signal output from the audio enhancement process.

18. The method according to claim 17, wherein: The audio enhancement process includes adding noise, adding reverberation, adding audio signals corresponding to speech or other interfering audio sources, or a combination thereof.

19. The method of claim 17 or 18, wherein training the at least one neural network comprises modulating masking data parameters.

20. The method of claim 19, wherein training the at least one neural network comprises maintaining a constant target audio signal during one or more time intervals during which the masking data parameters are modulated.

21. The method according to any one of claims 1 to 20, wherein: The input audio data includes an audio signal corresponding to speech.

22. The method according to any one of claims 1 to 21, wherein the control system is configured for speech representation learning (SRL).

23. The method of claim 22, wherein the at least one neural network comprises a SRL encoder.

24. The method of claim 23, wherein the SRL encoder comprises a convolutional encoder.

25. A method according to claim 23 or 24, wherein the convolutional encoder comprises a partial convolutional layer.

26. An apparatus comprising a control system configured to implement one or more of the methods according to claims 1-25.

27. A system configured to implement one or more of the methods according to claims 1-25.

Images