Spatial Representation Learning

Abstract

Some disclosed methods involve receiving multi-channel audio data including unlabeled multi-channel audio data; extracting audio feature data from the unlabeled multi-channel audio data; applying a spatial masking process to a portion of the audio feature data; applying a context encoding process to the masked audio feature data to generate a predicted spatial embedding in a latent space; obtaining a reference spatial embedding in the latent space; determining a loss function gradient based at least in part on a difference between the predicted spatial embedding and the reference spatial embedding; and updating the context encoding process according to the loss function gradient until one or more convergence metrics are achieved.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 315,344, filed March 1, 2022.

[0002] Technical Field The present disclosure relates to devices, systems and methods for determining spatial attributes of sound sources in a multi-channel audio signal. [Background technology]

[0003] Several methods, devices and systems are known for estimating source location signals in multi-channel audio signals, including methods that involve the use of pre-labeled training data. While existing devices, systems and methods can provide benefit in some situations, improved devices, systems and methods would be desirable.

[0004] Notation and Nomenclature Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used interchangeably to refer to any sound emitting transducer (or set of transducers) driven by a single speaker feed. A typical headphone set includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker signal may undergo different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to refer to performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has been subjected to pre-filtering or pre-processing before performing the operation).

[0006] Throughout this disclosure, including the claims, the term "system" is used broadly to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0008] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct connection or an indirect connection. Thus, if a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0009] As used herein, a "smart device" is generally an electronic device that is configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, near field communications, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., and can operate interactively and / or autonomously to some degree. Some notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bands, smart keychains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some characteristics of ubiquitous computing, such as artificial intelligence.

[0010] The term "smart audio device" is used herein to denote a smart device that is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and, optionally, includes or is coupled to at least one speaker and / or at least one camera) and is designed primarily or primarily to achieve a single purpose. For example, a TV can typically play (and is considered capable of playing) audio from program material, but in most cases, modern TVs run some operating system on which applications run locally, including applications for watching television. In this sense, a single-purpose audio device having a speaker and microphone is often configured to run local applications and / or services to directly use the speaker and microphone. Several single-purpose audio devices may be configured to be grouped together to achieve playback of audio over a zone or user-configured area.

[0011] One common type of multipurpose audio device is a smart audio device, such as a "smart speaker," that implements at least some aspects of the virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multipurpose audio device is configured to communicate. Such multipurpose audio devices may be referred to herein as "virtual assistants." A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to utilize multiple devices (different from the virtual assistant) for applications that are in some sense cloud-enabled or otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate over a network, such as the Internet. Virtual assistants may sometimes work together, for example in a discrete, conditionally defined manner. For example, two or more virtual assistants may cooperate in the sense that one of them, for example the one that is most confident that it heard the wake word, responds to the wake word. Connected virtual assistants may, in some implementations, form a kind of constellation, which may be managed by one main application that may be (or may implement) a virtual assistant.

[0012] Here, "wake word" is used broadly to mean any sound (e.g., a word spoken by a human being, or any other sound) that the smart audio device is configured to wake up in response to detecting ("listening to") that sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "awakening" refers to the device entering a state in which it waits for (i.e., listens for) a voice command. In some instances, what may be referred to herein as a "wake word" may include multiple words, e.g., a phrase.

[0013] Here, the term "wake word detector" refers to a device (or software including instructions for configuring a device) configured to continuously look for alignment between real-time audio (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that the wake word has been detected exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold tuned to provide a reasonable compromise between false accept and false reject rates. Following the wake word event, the device may enter a state (which may be referred to as an "awake" or "attentive" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0014] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and possibly video signals, that are intended to be listened to at least in part together. Examples include selections of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from a smart assistant, etc. In some instances, a content stream may contain multiple versions of at least a portion of an audio signal, e.g., the same dialogue in multiple languages. In such instances, only one version of the audio data or portion thereof is intended to be played at a time (e.g., the version corresponding to a single language). Summary of the Invention [Means for solving the problem]

[0015] At least some aspects of the present disclosure may be implemented via one or more methods. In some cases, the methods may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some methods may include receiving, by the control system, multi-channel audio data. The multi-channel audio data may be or include unlabeled multi-channel audio data. Some such methods may include extracting, by the control system, audio feature data from the unlabeled multi-channel audio data. Some such methods may include masking, by the control system, a portion of the audio feature data to generate masked audio feature data. The masking may be or include spatial masking.

[0016] Some such methods may include applying, by the control system, a context encoding process to the masked audio feature data to generate a predicted spatial embedding in a latent space. Some such methods may include obtaining, by the control system, a reference spatial embedding in the latent space. Some such methods may include determining, by the control system, a loss function gradient based at least in part on a difference between the predicted spatial embedding and the reference spatial embedding. Some such methods may include updating, by the control system, the context encoding process according to the loss function gradient until one or more convergence metrics are achieved.

[0017] In some examples, obtaining the reference spatial embedding in the latent space may include applying, by the control system, a context encoding process to the unmasked audio feature data corresponding to the masked audio feature data to generate the reference spatial embedding. According to some examples, obtaining the reference spatial embedding in the latent space may include a spatial unit discovery process. In some examples, the spatial unit discovery process may include clustering the spatial features into multiple granularities. According to some examples, the clustering may include applying an ensemble of k-means models with different codebook sizes.

[0018] In some examples, the spatial unit discovery process may include generating a library of codewords. In some such examples, each codeword may correspond to an acoustic zone of the audio environment. According to some examples, each codeword may correspond to a spatial location of a sound source relative to a microphone used to capture at least a portion of the multi-channel audio data. In some examples, each codeword may correspond to a covariance of signals corresponding to sounds captured by each microphone of a plurality of microphones used to capture the multi-channel audio data. According to some examples, the signal covariance may be represented by a microphone covariance matrix. Some disclosed methods may include updating the library of codewords according to an output of the context encoding process.

[0019] In some examples, the predicted spatial embedding may correspond to estimated cluster centroids in the latent space. According to some examples, the predicted spatial embedding may correspond to a representation of acoustic zones in the latent space.

[0020] According to some examples, the multi-channel audio data may include at least N channels of audio data and M channels of audio data, where N and M may be greater than or equal to 2 and may represent different value integers. In some examples, the multi-channel audio data may include audio data captured by two or more different types of microphone arrays.

[0021] In some examples, the control system may be configured to implement a neural network. According to some examples, the neural network may be trained according to a self-supervised learning process.

[0022] Some disclosed methods may include training a neural network implemented by the control system to implement a noise suppression function, a speech recognition function, a speaker identification function, a sound source separation function, a voice activity detection function, an audio scene classification function, a sound source localization function, a noise source recognition function, or a combination thereof, after the control system has been trained according to one of the disclosed methods. Some such disclosed methods may include implementing a noise suppression function, a speech recognition function, or a speaker identification function by the control system.

[0023] According to some examples, spatial masking may include concealing, corrupting, or removing at least a portion of spatial information of the multi-channel audio data. In some examples, spatial masking may include presenting audio data from channel A as being audio data from channel B and presenting audio data from channel B as being audio data from channel A during the masking time interval. According to some examples, spatial masking may include adding audio data corresponding to an artificial sound source in an artificial source acoustic zone during the masking time interval. Spatial masking may include reducing a number of uncorrelated channels of the multi-channel audio data during the masking time interval. According to some examples, spatial masking may include modifying an apparent acoustic zone of a sound source during the masking time interval.

[0024] In some examples, spatial masking may include increasing a number of correlated channels of the multi-channel audio data during the masking time interval. According to some examples, spatial masking may include combining two or more audio signals corresponding to two or more independent sound sources in a single channel during the masking time interval. In some examples, spatial masking may include modifying microphone signal covariance information of the multi-channel audio data during the masking time interval.

[0025] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0026] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices (e.g., a system including one or more devices) may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to implement some or all of the methods disclosed herein.

[0027] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0028] Like reference numbers and designations in the various drawings indicate like elements.

[0029] [Figure 1A] FIG. 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the disclosure. [Figure 1B] 1 shows an example of a speaker in an audio environment. [Diagram 2] 1 illustrates blocks of a spatial representation learning (SPRL) system according to an example. [Diagram 3] An example illustrates the spatial masking process. [Figure 4] 1 shows blocks of an SPRL system according to another example. [Diagram 5] 13 shows blocks of an SPRL system according to yet another example. [Figure 6] FIG. 1 is a flow diagram outlining an example of the disclosed method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Previous approaches to audio processing, especially in the context of speech processing, have relied on extracting an engineered set of features for a specific task and then using those features to perform the task in a supervised manner. This type of approach required expert domain knowledge, large amounts of labeled data, and modifying the features for each task.

[0031] The advent of machine learning (ML) and its advances in areas such as speech processing have provided the opportunity to represent speech in what is called a "latent space", where high-level attributes or distinct characteristics of the audio signal can be derived automatically from the data. Representations in latent space can be used to enable or improve a variety of use-case applications, including but not limited to sound event classification, speaker identification, and automatic speech recognition.

[0032] In recent years, self-supervised learning techniques have begun to be applied to various tasks, such as image data learning tasks. According to self-supervised learning techniques, latent spaces and feature representations are learned without prior knowledge of the task and without labeled training data. Self-supervised learning has recently been employed in speech analysis applications that use single-channel speech data, i.e., audio data corresponding to speech captured using a single microphone. However, little is known about the high-level representation of multi-channel audio data in the latent space, e.g., audio data corresponding to multiple sound sources captured using multiple microphones, potentially including speech. Many related questions, such as the speaker's location, how interfering speech or other interfering sound sources affect the speech representation, remain unanswered so far.

[0033] This disclosure provides examples of spatial representation learning (SPRL), which involves modeling high-level spatial attributes of one or more sound sources (such as voices) captured on a device that includes an array of microphones. In some examples of spatial representation learning, a spatial dimension is added to the embedding representation by learning spatial information corresponding to multi-channel audio data corresponding to speech. Localizing sound sources represented by multi-channel audio data and learning spatial attributes of these sources in latent space can improve a variety of learning and downstream audio processing tasks, including, but not limited to, noise suppression, speech recognition, and speaker identification.

[0034] FIG. 1A is a block diagram illustrating examples of components of a device capable of implementing various aspects of the disclosure. As with other figures provided herein, the types, numbers, and arrangements of elements shown in FIG. 1A are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements. According to some examples, device 100 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 100 may be or include one or more components of an office workstation, one or more components of a home entertainment system, and the like. For example, device 100 may be a laptop computer, a tablet device, a mobile device (such as a cellular phone), a smart home hub, a television, or another type of device.

[0035] According to some alternative implementations, apparatus 100 may be or include a server. In some such examples, apparatus 100 may be or include an encoder. In some examples, apparatus 100 may be or include a decoder. Thus, in some cases, apparatus 100 may be a device configured for use in an environment, such as a home environment, and in other cases, apparatus 100 may be a device configured for use in the "cloud," e.g., a server.

[0036] In this example, device 100 includes an interface system 105 and a control system 110. Interface system 105 may be configured to communicate with one or more other devices in an environment, in some implementations. The environment may be a home environment, in some examples. In other examples, the environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. Interface system 105 may be configured to exchange control information and associated data with other devices in the environment, in some implementations. The control information and associated data may relate to one or more software applications that device 100 is executing, in some examples.

[0037] The interface system 105, in some implementations, may be configured to receive or provide a content stream. In some examples, the content stream may include video data and audio data corresponding to the video data. The audio data may include, but is not limited to, an audio signal. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. The metadata may have been provided, for example, by what may be referred to herein as an “encoder.”

[0038] Interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, interface system 105 may include one or more wireless interfaces. Interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, a gesture sensor system, or combinations thereof. Thus, although some such devices are represented separately in FIG. 1A , such devices may correspond to aspects of interface system 105 in some examples.

[0039] In some examples, interface system 105 may include one or more interfaces between control system 110 and a memory system, such as optional memory system 115 shown in FIG. 1A. Alternatively or additionally, control system 110 may include a memory system in some cases. Interface system 105, in some implementations, may be configured to receive input from one or more microphones in the environment.

[0040] The control system 110 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0041] In some implementations, the control system 110 may reside in more than one device. For example, in some implementations, a portion of the control system 110 may reside in a device in one of the environments mentioned herein, and another portion of the control system 110 may reside in a device outside of the environment, such as a server, a mobile device (such as a smartphone or tablet computer). In other examples, a portion of the control system 110 may reside in a device in one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices of the environment. For example, the control system functionality may be shared by a leader device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 110 may reside in a device implementing a cloud-based service, such as a server, and another portion of the control system 110 may reside in another device implementing the cloud-based service, such as another server, a memory device, etc. The interface system 105 may also reside in more than one device in some examples.

[0042] In some implementations, the control system 110 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 110 may be configured to receive multi-channel audio data. The multi-channel audio data may be, or at least include, unlabeled multi-channel audio data. In some cases, not all of the multi-channel audio data may be labeled. According to some examples, the multi-channel audio data may include audio data corresponding to speech captured by different types of microphone arrays, audio data having different numbers of channels, or a combination thereof. In some examples, the control system 110 may be configured to extract audio feature data from the unlabeled multi-channel audio data. According to some examples, the control system 110 may be configured to mask a portion of the audio feature data to generate masked audio feature data. In some examples, the masking process may include spatial masking. Various examples of spatial masking are provided in this disclosure.

[0043] In some examples, the control system 110 may be configured to apply a context encoding process to the masked audio feature data to generate a predicted spatial embedding in the latent space. According to some such examples, the control system 110 may be configured to implement a neural network configured to apply the context encoding process.

[0044] According to some examples, the control system 110 may be configured to obtain a reference spatial embedding in the latent space and determine a loss function gradient based at least in part on the difference between the predicted spatial embedding and the reference spatial embedding. In some examples, the control system 110 may be configured to update the context encoding process according to the loss function gradient until one or more convergence metrics are achieved. Some examples of these processes are described below.

[0045] As noted elsewhere herein, the control system 110 may reside in a single device or in multiple devices, depending on the particular implementation. In some examples, all of the aforementioned processes may be performed by the same device. In some alternatives, the aforementioned processes may be performed by two or more devices. For example, the feature data extraction and masking processes may be performed by one device, and the remaining processing may be performed by one or more other devices, such as one or more devices (e.g., one or more servers) configured to implement a cloud-based service. In some alternatives, the feature data extraction may be performed by one device, and the masking process may be performed by another device.

[0046] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored in one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. The one or more non-transitory media may be present, for example, in optional memory system 115 and / or control system 110 shown in FIG. 1A. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may include, for example, instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may be executable by one or more components of a control system, such as, for example, control system 110 of FIG. 1A.

[0047] In some examples, the device 100 may include an optional microphone system 120 as shown in FIG. 1A. The optional microphone system 120 may include one or more microphones. According to some examples, the optional microphone system 120 may include an array of microphones. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to instructions from the control system 110. The array of microphones may be configured for receive-side beamforming, in some cases, e.g., according to instructions from the control system 110. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, the device 100 may not include the microphone system 120. However, in some such implementations, the device 100 may still be configured to receive microphone data corresponding to one or more microphones in one environment or corresponding to one or more microphones in another environment, via the interface system 110. In some such implementations, a cloud-based implementation of device 100 may be configured to receive microphone data captured at one or more microphones in the environment via interface system 110, or data corresponding to the microphone data (e.g., multi-channel audio data corresponding to speech).

[0048] According to some implementations, device 100 may include optional loudspeaker system 125 shown in FIG 1A. Optional loudspeaker system 125 may include one or more loudspeakers, which may also be referred to herein as a "speaker" or, more generally, an "audio reproduction transducer." In some examples (e.g., cloud-based implementations), device 100 may not include loudspeaker system 125.

[0049] In some implementations, the device 100 may include the optional sensor system 130 shown in FIG. 1A. The optional sensor system 130 may include one or more touch sensors, gesture sensors, motion detectors, cameras, eye tracking devices, or combinations thereof. In some implementations, the one or more cameras may include one or more freestanding cameras. In some examples, the one or more cameras, eye trackers, etc. of the optional sensor system 130 may be present in a television, a mobile phone, a smart speaker, or combinations thereof. In some examples, the device 100 may not include the sensor system 130. However, in some such implementations, the device 100 may still be configured to receive, via the interface system 110, sensor data for one or more sensors (cameras, eye trackers, etc.) present in or on other devices in the environment.

[0050] In some implementations, the device 100 may include an optional display system 135 as shown in FIG. 1A. The optional display system 135 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, the optional display system 135 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 135 may include one or more displays of a television, a laptop, a mobile device, a smart audio device, or another type of device. In some examples in which the device 100 includes a display system 135, the sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 135. According to some such implementations, the control system 110 may be configured to control the display system 135 to present one or more graphical user interfaces (GUIs).

[0051] According to some such examples, apparatus 100 may be or include a smart audio device, such as a smart speaker. In some such implementations, apparatus 100 may be or include a wake word detector. For example, apparatus 100 may be configured to implement (at least in part) a virtual assistant.

[0052] Figure IB illustrates an example of speakers in an audio environment. As with other figures provided herein, the types, numbers, and arrangements of elements shown in Figure IB are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0053] FIG. 1B illustrates an audio environment 150 in which audio device 154 detects sounds using microphones 164A, 164B, and 164C, thereby capturing multi-channel audio. Audio device 154 may be, for example, a smart audio device, such as a smart speaker. According to this example, audio device 154 is an instance of apparatus 100 of FIG. 1A. In this example, speakers 152 and 155 are located in acoustic zones 1 and 5. Speakers 152 and 155 speak to each other and to audio device 154 while an interfering noise source, in this example range hood 153, is operating in acoustic zone 3. In this example, the term "acoustic zone" refers to a spatial location within acoustic environment 150. Acoustic zones may be determined in various manners depending on the particular implementation. The acoustic zone of the speaker 155 may correspond, for example, to a volume in which the speaker 155 currently sits or stands, a volume corresponding to the size of the speaker's 155 head, a volume in which the speaker's 155 head moves during a remote conference, etc. In some such examples, the acoustic zone may be defined according to the location of the center or center of gravity of such a volume. The definition of the acoustic zones is advantageously such that each defined acoustic zone can be distinguished from all other defined acoustic zones according to the sounds emanating from the acoustic zone. For example, if acoustic zone A is adjacent to acoustic zone B, a microphone array such as microphones 164A, 164B, and 164C of FIG. 1B may desirably determine that the sounds emanating from acoustic zone A come from a different direction than the sounds emanating from acoustic zone B, for example according to the directions of arrival of the sounds emanating from acoustic zones A and B.

[0054] According to this example, control system 110 uses the multi-channel captured audio and the pre-trained SPRL model to perform multi-channel downstream tasks, such as real-time speaker identification, automatic speech recognition, one or more other downstream tasks, or a combination thereof. As noted elsewhere herein, control system 110 may reside in more than one device in some instances. In some examples, the downstream tasks may be performed, at least in part, by another device, such as a server.

[0055] The elements of FIG. 1B include the following: 150:The audio environment, in this example a room. 151: Table 152, 155: Talkers at spatial locations “acoustic zone 1” and “acoustic zone 5”. 153: Range hood at spatial location “acoustic zone 3”. 154: Audio Devices 164A, 164B, and 164C: Multiple microphones within or on the surface of device 154. 164, 157: Direct speech from speakers 152 and 155 to microphones 164A, 164B, and 164C. 110: A control system that resides at least partially within the audio device, parts of which may also reside elsewhere (such as one or more servers of a cloud-based service provider), that is configured to analyze audio captured by microphones 164A, 164B, and 164C.

[0056] FIG. 2 illustrates blocks of a spatial representation learning (SPRL) system according to an example. FIG. 2 illustrates an example of an SPRL system 200 in which hidden spatial representations are learned using a contextual encoder 206, sometimes referred to herein as a contextual network 206. In some examples, the hidden spatial representations may be learned according to a self-supervised training process. As with other figures provided herein, the types, numbers, and arrangements of elements shown in FIG. 2 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements. In this example, the blocks of the SPRL system 200 are implemented by an instance of the control system 110 of FIG. 1A. As noted elsewhere herein, the control system 110 may reside in two or more devices in some implementations. For example, the context encoder 206 may be implemented by one device, and other blocks (such as the audio feature extractor 202, the masking block 204, or both) may be implemented by one or more other devices.

[0057] In this example, the audio feature extraction block 202 is configured to extract audio features 203 from the multi-channel audio data 201. According to this example, the multi-channel audio data 201 is training data including unlabeled multi-channel audio data. In some examples, all of the multi-channel audio data 201 may be unlabeled multi-channel audio data. At least a portion of the multi-channel audio data 201 includes audio data corresponding to multiple sound sources, potentially including speech, captured using multiple microphones. In some implementations, the multi-channel audio data 201 includes audio data captured by two or more different types of microphone arrays, and in some examples includes audio data captured by many (e.g., 5 or more, 10 or more, 15 or more, etc.) different types of microphone arrays. According to some examples, the multi-channel audio data 201 may include at least N channels of audio data and M channels of audio data, where N and M are greater than or equal to 2 and represent different valued integers. For example, the multi-channel audio data 201 may include at least two-channel audio data and five-channel audio data.

[0058] The audio features 203 extracted by the audio feature extraction block 202 may be referred to herein as an audio feature vector. The audio features 203 may vary depending on the particular implementation. In some examples, the audio features 203 may simply be time samples. In other examples, the audio features 203 may be frequency bands or bin energies. According to some examples, the audio features 203 may be transform bins stacked across multiple channels.

[0059] In one optional example of the SPRL system 200, the gradient 212 of the loss function 210 with respect to the audio feature extraction parameters may be provided to the audio feature extraction block 202. Thus, in this example, the SPRL system 200 optionally includes an audio feature extraction block 202 configured to learn which audio feature extraction parameters are relatively more or relatively less effective and modify the audio feature extraction parameters accordingly. In this example, the extracted audio features 214A are extracted audio features for a particular time frame.

[0060] According to this example, the masking block 204 is configured to mask one or more parts or sections of the audio features 203 to generate masked audio feature data 205. In this example, the masked audio features 214M that are part of the masked audio feature data 205 are masked counterparts of the extracted audio features 214A. However, in some alternative examples, the masking process may be performed before the audio feature extraction process. In other words, in some examples, the masking block 204 may be configured to apply one or more types of masking processes to the multi-channel audio data 201.

[0061] In some examples, the masking block 204 may be configured to apply one or more types of spatial masking to the multi-channel audio data 201 or to one or more sections of the audio features 203. According to some examples, spatial masking may include concealing, corrupting, or removing at least a portion of the spatial information of the multi-channel audio data 201 or the audio features 203. Some additional examples of spatial masking are described below with reference to FIG.

[0062] The context encoder 206 may be or include a neural network implemented by a part of the control system 110, such as a transformer neural network or a reformer neural network. In this example, the context encoder 206 is configured to apply a context encoding process to the masked audio feature data 205 to generate a predicted spatial embedding 207 in a latent space, sometimes referred to herein as a “spatial representation” or a “hidden spatial representation”. In some examples, the hidden representation block 208 (also referred to herein as a spatial embedding block 208) may process multiple instances of the predicted spatial embedding 207 to generate a single predicted embedding 209. For example, in some implementations, the context encoder 206 may be or include a neural network. In some such implementations, different predicted spatial embeddings 207 may be output from different layers of a neural network. The spatial embedding block 208 may determine a single predicted embedding 209 from multiple predicted spatial embeddings 207, each generated by a different layer. In some such examples, the spatial embedding block 208 may apply a merging process, such as a pooling process, to the multiple predicted spatial embeddings 207 to generate a single predicted embedding 209. In some examples, the spatial embedding block 208 may generate a predicted embedding 209 that is a lower dimensional representation of the multiple predicted spatial embeddings 207. According to some examples, the spatial embedding block 208 may generate the single predicted embedding 209 according to one or more averaging processes. In some examples, the spatial embedding block 208 may implement one or more attention-based averaging processes.In some such examples, the spatial embedding block 208 may generate a single predicted embedding 209 according to at least one time-based averaging process, such as a process that operates on predicted spatial embeddings 207 corresponding to multiple frames of input data.

[0063] According to this example, the control system 110 is configured to obtain a reference spatial embedding 211 in the latent space. The reference spatial embedding 211 can be obtained in various ways, depending on the particular implementation. Some examples are described below with reference to Figures 4 and 5.

[0064] In this example, the control system 110 is configured to apply a loss function and determine a loss function gradient based at least in part on the difference between the predicted spatial embedding 209 (the predicted spatial embedding 209 of FIG. 2, which in this example is the current one output by the spatial embedding block 208 input to the loss function 210) and the reference spatial embedding (the reference spatial embedding 211 in this example). In the example of FIG. 2, element 213 indicates the gradient of the loss function with respect to the parameters of the context encoder 206. According to this example, the context encoder 206 has been trained according to a backpropagation technique, which involves working backwards from the losses with respect to the predicted spatial embedding 209 and the reference spatial embedding 211 to determine losses with respect to each of the trained parameters in the context encoder 206. This leads to the next step in the steepest descent algorithm, which may determine a more optimal set of parameters until convergence is reached. According to this example, optional element 212 indicates the gradient of the loss function with respect to the parameters of feature extractor 202, indicating that feature extractor 202 may also be trained according to the backpropagation technique.

[0065] According to some examples, the loss function may be the negative of a measure of similarity between the predicted spatial embedding 209 and the reference spatial embedding 211. The measure of similarity may be, for example, a measure of cosine similarity. In other examples, the loss function may be a mean squared error loss function, a mean absolute error loss function, a Huber loss function, a log-cosh loss function, or another loss function. In some examples, the loss function 210 is based only on portions of the predicted embedding that correspond to one or more masked regions of the masked audio feature data 205, such as the masked regions 215 of the masked audio features 214M, and the corresponding portions of the reference embedding 211. According to this example, the control system 110 is configured to update the context encoding process according to the loss function gradient 213 until one or more convergence metrics are achieved. In some examples, the control system 110 may be configured to determine that convergence has been achieved when the context encoding process achieves a state where the loss determined by the loss function settles within an error range around a final value or where the difference between the predicted spatial embedding 209 and the reference spatial embedding 211 is no longer decreasing.

[0066] In this example, the elements of FIG. 200: SPRL (Spatial Representation Learning) system. 201: Raw multi-channel input audio data. 202: Audio feature extraction block. 203: Stream of extracted audio features. 204: Masking Block 205: Stream of masked audio feature data. 206: A context encoder configured to predict masked audio features. 207: Predicted spatial embedding 207. This is, in this example, the output prediction after passing the masked audio feature data through the context encoder 206. 208: A spatial embedding block 208 configured to aggregate multiple predicted spatial embeddings 207 from the context encoder 206 to generate a predicted spatial embedding 209. 209: Predicted spatial embeddings. 211: Reference spatial embedding. 210: Loss function. 212: Gradient of the loss function with respect to the audio feature extraction parameters of the audio feature extraction block 202. 213: Gradient of the loss with respect to the parameters of the context encoder 206. 214A: Extracted audio features for a time frame. 214M: Masked counterpart of extracted audio feature 214A. 214P: Predicted embedding for 214A output by context encoder 206.

[0067] Figure 3 illustrates a spatial masking process according to one example. As with other figures provided herein, the types, numbers, and arrangements of elements shown in Figure 3 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements.

[0068] In this example, the masking block 204 is configured to implement a spatial masking process which involves adding, during the masking time interval, audio data corresponding to an artificial sound source in the artificial source acoustic zone. According to this example, the spatial masking process involves adding, during the masking time interval, audio data corresponding to an artificial sound source 302 which was not actually present when the multi-channel audio data corresponding to the real sound source 301 was captured. In this example, the real sound source 301 is a speech source, such as a person speaking, in the room 300 in which the multi-channel audio data was captured.

[0069] The masking time interval may vary depending on the particular implementation. In practice, it may be beneficial to select one or more masking time intervals corresponding to one or more types of interfering sounds that may mask the target signal (such as audio data corresponding to a human voice). For example, one masking time interval may be selected to correspond to a time interval of an interfering speech utterance or an interfering sequence of dog barks, which may be 1 second, 2 seconds, or more. Another masking time interval may be selected to correspond to a time interval of an interfering exclamation or an interfering dog bark, which may be less than 1 second in duration, e.g., 1 / 2 second. Another masking time interval may be selected to correspond to a time interval of an interfering door slamming sound, which may correspond to a time interval of a few milliseconds.

[0070] According to this example, the elements of FIG. 214A: Extracted audio features corresponding to real sound sources 310 during a time interval (for a time frame). 214M: Masked counterpart of extracted audio feature 214A. 300: The room 300 from which the multi-channel audio data currently being processed was captured. 301: A real sound source in the room 300. In this example, it is a speech source. 302: Artificial sound source 302. This may also be called an interference source, which was not actually present when the multi-channel audio data corresponding to the real sound source 301 was captured in the room 300. 354: An audio device containing an array of microphones. 364: An array of microphones within or on the surface of the audio device 354. 305: Direct sound from the artificial source 302 captured by the array of microphones 364. 306: Direct sound from real sound source 301 captured by an array of microphones 364. 307: Sound reflection from the walls of the room 300 corresponding to the artificial sound source 302. 308: Sound reflections from the walls of room 300 corresponding to sound from real sound source 301.

[0071] Sound reflections 307 from walls of the room 300 corresponding to the artificial sound source 302 may be simulated in various ways depending on the particular implementation. According to some examples, the sound reflections 307 may be simulated based on a library of sound recordings in real rooms. In some examples, the sound reflections 307 may be simulated based on a ray-traced acoustic model of a specified room. According to some examples, the sound reflections 307 may be simulated based on one or more parametric models of general reverberation (e.g., based on reverberation time, one or more absorption coefficients, and an estimated room size). In some examples, the sound reflections 307 may be simulated based on a "shoebox" method, such as the method described in "Analyzing Sound in Rooms with Sound," ... [Non-Patent Document 1] Steven M. Schimmel et al., "A Fast and Accurate "Shoebox" Room Acoustics Simulator," ICASSP-88, 1988 International Conference on May 2009, pp. 241-244

[0072] In some examples, the spatial masking process may involve changing the apparent acoustic zone of the sound source during the masking time interval. In other words, spatial masking may involve changing the direction, area or zone from which sound corresponding to the sound source appears to come during the masking time interval. In one such example, spatial masking may involve changing the area or zone from which sound corresponding to the real sound source 301 appears to come during the masking time interval.

[0073] In some examples, the spatial masking process may include temporarily switching audio channels. According to some such examples, the spatial masking process may include presenting audio data from channel A as being audio data from channel B and presenting audio data from channel B as being audio data from channel A during a masking time interval.

[0074] According to some examples, the spatial masking process may include reducing a number of uncorrelated channels of the multi-channel audio data 201 during the masking time interval. Alternatively or additionally, in some examples, the spatial masking process may include increasing a number of correlated channels of the multi-channel audio data 201 during the masking time interval. In some examples, the spatial masking process may include combining, in a single channel, two or more audio signals corresponding to two or more independent sound sources during the masking time interval.

[0075] In some examples, the spatial masking process may include modifying microphone signal covariance information of the multi-channel audio data 201 during the masking time interval. Some examples of microphone signal covariance information are described below.

[0076] FIG. 4 illustrates blocks of an SPRL system according to another example. FIG. 4 illustrates an example of an SPRL system 400 in which hidden spatial representations are learned using two context encoders, a student context encoder 406 and a teacher context encoder 416. In some examples, the hidden spatial representations may be learned according to a self-supervised training process. As with other figures provided herein, the types, numbers, and arrangements of elements shown in FIG. 4 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements. In this example, the blocks of the SPRL system 400 are implemented by an instance of the control system 110 of FIG. 1A. As noted elsewhere herein, the control system 110 may reside in more than one device in some implementations.

[0077] In this example, the multi-channel audio data 201, audio feature extraction block 202, audio features 203, masking block 204, masked audio feature data 205, predicted spatial embedding 207, hidden representation block 208, predicted spatial embedding 209 and loss function 210 may be as described above with reference to Figure 2, and thus the foregoing description will not be repeated here. According to this example, the student context encoder 406 is an instance of the context encoder 206 of Figure 2, and the reference spatial representation block 408 is an instance of the hidden representation block 208 of Figure 2.

[0078] The SPRL system 400 provides an example of how the reference spatial embedding 211 of FIG. 2 may be obtained. In this example, the teacher embedding 411 shown in FIG. 4 is an instance of the reference embedding 211 shown in FIG. 2. In this example, the teacher spatial embedding 411 is based on the unmasked audio features 203. According to this example, obtaining the teacher spatial embedding 411 in the latent space includes applying a context encoding process to the unmasked audio features 203 by the control system 110, in this example by a teacher context encoder 416, to generate the teacher spatial embedding 411. In some examples, the teacher context encoder 416 has the same configuration as the student context encoder 406. For example, the teacher context encoder 416 and the student context encoder 406 may include the same type(s) of neural network, the same number of layers, etc. In other examples, the teacher context encoder 416 may have a different configuration than the student context encoder 406. For example, the teacher context encoder 416 may have a simpler configuration, e.g., fewer layers, than the student context encoder 406. Because the tasks performed by the teacher context encoder 416 are generally simpler than the tasks performed by the student context encoder 406, a simpler configuration may be sufficient.

[0079] According to this example, the teacher context encoder 416 is configured to apply a context encoding process to the unmasked audio features 203 to generate a predicted spatial embedding 407 in the latent space. In this example, the teacher context encoder 416 is configured to apply the same context encoding process as the student context encoder 406. Thus, the predicted spatial embedding 407 corresponds to the predicted spatial embedding 207 and the reference spatial representation block 408 corresponds to the hidden representation block 208, except that the predicted spatial embedding 407 is based on the unmasked audio features 203 instead of the masked audio feature data 205. In this example, the student embedding 409 of FIG. 4 corresponds to the predicted spatial embedding 209 of FIG. 2.

[0080] FIG. 5 illustrates blocks of an SPRL system according to yet another example. As with other figures provided herein, the types, numbers, and arrangements of elements shown in FIG. 5 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and arrangements of elements. In this example, the blocks of SPRL system 500 are implemented by an instance of control system 110 of FIG. 1A. As noted elsewhere herein, control system 110 may reside in more than one device in some implementations.

[0081] FIG. 5 illustrates an example of an SPRL system 500 in which a clustering process is implemented for spatial unit discovery. In this example, a clustering process is used to generate a library of acoustic zone codewords 503, sometimes referred to herein as a codebook of acoustic zone codewords 503. According to some implementations, an initial codebook of codewords, each of which corresponds to an acoustic zone, may be computed prior to a training process, which may be a self-supervised training process. The initial codebook of acoustic zone codewords 503 may be populated, for example, by determining an acoustic zone directionality matrix for each frame of the multichannel audio data training set. These acoustic zone directionality matrices may be clustered to generate C clusters. In some examples, C may be 100. In alternative examples, C may be greater than 100 or less than 100. Each of the C clusters may be associated with a different acoustic zone. Thus, in some such examples, each codeword in the codebook of acoustic zone codewords 503 may correspond to only one of the C clusters and only one acoustic zone.

[0082] During the training process of the SPRL system 500, the acoustic zone codeword codebook 503 may be used to determine the reference embedding 211. For example, the acoustic zone codeword codebook 503 may be used to determine the reference embedding 211 by determining which cluster in the acoustic zone codeword codebook 503 is most similar to the extracted audio features corresponding to each time frame. In the example shown in FIG. 5, the acoustic zone codeword codebook 503 determines that the extracted audio features 214A are most similar to the embedding in the acoustic zone codeword codebook 503 corresponding to acoustic zone 2. Thus, the output 504 from the acoustic zone codeword codebook 503 is the embedding that corresponds to acoustic zone 2 in this particular example.

[0083] If necessary, the optional projection layer 505 may be configured to project the output 504 into a different dimensional space to ensure that the reference embedding 211 and the predicted embedding 209 are in the same dimensional space. For example, if the output 504 is a 768-dimensional value and the predicted embedding 209 is a 512-dimensional value, the optional projection layer 505 may be configured to project the output 504 from the 768-dimensional space into the 512-dimensional space.

[0084] In some implementations, the initial codebook 503 of acoustic zonal codewords may be updated during the training process according to a dynamic learning path 506. In the example shown in FIG. 5, the dynamic learning path 506 includes predicted spatial embeddings 207 that are used to update the codebook 503 of acoustic zonal codewords. In some examples, the predicted spatial embeddings 207 may result from one or more layers of the context encoder 206. In some alternative examples, the dynamic learning path 506 may include predicted embeddings 209.

[0085] The initial codebook of acoustic zone codewords 503 may be calculated according to different methods, depending on the particular implementation. In some examples, each codeword may correspond to a spatial location, area, or zone of a sound source relative to a microphone used to capture at least a portion of the multi-channel audio data 201. According to some examples, each codeword may correspond to a covariance of signals corresponding to sounds captured by each microphone of a plurality of microphones used to capture the multi-channel audio data. The covariance of signals corresponding to sounds captured by each microphone of a microphone array may include information about the direction from which acoustic waves from a sound source arrive at those microphones.

[0086] In some such examples, the covariance of signals corresponding to sounds captured by each microphone in a microphone array may be represented by a microphone covariance matrix. For example, if a microphone array includes three microphones, the covariance of those microphones may be determined by representing each sample of audio as a 1-by-3 column vector, with each value in the column vector corresponding to one of the three microphones. When the column vector is multiplied by its own transpose—essentially multiplying every microphone sample by the sample of the other microphone in the microphone array during the same time period—and the results are averaged over time, this produces a 3-by-3 microphone covariance matrix.

[0087] The covariance of signals corresponding to sounds captured by each microphone of a microphone array may be frequency dependent for a variety of reasons, including microphone spacing, the fact that the microphones involved are imperfect, the fact that sounds from a sound source reflect off walls differently at different frequencies, and the fact that sounds emitted by a sound source may have different directionality at different frequencies. To capture a more complete representation of spatial features in the multi-channel audio data 201, it is potentially advantageous to determine microphone covariance matrices for multiple frequency bands.

[0088] Some such examples may involve converting the microphone covariances expressed in the time domain into microphone covariances in each of a number of frequency bins--e.g., by a fast Fourier transform--and summing the results according to a set of frequency banding rules. In one such example, a microphone covariance matrix may be determined for each of 40 frequency bands, such as mel-spaced or log-spaced frequency bands. In other examples, microphone covariance matrices may be determined for more or fewer frequency bands.

[0089] In the 40-band example, we have 40 microphone covariance matrices that describe the frequency-dependent microphone covariance in each of the 40 bands. In the above example, each microphone covariance matrix contains 9 complex numbers, each of which has a real and an imaginary part. More generally, the microphone covariance matrix may be represented as an N×N complex matrix that is Hermitian, meaning that the N diagonal elements of the matrix are real, and further that the (N-1)*N / 2 complex values ​​in the upper triangle of the matrix are the complex conjugates of the lower triangle. Thus, the matrix has N real numbers and (N-1)*N / 2 complex numbers that represent all the degrees of freedom of the matrix. Enumerating the real and imaginary parts gives us a total of N+(N-1)×N=N+N×NN=N×N real numbers, which represent all the degrees of freedom. Thus, in summary, a complex Hermitian N×N matrix can be reduced to N real numbers.

[0090] In this example of three microphones, the 3×3 complex Hermitian covariance matrix in each band can be expressed as 9 real numbers. These 40×9 real numbers are an example of spatial acoustic features 203.

[0091] Alternatively or additionally, in some examples, each microphone covariance matrix may be factor-normalized with respect to signal level. According to some examples of 3×3 microphone covariance matrices, one out of nine matrix values ​​may correspond to a level, so the normalization may reduce each matrix to a 40×8 matrix.

[0092] Thus, each frame of audio may be represented by a combination of, for example, 40 microphone covariance matrices that characterize spatial information corresponding to that frame of audio across each of 40 different frequency bands. The microphone covariance matrices may be combined into a single microphone covariance matrix, such as a single 40×9 matrix, a single 40×8 matrix, etc. According to some examples, whatever the size of the microphone covariance matrices, the microphone covariance matrices for each sample of multi-channel audio data in the training set may be clustered into C clusters, for example according to a k-means clustering process. Each of the clusters may be associated with a distinct set of spatial characteristics. Each cluster may be assigned a number, for example, from cluster 0 to cluster 99, each of which may be associated with a different acoustic zone. The resulting C clusters may be used as an initial codebook 503 for acoustic zone codewords.

[0093] In some examples, the spatial unit discovery process may include clustering spatial features into multiple levels of granularity. For example, the spatial unit discovery process may include clustering spatial features using different numbers of audio frames. In some such examples, instead of having one number between 0 and 99, or more generally between 0 and (C-1)-1, for each frame of audio, there are K numbers between 0 and 99. One such clustering model may include averaging over 10 audio frames to generate a microphone covariance matrix at one level of granularity, averaging over 100 audio frames (or a different number greater than 10) to generate a microphone covariance matrix at another level of granularity, averaging over 1000 audio frames (or a different number greater than the previous number) to generate a microphone covariance matrix at another level of granularity, etc.

[0094] The initial codebook 503 of acoustic zone codewords includes embeddings associated with acoustic zones that were created according to a self-supervised process from "unlabeled" multi-channel audio data 201, as that term is used herein. "Labeled" multi-channel audio data can include, for example, human-labeled multi-channel audio data or multi-channel audio data with associated direction of arrival (DOA) metadata. According to this example, the multi-channel audio data 201 used to construct the initial codebook 503 of acoustic zone codewords did not include such labels.

[0095] 6 is a flow diagram outlining an example of the disclosed method. The blocks of method 600, as well as other methods described herein, are not necessarily performed in the order shown. According to some examples, one or more blocks may be performed in parallel. Furthermore, some similar methods may include more or fewer blocks than shown and / or described. In this example, method 600 involves training an SPRL system.

[0096] Method 600 may be performed by an apparatus or system such as apparatus 100 shown in FIG. 1A and described above. In some examples, apparatus 100 includes at least control system 110 shown in FIG. 1A and FIG. 2 and described above. In some examples, at least some aspects of method 600 may be performed by one or more devices in an audio environment, for example, by an audio system controller (such as what may be referred to herein as a smart home hub), or by another component of an audio system, such as a television, a television control module, a laptop computer, a mobile device (such as a cellular phone), etc. However, in some implementations, at least some blocks of method 600 may be performed by one or more devices configured to implement a cloud-based service, such as one or more servers.

[0097] In this example, block 605 involves receiving, by the control system, multi-channel audio data. In this example, the multi-channel audio data includes unlabeled multi-channel audio data. In some examples, the multi-channel audio data may include only unlabeled multi-channel audio data. Block 605 may obtain the frame of audio data from a memory device that stores the multi-channel audio data 201 shown in FIG. 2, FIG. 4, or FIG. 5, for example.

[0098] According to this example, block 610 involves extracting audio feature data from the unlabeled multi-channel audio data by the control system. According to some examples, in block 610, audio features 203 may be extracted by audio feature extraction block 202.

[0099] In this example, block 615 involves masking, by the control system, a portion of the audio feature data to generate masked audio feature data. According to this example, block 615 involves one or more types of spatial masking. In some examples, block 615 may include the masking block 204 masking one or more sections of the audio feature 203 to generate masked audio feature data 205.

[0100] According to this example, block 620 includes applying, by the control system, a context encoding process to the masked audio feature data to generate a predicted spatial embedding in the latent space. In some examples, block 620 may include the context encoder 206 applying a context encoding process to the masked audio feature data 205 to generate a predicted spatial embedding in the latent space 207.

[0101] In this example, block 625 includes obtaining, by the control system, a reference spatial embedding in the latent space. The reference spatial embedding 211 may be obtained in various manners, depending on the particular implementation. In some examples, the reference spatial embedding 211 may be obtained as described herein with reference to FIG. 4 or as described herein with reference to FIG. 5.

[0102] According to this example, block 630 includes determining, by the control system, a loss function gradient based at least in part on the difference between the predicted spatial embedding and the reference spatial embedding. In some examples, the loss function gradient base can be based on the loss function 210 disclosed herein or a similar loss function. According to some examples, the loss function can be the negative of a measure of similarity between the predicted spatial embedding 209 and the reference spatial embedding 211. In some examples, the loss function can be applied only to portions of the predicted embedding that correspond to one or more masked regions of the masked audio feature data 205, such as the masked region 215 of the masked audio features 214M, and the corresponding portions of the reference embedding 211.

[0103] In this example, block 635 includes updating, by the control system, the context encoding process according to the loss function gradient until one or more convergence metrics are achieved. In some examples, the context encoding process implemented by the context encoder 206 may be updated in block 635.

[0104] The method 600 may include implementing various types of spatial masking depending on the particular implementation. In some examples, the spatial masking may include concealing, corrupting, or removing at least a portion of spatial information of the multi-channel audio data. According to some examples, the spatial masking may include swapping channels during the masking time interval. The spatial masking may include, for example, presenting audio data from channel A as audio data from channel B and presenting audio data from channel B as audio data from channel A during the masking time interval. In some examples, the spatial masking may include changing an apparent acoustic zone of a sound source during the masking time interval. According to some examples, the spatial masking may include adding audio data corresponding to an artificial sound source in an artificial sound source acoustic zone during the masking time interval. In some examples, the spatial masking may include reducing a number of uncorrelated channels of the multi-channel audio data during the masking time interval. Alternatively or additionally, spatial masking may include increasing a number of correlated channels of the multi-channel audio data during the masking time interval. In some examples, spatial masking may include combining two or more audio signals corresponding to two or more independent sound sources in a single channel during the masking time interval. According to some examples, spatial masking may include modifying microphone signal covariance information of the multi-channel audio data during the masking time interval.

[0105] In some examples, obtaining the reference spatial embedding in the latent space may include applying, by the control system, a context encoding process to unmasked audio feature data corresponding to the masked audio feature data to generate the reference spatial embedding. The description of FIG. 4 provides some relevant examples.

[0106] According to some examples, obtaining the reference spatial embedding in the latent space may include a spatial unit discovery process. The description of FIG. 5 provides some relevant examples. In some such examples, the spatial unit discovery process may include a clustering process. According to some such examples, the spatial features may be clustered according to multiple granularities. In some examples, the clustering may include applying an ensemble of k-means models with different codebook sizes.

[0107] In some examples, the spatial unit discovery process may include generating a library of codewords. According to some such examples, each codeword may correspond to an acoustic zone of the audio environment. In some examples, each codeword may correspond to a spatial location of a sound source relative to a microphone used to capture at least a portion of the multi-channel audio data. According to some examples, each codeword may correspond to a covariance of signals corresponding to sounds captured by each microphone of a plurality of microphones used to capture the multi-channel audio data. In some examples, the signal covariance is represented by a microphone covariance matrix. According to some examples, the method 600 may include updating the library of codewords according to an output of the context encoding process.

[0108] According to some examples, the predicted spatial embedding may correspond to estimated cluster centroids in the latent space. In some examples, the predicted spatial embedding may correspond to a representation of acoustic zones in the latent space.

[0109] In some examples, the multi-channel audio data may include audio data captured by different types of microphone arrays, e.g., two or more different types of microphone arrays. According to some examples, the multi-channel audio data may include various numbers of channels. For example, the multi-channel audio data may include two channels of audio data, three channels of audio data, five channels of audio data, six channels of audio data, seven channels of audio data, or various combinations thereof. More generally, in some examples, it may be said that the multi-channel audio data may include at least N channels of audio data and M channels of audio data, where N and M are 2 or greater and represent different valued integers.

[0110] According to some examples, the method 600 may include a self-supervised learning process. In some such examples, the method 600 may include training a neural network according to a self-supervised learning process. In some examples, the method 600 may include training a neural network implemented by the control system to implement one or more types of “downstream” audio processing functions after the control system has been trained according to at least blocks 605-635 of FIG. 6 . The downstream audio processing functions may include noise suppression functions, speech recognition functions, speaker identification functions, or combinations thereof. In some examples, the method 600 may include implementing the downstream audio processing functions by the control system. For example, the method 600 may include implementing a noise suppression function, a speech recognition function, a speaker identification function, or combinations thereof.

[0111] Some aspects of the disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to asserted data.

[0112] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform the necessary processing on the audio signal(s), including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, the system also including other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0113] Another aspect of the disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., code executable to perform the examples) for performing one or more examples of the disclosed methods or steps thereof.

[0114] While specific embodiments of and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the present disclosure as described and claimed herein. While certain forms of the present disclosure have been illustrated and described, it will be understood that the present disclosure is not limited to the specific embodiments described and illustrated, or to the specific manner described.

Claims

1. (a) receiving, by a control system, multi-channel audio data, the multi-channel audio data including unlabeled multi-channel audio data; (b) extracting, by the control system, audio feature data from the unlabeled multi-channel audio data; and (c) masking, by the control system, a portion of the audio feature data to generate masked audio feature data, the masking including spatial masking; and (d) applying, by the control system, a context encoding process to the masked audio feature data to generate a predicted spatial embedding in a latent space; (e) obtaining, by the control system, a reference spatial embedding in the latent space; (f) determining, by the control system, a loss function gradient based at least in part on a difference between the predicted spatial embedding and the reference spatial embedding; (g) updating, by the control system, the context encoding process according to the loss function gradient until one or more convergence metrics are achieved. method.

2. 2. The method of claim 1 , wherein obtaining a reference spatial embedding in the latent space comprises applying, by the control system, a context encoding process to unmasked audio feature data corresponding to the masked audio feature data to generate the reference spatial embedding.

3. The method of claim 1 , wherein obtaining a reference spatial embedding in the latent space comprises a spatial unit discovery process.

4. The method of claim 3 , wherein the spatial unit discovery process includes clustering spatial features into multiple granularities.

5. The method of claim 4 , wherein the clustering comprises applying an ensemble of k-means models with different codebook sizes.

6. The method of claim 3 , wherein the spatial unit discovery process includes generating a library of code words, each code word corresponding to an acoustic zone of the audio environment.

7. The method of claim 6 , wherein each codeword corresponds to a spatial location of a sound source relative to a microphone used to capture at least a portion of the multi-channel audio data.

8. 7. The method of claim 6, wherein each codeword corresponds to a covariance of signals corresponding to sounds captured by each microphone of a plurality of microphones used to capture the multi-channel audio data.

9. The method of claim 8 , wherein the covariance of a signal is represented by a microphone covariance matrix.

10. The method of claim 6 , further comprising updating the library of codewords according to an output of the context encoding process.

11. The method of claim 4 , wherein the predicted spatial embedding corresponds to estimated cluster centroids in the latent space.

12. The method of claim 1 , wherein the predicted spatial embedding corresponds to a representation of acoustic zones in the latent space.

13. The method of claim 1 , wherein the multi-channel audio data includes at least N channels of audio data and M channels of audio data, where N and M are greater than or equal to 2 and represent different value integers.

14. The method of claim 1 , wherein the multi-channel audio data comprises audio data captured by two or more different types of microphone arrays.

15. 10. The method of claim 1, further comprising, after the control system has been trained according to at least steps (a) through (g), training a neural network implemented by the control system to implement a noise suppression function, a speech recognition function, a speaker identification function, a sound source separation function, a voice activity detection function, an audio scene classification function, a sound source localization function, a noise source recognition function, or a combination thereof.

16. The method of claim 15 , further comprising implementing, by the control system, the noise suppression functionality, the speech recognition functionality, or the speaker identification functionality.

17. The method of claim 1 , wherein the spatial masking comprises concealing, corrupting, or removing at least a portion of spatial information of the multi-channel audio data.

18. 2. The method of claim 1, wherein the spatial masking comprises presenting audio data from channel A as being audio data from channel B and presenting audio data from channel B as being audio data from channel A during a masking time interval.

19. The method of claim 1 , wherein the spatial masking comprises modifying the apparent acoustic zone of a sound source during the masking time interval.

20. The method of claim 1 , wherein the spatial masking comprises adding audio data corresponding to an artificial sound source in an artificial source acoustic zone during the masking time interval.

21. The method of claim 1 , wherein the spatial masking comprises reducing a number of uncorrelated channels of the multi-channel audio data during the masking time interval.

22. The method of claim 1 , wherein the spatial masking comprises increasing a number of correlated channels of the multi-channel audio data during the masking time interval.

23. The method of claim 1 , wherein the spatial masking comprises combining, during the masking time interval, two or more audio signals corresponding to two or more independent sound sources in a single channel.

24. The method of claim 1 , wherein the spatial masking comprises modifying microphone signal covariance information of the multi-channel audio data during the masking time interval.

25. The method of claim 1 , wherein the control system is configured to implement a neural network.

26. The method of claim 1 , wherein operations (a) through (g) involve a self-supervised learning process.

27. 27. Apparatus comprising a control system configured to implement a method according to any one of claims 1 to 26.

28. 27. A system configured to implement the method of any one of claims 1 to 26.

Images